We Keep Coding

How to properly separate concerns in my architecture without designing a spacecraft?

In my last question I posted some sample code on how I was trying to achieve separation of concerns. I received some ok advice, but I still just don't "get it" and can't figure out how to design my app to properly separate concerns without designing the next space shuttle.
The site I am working on (slowly converting from old ASP section by section) is moderately sized with several different sections including a store (with ~100 orders per day) and gets a decent amount of traffic (~300k uniques/month). I am the primary developer and there might be at most 2-3 devs that will also work on the system.
With this in mind, I am not sure I need full enterprise level architecture (correct me if i am wrong), but since I will be working on this code for the next few years, I want it to perform well and also be easy to extend as needed. I am learning C# and trying to incorporate best practices from the beginning. The old ASP site was a spaghetti mess and I want to avoid that this time around.
My current stab at doing this ended up being a bunch of DTOs with services that validate and make calls to a DAL layer to persist. It was not intentional, but I think the way it is setup now is a perfect anemic domain model. I have been trying to combat this by turning my BLL to domain objects and only use the DTOs to transfer data between the DAL and BO, but it is just not working. I also had all my dtos/blls split up according to the database tables / functionality (eg - YouTube style app - I have separate DTO/BLL/DAL for segments, videos, files, comments, etc).
From what I have been reading, I need to be at least using repositories and probably interfaces as well. This is great, but I am unsure how to move forward. Please help!

From what I can see you have four points that need addressing:
(1) "With this in mind, I am not sure I need full enterprise level architecture"
Lets deal with the high level fluff first. It depends on what you mean by "full enterprise level architecture", but the short answer is "Yes" you need to address many aspects of the system (and it will depend on the context of the system as to what the main ones are). If nothing else, the keys ones would be Change and Supportability. You need to structure the application in a way that supports changes in the future (logical and physical separation of concerns (Dependency Injection is a great for the latter); modular design, etc).
(2) "How to properly separate concerns in my architecture without designing a spacecraft?"
I like this approach (it's an article I wrote that distilled everything I had learnt up to that point) - but here's the gist:
Looking at this you'll have a minimum of six assemblies - and that's not huge. If you can break your system down (separate concerns) into these large buckets it should go a long way to giving what you need.
(3) Detail
Separating concerns into different layers and classes is great but you need to go further than that if you want to be able to effectively deal with change. Dependency Inversion (DI) is a key tool to use here. When I learnt DI it was a hand-rolled affair (as shown in the previous link) but there are lots of frameworks (etc) for it now. If you're new to DI (and you work in .Net) the article will step you through the basics.
(4) How to move forward
Get a simple vertical slice (UI all the way to the DB) working using DI, etc. As you do this you'll also be building the bones of the framework (sub-systems and major plumbing) that your system will use.
Having got that working start on a second slice; it's at this point that you should uncover any places where you're inadvertently not reusing things you should be - this is the time to change those before you build slices 3,4 and 5 - before there's too much rework.
Updates for Comments:
Do you you think I should completely
drop web forms and take up MVC from
scratch or just with what I know for
now?
I have no idea, but for the answer to be 'yes' you'd need to be able to answer these following questions with 'yes':
We have the required skills and experience to use and support MVC.
We have time to make the change (there is clear benefit in making this change).
We know MVC is better suited for our needs.
Making this change does not put successful delivery at risk.
...do I need to move to projects and
setup each of these layers as a
separate project?
Yes. Projects map 1-to-1 with assemblies, so get the benefits of loose-coupling you'll definitely want to separate things that way, and be careful how you set references.
when you refer to POCOs, are you meaning just DTOs or rich domain objects?
DTO not Rich Domain Object. BUT, people seem yo use the terms POCO and DTO interchangeably when strictly speaking they aren't - if you're from the Martin Fowler school of thought. In his view a DTO would be a bunch of POCO's (or other objects(?)) parcelled together for sending "across the wire" so that you only make one call to some external system and not lots of calls.
Everyone says I should not expose my
data structures to my UI, but I say
why not?
Managing dependencies. What you don't want is for you UI to reference the physical data structure because as soon as that changes (and it will) you'll be (to use the technical term) screwed. This is the whole point of layering. What you want to do is have the UI depend on abstractions - not implementations. In the 5-Layer Architecture the POCOs are safe to use for that because they are an abstract / logical definition of 'some thing' (a business concept) and so they should only change if there is a business reason - so in that sense they are fairly stable and safer to depend on.

If you are in the process of rewriting your eCommerce site, you should atleast consider replacing it with a standard package.
There are many more such packages available today. So although the decision to build the original site may have been correct, it is possible that building a custom app is no longer the correct decision.
There are several eConmmerce platforms listed here: Good e-commerce platform for Java or .NET
It should cost much less than the wages of 2-3 developers.

in .net, what programming model would be good for prototyping, but then reusable for production (for business logic/data access layers)

In .NET land what would be a good approach for quick prototyping of a concept (i.e. development just on my PC) that could then be extended out to product (users across LAN/WAN), BUT in a fashion that the model/business logic code and data access layer code can be used as is?
One thought for example I had as to do:
(a) WinForms with business logic and Entity Framework layer to SQL Server Express on my PC, then
(b) Go then to ASP.net (using the business logic / data library) with SQL Server/IIS
Any comments? Other suggestions?

I would recommend trying a layered approach:
put your entity data model and validation classes into a separate assembly
put additional business logic into a separate business logic assembly
put your services (WCF or WCF Data Services) into their own assembly
All those base layers are pretty much independent of what you choose as the UI frontend technology. You can make choices here (e.g. Linq-to-SQL vs. Entity Framework for your data access; do you need a WCF-based service layer, or does your app use direct DB access?) more or less independent of what you put on top of that for the UI layer.
And on top of those base assemblies:
create your UI either as Winforms app, or as an ASP.NET (Webforms or MVC) web app (or both)
If you have layers and if you architect them well, you can reuse a large portion of your code and business rules.
Try to put only stuff that's specific for each UI technology (Winforms vs. ASP.NET) into those frontend presentation assemblies. Keep all the common business rules, validation rules, access and service layers separate.
And again: it seems you believe that "going ASP.NET" excludes using WCF/WCF Data Services - not at all ! You can easily use data from a WCF service in an ASP.NET app. You're not losing anything by layering your business and services layers - those can be easily reused in both Winforms and ASP.NET apps!

A few comments:
Prototyping as an approach to developing production quality software can be problematic, as the very nature of prototyping can mean that the software and design quality are not that great. Prototypes are not meant to be great quality by definition.
If the goals are to get some feedback from the intended customer\user during the early stages, it can often be a good idea to not prototype full vertical slices (e.g. UI -> Business -> DB), but work with UI mockups which can be used to explore ideas with users. The mockup is flexible and easy to change, but is not fully functional e.g. does not have business logic or persistence. This approach allows users to get some idea of functionality and be involved in the design and requirements gathering process. It will be quick to change the mocks as requirements change, especially as there is no business logic or database code that has to be changed with it. An example of a UI mocking tool is Balsamiq:
http://www.balsamiq.com/
If one of the overall goals of prototyping is to investigate and explore different technology choices, these can be done in isolation of UI mockups, and in a more abstract fashion i.e. pure technology investigation rather than geared at delivering the exact needs of a prototype which can be changing massively. The UI mockups may change a lot via user feedback, so having technology investigation as a different activity can make this process simpler i.e. there is less coupling as things will change a lot during the early discovery phase, and having to change the "backend" constantly because UI ideas are developing so rapidly, will slow you down.
In terms of speeding up the pace of software development, leverage third party libraries. If you are using a database for persistence, look at ORM solutions which could massively reduce the work needed to develop a data access layer e.g. nHibernate. Depending on what UI technology you use, look at third party control libraries.
In the industry, the approach that has very much replaced prototype driven development over the years is: Agile. It seeks to tackle the changing needs of users head-on by always striving to deliver features, but has a focus on developing high quality software through techniques such as TDD.

Best practices for how to Layer a ASP.NET/C# web app

I have been working on an ASP.NET/C# web app for some time and its size has gotten way to large for how it is being programmed. It has become very hard to maintain and getting harder quickly, something that used to take an 1hr to update now takes about 3-4hrs.
I believe that reworking the app to use different layers would help solve many of these problems. However the more I read the more it seems that everyone does it differently, but achieve mostly the same goals.
I have seen layers such as Presentation/UI, DB, Business, Services, ect. It appears that a 3 layer may be the best but I am unsure.
What layers should I have in a web app and what should each include or be limited to?
Words from previous experience are most appreciated.

I believe the common approach is to have 3 layers: presentation, business logic and data access. It should provide for a good foundation.
Having said that I need to point out that division into layers may not help very much with ASP.NET WebForms project. The biggest issue of this framework is its code behind which lures developers into creating monster pages that talk to all layers and services simultaneously to fetch the data to display. The cure is to work out this presentation layer first and let the code only interact with one specific layer (most usually, the business logic). When (and if) this is done, then the project may be successfully refactored.

Your correct in saying everyone does it a little differently. This is how we do it:
Domain
Model - class objects such as Customer, Account, etc.
LookUp - value objects such as AccountType, IndentityType, etc.
Repositories or DataAccessObjects current we use LinqToSql and SQLClient on our older applications.
Service
Services for each of the models or Aggregates. You could also call this the application layer. The idea is we could change to a different UI and it would involve as little as possible code changes.
UI
Currently we are using Asp.net and MVC in our newer applications
The idea is that we can insert different "things" into the layers and not effect the others. ex. If I start using EntityToSql the UI or the Service layer is none the wiser. It just knows it creates an IRepository and calls the FindAll() method. We are also converting older applications to MVC and before we do that we seperate them into these layers so when something else comes out we would like to implement our Service and Domain layers don't have to be changed.
It's important to always be asking your self where should this live? Logic in your UI should really be confined to UI logic. The important thing is to have everyone on your team understand your way and be willing to implement it...

Mainly it is just managing dependencies.
Good examples are Sharp Architecture projects and Arc projects. You may download some open-source example applications for gaining more insight.

Seperation of concern might be a better way of looking at your web application and breaking things up. For say, if you have one page that is doing data access, validation and displaying of items you could take each one of those items, and create it's own class so that you have only one type of thing happening in each class. Then as you go through and start to refactor your code you can then start grouping same type classes into the same type folders, namespaces and projects.
Good luck, you've got a lot of work ahead of you.

Check out the Microsoft MVC model for ASP.NET. The issues you described are very common to ASP.NET applications, and refactoring it to match the MVC model might be the way to go. Don't get me wrong - it'll definitely take some work - but IMHO, it'll be worth it.

I believe it's heavily dependent on the requirements and whats needed. There is many types of design patterns of ways to structure a app. I've found it helpful to fully digest what I'm trying to accomplish from the requirements and find a match to patterns on this website. Give or take unique circumstances.
http://www.blackwasp.co.uk/DesignPatternsArticles.aspx
This site actually breaks it down design pattern by design pattern. Allowing you to make a good match based on your requirements and scalability needs.

Traditional layering is only part of the solution. Since that was your question, though, I know many very large sites that layer by presentation, business logic and data access.
However, there's much more that you could / should do, including:
Tiering and division-of-labor: partition business logic between the web tier and the data tier in a maintainable way (usually implies using stored procedures instead of dynamic SQL)
Refactoring your application. In many sites, pages get large and unmanageable due to insufficient attention to continual refactoring. Use approaches such as custom user controls, Master pages, common page base class, ASP.NET skins, page adapters, control adapters, HttpModules, etc, to move your application-wide logic into centrally manageable locations.
Careful adherence to OO design principles.
Strategic use of events to help ensure consistent separation of concern.

Need an example of a "Good Line of Business Application" for .NET that uses ORM

I'm trying to move toward TDD, ORM, Mocking, ect. I need a good example of a line of business app that uses an ORM preferably NHibernate.
It has to be open source and use the repository pattern.
I learn best by example, I have played around with the repository pattern and unit of work pattern but not in any meaningful applications.
I'm familiar with IoC (I use unity), WCF, Workflow Foundation, WPF, Smart Client Software Factory, Webclient Software Factory, ect.
I've learned all the "basics" (they are pretty advanced principals to be called basics, IMO) I just can't seem to put it all together.
The applications we write follow all "best practices" as far as architecture, we have a business logic layer, data access layer, MVP, MVVP, MVC, ect. but there is never any code in our BLL's besides
return dal.GetBlahBlahBlah();
I have to ask myself where is all my business logic???
Probably 95% of our data access is through stored procedures and I have to assume that its all if the database. Some of these SP's are huge and have lots and lots of if statements, case statement, and the occasional cursor.
As mentioned above I know how to use all of these cool technologies but it seems like the only thing I'm using them for is to make a really, really overcomplicated, overly architecture'd reporting tool for sql server.
If ALT.NET is the better way, if having all of your business logic in the code is the better way, there has to be an open source application out there that puts it all together in all the right ways

I haven't come across any LOB applications but I have heard that Cuyahoa is an excellent example of how to use NHibernate.
I am familiar with the code in Suteki Shop, an e-commerce platform using ASP.Net MVC and Linq-To-Sql which is active at the moment and being re-factored very well and should provide you with some insight in to what you're attempting.
Penultimately, there is Rob Conery's MVC StoreFront. Rob is the master of screencasts and presents a great insight in to learning TDD.
My final link is summer of nhibernate another series of screencasts this time specifically on using NHibernate

How would you architect a desktop application in C# 3.0

I've created a simple desktop application in C# 3.0 to learn some C#, wpf and .Net 3.5.
My application essentially reads data from a csv file and stores it in a SQL server CE database. I use sqlmetal to generate the ORM code for the database.
My first iteration of this app is ugly as hell and I'm in the process of refactoring it.
Which brings me to my question. How would you architect a desktop database app in C#?
What are the best practices?
Do you create a Database Abstraction Layer (DAL) which uses the sqlmetal generated code? Or is the generated code enough of an abstraction?
If you use DAL pattern, do you make it a singleton or a static member?
Do you use the View-Model-ModelView pattern with the DAL pattern?
Apologies if this seems like a long open ended question, but I have been giving this a lot of thought recently.
I see a lot of examples on how to architect an enterprise n-tier app in C# but not that many on architecting standalone desktop apps.

I would start with the Composite Application Guidance for WPF (cough PRISM cough) from Microsoft's P&P team. With the download comes a great reference application that is the starting point for most of my WPF development today.
The DotNetRocks crew just interviewed Glenn Block and Brian Noyes about this if you're interested in hearing more from them.
Even better, Prism is not nearly as heavy as the CAB was, if you're familiar at all with that from the WinForms days.

The answer is "it depends" as always.
A few things to think about:
You may want to make this fat client app a web app (for example) at some point. If so, you should be sure to keep separation between the business layer (and below) and the presentation. The simplest way to do this is to be sure all calls to the business logic go through an interface of some kind. A more complex way is to implement a full MVC setup.
Another thing you may consider is making the data access layer independent of the business logic and user interface. By this I mean that all calls from business logic into the DAL should be generic "get me this data" rather than "get me this data from SQL" or even worse "run this SQL statement". In this way, you can replace your DAL with one that accesses a different database, XML files, or even something icky like flat files.
In short, separation of concerns. This allows you to grow in the future by adding a different UI, segmenting all three areas into their own tier, or changing the relevant technology.

Before architecting anything you should define requirements for your app.
It's a common error of beginner developers - starting writing code ahead of thinking about how it would perform. My advice will be to try to describe some feature of you application. It will help you to feel how it should be implemented.
As for useful learning resources I would highly recommend you to take a look at CompositeWPF it's a project designed specifically to teach developers best practices of desktop app development.

I'd start with Jeremy Miller's Build Your Own Cab series.
I was an early CAB adopter. I learned a lot from digging into that technology and reading all the .NET blogs about application architecture.
But recently I had a chance to start a new project, and instead of using CAB I went with StructureMap & NHibernate and borrowed some of the patterns that Jeremy uses (in particular, his way of handling event aggregation). The result was a really simplified, hand-tooled framework that does everything I need and I love working with it.
As to the specifics of your question: I use a Repository for data access. I initially wrote some ADO.NET code and used data readers and mapped my objects. But that got old real fast, so I grabbed NHibernate and was really pleased. The repositories use NHibernate for data access, and my data access needs are pretty simple in this particular app.
I have a service layer (exposed via WCF, Duplex channels) that utilizes the repositories. My app is basically client-server with real time updating (and I know your question was just about clients, but I would use the same technologies and patterns). O
n the client side I utilize MVP with StructureMap for IoC and some very simple event aggregation strategies for cross-class communications. I code to interfaces for just about everything. The only other thing I did was borrow from the CAB the idea of a flexible "Workspace" for dynamically displaying views. I wrote my own Workspace interface though and implemented my own DeckWorkspace and TableWorkspace for use in my app (these were really simple things to write).
A lot of my decisions in this most recent application were the result of experience and pain I felt using other frameworks and tools. I made different decisions this time around. Maybe the only way to really understand how to architect an application is to feel the pain of doing it wrong beforehand.

I would say yes, it could easily be structured towards smaller applications. There is a learning curve towards getting started, but honestly, it helped me understand WPF better than attempting to start from scratch. After starting a project with CompositeWPF and then starting another project without it, I found myself attempting to duplicate features of CompositeWPF on my own because I missed those features! :)