We Keep Coding

How to change persistence layer at runtime in C#

I am embarking on a new project and I need some guidance from veteran architects/design pattern gurus!
My new project needs to have a number of persistence layers whereby the client can decide at runtime where the data will be stored, for example, in house SQL database, MS Exchange or Google storage.
The functionality will essentially be the same just the storage/implementation of each will be different.
What I'm not looking for a here is how you do it just a pointer to the best patterns to use to serve my purpose whilst still providing flexibility down the road as their will be CHANGE. I am trying to avoid concrete implementations that will inevitably lead to some nasty code smells.
I know it will involve some kind of DI along the way but any pointers here would be greatly appreciated.

There is nothing special with your case really, so if you would follow standard practices with DI and use container to ease your task like SimpleInjector that will do the trick. The main point for you should be to not depend on concrete classes but on abstraction and that's where DI-container will help you organize this.
E.g. if you plan to save user you might have some IUserRepository with a method SaveUser. Then you will implement SqlUserRepository, GoogleStorageRepository, etc. The same goes for any other data access layer interface. If you just do that, you will need to configure your DI in a way where you can supply the required repository at a runtime based on your needs. Do not forget to never depend on GoogleStorageRepository, etc. directly, but only on a common interface. I would create a project for interfaces (and corresponding BI data model that DL will be aware of) and a project per each implementation as well to separate it even further.

Repository pattern is all about creating a separation between the persistence layer and the business layer.
Many examples on the web demonstrates it incorrectly by just using it as an wrapper over their data entities. That is incorrect. The design of a repository class/interface should be driven by the business requirements and not from how the first data store looks like.
Thus it's a perfect pattern for your use case. You define an repository interface from the business layer perspective and then create an implementation for each data store like MSSQL. I even put that interface in my business layer to further demonstrate that perspective.

DDD - Duplication between service layer and repositories

I've worked on several applications that try to adhere to DDD principles, I noticed that we end up with situations where there is duplication between the Service Layer and the repositories that feels like a code smell.
For most of the operations in the Service layer, it seems like it is a direct mapping to CRUD operations, GetAll, GetById, Create, Delete etc.. the flow of the architecure is within these lines: I have a controller calling a Service layer that calls a Repository that calls an ORM which talks to the Backend ..
So for example GetAll would exist in both SL and Repository. Now, if we have a change/business requirement that GetAll should ignore certain items, how am I supposed to do it, should I ignore these in the repository, or that's business logic that should go in the Service Layer? Wouldn't life be easier if we just had a Service Layer calling the ORM directly?
To Sum up: I understand the Service Layer could abstract some businees logic but when - in most cases - it is dealing with simple CRUD operations, wouldn't it be easier to just get rid of the Repository? But, what if the SL also contains some methods with complicated business logic, should these go through a repository? From good design perspective, Should I favor consistency and always go through repository or just keep it simple and only use a repository when it is not a simple one-to-one mapping to a CRUD operation.
PS: I realize there are seemingly similar questions but didn't find any satisfying answer

I noticed that we end up with situations where there is duplication
between the Service Layer and the repositories that feels like a code
smell.
It isn't a code smell since they do different things.
You should keep in mind that Domain or Application Services reside in a different layer than Repository implementations. Layers are there for a reason - objects in different layers don't have the same responsibilities and don't talk to the same neighbors. Repository implementations are tightly coupled to the means of persistence of your objects. They might generate SQL statements and talk to a relational database, they might talk to your ORM... the important thing is that they know about the way your objects are persisted, which is not true of Application Services.
If your Services layer was to call the ORM directly, it would really do 2 big things, breaking the Single Responsibility Principle. It would also be more difficult to change your ORM for another one or for a different means of persistence.
So for example GetAll would exist in both SL and Repository. Now, if
we have a change/business requirement that GetAll should ignore
certain items, how am I supposed to do it, should I ignore these in
the repository, or that's business logic that should go in the Service
Layer?
If GetAll() ignores certain items, I strongly suggest renaming it both in the Service and the Repository to reflect that, eg : GetAllAllowedToUser(), GetAllBut...(). Thus the contract of the method will be clear and you'll avoid misunderstandings about what it's supposed to return. Plus you'll be able to keep the original genuine GetAll() method which could still be of some use.

in most cases - it is dealing with simple CRUD operations, wouldn't it
be easier to just get rid of the Repository
IMHO, I wouldn't say to get rid of Repository. I would say that if you are doing CRUD you don't need DDD (at all). If you read Fowler's enterprise patterns or Evans they both say that DDD is only of use when you have domain logic that is significantly complex. CRUD is not complex and therefore no DDD needed.
What you describe is a code smell. But I don't think it is a smell with DDD. You are just seeing an over-engineered piece of code.

+1 for Dtryon, also:
Now, if we have a change/business requirement that GetAll should ignore certain item
Not directly related and I know you just used it as an example, but I've seen this exact thing. Please don't end up with methods called GetAll that do not get all. Keep GetAll, have it GetAll, then have GetAllLive, or GetAllAvailable or something like that as well which does what it says it does

Maybe the "Finder pattern"(dont know if this is the right term) can solve your problem. According to CQS(Command-Query-Separation) principle, (IMO,) query operations are not "business logic" at all. We can write some specific "Finder"s in the Infrastructure Layer to perform various queries and let all non-query operations(business logic) stay in the Service Layer, then on the client side we treat the finders the same as services.
Sorry for my language :-(.

C# Data Layer and Dto's

I have recently joined a company that using typed datasets as their 'Dto'. I think they are really rubbish and want to change it to something a little more modern and user friendly. So, I am trying to update the code so that the data layer is more generic, i.e. using interfaces etc, the other guy does not know what a Dto is and we are having a slight disagreement about how it should be done.
Without trying to sway people to my way of thinking, I would like to get impartial answers from you people as to what layers the Dto can be present in. All layers; DAL, BL and Presentation or a small sub set within these layers only.
Also, whether IList objects should or should not be present in the DAL.
Thanks.

It really depends on your architecture.
For the most point you should try to code to interfaces then it doesn't really matter what your implementation is. If you return ISomething it could be your SomethingEntity or your SomethingDTO but your consuming code doesn't care less as long as it implements the interface.
You should be returning an IList/ICollection/IEnumerable over a concrete collection or array.
Properties should not return arrays
Do not expose generic lists
What you should try to do first is separate your code and make it loosely coupled by inserting some interfaces between your layers such as a repository for your DataAccess layer. Your repository then returns your entities encapsulated by an interface. This will make your code more testable and allow you to mock more easily. Once you have your tests in place you can then start to change the implementations with less risk.
If you do start to use interfaces I would suggest integrating an IoC such as Windsor sooner rather than later. If you do it from the get go it will make things easier later on.

One thing is DataSets are poor to achieve interoperability. Even typed datasets are also not so compatible when it comes to consuming typed datasets from a non .net client. Refer this link. If you have to achieve interoperability then fight hard for DTOs otherwise try to make your team understand DTOs over a period of time because datasets are not so bad after all.
On part of interfaces, yes you should be exposing interfaces. For example - If you are returning List<T> from DAL, instead you should return IList<T>. Some people go to extent of returning only IEnumerable<T> because all you need is capability to enumerate. But then while doing it don't become astronaut architect.
In my applications I have figured out that returning IList<T> instead of List<T> pollutes my code base with codes like this:
//consider personCollection as IList<Person>
(personCollection as List<Person>).ForEach(//Do Something)
So I personally try to maintain a balance between returning interface or concrete object. If you ask me what I am doing right now, then I will tell you I am returning List<T>. I am influenced to not to become astronaut architect.

I always use DTOs, never DataTable. But I only use them to transfer from BL to DL and the other way around. My presentation layers are often only aware of the business and service layers in case of service oriented.
The benefits I can see to use DTOs rather than datatables:
easy refactoring
easy diagram production
cleaner more readable code, especially in the DAL's unit tests

By definition a DTO is a data transfer object, used to (wait for it) transfer data from one layer to another.
DTOs can be used across all layers and I have used them well with web services.

Business Objects and Data Layer

This site has provided me with many useful answers, however after a hours search I haven't found anything that specifically answers my needs. So here goes...
The company I'm working for is in the process of designing a new Business Objects Layer and a Data Access Layer - these will reside in separate assemblies.
The problem is I'm having a hard time getting my head around the interaction between these two layers - specifically, should the DAL know about the BOL, I've read numerous articles that have said the dependency order should go something like this:
GUI / Presentation --> BOL ---> DAL
But as far as I can see, the DAL needs a reference to the BOL in order to be able to 'return' objects to the BOL layer.
I'm going for a intermediate assembly between the BOL and DAL which will be basically a thin layer filled with interfaces to decouple those two DLL's, so the framework can use different DALs if the need arises.
This lead me to the idea of introducing another thin layer with a bunch of interfaces that the BOs implement, then when the BOL calls the DAL interface, it passes it an object which implements one of these BO interfaces and then the DAL proceeds to populate the object. This removes all dependencies between the BOL and the DAL - however, I'm finding it hard to justify it to be honest.
Ideally we would like to use an ORM as it just removes the need to write CRUD stuff but our clients have a habit of fiddling with column lengths on their database and this is the cause of most of our errors to-date using the strongly typed DataTables. I've heard Linq2SQL also stores column lengths at compile time, not sure if NHibernate does or not (but, I'm not sure if our Database Schema is designed cleanly enough for NHibernate, pitfalls of dealing with legacy systems).
So yea, any insight on the relationship between a BOL and a DAL would be very much welcome - apologies if the above is poorly written, if anyone needs clarification I'll be happy to provide more detail.
Marlon

The way the I do this is the BO expects a DataReader or a DataContext or whatever back from the DAL, not the actual formed object. It is then the job of the BO layer to take and fill itself from the object that came back. The DAL isn't returning back a completed BO. The main thing to remember is that changing something in the BO layer shouldn't cause issues for the DAL layer, but changing something in the DAL layer could cause issues for the BO layer.
A short example of what I typically do
In the BO layer
FillData(){
DataReader dr = DataLayer.GetData("SomePropertyForAStoreProcedure");
If dr.Read(){
Property1 = dr.GetValue("Property1");
//So on and so forth
}
}
In the DAL
DataReader GetData(String SPProperty){
}

take a look at SubSonic http://subsonicproject.com/ it does most of the data access tedious work for you and it's easier than most ORMs out there

The DAL needs a reference to the BOL so that it can populate the objects. What you do not want to have is any reference or coupling from the BOL back to the DAL - doing so causes your BOL to be coupled to a specific database implementation. When you think about it this makes sense. Your DAL knows details about the business objects down to the level of properties and how to retrieve their data from the database - of course the DAL is inherently tightly coupled to the BOL. So the reference that way is fine. And if you think about it what is on the other side? The database. "Tightly coupling" going from your object data to your database? Yeah, it is pretty darn tight. The concept is not very meaningful even.
It is all the other direction where you need to decouple. So yes as long as there is no direct coupling from the DAL into the BOL you can change your data platform anyway you want.
Not much point in creating interfaces for BOs and passing them to DAL in this scenario. You might sometimes need to go the other way however. As a rule business objects should not have to know anything about how they are either created or persisted.
In practice even with most ORMs, for example, creating a business layer completely free of any sort of persistence artifacts can become very difficult, sometimes effectively not possible. So occasionally you have something that is just too difficult to work around though, and you might find that strictly avoiding having any data knowledge in BOs is leading you to over complexity that is degrading rather than adding value.
If you feel like there is no better way and you need to have something persisted from within the BOL, create a simple interface so that the DAL functionality can be passed into the BOL. That way you can still keep the BOL decoupled from the specific database implementation at least.
Also, although it is a lot of additional work, unless this is a very simple throwaway app, I strongly recommend that you also add another layer between the UI and the BOL. The MVP (Model-View-Presenter) pattern is a general purpose design pattern for reducing coupling between the core app and the UI. There are a lot of variants on presenters, don't get too caught up in the specific details, just start off with the simple MVP if you have never used it.
The patterns is not that hard, it is just that UI itself is so messy that it may take you at least a couple of major iterations / applications before you feel like the code you are writing at any time is systematically and methodically working to decouple the UI. Just keep working at it, start to acquire an arsenal of techniques, and don't get hung up on the fact that you really have not achieved a sharp clean separation yet. Anything and everything you learn and can do that even contributes a little to creating creating a well defined boundary at the UI is a big step in the right direction.

The 'correct' approach is going to vary depending on business needs. To be honest, there are many projects where I feel the old style ado recordsets incurred less development time and were easier to maintain than many of the ORM's out now. Take some time to identify what your needs are, and remember that development time and maintainability are design goals that should be properly weighed as well.

It also depends on if/what library/ORM (Object-Relational Mapper) you use. When using a (good) ORM, the DAL should be a very remote concern, because it is almost completely hidden by the ORM; however, best practices dictate that even then, for medium to large size applications, you should introduce another layer between the BOL and ORM, usually DTO (Data Transfer Objects). DTOs can also be used without an ORM, as they are just dumb objects defined in a separate library, and the DAL can be responsible for persisting them (transforming them from/to database structures), while the BOL can query the DAL and receive those objects.
Decoupling the layers can be achieved in a variety of ways, most commonly through interfaces and/or MEF or another DI/IOC framework. Any such technique achieves more than sufficient decoupling if used effectively.
Also, depending on the technology used, as Sisyphus said, one of the layered architectural patterns will help separate concerns nicely: MVC, MVP, MVVM etc. I personally recommend MVVM with WPF (desktop) or Silverlight (web) but I'm highly biased - i.e. I love both of them to death :)

These are my findings,
1. Use interfaces
2. Use DTOs [Data Transfer Objects] between DAL & BLL
3. Split BLL into two,
a. BLL
b. Service Layer
4. Use Inversion of Control (IoC) container for keep coupling as low as possible.

Generic business layer design with repository pattern with c# (how)

I am using a repository design with web applications (repository (data layer) exposing model (objects) to the business layer which is then consumed to the data layer (ui). Objects or lists of objects are passed between the layers with this type of implementation.
I am finding my business layer is a becoming a series of manager type classes which all have common GetAll, GetById, Save, Delete type methods. This is very common with a number of very small simple objects. This is the area of concern or opportunity for improvement (the series of smaller business manager classes). I am looking for options to avoid the whole series of smaller business manager classes mapping to the smaller objects which only do get/save/delete object.
The bigger objects which are closer to the functionality to the application have a number of methods in addition to the get/save/delete type methods (these manager classes are ok).
I am thinking there is a design pattern or implementation which will allow me to have one manager class which resides in the business layer which would accept an object as a parameter of a particular object type and the get/save/delete methods respectively know the type of repository object to spin up and pass the object to it for its operation.
The benefit here would be that I can have one generic manager class to pass save/delete/get's for smaller type objects to the appropriate repository class thereby reducing the many smaller manager classes.
Ideas on how to accomplish this?
thx

I would not go that way. The business layer classes can be as simple as code that forwards to the data layer, and it is true that they can be annoying to write, but they exist for a couple of reasons: validation, security, taking some actions based on business rules.
If you try to make a generic business layer, it will hard to include all the various things that a business class could do. The generic business layer will become much more complex than the one you currently have. Testing will be much harder. Adding a new business rule will be hard, too.
Sorry, this is not what you wanted to read, but I have already gone the route of generic systems and have always had lots of regrets.

The idea behind a repository (or a dao), is to further abstract data access concerns away from the business layer in order to simplify that layers focus on the "business" of a given domain.
That said, there are many common plumbing type of concerns that are reuseable across different applications, some of which do lend themselves to a supertype in the business layer. Consider the cross cutting concern of being able to retrieve a given business entity by some Id from a database, and you might come to the conclusion that it is in fact useful to have an Id property in a business layer supertype. It might even be useful if entities considered that Id when determining equality. Etc..
Now I do believe that Timores is right in principal and trying to write one application that fits all domains is both incredibly painful and totally fruitless, but I also beleive the the art of the profession is knowing how to use a variety of tools and when to apply which one, and having some core infrastructure code should be in your tool belt.
For a good idea of a framework concept for a web app that has been road tested, take a look at SharpArch.
HTH,
Berryl