We Keep Coding

Is repository pattern only to be used with Entity Framework?

I have been using Entity Framework and the repository pattern for some time now.
I was asked the other day to write a data layer without using Entity Framework, just plain old ADO.NET. I was wondering what would be the best approach for this? Do I also use a repository pattern for my CRUD operations using plain old ADO.NET?
If I go to Codeplex and search for repository pattern then 99.9% of all the sample projects use Entity Framework. Is there a different pattern that needs to be used if I use plain ADO.NET with stored procedures?

No, the repository pattern is used extensively outside of the Entity Framework, and is an all round useful way of handling data access.
From MSDN
It centralizes the data logic or Web service access logic.
It provides a substitution point for the unit tests.
It provides a flexible architecture that can be adapted as the overall design of the application evolves.
http://msdn.microsoft.com/en-us/library/ff649690.aspx
Other benefits:
Simple to add logic in the repository, such as caching results over a web request
Common query's can be added, such as userRepository.FindByEmailAddress(emailAddress);
Repository can be changed out with another, such as switching a dabase to a web service with minimal effort

I don't think this is the right way. But there are some assumptions
Adding a Repository pattern on top of EF code. This keep distances you from the features of your ORM. The Entity Framework is already an abstraction layer over your database.
If you want to use the Dependency Injection and Test Driven Development over EF then you follow the Repository Pattern. By using RP your code become testable and Injectable / maintainable.
Out of the box EF is not very testable, but it's quite easy to make a mockable version of the EF data context with an interface that can be injected.
If we don’t want our code to be testable or injectable then just don’t use RP.
I saw a blog post: http://www.nogginbox.co.uk/blog/do-we-need-the-repository-pattern

Martin Fowler's "Patterns of Enterprise Architecture", provides the following definition for a repository:
Mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.
A common way to implement that in C# is to have a generic Repository<T> class where T is a persistent object that implements IQueryable<T> and provides additional methods like Add(entity), Remove(entity).
It would be very difficult to implement without an ORM. You can make a simpler repository that takes SQL statements as WHERE conditions but it can get messy.
Numerous examples use concrete repository classes for each type with different persistence methods. But those are just disguised DAO classes.

DDD - Duplication between service layer and repositories

I've worked on several applications that try to adhere to DDD principles, I noticed that we end up with situations where there is duplication between the Service Layer and the repositories that feels like a code smell.
For most of the operations in the Service layer, it seems like it is a direct mapping to CRUD operations, GetAll, GetById, Create, Delete etc.. the flow of the architecure is within these lines: I have a controller calling a Service layer that calls a Repository that calls an ORM which talks to the Backend ..
So for example GetAll would exist in both SL and Repository. Now, if we have a change/business requirement that GetAll should ignore certain items, how am I supposed to do it, should I ignore these in the repository, or that's business logic that should go in the Service Layer? Wouldn't life be easier if we just had a Service Layer calling the ORM directly?
To Sum up: I understand the Service Layer could abstract some businees logic but when - in most cases - it is dealing with simple CRUD operations, wouldn't it be easier to just get rid of the Repository? But, what if the SL also contains some methods with complicated business logic, should these go through a repository? From good design perspective, Should I favor consistency and always go through repository or just keep it simple and only use a repository when it is not a simple one-to-one mapping to a CRUD operation.
PS: I realize there are seemingly similar questions but didn't find any satisfying answer

I noticed that we end up with situations where there is duplication
between the Service Layer and the repositories that feels like a code
smell.
It isn't a code smell since they do different things.
You should keep in mind that Domain or Application Services reside in a different layer than Repository implementations. Layers are there for a reason - objects in different layers don't have the same responsibilities and don't talk to the same neighbors. Repository implementations are tightly coupled to the means of persistence of your objects. They might generate SQL statements and talk to a relational database, they might talk to your ORM... the important thing is that they know about the way your objects are persisted, which is not true of Application Services.
If your Services layer was to call the ORM directly, it would really do 2 big things, breaking the Single Responsibility Principle. It would also be more difficult to change your ORM for another one or for a different means of persistence.
So for example GetAll would exist in both SL and Repository. Now, if
we have a change/business requirement that GetAll should ignore
certain items, how am I supposed to do it, should I ignore these in
the repository, or that's business logic that should go in the Service
Layer?
If GetAll() ignores certain items, I strongly suggest renaming it both in the Service and the Repository to reflect that, eg : GetAllAllowedToUser(), GetAllBut...(). Thus the contract of the method will be clear and you'll avoid misunderstandings about what it's supposed to return. Plus you'll be able to keep the original genuine GetAll() method which could still be of some use.

in most cases - it is dealing with simple CRUD operations, wouldn't it
be easier to just get rid of the Repository
IMHO, I wouldn't say to get rid of Repository. I would say that if you are doing CRUD you don't need DDD (at all). If you read Fowler's enterprise patterns or Evans they both say that DDD is only of use when you have domain logic that is significantly complex. CRUD is not complex and therefore no DDD needed.
What you describe is a code smell. But I don't think it is a smell with DDD. You are just seeing an over-engineered piece of code.

+1 for Dtryon, also:
Now, if we have a change/business requirement that GetAll should ignore certain item
Not directly related and I know you just used it as an example, but I've seen this exact thing. Please don't end up with methods called GetAll that do not get all. Keep GetAll, have it GetAll, then have GetAllLive, or GetAllAvailable or something like that as well which does what it says it does

Maybe the "Finder pattern"(dont know if this is the right term) can solve your problem. According to CQS(Command-Query-Separation) principle, (IMO,) query operations are not "business logic" at all. We can write some specific "Finder"s in the Infrastructure Layer to perform various queries and let all non-query operations(business logic) stay in the Service Layer, then on the client side we treat the finders the same as services.
Sorry for my language :-(.

MVC, ORM, and data access patterns

I think I've hit that "paralysis by analysis" state.
I have an MVC app, using EF as an ORM.
So I'm trying to decide on the best data access pattern, and so far I'm thinking putting all data access logic into controllers is the way to go.. but it kinda doesn't sound right.
Another option is creating an external repository, handling data interactions.
Here's my pros/cons:
If embedding data access to controllers, I will end up with code like this:
using (DbContext db = new DbContext())
{
User user = db.Users.Where(x=>x.Name == "Bob").Single();
user.Address.Street = "some st";
db.SaveChanges();
}
So with this, I get full benefits of lazy loading, I close connection right after I'm done, I'm flexible on where clause - all the niceties.
The con - I'm mixing a bunch of stuff in a single method - data checking, data access, UI interactions.
With Repository, I'm externalizing data access, and in theory can just replace repos if I decide to use ado.net or go with different database.
But, I don't see a good clean way to realize lazy loading, and how to control DbContext/connection life time.
Say, I have IRepository interface with CRUD methods, how would I load a List of addresses that belong to a given user ? Making methods like GetAddressListByUserId looks ugly, wrong,
and will make me to create a bunch of methods that are just as ugly, and make little sense when using ORM.
I'm sure this problem been solved like million times, and hope there's a solution somewhere..
And one more question on repository pattern - how do you deal with objects that are properties ? E.g. User has a list of addresses, how would you retrieve that list ? Create a repository for the address ? With ORM the address object doesn't have to have a reference back to user, nor Id field, with repo - it will have to have all that. More code, more exposed properties..

The approach you choose depends a lot on the type of project you are going to be working with. For small projects where a Rapid Application Development (RAD) approach is required, it might almost be OK to use your EF model directly in the MVC project and have data access in the controllers, but the more the project grows, the more messy it will become and you will start running into more and more problems. In case you want good design and maintainability, there are several different approaches, but in general you can stick to the following:
Keep your controllers and Views clean. Controllers should only control the application flow and not contain data access or even business logic. Views should only be used for presentation - give it a ViewModel and it will present it as Html (no business logic or calculations). A ViewModel per view is a pretty clean way of doing it.
A typical controller action would look like:
public ActionResult UpdateCompany(CompanyViewModel model)
{
if (ModelState.IsValid)
{
Company company = SomeCompanyViewModelHelper.
MapCompanyViewModelToDomainObject(model);
companyService.UpdateCompany(company);
return RedirectToRoute(/* Wherever you go after company is updated */);
}
// Return the same view with highlighted errors
return View(model);
}
Due to the aforementioned reasons, it is good to abstract your data access (testability, ease of switching the data provider or ORM or whatever, etc.). The Repository pattern is a good choice, but here you also get a few implementation options. There's always been a lot of discussion about generic/non-generic repositories, whether or not one should return IQueryables, etc. But eventually it's for you to choose.
Btw, why do you want lazy loading? As a rule, you know exactly what data you require for a specific view, so why would you choose to fetch it in a deferred way, thus making extra database calls, instead of eager loading everything you need in one call? Personally, I think it's okay to have multiple Get methods for fetching objects with or without children. E.g.
public class CompanyRepository
{
Get(int Id);
Get(string name);
GetWithEmployees(int id);
...
}
It might seem a bit overkill and you may choose a different approach, but as long as you have a pattern you follow, maintaining the code is much easier.

Personally I do it this way:
I have an abstract Domain layer, which has methods not just CRUD, but specialized methods, for example UsersManager.Authenticate(), etc. It inside uses data access logic, or data-access layer abstraction (depending on the level of abstraction I need to have).
It is always better to have an abstract dependency at least. Here are some pros of it:
you can replace one implementation with another at a later time.
you can unit test your controller when needed.
As of controller itself, let it have 2 constructors: one with an abstract domain access class (e.g. facade of domain), and another (empty) constructor which chooses the default implementation. This way your controller lives well during web application run-time (calling empty constructor) and during the unit-testing (with mock domain layer injected).
Also, to be able to easily switch to another domain at a later time, be sure to inject the domain creator, instead of domain itself. This way, localizing the domain layer construction to the domain creator, you can switch to another implementation at any time, by just reconstructing the domain creator (by creator I mean some kind of factory).
I hope this helps.
Addition:
I would not recommend having CRUD methods in domain layer, because this will become a nightmare whenever you rich the unit-testing phase, or even more, when you need to change the implementation to the new one at a later time.

It really comes down to where you want your code. If you need to have data access for an object you can put it behind an IRepository object or in the controller doesn't matter: you will still wind up with either a series of GetByXXX calls or the equivilent code. Either way you can lazy load and control the lifetime of the connection. So now you need to ask yourself: where do I want my code to live?
Personally, I would argue to get it out of the controller. By that I mean moving it to another layer. Probably using an IRespository type of pattern where you have a series of GetByXXX calls. Sure they are ugly. Wrong? I would argue otherwise. At least they are all contained within the same logical layer together rather than being scattered throughout the controllers where they are mixed in with validation code, etc.

Is DTO plus UnitOfWork pattern a good approach to design a DAL for a web application?

I'm implementing a DAL using entity framework. On our application, we have three layers (DAL, business layer and presentation). This is a web app. When we began implementing the DAL, our team thought that DAL should have classes whose methods receive a ObjectContext given by services on the business layer and operate over it. The rationale behind this decision is that different ObjectContexts see diferent DB states, so some operations can be rejected due to problems with foreign keys match and other inconsistencies.
We noticed that generating and propagating an object context from the services layer generates high coupling between layers. Therefore we decided to use DTOs mapped by Automapper (not unmanaged entities or self-tracking entities arguing high coupling, exposing entities to upper layers and low efficiency) and UnitOfWork. So, here are my questions:
Is this the correct approach to design a web application's DAL? Why?
If you answered "yes" to 1., how is this to be reconciled the concept of DTO with the UnitOfWork patterns?
If you answered "no" to 1., which could be a correct approach to design a DAL for a Web application?
Please, if possible give bibliography supporting your answer.
About the current design:
The application has been planned to be developed on three layers: Presentation, business and DAL. Business layer has both facades and services
There is an interface called ITransaction (with only two methods to dispose and save changes) only visible at services. To manage a transaction, there is a class Transaction extending a ObjectContext and ITransaction. We've designed this having in mind that at business layer we do not want other ObjectContext methods to be accessible.
On the DAL, we created an abstract repository using two generic types (one for the entity and the other for its associated DTO). This repository has CRUD methods implemented in a generic way and two generic methods to map the DTOs and entities of the generic repository with AutoMapper. The abstract repository constructor takes an ITransaction as argument and it expects the ITransaction to be an ObjectContext in order to assign it to its proctected ObjectContext property.
The concrete repositories should only receive and return .net types and DTOs.
We now are facing this problem: the generic method to create does not generate a temporal or a persistent id for the attached entities (until we use SaveChanges(), therefore breaking the transactionality we want); this implies that service methods cannot use it to associate DTOs in the BL)

There are a number of things going on here...The assumption I'll make is that you're using a 3-Tier architecture. That said, I'm unclear on a few design decisions you've made and what the motivations were behind making them. In general, I would say that your ObjectContext should not be passed around in your classes. There should be some sort of manager or repository class which handles the connection management. This solves your DB state management issue. I find that a Repository pattern works really well here. From there, you should be able to implement the unit of work pattern fairly easily since your connection management will be handled in one place. Given what I know about your architecture, I would say that you should be using a POCO strategy. Using POCOs does not tightly couple you to any ORM provider. The advantage is that your POCOs will be able to interact with your ObjectContext (probably via Repository of some sort) and this will give you visibility into change tracking. Again, from there you will be able to implement the Unit of Work (transaction) pattern to give you full control over how your business transaction should behave. I find this is an incredibly useful article for explaining how all this fits together. The code is buggy but accurately illustrates best practices for the type of architecture you're describing: Repository, Specification and Unit of Work Implementation
The short version of my answer to question number 1 is "no". The above link provides what I believe to be a better approach for you.

I always believed that code can explain things better than worlds for programmers. And this is especially true for this topic. Thats why I suggest you to look at the great sample application in witch all consepts you expecting are implemented.
Project is called Sharp Architecture, it is centered around MVC and NHibernate, but you can use the same approaches just replacing NHibernate parts with EF ones when you need them. The purpose of this project is to provide an application template with all community best practices for building web applications.
It covers all common and most of the uncommon topics when using ORM's, managing transactions, managing dependencies with IoC containers, use of DTOs, etc.
And here is a sample application.
I insist on reading and trying this, it will be a real trasure for you like it was for me.

You should take a look what dependency injection and inversion of control in general means. That would provide ability to control life cycle of ObjectContext "from outside". You could ensure that only 1 instance of object context is used for every http request. To avoid managing dependencies manually, I would recommend using StructureMap as a container.
Another useful (but quite tricky and hard to do it right) technique is abstraction of persistence. Instead of using ObjectContext directly, You would use so called Repository which is responsible to provide collection like API for Your data store. This provides useful seam which You can use to switch underlying data storing mechanism or to mock out persistence completely for tests.
As Jason suggested already - You should also use POCO`s (plain old clr objects). Despite that there would still be implicit coupling with entity framework You should be aware of, it's much better than using generated classes.
Things You might not find elsewhere fast enough:
Try to avoid usage of unit of work. Your model should define transactional boundaries.
Try to avoid usage of generic repositories (do note point about IQueryable too).
It's not mandatory to spam Your code with repository pattern name.
Also, You might enjoy reading about domain driven design. It helps to deal with complex business logic and gives great guidelines to makes code less procedural, more object oriented.

I'll focus on your current issues: To be honest, I don't think you should be passing around your ObjectContext. I think that is going to lead to problems. I'm assuming that a controller or a business service will be passing the ObjectContext/ITransaction to the Repository. How will you ensure that your ObjectContext is disposed of properly down stream? What happens when you use nested transactions? What manages the rollbacks, for transactions down stream?
I think your best bet lies in putting some more definition around how you expect to manage transactions in your architecture. Using TransactionScope in your controller/service is a good start since the ObjectContext respects it. Of course you may need to take into account that controllers/services may make calls to other controllers/services which have transactions in them. In order to allow for scenarios where you want full control over your business transactions and the subsequent database calls, you'll need to create some sort of TransactionManager class which enlists, and generally manages transactions up and down your stack. I've found that NCommon does an extraordinary job at both abstracting and managing transactions. Take a look at UnitOfWorkScope and TransactionManager classes in there. Although I disagree with NCommon's approach of forcing the Repository to rely on the UnitOfWork, that could easily be refactored out if you wanted.
As far as your persistantID issue goes, check this out

How many levels of abstraction do I need in the data persistence layer?

I'm writing an application using DDD techniques. This is my first attempt at a DDD project. It is also my first greenfield project and I am the sole developer. I've fleshed out the domain model and User interface. Now I'm starting on the persistence layer. I start with a unit test, as usual.
[Test]
public void ShouldAddEmployerToCollection()
{
var employerRepository = new EmployerRepository();
var employer = _mockery.NewMock<Employer>();
employerRepository.Add(employer);
_mockery.VerifyAllExpectationsHaveBeenMet();
}
As you can see I haven't written any expectations for the Add() function. I got this far and realized I haven't settled on a particular database vendor yet. In fact I'm not even sure it calls for a db engine at all. Flat files or xml may be just as reasonable. So I'm left wondering what my next step should be.
Should I add another layer of abstraction... say a DataStore interface or look for an existing library that's already done the work for me? I'd like to avoid tying the program to a particular database technology if I can.

With your requirements, the only abstraction you really need is a repository interface that has basic CRUD semantics so that your client code and collaborating objects only deal with IEmployerRepository objects rather than concrete repositories. You have a few options for going about that:
1) No more abstractions. Just construct the concrete repository in your top-level application where you need it:
IEmployeeRepository repository = new StubEmployeeRepository();
IEmployee employee = repository.GetEmployee(id);
Changing that in a million places will get old, so this technique is only really viable for very small projects.
2) Create repository factories to use in your application:
IEmployeeRepository repository = repositoryFactory<IEmployee>.CreateRepository();
IEmployee employee = repository.GetEmployee(id);
You might pass the repository factory into the classes that will use it, or you might create an application-level static variable to hold it (it's a singleton, which is unfortunate, but fairly well-bounded).
3) Use a dependency injection container (essentially a general-purpose factory and configuration mechanism):
// A lot of DI containers use this 'Resolve' format.
IEmployeeRepository repository = container.Resolve<IEmployee>();
IEmployee employee = repository.GetEmployee(id);
If you haven't used DI containers before, there are lots of good questions and answers about them here on SO (such as Which C#/.NET Dependency Injection frameworks are worth looking into? and Data access, unit testing, dependency injection), and you would definitely want to read Martin Fowler's Inversion of Control Containers and the Dependency Injection pattern).

At some point you will have to make a call as to what your repository will do with the data. When you're starting your project it's probably best to keep it as simple as possible, and only add abstraction layers when necessary. Simply defining what your repositories / DAOs are is probably enough at this stage.
Usually, the repository / repositories / DAOs should know about the implementation details of which database or ORM you have decided to use. I expect this is why you are using repositories in DDD. This way your tests can mock the repositories and be agnostic of the implementation.

I wrote a blog post on implementing the Repository pattern on top of NHibernate, I think it will benefit you regardless of whether you use NHibernate or not.
Creating a common generic and extensible NHiberate Repository

One thing I've found with persistence layers is to make sure that there is a spot where you can start doing abstraction. If you're database grows, you might need to start implementing sharding and unless there's already an abstraction layer already available, it can be difficult to add one later.

I believe you shouldn't add yet another layer below the repository classes just for the purpose of unit testing, specially if you haven't chosen your persistence technology. I don't think you can create an interface more granular than "repository.GetEmployee(id)" without exposing details about the persistence method.
If you're really considering using flat text or XML files, I believe the best option is to stick with the repository interface abstraction. But if you have decided to use databases, and you're just not sure about the vendor, an ORM tool might be the way to go.