We Keep Coding

Entity Framework dependencies loading

I have a long-time burning question about how to avoid null errors with data queried via Entity Framework (version 6 - not Core yet, sadly).
Let's say you have a table Employees, and it has a relationship with another table, EmployeePayments (one employee has many employee payments).
On your Employee domain object you create a property TotalPayments which relies on you having loaded the EmployeePayments for that object.
I try to ensure that any time I do a query, I "include" the dependency, for example:
var employees = context.Employees.Include(e => e.EmployeePayments);
The problem is, I have a lot of queries around the place (I use the generic repository pattern, so I call repository functions like GetAll or GetSingle from my service library), and so that's a lot of places to remember to add the includes. If I don't include them, I run the risk of having a null exception if the TotalPayments property is used.
What's the best way to handle this?
Note 1: we have a lot of tables and I don't really want to have to revert to using specific repositories for each one, we take advantage of the generic repository in a lot of ways.... but I will be happy to hear strong arguments for the alternative :)
Note 2: I do not have lazy loading turned on, and don't plan on turning it on, for performance reasons.

This is one reason I consider the Generic Repository an anti-pattern for EF. I use a repository pattern, but scope it like I would a controller. I.e. a CreateOrderController would have a CreateOrderRepository. This repository would provide access to all relevant entities via IQueryable. Common stuff like lookups etc. would have their own secondary repository. Using generic repositories that are geared to working with a single entity type mean adding references to several repositories to do specific things and running into issues like this when attempting to load entities. Sometimes you want related data, sometimes you don't. Simply adding convenient methods in top level entities effectively "breaks" that an object should always be considered complete or complete-able without relying on lazy-loading which brings significant performance costs.
Having repositories return IQueryable avoids many of the problems by giving control to the calling code how entities are consumed. For instance I don't put helper methods in the entities, but rather code needing to populate a view model relies on Linq to build the view model. If my view model wants a sum of payments for an employee, then my repository returning IQueryable can do the following:
public IQueryable<Employee> GetEmployeeById(int employeeId)
{
return Context.Employees.Where(x => x.EmployeeId == employeeId);
}
then in the controller / service:
using (var contextScope = ContextScopeFactory.Create())
{
var employeeViewModel = EmployeeRepository.GetEmployeeById(employeeId)
.Select(x => new EmployeeSummaryViewModel
{
EmployeeId = x.EmployeeId,
EmployeeName = x.LastName + ", " + x.FirstName,
TotalPayments = x.Payments.Where(p => p.IsActive).Sum(p => p.Amount)
}).Single();
}
I use a repository because it is easier to mock out than the DbContext and it's DbSets. For Synchronous code I just have the mock to populate and return List<Employee>().AsQueryable(). For Async code I need to add a wrapper for an Async List.
This pattern may go against more traditional views of a repository and separation of concerns that the calling code needs to "know" about the entities, that EF-isms are leaked. However, no matter what approach you try to rationalize to get around the inefficiencies of trying to "hide" EF behind a repository, either you will be left with very inefficient code where repositories return pre-populated DTOs or contain dozens of near identical methods to return different DTOs (or worse, entities in various degrees of completeness) or you are adding complexities like passing in magic strings or expression trees into your methods to tell EF how to filter, how to sort, what to include, paging, etc. Passing in expressions or strings requires the calling code to "know" about the entities and leaks EF restrictions. (Passed in expressions / strings still have to be able to be ultimately understood by EF)
So this may not be a viable answer to the current state of your project, but it might be worth looking into whether your dependency on the repositories can be better managed without splitting them with the Generic pattern, and/or leveraging EF's excellent IQueryable / Linq capabilities to let your controllers/services project the entities into view models / DTOs rather than embedding these reduce elements in the entities themselves.

Repository and query objects pattern. How to implement complex queries

I have read a lot of posts of the repository pattern but there are a few practical problems that they doesn't seem to solve or explain. This is what I understand about this two patterns:
The repository and the query pattern are complementary: Query objects represents business logic (WHERE clauses) and the repository pattern has a Get(IPredicate) method that takes a query object and returns a SELECT WHERE result
The repository should not have business logic: All the business logic must go on the query objects
Currently I have a class that wraps each logical object (which almost always is a single entity object) that implements multiple "Get" methods that implement most complex queries (joins, groupBy, etc...), this isn't a good pattern because classes tend to grow a lot because of boilerplate code for similar queries and its public methods are dependent on context that this class will be used, thus, making this classes unusable for multiple projects that depends on the same database, which is my primary goal for this project refactoring.
How queries that are more complex than a single SELECT WHERE are implemented with this two patterns without leaking business logic into the repository?
Or if this business logic doesn't fit into the repository nor the query objects
where does this logic fit?

The Repository pattern works well for standard CRUD applications. Where you need to implement the classic set of create, read, update and delete operations against a single table in a database. In that case you create a repository for each table and allow the read operation to have extra values so that filtering can be applied.
At the next level up you have the Unit of Work pattern. These are used to span multiple repositories and perform business operations. So for example, you would read values from multiple repositories, perform calculations and then write back changes to multiple repositories. All of that would occur inside a transaction so that you always have a consistent state in the database.
The problem is when you have complex queries that span multiple tables. In that case you would place the query into the repository that is the first table in the query from clause. Then you would need to provide parameters to that repository method so it can be parameterised as needed.

There are quite a few implementations of repository patterns and unit of work flying around on the internet. Some of them are quite simple where developer basically implements his own for each table himself manually, some are generic but not advanced, and some are really cool, generic and still offer you the ability to do a decent where, projection and the like.
An example of an in my opinion good implementation can be found here :
https://genericunitofworkandrepositories.codeplex.com/
It is targetting MVC, which is shown by the interface. I am focussing on WPF applications so I needed to tune it a bit. But the ideas of this unit of work implementation are pretty good.
There is down side to this implementation. Because it is relying on some advanced LINQ and EF functionality one could argue that your underlying access layer is infecting the repository layer and the layers using the repositories.
The point being that when for instance you want to move away from EF, chances are that you would have to change the interface of your repositories.
To show the power of this library some code snippets to prove this :
_fotoRepository = unitOfWork.RepositoryAsync<Foto>();
var fotos = await _fotoRepository
.Query(r => r.BestelBonId == bestelBonId || werkstukids.Contains(r.WerkstukMetBewerkingenId.Value))
.SelectAsync()
.ConfigureAwait(false);
or using projection:
IRepository<Relatie> relatieRepository = unitOfWork.RepositoryAsync<Relatie>();
var relatiesOverviewsEnumerable = relatieRepository
.Query()
.NoTracking()
.OrderBy(q => q.OrderBy(d => d.RelatieId))
.Select(b => new RelatieOverview
{
RelatieId = b.RelatieId,
Naam = b.Naam,
BTW = b.BTW,
HoofdAdres = b.Adressen.FirstOrDefault(a => a.AdresTypeId == HoofdadresType)
});
_relatieOverviews = new ObservableCollection<RelatieOverview>(relatiesOverviewsEnumerable);

Entity Framework vs repository pattern

I'm wondering about usefulness of repository pattern in project using Entity Framework. There are opposite claims about this - some say that EF is implementation of repository and Unit of Work pattern itself, so there is no need to wrap it in next abstraction layer, some argue that it has advantages like separation of DAL and BL and easier creation of unit tests. In my experience, I often come across the following approach (generally, not only in EF projects):
Repository (DAL) <-> Service (BL) <-> Controller
Repository + Service + Type = Model
Repository has methods responsible for data access only i.e.:
public interface IUsersRepository
{
IEnumerable<User> GetAll();
User Get(int id);
User GetByLogin(string login);
void Update(User item);
void Delete(int id);
void Delete(User item);
// ...
}
Often, there is used generic repository instead, which methods receive functions for filtering, sorting etc.
Service in turn uses repository and contains business logic i.e.:
public interface IUsersService
{
// ...
bool VerifyPassword(string login, string password);
void ChangePassword(string login, string password);
// ...
}
As I understand, service shouldn't have any DAL operations - this means that we shouldn't i.e. return IQueryable collections from repository, because then query will be executed outside repository and unit tests will not be fully reliable (differences between LINQ-to-Entities and LINQ-to-Objects).
I see a problem with queries efficiency here - we will often get more data from database than needed and then filter it in memory.
For example, let's consider bool VerifyPassword(string login, string password) method.
In this case we need to get whole User entity from database, which can have 50 properties for example, for password verification purpose only. We can, of course, create many methods in repository like :
string GetPasswordHash(string login)
or
bool VerifyPassword(string login, string passwordHash)
{
return db.Users.Any(x => x.Login = login && x.Password = passwordHash);
}
without need to get whole entity from database, but in my opinion this can be a "little" overhead.
We can also move VerifyPassword function from service to repository - then we should ask ourself a question whether two layers - repository and service are needed. But if we merge them, we will lose benefits of separating DAL and BL layers - unit tests would be integration tests in reality. So maybe it would be simplier (and better) to keep it all in controller and inject mocked DbContext or use something like Effort for unit tests?
I will be grateful for your answers how you see this and how you solve this problem in your projects.
UPDATE
I understand that repository pattern allows to easily change data source/provider to other like nHibernate, LINQ-to-SQL, plain ADO and so on.
But could you tell me how do you implement sorting and paging in your repository API? I mean what's the best way to pass sorting and paging specification to repository? I see that some passes function predicates for LINQ but this will tightly couple repository with LINQ - using it with plain ADO, stored procedures etc. would be problematic. Creating many methods like GetUsersOrderedByNameDesc() etc. is crazy in my opinion. So maybe create your own class for specification, forge and pass sorting/paging criterias, and process it in repository? If yes, can you provide me some example of implementation?

I will try to attempt your core question. Here are my two cents.
Why repository vs entity framework. You will need to answer some of the questions I propose you ask when designing or picking which one:
Is there a chance your project will have different repositories it will talk to? If yes, then repository pattern over entity framework makes sense.
Having a repository pattern will also allow you to set up your test framework more easily where you can have your mockup db that the repository pattern (DAL) will use.
You may also want to consider the changes to your data model can percolate back to changes in your business logic with the entity framework.

The responsibilities of my service and repository layer

The other day I asked this question:
Should the repository layer return data-transfer-objects (DTO)?
The answer (well by just one person, but I already had a hunch that it wasn't a good idea) was that no, the repository later should not have to deal with the DTO objects (their purpose is purely to be sent over the wire) and the service layer should deal with that.
Now I've come up with a construction in the meantime that I need your opinion on. The idea is that, when it makes sense to do so, the repository layer can return an interface type I've defined called IProjectable. This wraps the query (the repository layer does not execute the query yet) but does not allow the consumer to change the query (it's not IQueryable), just to perform projection operations on it (so far for me only First and ToPagedList) that would perform the projection and actually execute the query.
So something like this in the repository:
public IProjectable<User> GetUser(int id)
{
var query = from u in Set<User>()
where u.UserID == id
select u;
return query.AsProjectable();
}
And in the service layer something like this:
var dto = repository.GetUser(16).Single(u => new SimpleUserDto
{
FullName = u.FirstName + " " + u.LastName,
DisplayAddress = u.Address.Street + u.Address.HouseNumber,
OrderCount = u.Orders.Count()
});
return dto;
Am I correct in saying that doing the actual data access here is still the responsibility of the repository layer (as it should be) and that the projection to a serializable form is the responsibility of the service layer (as it should be)?
The only other way I see to do this efficiently (returning a User from the repository and doing the Count() on his Orders in the service layer would result in an extra query to the database) is to define a type that has all these properties and return it from the repository layer and just don't call it "Dto", which seems silly as it would be identical to the DTO just not named the same for the sake of "purity". This way, it seems, I can have my cake and eat it for the most part too.
The downside I see is that you can get a mismatch where the service layer performs projections that can't actually be translated to SQL which it shouldn't have to worry about, or where it performs such complex projections that makes it questionable what layer is doing the actual data access.
I'm using Entity Framework 4 by the way, if it matters.

Am I correct in saying that doing the
actual data access here is still the
responsibility of the repository layer
(as it should be) and that the
projection to a serializable form is
the responsibility of the service
layer (as it should be)?
Yes you are, the service layer still has no idea how the actual DataAccess is being performed (as it should not have not). Are the calls send to SQL? is there a caching layer in between?
The downside I see is that you can get
a mismatch where the service layer
performs projections that can't
actually be translated to SQL which it
shouldn't have to worry about, or
where it performs such complex
projections that makes it questionable
what layer is doing the actual data
access.
For this problem i use a pipeline pattern which basicly is just a set of extension methods over IProjectable which can perform tested projections. Next, in your serviceLayer you can just write your query using a composition of these pipeline methods, for example:
var users = repository.GetUsers().FilterByName("Polity").OrderByAge().ToTransferObjects();

One of developers I most respect ayende (http://ayende.com/Blog/Default.aspx) said: "ORM is your repository" video here -> http://community.devexpress.com/blogs/seth/archive/2011/03/09/interview-with-ayende-rahien-aka-oren-eini.aspx
Question is do you really need Repository pattern?
Just my opinion :)

Does Queryability and Lazy Loading in C# blur the lines of Data Access vs Business Logic?

I am experiencing a mid-career philosophical architectural crisis. I see the very clear lines between what is considered client code (UI, Web Services, MVC, MVP, etc) and the Service Layer. The lines from the Service layer back, though, are getting more blurred by the minute. And it all started with the ability to query code with Linq and the concept of Lazy loading.
I have created a Business Layer that consists of Contracts and Implementations. The Implementations then could have dependencies to other Contracts and so on. This is handled via an IoC Container with DI. There is one service that handles the DataAccess and all it does is return a UnitOfWork. This UnitOfWork creates a transaction when extantiated and commits the data on the Commit method. [View this Article (Testability and Entity Framework 4.0)]:
public interface IUnitOfWork : IDisposable {
IRepository<T> GetRepository<T>() where T : class;
void Commit();
}
The Repository is generic and works against two implementations (EF4 and an InMemory DataStore). T is made up of POCOs that get generated from the database schema or the EF4 mappings. Testability is built into the Repository design. We can leverage the in-memory implementation to assert results with expectations.
public interface IRepository<T> where T : class {
IQueryable<T> Table { get; }
void Add(T entity);
void Remove(T entity);
}
While the Data Source is abstracted, IQueryable still gives me the ability to create queries anywhere I want within the Business logic. Here is an example.
public interface IFoo {
Bar[] GetAll();
}
public class FooImpl : IFoo {
IDataAccess _dataAccess;
public FooImpl(IDataAccess dataAccess) {
_dataAccess = dataAccess;
}
public Bar[] GetAll() {
Bar[] output;
using (var work = _dataAccess.DoWork()) {
output = work.GetRepository<Bar>().Table.ToArray();
}
return output;
}
}
Now you can see how the queries could get even more complex as you perform joins with complex filters.
Therefore, my questions are:
Does it matter that there is no clear distinction between BLL and the DAL?.
Is queryability considered data access or business logic when behind a Repository layer that acts like an InMemory abstraction?
Addition: The more I think about it, maybe the second question was the only one that should have been asked.

I think the best way to answer your questions is to step back a moment and consider why separation between business logic layers and data access layers is the recommended practice.
In my mind, the reasons are simple: keep the business logic separate from the data layer because the business logic is where the value is, the data layer and business logic will need to change over time more or less independently of each other, and and the business logic needs to be readable without having to have detailed knowledge of what all the data access layer does.
So the litmus test for your query gymnastics boils down to this:
Can you make a change to the data schema in your system without upsetting a significant portion of the business logic?
Is your business logic readable to you and to other C# developers?

1. Only if you care more about philosophy than getting stuff done. :)
2. I'd say it's business logic because you have an abstraction in between. I would call that repository layer part of DAL, and anything that uses it, BL.
But yeah, this is blurry to me as well. I don't think it matters though. The point of using patterns like this is to write a clean, usable code that easy to communicate at the same time, and that goal is accomplished either way.

1.Does it matter that there is no clear distinction between BLL and the DAL?.
It sure does matter! Any programmer that uses your Table property needs to understand the ramifications (database roundtrip, query translation, object tracking). That goes for programmers reading the business logic classes as well.
2.Is queryability considered data access or business logic when behind a Repository layer that acts like an InMemory abstraction?
Abstraction is a blanket that we hide our problems under.
If your abstraction is perfect, then the queries could be abstractly considered as operating against in-memory collections and therefore they are not data access.
However, abstractions leak. If you want queries that make sense in the data world, there must be effort to work above and beyond the abstraction. That extra effort (which defeats abstraction) produces data access code.
Some examples:
output = work.GetRepository<Bar>().Table.ToArray();
This is code is (abstractly) fine. But in the data world it results in scanning an entire table and is (at least generally) dumb!
badquery = work.GetRepository<Customer>().Table.Where(c => c.Name.Contains("Bob")).ToArray();
goodquery = work.GetRepository<Customer>().Table.Where(c => c.Name.StartsWith("Bob")).ToArray();
Goodquery is better than bad query when there's an index on Customer.Name. But that fact is not available to us unless we lift the abstraction.
badquery = work.GetRepository<Customer>().Table
.GroupBy(c => c.Orders.Count())
.Select(g => new
{
TheCount = g.Key,
TheCustomers = g.ToList()
}).ToArray();
goodquery = work.GetRepository<Customer>().Table
.Select(c => new {Customer = c, theCount = c.Orders.Count())
.ToArray()
.GroupBy(x => x.theCount)
.Select(g => new
{
TheCount = g.Key,
TheCustomers = g.Select(x => x.Customer).ToList()
})
.ToArray();
goodquery is better than bad query since badquery will requery the database by group key, for each group (and worse, it is highly unlikely there is an index to help with filtering customers by c.Orders.Count() ).
Testability is built into the Repository design. We can leverage the InMemory implementation to assert results with expectations.
Be under no illusions that your queries are being tested if you actually run them against in-memory collections. Those queries are untestable unless a database is involved.