We Keep Coding

Entity Framework repostory pattern GetAll() too slow

I'm using repository layer. My problem here is GetAll() method is too slow when join a table with large records. It is taking 40 seconds to run a simple query.
IGenericRepository:
public interface IGenericRepository<TEntity>
{
TEntity FindBy(Expression<Func<TEntity, bool>> predicate);
IEnumerable<TEntity> GetAll();
TEntity GetById(int id);
TEntity Insert(TEntity entity);
TEntity Update(TEntity entity);
void Delete(object id);
void Save();
}
GenericRepository:
public class GenericRepository<TEntity> : IGenericRepository<TEntity> where TEntity : class
{
private MyStoreContext _dbContext;
protected DbSet<TEntity> DbSet;
public GenericRepository()
{
_dbContext = new MyStoreContext ();
DbSet = _dbContext.Set<TEntity>();
}
public TEntity FindBy(Expression<Func<TEntity, bool>> predicate)
{
return DbSet.Where(predicate).SingleOrDefault();
}
public IEnumerable<TEntity> GetAll()
{
return DbSet.AsNoTracking();
}
public TEntity GetById(int id)
{
return DbSet.Find(id);
}
public TEntity Insert(TEntity entity)
{
DbSet.Add(entity);
Save();
return entity;
}
public TEntity Update(TEntity obj)
{
DbSet.Attach(obj);
_dbContext.Entry(obj).State = EntityState.Modified;
Save();
return obj;
}
public void Delete(object id)
{
TEntity entityToDelete = DbSet.Find(id);
Delete(entityToDelete);
}
public void Delete(TEntity entityToDelete)
{
if (_dbContext.Entry(entityToDelete).State == EntityState.Detached)
{
DbSet.Attach(entityToDelete);
}
DbSet.Remove(entityToDelete);
Save();
}
public void Save()
{
try
{
_dbContext.SaveChanges();
}
catch (DbEntityValidationException dbEx)
{
foreach (var validationErrors in dbEx.EntityValidationErrors)
{
foreach (var validationError in validationErrors.ValidationErrors)
{
System.Console.WriteLine("Property: {0} Error: {1}", validationError.PropertyName, validationError.ErrorMessage); // you just put the log to know the errors
}
}
}
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
if (_dbContext != null)
{
_dbContext.Dispose();
_dbContext = null;
}
}
}
}
Linq:
var conceptosDetalle = from pf in _parFactfRepository.GetAll()
join inve in _inveRepository.GetAll() on pf.CVE_ART equals inve.CVE_ART
where inve.STATUS == "A" && pf.CVE_DOC == cveDoc
orderby inve.CTRL_ALM, inve.CVE_ART
select new MyViewModel()
{
CTRL = inve.CTRL_ALM,
CVE_ART = inve.CVE_ART,
UNID = "PIEZA",
CANT = pf.CANT,
DESCR = inve.DESCR,
PREC = pf.PREC,
DESC1 = pf.DESC1,
TOTIMP4 = pf.TOTIMP4
};
The query returns 10 records. parFactfRepository contains 992590 rows, inveRepository contains 41908 rows.
What i'm doing wrong?

That's because you're mixing and matching repository-based queries and LINQ queries. Rather than doing a true join, you're fetching all the rows for each table and then joining them in-memory.
The easiest way to fix this is probably to just return IQueryable<TEntity> rather than IEnumerable<TEntity> from your GetAll method. Using IEnumerable<TEntity> forces the query to evaluate. If you're going to return IEnumerable<TEntity> your data should be fully-baked, i.e. no further alterations to the query are necessary (including joins).
That said, this is yet one more failing of trying to use the repository pattern with EF. If you aren't very careful, you end up introducing logical errors like this that aren't obvious as to why they are happening. Entity Framework already implements the repository pattern; that is what a DbSet is. If you want an abstraction over that, introduce a service layer. With that, you'd simply have a method like:
public IEnumerable<MyViewModel> GetConceptosDetalle()
{
...
}
And that method would contain this entire query (using EF directly, rather than your completely unnecessary repository layer). That way, your application simply calls a method that returns the data it needs, and that service layer contains all the logic. That's true abstraction. With a repository, you're bleeding logic all over your codebase.
Note: I made it return MyViewModel simply for ease of explanation, but in reality, you should return some sort of DTO, which you could then map to your view model. It would be bad idea to leak view business logic into a service layer.

Specifying deferred action before IQueryable has been evaluated

I would like to perform some post-processing on an entity that is returned from an IQueryable, but I would like to specify that post-processing before evaluating the query. So for example:
IQueryable<Client> GetClients()
{
return _context.Set<Client>()
.PostProcess(c => c.MobileNumber = c.MobileNumber.Replace(" ", ""));
}
Client GetClient(int id)
{
return GetClients()
.FirstOrDefault(c => c.Id == id);
}
In the above example I would want every Client returned from the above two methods to have all spaces removed from their mobile number. Is this possible?

Edit: This is a bad idea
It works in the simple case stated in the OP, but any further IQueryable extensions (such as a .Where() clause) will discard the post-processing behaviour. I'm not sure it's technically possible to do what I want to do, and now I come to think of it I'm not even sure what the semantics would be beyond the simple case. Still, the simple case works and is quite handy to keep things fluent...
Original:
I thought I would share my approach for this in case someone finds it useful (or has a comment on why it's a bad idea). The solution uses two decorators to defer the post-processing action until the query is executed, and an extension method to set them up:
public static class QueryableExtensions
{
public static IQueryable<T> PostProcess<T>(this IQueryable<T> source, Action<T> postProcessor) where T : class
{
return new QueryableWrapper<T>(source, postProcessor);
}
// wraps IQueryProvider.Execute methods with post-processing action
class QueryProviderWrapper<T> : IQueryProvider where T : class
{
private readonly IQueryProvider _wrapped;
private readonly Action<T> _postProcessor;
public QueryProviderWrapper(IQueryProvider wrapped, Action<T> postProcessor)
{
_wrapped = wrapped;
_postProcessor = postProcessor;
}
public IQueryable CreateQuery(Expression expression)
{
return _wrapped.CreateQuery(expression);
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
return _wrapped.CreateQuery<TElement>(expression);
}
public object Execute(Expression expression)
{
var result = _wrapped.Execute(expression);
var asT = result as T;
if (asT != null)
_postProcessor(asT);
return result;
}
public TResult Execute<TResult>(Expression expression)
{
var result = _wrapped.Execute<TResult>(expression);
var asT = result as T;
if (asT != null)
_postProcessor(asT);
return result;
}
}
// wraps IQueryable.GetEnumerator() with post-processing action
class QueryableWrapper<T> : IQueryable<T> where T : class
{
private readonly IQueryable<T> _wrapped;
private readonly Action<T> _postProcessor;
private readonly IQueryProvider _provider;
public QueryableWrapper(IQueryable<T> wrapped, Action<T> postProcessor)
{
_wrapped = wrapped;
_postProcessor = postProcessor;
_provider = new QueryProviderWrapper<T>(_wrapped.Provider, postProcessor);
}
public Expression Expression
{
get { return _wrapped.Expression; }
}
public Type ElementType
{
get { return _wrapped.ElementType; }
}
public IQueryProvider Provider
{
get { return _provider; }
}
public IEnumerator<T> GetEnumerator()
{
return _wrapped
.AsEnumerable()
.Do(_postProcessor)
.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
}
The GetEnumerator() decorator uses the .Do() extension method from Interactive Extensions which is like a cross between Select and ForEach: it is lazily invoked, but takes an Action<T> and returns T

Decide Insert or Update on Entity Framework

I wrote this function to save data to EF4 using POCOs classes:
public void Guardar(Pedidos myPedido)
{
using (var context = new OhmioEntities())
{
if (myPedido.ID_Pedido == 0)
{
context.Pedidos.AddObject(myPedido);
}
else
{
context.Pedidos.Attach(myPedido);
context.ObjectStateManager.ChangeObjectState(myPedido, System.Data.EntityState.Modified);
}
context.SaveChanges();
}
}
Now i want to write this in a generic way on a base class. Is there a way to decide if i need to do UPDATE or INSERT without using the ID? (ID_Pedido in this case), because the name on key field change on every object type. The rest of the code is generic. I'm traing to know if i need to use AddObject (new) or Attach(exist).
Thanks you!

look to the method InsertOrUpdate! You can make this repository more generic; For example you can create an Entity base class and use it in a generic Approach.
public class Employee
{
public int Id { get; set; }
public string FullName { get; set; }
}
Now using this we will have a simple context class
public class HRContext : DbContext
{
public DbSet<DomainClasses.Employee> Employees { get; set; }
}
After that, define the repository interface IEmployeeRepository
public interface IEmployeeRepository : IDisposable
{
IQueryable<Employee> All { get; }
IQueryable<Employee> AllIncluding(params Expression<Func<Employee, object>>[] includeProperties);
Employee Find(int id);
void InsertOrUpdate(Employee employee);
void Delete(int id);
void Save();
}
Then the Repository class called EmployeeRepository
public class EmployeeRepository : IEmployeeRepository
{
HRContext context = new HRContext();
public IQueryable<Employee> All
{
get { return context.Employees; }
}
public IQueryable<Employee> AllIncluding(params Expression<Func<Employee, object>>[] includeProperties)
{
IQueryable<Employee> query = context.Employees;
foreach (var includeProperty in includeProperties) {
query = query.Include(includeProperty);
}
return query;
}
public Employee Find(int id)
{
return context.Employees.Find(id);
}
public void InsertOrUpdate(Employee employee)
{
if (employee.Id == default(int)) {
// New entity
context.Employees.Add(employee);
} else {
// Existing entity
context.Entry(employee).State = EntityState.Modified;
}
}
public void Delete(int id)
{
var employee = context.Employees.Find(id);
context.Employees.Remove(employee);
}
public void Save()
{
context.SaveChanges();
}
public void Dispose()
{
context.Dispose();
}
}
I get the soruce code from :
http://blogs.msdn.com/b/wriju/archive/2013/08/23/using-repository-pattern-in-entity-framework.aspx
for example for a generic repository:
public interface IGenericRepository<T> where T : class {
IQueryable<T> GetAll();
IQueryable<T> FindBy(Expression<Func<T, bool>> predicate);
void Add(T entity);
void Delete(T entity);
void Edit(T entity);
void Save();
}
Where T is the base entity for all your entities.
here is the complete generic example:
http://www.tugberkugurlu.com/archive/generic-repository-pattern-entity-framework-asp-net-mvc-and-unit-testing-triangle

I've Found it myself! In case anyone face the same problem, here is the solution. I wrote this method:
public string getKey<T>() where T :new()
{
T _obj = new T();
return context.CreateEntityKey(_obj.ToString().Split('.')[2], _obj).EntityKeyValues[0].Key;
}
Wich return the first Primary key of the object (in my case that's enough)
And use it like this:
string sKey = getKey<GruposComerciales>();
Now i can write a generic saveorupdate method on my repository. Thank you!!!

You can query all parts of the primary key via Metadataworkspace
IDictionary<string, ICollection<EdmMember>> dict = // create instance ...
MetadataWorkspace.GetItems<EntityContainer>(DataSpace.CSpace)
.First()
.BaseEntitySets
.ToList()
.ForEach(s => dict.Add(s.ElementType.Name, s.ElementType.KeyMembers));
With this I put the defined propertys of the primary key into a dictionary for later use.

How do I implement IEnumerable to enumerate an ObjectSet correctly?

My main goal is to override the DeleteObject method of ObjectSet. I'm using EntityFramework 6 with an ObjectContext that I migrated from EF 4.x. EDIT: I modified the T4 template that created the ObjectContext to associate all the ObjectSets on the context to use MyObjectSet instead of the standard, but I want to use it the same way: context.HijackedSet.Where(i=> i.blah == 'blah')
EDIT: I am only having issues with nested ObjectSet calls such as
(from p in context.SomeTable
let poco = new MyExcellentPOCO
{
Name = p.Name,
OtherProperty = context.UnrelatedTable.Where(i=> i.LimitingFactor = x)
}
select poco).ToList()
Unfortunately ObjectSet uses an internal constructor I was not able to inherit from it. Instead I created my own version with an ObjectSet property, and passes through to the property for all of the ObjectSet Methods
Code:
public class MyObjectSet<TEntity> : IObjectSet<TEntity>, IQueryable<TEntity>, IEnumerable<TEntity>, IQueryable, IEnumerable where TEntity : class
{
public MyObjectSet(ObjectSet<TEntity> objectSet)
{
ObjectSet = objectSet;
}
public ObjectSet<TEntity> ObjectSet { get; set; }
public void DeleteObject(TEntity entity)
{
if (entity is IEnforceLogicalDelete)
{
((IEnforceLogicalDelete)entity).IsDeleted = true;
}
else
{
ObjectSet.DeleteObject(entity);
}
}
public void AddObject(TEntity entity)
{
ObjectSet.AddObject(entity);
}
public void Attach(TEntity entity)
{
ObjectSet.Attach(entity);
}
public void Detach(TEntity entity)
{
ObjectSet.Detach(entity);
}
public IEnumerator<TEntity> GetEnumerator()
{
return ((IEnumerable<TEntity>)ObjectSet).GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return ((IEnumerable<TEntity>)ObjectSet).GetEnumerator();
}
public Type ElementType
{
get { return typeof(TEntity); }
}
public Linq.Expressions.Expression Expression
{
get
{
return ObjectSet.AsQueryable<TEntity>().Expression;
}
}
public IQueryProvider Provider
{
get { return ObjectSet.AsQueryable<TEntity>().Provider; }
}
public ObjectQuery<TEntity> Include(string path)
{
return ObjectSet.Include(path);
}
}
MyObjectSet as it is implemented on the context:
public MyObjectSet<Address> Addresses
{
get
{
if ((_Addresses == null))
{
_Addresses = new MyObjectSet<Address>(base.CreateObjectSet<Address>("Addresses"));
}
return _Addresses;
}
}
private MyObjectSet<Address> _Addresses;
I inspected the ObjectQuery class (ObjectSet inherits from ObjectQuery) and it implements IEnumerator IEnumerable.GetEnumerator() as return ((IEnumerable<T>)this).GetEnumerator();. Does my use of the property change the result? return ((IEnumerable<TEntity>)ObjectSet).GetEnumerator();

Can a DbContext enforce a filter policy?

I would like to pass a value to the ctor of a DbContext and then have that value enforce "filtering" on the related DbSets. Is this possible...or is there a better approach?
Code might look like this:
class Contact {
int ContactId { get; set; }
int CompanyId { get; set; }
string Name { get; set; }
}
class ContactContext : DbContext {
public ContactContext(int companyId) {...}
public DbSet<Contact> Contacts { get; set; }
}
using (var cc = new ContactContext(123)) {
// Would only return contacts where CompanyId = 123
var all = (from i in cc.Contacts select i);
// Would automatically set the CompanyId to 123
var contact = new Contact { Name = "Doug" };
cc.Contacts.Add(contact);
cc.SaveChanges();
// Would throw custom exception
contact.CompanyId = 456;
cc.SaveChanges;
}

I decided to implement a custom IDbSet to deal with this. To use this class, you pass in a DbContext, a filter expression, and (optionally) an Action to initialize new entities so they meet the filter criteria.
I've tested enumerating the set and using the Count aggregate functions. Both of them modify the SQL that is generated so they should be much more efficient than filtering on the client.
using System;
using System.Collections;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Data.Entity;
using System.Linq;
using System.Linq.Expressions;
namespace MakeMyPledge.Data
{
class FilteredDbSet<TEntity> : IDbSet<TEntity>, IOrderedQueryable<TEntity>, IOrderedQueryable, IQueryable<TEntity>, IQueryable, IEnumerable<TEntity>, IEnumerable, IListSource
where TEntity : class
{
private readonly DbSet<TEntity> Set;
private readonly IQueryable<TEntity> FilteredSet;
private readonly Action<TEntity> InitializeEntity;
public FilteredDbSet(DbContext context)
: this(context.Set<TEntity>(), i => true, null)
{
}
public FilteredDbSet(DbContext context, Expression<Func<TEntity, bool>> filter)
: this(context.Set<TEntity>(), filter, null)
{
}
public FilteredDbSet(DbContext context, Expression<Func<TEntity, bool>> filter, Action<TEntity> initializeEntity)
: this(context.Set<TEntity>(), filter, initializeEntity)
{
}
private FilteredDbSet(DbSet<TEntity> set, Expression<Func<TEntity, bool>> filter, Action<TEntity> initializeEntity)
{
Set = set;
FilteredSet = set.Where(filter);
MatchesFilter = filter.Compile();
InitializeEntity = initializeEntity;
}
public Func<TEntity, bool> MatchesFilter { get; private set; }
public void ThrowIfEntityDoesNotMatchFilter(TEntity entity)
{
if (!MatchesFilter(entity))
throw new ArgumentOutOfRangeException();
}
public TEntity Add(TEntity entity)
{
DoInitializeEntity(entity);
ThrowIfEntityDoesNotMatchFilter(entity);
return Set.Add(entity);
}
public TEntity Attach(TEntity entity)
{
ThrowIfEntityDoesNotMatchFilter(entity);
return Set.Attach(entity);
}
public TDerivedEntity Create<TDerivedEntity>() where TDerivedEntity : class, TEntity
{
var entity = Set.Create<TDerivedEntity>();
DoInitializeEntity(entity);
return (TDerivedEntity)entity;
}
public TEntity Create()
{
var entity = Set.Create();
DoInitializeEntity(entity);
return entity;
}
public TEntity Find(params object[] keyValues)
{
var entity = Set.Find(keyValues);
if (entity == null)
return null;
// If the user queried an item outside the filter, then we throw an error.
// If IDbSet had a Detach method we would use it...sadly, we have to be ok with the item being in the Set.
ThrowIfEntityDoesNotMatchFilter(entity);
return entity;
}
public TEntity Remove(TEntity entity)
{
ThrowIfEntityDoesNotMatchFilter(entity);
return Set.Remove(entity);
}
/// <summary>
/// Returns the items in the local cache
/// </summary>
/// <remarks>
/// It is possible to add/remove entities via this property that do NOT match the filter.
/// Use the <see cref="ThrowIfEntityDoesNotMatchFilter"/> method before adding/removing an item from this collection.
/// </remarks>
public ObservableCollection<TEntity> Local { get { return Set.Local; } }
IEnumerator<TEntity> IEnumerable<TEntity>.GetEnumerator() { return FilteredSet.GetEnumerator(); }
IEnumerator IEnumerable.GetEnumerator() { return FilteredSet.GetEnumerator(); }
Type IQueryable.ElementType { get { return typeof(TEntity); } }
Expression IQueryable.Expression { get { return FilteredSet.Expression; } }
IQueryProvider IQueryable.Provider { get { return FilteredSet.Provider; } }
bool IListSource.ContainsListCollection { get { return false; } }
IList IListSource.GetList() { throw new InvalidOperationException(); }
void DoInitializeEntity(TEntity entity)
{
if (InitializeEntity != null)
InitializeEntity(entity);
}
}
}

EF doesn't have any "filter" feature. You can try to achive something like that by inheriting custom DbSet but I think it will still be problematic. For example DbSet directly implements IQueryable so there is probably no way how to include custom condition.
This will require some wrapper which will handle these requirements (can be repository):
Condition in select can be handled by wrapping method around DbSet which will add Where condition
Insert can be handled by wrapping method as well
Update must be handled by overriding SaveChanges and using context.ChangeTracker to get all updated entities. Then you can check if entities were modified.
By wrapper I do not mean custom DbSet implementation - that is too complex:
public class MyDal
{
private DbSet<MyEntity> _set;
public MyDal(DbContext context)
{
_set = context.Set<MyEntity>();
}
public IQueryable<MyEntity> GetQuery()
{
return _set.Where(e => ...);
}
// Attach, Insert, Delete
}