Caching and lazy loading with entity framework - c#

let's say I have an application, for example a web site, where my objectcontext leaves during the time of a request. Some datas I load with EF should be cached to avoid to read in DB and improve performance.
Ok, I read my datas with EF, I put my object in cache (says AppFabric, not in memory cache), but related datas that can be lazy loaded are now null (and access to this property results in a nullreferenceexception). I don't want to load everything in one request, because it's going to be too long, so I want to keep the loading on demand and as soon as it's read, I would like to complete the cache with the new fetched datas.
Note :
only read operations, no create/update/delete.
Don't want to use second level cache like "EF Provider Wrappers" made by Jarek Kowalski
How can I do that ?
EDIT : I've built this samples with northwind database, it's working :
class Program
{
static void Main(string[] args)
{
// normal use
List<Products> allProductCached = null;
using (NORTHWNDEntities1 db = new NORTHWNDEntities1())
{
allProductCached = db.Products.ToList().Clone<DbSet<Products>>();
foreach (var product in db.Products.Where(e => e.UnitPrice > 100))
{
Console.WriteLine(product.ProductName + " => " + product.Suppliers.CompanyName);
}
}
// try to use cache, but missing Suppliers
using (NORTHWNDEntities1 db = new NORTHWNDEntities1())
{
foreach (var product in allProductCached.Where(e => e.UnitPrice > 100))
{
if (product.Suppliers == null)
product.Suppliers = db.Suppliers.FirstOrDefault(s => s.SupplierID == product.SupplierID).Clone<Suppliers>();
Console.WriteLine(product.ProductName + " => " + product.Suppliers.CompanyName);
}
}
// try to use full cache
using (NORTHWNDEntities1 db = new NORTHWNDEntities1())
{
foreach (var product in allProductCached.Where(e => e.UnitPrice > 100))
{
Console.WriteLine(product.ProductName + " => " + product.Suppliers.CompanyName);
}
}
}
}
public static class Ext
{
public static List<Products> Clone<T>(this List<Products> list)
{
return list.Select(obj =>
new Products
{
ProductName = obj.ProductName,
SupplierID = obj.SupplierID,
UnitPrice = obj.UnitPrice
}).ToList();
}
public static Suppliers Clone<T>(this Suppliers obj)
{
if (obj == null)
return null;
return new Suppliers
{
SupplierID = obj.SupplierID,
CompanyName = obj.CompanyName
};
}
}
The problem is that I have to copy everything (without missing a property) and test everywhere if the property is null and load the needed property. My code is of course more and more complex, so that will be a problem if I miss something. No other solution ?

You cannot access the database in EF without an ObjectContext or a DbContext.
You can still use caching effectively, even if you don't have the original context any more.
Maybe your scenario is something like this... Imagine that you have some reference data that you use frequently. You do not want to hit the database each time you need it, so you store it in a cache. You also have per-user data that you don't want to cache. You have navigation properties from your user data to your reference data. You want to load your user data from the database, and have EF automatically "fix up" the navigation properties to point to the reference data.
For a request:
Create a new DbContext.
Retrieve reference data from the cache.
Make a deep copy of the reference objects. (You probably don't want to have the same entities attached to multiple contexts simultaneously.)
Attach each of the reference objects to the context. (e.g. with DbSet.Attach())
Execute whatever queries are required to load the per-user data. EF will automatically "fix up" the references to the reference data.
Identify newly loaded entities that could be cached. Ensure that they contain no references to entities that should not be cached, then save them to the cache.
Dispose of the context.
Cloned Objects and Lazy Loading
Lazy loading in EF is usually accomplished using dynamic proxies. The idea is that you make all properties that could potentially be loaded dynamically virtual. Whenever EF creates an instance of your entity type, it actually substitutes a derived type instead, and that derived type has the lazy loading logic in its overridden version of your properties.
This is all well and good, but in this scenario you are attaching entity objects to the context that were not created by EF. You created them, using a method called Clone. You instantiated the real POCO entity, not some mysterious EF dynamic proxy type. That means you won't get lazy loading on these entities.
The solution is simple. The Clone method must take an additional argument: the DbContext. Don't use the entity's constructor to create a new instance. Instead, use DbSet.Create(). This will return a dynamic proxy. Then initialize its properties to create a clone of the reference entity. Then attach it to the context.
Here is the code you might use to clone a single Products entity:
public static Products Clone(this Products product, DbContext context)
{
var set = context.Set<Products>();
var clone = set.Create();
clone.ProductName = product.ProductName;
clone.SupplierID = product.SupplierID;
clone.UnitProce = product.UnitPrice;
// Initialize collection so you don't have to do the null check, but
// if the property is virtual and proxy creation is enabled, it should get lazy loaded.
clone.Suppliers = new List<Suppliers>();
return clone;
}
Code Sample
namespace EFCacheLazyLoadDemo
{
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity;
using System.Linq;
class Program
{
static void Main(string[] args)
{
// Add some demo data.
using (MyContext c = new MyContext())
{
var sampleData = new Master
{
Details =
{
new Detail { SomeDetail = "Cod" },
new Detail { SomeDetail = "Haddock" },
new Detail { SomeDetail = "Perch" }
}
};
c.Masters.Add(sampleData);
c.SaveChanges();
}
Master cachedMaster;
using (MyContext c = new MyContext())
{
c.Configuration.LazyLoadingEnabled = false;
c.Configuration.ProxyCreationEnabled = false;
// We don't load the details here. And we don't even need a proxy either.
cachedMaster = c.Masters.First();
}
Console.WriteLine("Reference entity details count: {0}.", cachedMaster.Details.Count);
using (MyContext c = new MyContext())
{
var liveMaster = cachedMaster.DeepCopy(c);
c.Masters.Attach(liveMaster);
Console.WriteLine("Re-attached entity details count: {0}.", liveMaster.Details.Count);
}
Console.ReadKey();
}
}
public static class MasterExtensions
{
public static Master DeepCopy(this Master source, MyContext context)
{
var copy = context.Masters.Create();
copy.MasterId = source.MasterId;
foreach (var d in source.Details)
{
var copyDetail = context.Details.Create();
copyDetail.DetailId = d.DetailId;
copyDetail.MasterId = d.MasterId;
copyDetail.Master = copy;
copyDetail.SomeDetail = d.SomeDetail;
}
return copy;
}
}
public class MyContext : DbContext
{
static MyContext()
{
// Just for demo purposes, re-create db each time this runs.
Database.SetInitializer(new DropCreateDatabaseAlways<MyContext>());
}
public DbSet<Master> Masters { get { return this.Set<Master>(); } }
public DbSet<Detail> Details { get { return this.Set<Detail>(); } }
}
public class Master
{
public Master()
{
this.Details = new List<Detail>();
}
public int MasterId { get; set; }
public virtual List<Detail> Details { get; private set; }
}
public class Detail
{
public int DetailId { get; set; }
public string SomeDetail { get; set; }
public int MasterId { get; set; }
[ForeignKey("MasterId")]
public Master Master { get; set; }
}
}
Here is a sample model, different from yours, that shows how to get this working in principle.

Related

ef core attach same entity multiple times

I'm sure this was asked before, I don't know what to search for, so it's probably duplicate.
I have code that adds new entity to database. This entity has reference to another entity(Role), and I get it via service. Service creates another instance of dbContext, so I have to attach role to the context after I fetch it. The problem is, when I try to attach two same roles, I get this exception:
'Role' cannot be tracked because another instance with the same key value for {'Id'} is already being tracked. When attaching existing entities, ensure that only one entity instance with a given key value is attached. Consider using 'DbContextOptionsBuilder.EnableSensitiveDataLogging' to see the conflicting key values.'
How should I do it? Code below:
using (var context = new TenantContext(schemaName, connectionString))
{
ApprovalTemplates templates = new ApprovalTemplates();
ApprovalTemplate template = new ApprovalTemplate();
template.Approvers = new List<StageTemplate>();
foreach (var stage in request.Stages)
{
var temp = new StageTemplate();
temp.Order = stage.Order;
temp.Name = stage.Name;
var role = roleService.GetById(stage.RoleId, schemaName);//here I get the role
temp.AvailableActions = new List<ApprovalActionTemplate>();
foreach (var actionId in stage.Actions)
temp.AvailableActions.Add(context.ApprovalActions.First(a => a.Id == actionId));
//when I try to add already attached role, exception is thrown
context.TenantRoles.Attach(role);
temp.Role = role;
template.Approvers.Add(temp);
}
templates.PRApprovalTemplate = template;
context.ApprovalTemplates.Add(templates);
context.SaveChanges();
}
I would share potential approach for this and similar cases with Attach - the rule is very simple, you should never attach Entity with the same Id twice. Good point that there is an easy way to check if it's already attached and if it's attached, you can just use that entity, so best way is to always check local entities before attaching any Entity.
For your case in place of
var role = roleService.GetById(stage.RoleId, schemaName);//here I get the role
it may be:
var localRole = context.Set<TenantRole>().Local.FirstOrDefault(entry => entry.Id.Equals(stage.RoleId));
if (localRole == null)
{
localRole = new TenantRole
{
Id = stage.RoleId,
};
Context.TenantRoles.Attach(localRole);
}
...
temp.Role = localRole;
Because if you know RoleId, you do not need to make DB call just to attach TenantRole to the Context.
Given code works fine, but once someone have many-many places like this, it's becomes to heavy. Potential solution for this would be creating extension method for your Context:
public static class RepositoryExtensions
{
public static T LocalContextEntitiesFinder<T>(this TenantContext context, Guid id) where T : class, ISomeInterfaceThatAllYourDBModelsImplements, new()
{
var localObj = context.Set<T>().Local.FirstOrDefault(entry => entry.Id.Equals(id));
if (localObj != null)
{
return localObj;
}
localObj = new T
{
Id = id
};
context.Set<T>().Attach(localObj);
return localObj;
}
}
So you will be able to re-write your code to something like:
...
temp.Role = context.LocalContextEntitiesFinder<TenantRole>(id: stage.RoleId);
...
To make it work, you should add interface ISomeInterfaceThatAllYourDBModelsImplements similar to this (in place of Guid you can use any other type you like):
public interface ISomeInterfaceThatAllYourDBModelsImplements
{
public Guid Id { get; set; }
}
And update TenantRole
public class TenantRole: ISomeInterfaceThatAllYourDBModelsImplements
{
[Key]
public Guid Id { get; set; }
...
I hope this may help somebody.

EF6: Possibility to use Navigation Property value outside of the context

I use a using block to select a dataset from the database with Entity Framework:
protected StagingProcessingEngineStatus GetProcessingStatusEntryById (Guid processingId) {
using (var context = new ProcessingEntities()) {
var matchedPlan = context.StagingProcessingEngineStatus.FirstOrDefault(plan => plan.ProcessingId == processingId && plan.Owner == this.UserCredentials.UserId.ToString());
return matchedPlan;
}
}
The StagingProcessingEngineStatus entity contains a navigation property to another entity called StagingRefProcessingEngineLibrary:
public partial class ProcessingEntities : DbContext {
public virtual DbSet<StagingProcessingEngineStatus> StagingProcessingEngineStatus { get; set; }
}
public partial class StagingProcessingEngineStatus {
public virtual StagingRefProcessingEngineLibrary StagingRefProcessingEngineLibrary { get; set; }
}
Now I try to use the return value of the GetProcessingStatusEntryById method to access the content of the StagingRefProcessingEngineLibrary property of StagingProcessingEngineStatus.
But the gives me the following exception:
The ObjectContext instance has been disposed and cannot be used for
operations anymore, which require a connection.
Is there a possibility to access the value of the navigation property outside of the using block like below?
var stagingPlan = this.GetProcessingStatusEntryById(processingId);
var library = GetProcessingLibraryByXmlString(stagingPlan.StagingRefProcessingEngineLibrary.LibraryXml);
Either do not dispose the context (remove the using) to keep lazy loading functional, or use eager loading (.Include(e => e.StagingRefProcessingEngineLibrary)

Entity Framework ICollection Navigation Properties null only when testing

I'm using Entity Framework 6.1.3 in an MVC 5 project, and I'm having issues with ICollection navigation properties being null when unit testing. I am hitting an actual test database in SQL Express, so this may be more like an integration test for you purists. Regardless of what you call it, it is a problem that I would like to solve.
I have read many answers to similar sounding questions, but none of them seem to hit the same problem that I am having here. I understand EF for the most part, I have lazy loading enabled, my classes are public, and I'm using virtual on my navigation properties.
Here is a simplified example of what I am trying to do:
Models:
public class Session
{
public int Id { get; set; }
public string Name { get; set; }
// Navigation Property
public virtual ICollection<File> Files { get; set; }
}
public class File
{
public int Id { get; set; }
public string Name { get; set; }
public int SessionId { get; set; }
public virtual Session Session { get; set; }
}
Test methods:
[TestMethod]
public void Test_TotalFileCount1()
{
ApplicationDbContext context = new ApplicationDbContext();
// Create session with no files
var session = new Session() { Name = "Session1" };
context.Sessions.Add(session);
context.SaveChanges();
// This line blows up because session.Files == null
Assert.AreEqual(0, session.Files.Count);
}
[TestMethod]
public void Test_TotalFileCount2()
{
ApplicationDbContext context = new ApplicationDbContext();
// Create session
var session = new Session() { Name = "Session2" };
context.Sessions.Add(session);
context.SaveChanges();
// Create file for session
var file = new File() { Name = "File1", Session = session };
context.Files.Add(file)
context.SaveChanges();
// This test passes because session.Files is a
// collection of one file
Assert.AreEqual(1, session.Files.Count);
}
The first test above fails because session.Files throws an ArgumentNullException. However, when I call this same code in the full MVC application, session.Files is not null and instead is an empty collection with Count = 0. The second test passes because session.Files is a collection of one File as I would expect. The navigation properties are clearly doing what they're supposed to in the second case, but not in the first case.
Why is EF behaving like this?
I was able to get around this problem by initializing Files as an empty list in the constructor. I know I could do this conditionally in the getter instead, but I don't think I should have to do either of these things because it just works when it's running normally.
public Session()
{
this.Files = new List<File>();
}
Does anyone have any insight into what is going on here?
If you are working with lazy loading enabled and if you want that the navigation property gets populated after adding the object with the foreign key property to the context you must use the Create method of DbSet (instead of instantiating the object with new):
var session = context.Sessions.Create();
With active lazy loading this will create a proxy object which ensures that the navigation property gets loaded.
[TestMethod]
public void Test_TotalFileCount1()
{
ApplicationDbContext context = new ApplicationDbContext();
// Create session with no files
var session = context.Sessions.Create();
session.Name = "Session1";
context.Sessions.Add(session);
context.SaveChanges();
// This line blows up because session.Files == null
Assert.AreEqual(0, session.Files.Count);
}
[TestMethod]
public void Test_TotalFileCount2()
{
ApplicationDbContext context = new ApplicationDbContext();
// Create session
var session = context.Sessions.Create();
session.Name = "Session2";
session.Files = new List<File>()
{
new File() { Name = "File1" }
};
context.Sessions.Add(session);
context.SaveChanges();
// This test passes because session.Files is a
// collection of one file
Assert.AreEqual(1, session.Files.Count);
}
See more about Proxies

Database connection errors on EF

I am very new to entity framework and I am having a problem with a web api based site (connected to mssql) that I am writing. I keep getting seemingly random errors (mostly seeming to be database related). These errors happen most often when the site is first published but they do sometimes happen when it has been hours since the last publish. A selection of the errors:
Invalid operation. The connection is closed.
There is already an open DataReader associated with this Command which must be closed first.
The connection was not closed. The connection's current state is connecting.
The context cannot be viewed while the model is being created
Underlying provider failed to open
My context looks like this:
public class Context : DbContext
{
public Context() : base("name=DefaultConnection")
{
}
public override int SaveChanges()
{
DateTime now = DateTime.Now;
foreach (ObjectStateEntry entry in (this as IObjectContextAdapter).ObjectContext.ObjectStateManager.GetObjectStateEntries(EntityState.Added | EntityState.Modified))
{
if (!entry.IsRelationship)
{
IHasUpdated updated = entry.Entity as IHasUpdated;
if (updated != null)
updated.updated = now;
}
}
return base.SaveChanges();
}
public DbSet<Branch> Branches { get; set; }
public DbSet<Company> Companies { get; set; }
public DbSet<User> Users { get; set; }
public DbSet<UsefulLink> UsefulLinks { get; set; }
}
There are many more DbSets than this. Should I be creating a separate context for each?
One of my basic controllers:
public class UsefulLinksController : ApiController
{
private Context db = new Context();
[ResponseType(typeof(UsefulLinksWrapper))]
public IHttpActionResult GetUsefulLinks([FromUri]UsefulLinkParams prams)
{
UsefulLinksWrapper wrapper = new UsefulLinksWrapper();
Meta meta = new Meta();
IQueryable<UsefulLink> query = db.UsefulLinks;
if (prams.sortBy == null)
{
prams.sortBy = "ID";
}
// Paging
query = query.OrderBy(prams.sortBy + " " + prams.sortDirection).Skip(prams.offset - 1).Take(prams.limit);
List<UsefulLink> data = query.ToList();
meta.totalCount = query.Count();
meta.offset = 1;
meta.limit = prams.limit;
wrapper.meta = meta;
wrapper.data = data;
return Ok(wrapper);
}
protected override void Dispose(bool disposing)
{
if (disposing)
{
db.Dispose();
}
base.Dispose(disposing);
}
private bool UsefulLinkExists(int id)
{
return db.UsefulLinks.Count(e => e.ID == id) > 0;
}
}
I don't seem to see these errors when I run the site locally though there are two of us hitting it when it is published so perhaps the issue stems from multiple users?
Chris, I notice in your controller you are sharing your db context with all of the methods in your controller class.
This is generally not a best practice in Entity Framework (see: EntityFramework 4 ObjectContext Lifetime). You should keep your context alive as briefly as possible. Leaving the context alive to share across multiple methods could result in many of the errors that you list above.
I would recommend trying to instantiate a new instance of the context, instead, wherever it is used and quickly disposing of it.
This should generally result in more stable behavior.
So the below:
class SomeClass
{
private context = new Context(); //sharing your context with all methods
public someMethod()
{
context.doSomething;
}
public someMethod2()
{
context.doSomething;
}
}
should become:
class SomeClass
{
public someMethod()
{
Context context = new Context(); //now your context is declared and disposed of within each method
context.doSomething;
}
public someMethod2()
{
Context context = new Context(); //now your context is declared and disposed of within each method
context.doSomething;
}
}
Or even better, you can use a using construct to ensure that your context is properly disposed of:
class SomeClass
{
public someMethod3()
{
using(Context context = new Context()) //now wrapping the context in a using to ensure it is disposed
{
context.doSomething;
}
}
}
I would recommend trying the above changes and seeing if your behavior becomes more stable.
Since I do not know how your page uses the methods UsefulLinksController and in which order, I would say UsefulLinkExists is perhaps the culprit due to lazy loading
Lazy loading means delaying the loading of related data until you
specifically request it
Which would explain why your "reader" remains "open".
Try:
return db.UsefulLinks.ToList().Count(e => e.ID == id) > 0;
In any case, you can disable lazy loading by default in the context constructor as such noted here:
public MyEntitiesContext() : base("name=MyEntitiesContext", "MyEntitiesContext")
{
this.ContextOptions.LazyLoadingEnabled = false;
OnContextCreated();
}
As far as I know, it applies to EF4 and up.

Keeping encapsulation in while keeping repositories out, entity framework, code first

I have a situation, which I will try to greatly simplify, in which a method on an object should create a new persistent object of a different class. How can I do this while maintaining repository ignorance and encapsulation at the same time?
In this example, we have widgets and containers of widgets. When a widget is sold or purchased, a WidgetEvent is created (somewhere else) and added to the list of WidgetEvents for the container. We can always query the container for the number of pieces on hand by summing the collection of WidgetEvents.At some point, a worker calls and says the container is empty even though there should still be some widgets in the container. In that case, we call the "SetComplete" method which creates a final WidgetEvent to zero out the container.
public class TestContext : DbContext
{
public DbSet<WidgetEvent> WidgetEvents { get; set; }
public DbSet<WidgetContainer> WidgetContainers { get; set; }
}
public class WidgetEvent
{
public int Id { get; set; }
public int Amount {get;set;}
public WidgetContainer Container {get;set;}
}
public class WidgetContainer
{
public int Id { get; set; }
public virtual ICollection<WidgetEvent> WidgetEvents {get;set;}
public int GetPiecesOnHand()
{
return WidgetEvents.Sum(a=> a.Amount);
}
public void SetComplete()
{
if (GetPiecesOnHand() != 0)
{
WidgetEvents.Add(new WidgetEvent() { Amount = -GetPiecesOnHand() });
}
}
}
It seems to me that proper encapsulation would keep the creation of this event within the class definition. It makes it more understandable to someone reading the code. But I also see there is no way to make this new WidgetEvent persistent without introducing some repository knowledge into the class.
How should I do this? You can assume that I have a Factory for the WidgetEvents somewhere else. In case you were wondering, SaveChanges does not realize that the WidgetEvent has been created.
Edit: This returns
25
25
0
0
which is correct. I guess this leaves me a bit confused about track changes, but its good to know this works. I did something like this a couple of days ago and thought it did not work.
using (var context = new TestContext())
{
WidgetContainer acontainer = new WidgetContainer();
acontainer.WidgetEvents = new List<WidgetEvent>();
context.WidgetContainers.Add(acontainer);
acontainer.WidgetEvents.Add(new WidgetEvent() { Container = acontainer, Amount = 25 });
Console.WriteLine(acontainer.GetPiecesOnHand());
context.SaveChanges();
}
using (var context = new TestContext())
{
WidgetContainer acontainer = context.WidgetContainers.Find(1);
Console.WriteLine(acontainer.GetPiecesOnHand());
acontainer.SetComplete();
Console.WriteLine(acontainer.GetPiecesOnHand());
context.SaveChanges();
}
using (var context = new TestContext())
{
WidgetContainer acontainer = context.WidgetContainers.Find(1);
Console.WriteLine(acontainer.GetPiecesOnHand());
}
It should actually work if the WidgetContainer is attached to (=tracked by) the Entity Framework context without giving the class a reference to a context (or a repository).
For example a code like this...
var widgetContainer = context.WidgetContainers.Find(1);
// or:
// var widgetContainer = context.WidgetContainers.Create();
// widgetContainer.Id = 1;
// context.WidgetContainers.Attach(widgetContainer);
widgetContainer.SetComplete();
context.SaveChanges();
...should insert the new WidgetEvent into the database with a foreign key relationship to the widgetContainer because EF change tracking will recognize the new event that has been added to the WidgetEvents collection when SaveChanges is called.
The important piece is that the widgetContainer must be attached to the context before SetComplete is called.

Categories