I have a very basic example C# program which uses EF6 with DbContext and which contains two classes:
MyTable1 1...n MyTable2
I have the following code:
// Creation of instances and saving them to the database.
using (var context = new EFTestEntities()) {
var myTable1 = new MyTable1() { ID = 1 };
var myTable2 = new MyTable2() { ID = 2 };
myTable1.MyTable2.Add(myTable2);
context.MyTable1.Add(myTable1);
context.SaveChanges();
}
// Getting the above objects in another context.
// This part here is what my question is about.
using (var context = new EFTestEntities()) {
var myTable1 = context.MyTable1.Where(e => e.ID == 1).FirstOrDefault();
var myTable2 = context.MyTable2.Where(e => e.ID == 2).FirstOrDefault();
var myTable2AssignedToMyTable1 = myTable1.MyTable2.FirstOrDefault();
}
My question is about the second part of the code above.
When lazy loading is enabled, I get three database calls. One for each line. The variable myTable2AssignedToMyTable1 contains then myTable2 - which is correct.
When lazy loading is disabled, I get two database calls. One for the first and one for the second line. The variable myTable2AssignedToMyTable1 contains as well myTable2 - without a third database call. This makes sense to me because myTable2 was loaded in line 2.
But: If that third database call is not needed, why is it made when lazy loading is enabled?
When I check the database calls in SQL Server Profiler with and without lazy loading I realize that they are exactly the same (apart from the third call). The data from the third call is the same as in the first or second call. So when the third call is happening, the desired data are already in the context. When lazy loading is disabled, EF realizes it and uses the already loaded data. When lazy loading is enabled it does not realize it and fetches the same data from the database again. Why?
Is it faster than finding the correct instance of MyTable2 in the context?
Or is it faster than always searching for the correct instance in the context first and then make the database call? One can assume that in most cases the desired data are not already in the context. So I guess the EF designers decided to not check the context first. In most cases it will be a miss any way.
But these are only my assumptions. Knowing the real reason would be interesting.
Related
I have the below code that fetches the data from service and attaches to the objects that was loaded from local database by EntityFramework 6!
We had the UserJob details before in our local database, However we decided to move it to the service later and migrated all the UserJobs to new micro service.
We are in the middle of this move and now we are saving the User jobs in both local database and in the new service!
When we read from micro service and update the local database as well, the code treats its a new job, since the JobId is no longer available in new service, but we have same creation date and user id to find the JobId available in local database! I set it now correctly.
private void IncludeUserJobOnUser(UserJob userJob, MyDBContext db)
{
var aspnetUser = new EFDataReader().GetUser(userJob.userId, db);
if (userJob != null && aspnetUser != null)
{
userJob.JobId = new EFDataReader().GetUserJobId(userJob.userId, userJob.CreationDate, db);
DbEntityEntry entry = db.Entry(userJob);
entry.State = EntityState.Unchanged;
if (aspnetUser.UserJobs.Any(x => x.JobId != userJob.JobId)) // Here is the problem
{
aspnetUser.UserJobs.Add(userJob);
}
userJob.AspNetUser = aspnetUser;
}
}
PROBLEM:
I am trying to set the navigational property with the objects retrieved from the microservice. However, EF6 always hits the database and populate them from local database instead adding the entity came from microservice.
Any idea how to solve the above?
If you mean that by accessing "aspnetUser.UserJobs" you are seeing a query kicked off to load the UserJobs as part of the Any() check, that would be Lazy Loading. If the UserJobs were already eager loaded when the data reader loaded the user then this wouldn't happen. If the user hasn't loaded the jobs, how would the code know if the user had that job or not?
Code like this can be dangerous at runtime if not careful because you are passing entity references around to be associated with other entities where the scope of the DbContext that may, or may not still be tracking these entities is not clearly defined. Provided that "db" DbContext instance is the single overarching DbContext for retrieving all entities (including UserJob) are loaded from or associated to then you should be ok, but when I read things like "objects retrieved from the microservice" that sends up a bit of a red flag as Microservices would be scoping their own DbContext so returning entities from one would not be known by the current DbContext whose entities you are associating such results with.
I'm kinda confused about recognizing a disconnected scenario and a connected scenario, I've searched the internet but I couldn't find any real answer to my questions, I'm kinda confused about entities tracking system, connected and disconnected scenarios, when should I use the Attach method and also in differences between using the Entry(entity).State = EntityState.Deleted and Remove(entity) method, and while I was searching about the last one, most of the time, they were thought identical, but it didn't match with the test that I did and what I expected
I just made a simple console app to test the differences, and how it works is that I make a person completely outside of the context instantiation scope and then pass it to the AddPerson method, because I think this makes a disconnected scenario, right? because the Remove method will complain about why I haven't attached the entity first, so I think that tells us that we're in a disconnected scenario, I'm not sure tho
This is the app:
class Program
{
static void Main(string[] args)
{
Person person = new Person()
{
PersonID = 1,
Name = "John",
Family = "Doe"
};
using (var context = new MyContext())
{
// Why this one requires attaching but the code below doesn't
context.Person.Attach(person);
context.Person.Remove(person);
context.SaveChanges();
// This method of deleting works fine without the entity being attached
context.Entry(person).State = EntityState.Deleted;
context.SaveChanges();
var people = context.Person.ToList();
foreach (var p in people)
{
Console.WriteLine($"PersonID: {p.PersonID} | Name: {p.Name} | Family: {p.Family}");
}
}
Console.ReadKey();
}
}
so for the Remove method, I have to Attach the entity first, otherwise, it will throw an exception, BUT when I use the Entry(person).state = EntityState.Deleted without attaching it, it works fine, and deletes the person, now why is that, isn't this a big difference? why is it not said anywhere, I've read some websites and some other similar questions on Stackoverflow too, but this wasn't said anywhere, and for the most part, these two were presumed to be the same, and do the same thing, yes they both delete the entity, but how can we describe what happened in this test, isn't this a difference between these two?
I have two questions but I think they're related to each other, so I'm just going to ask both of them here:
When does exactly a disconnected scenario happen, and how can I recognize it, does it depend on the scope of the context instantiation, or on retrieving the entity directly from the context and then modifying it (with no need to attach it), or using an entity from outside of the context (like passing it from another scope to our context as a parameter, as I did in my test)?
Why does the Remove method requires attaching but the EntityState.Deleted doesn't, but they're presumed identical? why should I even bother to attach the entity first, while setting the state to deleted works without needing to attach, so When to use each of them?
Basically, The way I assume that how all these work (with my current understanding of Entity Framework which is probably wrong) is that when you're in a disconnected scenario, you have to attach your entity first, but then setting the state to EntityState.Deleted doesn't need attaching, so then why does the Remove method exists at all, we could use the other way of deleting all the time.
EDIT:
Based on the second code block in the accepted answer, I wrote this test, to figure out how it's working, you said that the otherPersonReference is equal to having a Attach(Person) but when I first attach the person and try to use EntityState.Deleted It works then too, and it'll delete it, but you said that it would fail, I'm a little confused :s
class Program
{
static void Main(string[] args)
{
Person person = new Person()
{
PersonID = 3,
Name = "John",
Family = "Doe"
};
using (var context = new MyContext())
{
//var pr = context.Person.Single(p => p.PersonID == 3);
context.Person.Attach(person);
context.Entry(person).State = EntityState.Deleted;
context.SaveChanges();
}
Console.ReadKey();
}
}
if I uncomment the pr variable line and then comment the context.Person.Attach(person) then setting the EntityState to Deleted would fail and it'll throw an exception as expected
Setting context.Entry(person).State tells EF to start tracking the "person" instance if it isn't already tracking it. You would get an error if the DbContext was already tracking an instance for the same record.
For example, you can try the following:
var person = new Person { Id = 100 }; // assume an existing record with ID = 100;
using (var context = new AppDbContext())
{
context.Entry(person).State = EntityState.Deleted;
context.SaveChanges();
}
This works as you expect... However, if you were to have code that did this:
var person = new Person { Id = 100 }; // assume an existing record with ID = 100;
using (var context = new AppDbContext())
{
var otherPersonReference = context.Persons.Single(x => x.Id == 100);
context.Entry(person).State = EntityState.Deleted;
context.SaveChanges();
}
Your attempt to use context.Entry(person).State = EntityState.Deleted; would fail because the context is now already tracking an entity with that ID. It's the same behaviour as if you were to try and call Attach(person).
When dealing with short-lived DbContexts (such as when using using() blocks) and single entity operations, it can be reasonably safe to work with detached entity references, but this will get a lot more "iffy" once you start dealing with multiple possible entity references (I.e. working with lists or objects sharing references etc.) and/or calls across a DbContext which may already be tracking entity references from previous operations / iterations.
Edit: Working with detached references can be problematic and you need to take extra care when doing so. My general recommendation is to avoid it wherever possible. The approach I recommend when dealing with entities is that you should never pass an entity outside of the scope of the DbContext that read it. This means leveraging a ViewModel or DTO to represent entity-sourced details outside the scope of the DbContext. A detached EF Entity
can certainly work, but with a DTO it is explicitly clear that the data cannot be confused with a tracked entity. When it comes to performing operations like a Delete, you only really need to pass the ID.
For example, leveraging Automapper to help translate between DTOs and entities:
PersonDTO AddPerson(PersonDTO details)
{
if(details == null)
throw new ArgumentNullException("details");
using (var context = new AppDbContext())
{
// TODO: Add validations such as verifying unique name/dob etc.
var person = Mapper.Map<Person>(details); // Creates a new Person.
context.Persons.Add(person);
context.SaveChanges();
details.PersonId = person.PersonId; // After SaveChanges we can retrieve the new row's ID.
return details;
}
}
PersonDTO UpdatePerson(PersonDTO details)
{
if(details == null)
throw new ArgumentNullException("details");
using (var context = new AppDbContext())
{
var existingPerson = context.Persons.Single(x => x.PersonId == details.PersonId); // Throws if we pass an invalid PersonId.
Mapper.Map(details, existingPerson); // copies values from our DTO into Person. Mapping is configured to only copy across allowed values.
context.SaveChanges();
return Mapper.Map<PersonDTO>(existingPerson); // Return a fresh, up to date DTO of our data record.
}
}
void DeletePerson(int personId)
{
using (var context = new AppDbContext())
{
var existingPerson = context.Persons.SingleOrDefault(x => x.PersonId == details.PersonId);
if (existingPerson == null)
return; // Nothing to do.
// TODO: Verify whether the current user should be able to delete this person or not. (I.e. based on the state of the person, is it in use, etc.)
context.Persons.Remove(existingPerson);
context.SaveChanges();
}
}
In this example a Person entity does not ever leave the scope of a DbContext. The trouble with detached entities is that whenever passing an entity around to other methods and such, those methods might assume they are working with attached, complete or complete-able (i.e. through lazy loading) entities. Was the entity loaded from a DbContext that is still "alive" so if if the code wants to check person.Address that data is either eager loaded and available, or lazy-loadable? vs. #null which could mean the person does not have an address, or that without a DbContext or lazy loading we cannot determine whether it does or not. As a general rule if a method is written to accept an entity, it should always expect to have a complete, or complete-able version of that entity. Not a detached "maybe complete, maybe not" instance, not a "new"ed up instance of a class that has some arbitrary values populated, (rather than an entity representing a data row) and not a deserialized block of JSON coming from a web client. All of those can be typed as a "Person" entity, but not a Person entity.
Edit 2: "Complete" vs. "Complete-able"
A Complete entity is an entity that has all related entities eager loaded. Any method that accepts a Person should be able to access any property, including navigation properties, and receive the true value. If the Person has an Address, then a #null address should only ever mean that person does not have an address (if that is valid), not "that person does not have an address, or it just wasn't loaded." This also goes for cases where you might have a method that accepts an entity, which you haven't loaded, but want to substitute with a entity class populated with an ID and whatever data you might have on hand. That incomplete "entity" could find itself sent to other methods that expect a more complete entity. Methods should never need to guess at what they receive.
A Complete-able entity is an entity where any related entities within that entity can be lazy loaded if accessed. The consuming method doesn't need to determine whether properties are available or not, it can access Person.Address and it will always get an Address if that person is supposed to have one, whether the caller remembered to eager load it or not.
Where methods are using tightly scoped DbContexts (using()) if you return an entity then there is no way that you can guarantee later down the call-chain that this entity is complete-able. Today you can make the assurance that all properties are eager-loaded, but tomorrow a new relationship could be added leaving a navigation property somewhere within the object graph that might not be remembered to be eager-loaded.
Eager loading is also expensive, given to ensure an entity is "complete", everything needs to be loaded, whether the consumers ever need it or not. Lazy Loading was introduced to facilitate this, however, in many cases this is extremely expensive leading to a LOT of chatter with the database and the introduction of performance costs when the model evolves. Elements like serialization (a common problem in web applications) touch every property by default leading to numerous lazy load calls for every entity sent.
DTOs/ViewModels are highly recommended when data needs to leave the scope of a DbContext as it ensures only the data a consumer needs is loaded, but equally importantly, as a model may evolve, you avoid lazy loading pitfalls. Serializing a DTO rather than an Entity will ensure those new relationships don't come into play until a DTO is updated to actually need that data.
How would you Upsert without select? the upsert would be a collection of entities received by a method which contains DTOs that may not be available in the database so you can NOT use attach range for example.
One way theoretically is to load the ExistingData partially with a select like dbContext.People.Where(x => x exists in requested collection).Select(x => new Person { Id = x.Id, State = x.State }).ToList() which just loads a part of the entity and not the heavy parts. But here if you update one of these returned entityItems from this collection it will not update because of the new Person its not tracking it and you also cannot say dbContext.Entry<Person>(person).State = Modified because it will throw an error and will tell you that ef core is already "Tracking" it.
So what to do.
One way would be to detach all of them from the ChangeTracker and then do the state change and it will do the update but not just on one field even if you say dbContext.Entry<Person>(person).Property(x => x.State).Modified = true. It will overwrite every fields that you haven't read from the database to their default value and it will make a mess in the database.
The other way would be to read the ChangeTracker entries and update them but it will also overwrite and it will consider like everything is chanaged.
So techinically I don't know how ef core can create the following SQL,
update People set state = 'Approved' where state != 'Approved'
without updating anything else. or loading the person first completely.
The reason for not loading your data is that you may want to update like 14000 records and those records are really heavy to load because they contain byte[] and have images stored on them for example.
BTW the lack of friendly documentation on EFCore is a disaster compare to Laravel. Recently it has cost us the loss of a huge amount of data.
btw, the examples like the code below will NOT work for us because they are updating one field which they know that it exists in database. But we are trying to upsert a collection which some of those DTOs may not be available in the database.
try
{
using (var db = new dbContext())
{
// Create new stub with correct id and attach to context.
var entity = new myEntity { PageID = pageid };
db.Pages.Attach(entity);
// Now the entity is being tracked by EF, update required properties.
entity.Title = "new title";
entity.Url = "new-url";
// EF knows only to update the propeties specified above.
db.SaveChanges();
}
}
catch (DataException)
{
// process exception
}
Edit: The used ef core version is #3.1.9
Fantastic, I found the solution (You need to also take care about your unit tests).
Entityframework is actually working fine it can be just a lack of experience which I'm documenting here in case anyone else got into the same issue.
Consider that we have an entity for Person which has a profile picture saved as Blob on it which causes that if you do something like the following for let's say 20k people the query goes slow even when you've tried to have enough correct index on your table.
You want to do this query to update these entities based on a request.
var entityIdsToUpdate = request.PeopleDtos.Select(p => p.Id);
var people = dbContext.People.Where(x => entityIdsToUpdate.Contains(x.Id)).ToList();
This is fine and it works perfectly, you will get the People collection and then you can update them based on the given data.
In these kind of updates you normally will not need to update images even if you do, then you need to increase the `TimeOut1 property on your client but for our case we did not need to update the images.
So the above code will change to this.
var entityIdsToUpdate = request.PeopleDtos.Select(p => p.Id);
var people = dbContext.People
.Select(p => new Person {
Id = p.Id,
Firstname = p.Firstname,
Lastname = p.Lastname,
//But no images to load
})
.Where(p => entityIdsToUpdate.Contains(p.Id)).ToList();
But then with this approach, EntityFramework will lose the track of your entities.
So you need to attach it like this and I will tell you how NOT to attach it.
This is the correct way for a collection
dbContext.People.AttachRange(people); //These are the people you've already queried
Now DO NOT do this, you may want to do this because you get an error from the first one from EntityFramework which says the entity is already being tracked, trust it because it already is. I will explain after the code.
//Do not do this
foreach(var entry in dbContext.ChangeTracker.Entries())
{
entry.State = EntityState.Detached;
}
//and then on updating a record you may write the following to attach it back
dbContext.Entry(Person).State = EntityState.Modified;
The above code will cause EntityFramework not to follow the changes on the entities anymore and by the last line you will tell it literally everything edited or not edited is changed and will cause you to LOSE your unedited properties like the "image".
Note: Now what can u do by mistake that even messes up the correct approach.
Well since you are not loading your whole entity, you may assume that it is still fine to assign values to the unloaded ones even if the value is not different than the one in the database. This causes entity framework to assume that something is changed and if you are setting a ModifiedOn on your records it will change it for no good reason.
And now about testing:
While you test, you may get something out from database and create a dto from that and pass the dto with the same dbContext to your SystemUnderTest the attach method will throw an error here which says this entity is already bein tracked because of that call in your test method. The best way would be create a new dbContext for each process and dispose them after you are done with them.
BTW in testing it may happen that with the same dbContext you update an entity and after the test you want to fetch if from the database. Please take note that this one which is returning to you is the "Cached" one by EntityFramework and if you have fetched it in the first place not completely like just with Select(x => ) then you will get some fields as null or default value.
In this case you should do DbContext.Entry(YOUR_ENTRY).Reload().
It is a really complete answer it may not directly be related to the question but all of the things mentioned above if you don't notice them may cause a disaster.
Background: I'm using EF6 and Database First.
I'm run into a scenario that has me perplexed. After creating a new object, populating the Navigation Properties with new objects, and calling SaveChanges, the navigation properties are reset. The first line of code that references the navigation property after the SaveChanges call will end up re-fetching the data from the database. Is this expected behavior, and can someone explain why it behaves this way? Here is a sample code block of my scenario:
using (DbContext context = new DbContext) {
Foo foo = context.Foos.Create();
context.Foos.Add(foo);
...
Bar bar = context.Bars.Create();
context.Bars.Add(bar);
...
FooBar foobar = context.FooBars.Create();
context.FooBars.Add(foobar)
foobar.Foo = foo;
foobar.Bar = bar;
//foo.FooBars is already populated, so 1 is returned and no database query is executed.
int count = foo.FooBars.Count;
context.SaveChanges();
//This causes a new query against the database - Why?
count = foo.FooBars.Count;
}
I can't say 100%, but I doubt this behavior is made specifically.
As for why it behaves this way, the root of the issue is that DbCollectionEntry.IsLoaded property is false until the navigation property is either implicitly lazy loaded or explicitly loaded using Load method.
Also seems that lazy loading is suppressed while the entity is in Added state, that's why the first call does not trigger reload. But once you call SaveChanges, the entity state becomes Unmodified, and although the collection is not really set to `null' or cleared, the next attempt to access the collection property will trigger lazy reload.
// ...
Console.WriteLine(context.Entry(foo).Collection(e => e.FooBars).IsLoaded); // false
//foo.FooBars is already populated, so 1 is returned and no database query is executed.
int count = foo.FooBars.Count;
// Cache the collection property into a variable
var foo_FooBars = foo.FooBars;
context.SaveChanges();
Console.WriteLine(context.Entry(foo).State); // Unchanged!
Console.WriteLine(context.Entry(foo).Collection(e => e.FooBars).IsLoaded); // false
// The collection is still there, this does not trigger database query
count = foo_FooBars.Count;
// This causes a new query against the database
count = foo.FooBars.Count;
If you want to avoid the reload, the workaround is to explicitly set IsLoaded property to true.
// ...
context.SaveChanges();
context.Entry(foo).Collection(e => e.FooBars).IsLoaded = true;
context.Entry(bar).Collection(e => e.FooBars).IsLoaded = true;
// No new query against the database :)
count = foo.FooBars.Count;
Why would you expect it not to re-query the database? EF isn't holding a complete cached copy of your database. What if you have some trigger(s) doing a row insert after something you changed or some column that serves as a function column that EF doesn't know the exact calculation for? It needs to get the most recent data otherwise your Count won't be correct.
I think the issue is the concept of 'context' is being confused. You are in a 'connected' state when you are in the context of EF. It doesn't always care to reuse your data smartly. It knows "I exist as an extension of the database level through a context set up to talk to it." If you have an object you create or one already existing say 'FooBars' and you do this:
foo.Foobars.(some operation)
If foo is your context and Foobars is some object off of it, it is referenced as an extension of context. If you want to realize reuse without incurring roundtrip to the database collect an object or objects outside the scope of the context like:
var foobars= new Foobars;
using(var foo = new new DbContext)
{
foobars = foo.FooBars;
}
var cnt = foobars.Count();
Generally speaking with EF, I see a lot of people do something like have a whole 'using(var context = new Efcontext())' over an entire method that may be long and do this process over and over everywhere. It is not bad out of the box per say, but doing this everywhere I would simply say: "Do you need to keep a db connection open over and over again?" Sometimes you do, most of the time you do not though.
I am facing the exception The ObjectContext instance has been disposed and can no longer be used for operations that require a connection even after using the Include method.
Here the function that retrieve the entities:
public List<Entity.CapacityGrid> SelectByFormula(string strFormula, int iVersionId)
{
// declaration
List<Entity.CapacityGrid> oList;
// retrieve ingredients
oList = (from Grid in _Dc.CapacityGrid.Include("EquipmentSection")
join Header in _Dc.CapacityHeader
on Grid.HeaderId equals Header.HeaderId
where Header.Formula == strFormula
&& Header.VersionId == iVersionId
select Grid).ToList();
// return
return oList;
Here the usage of the function:
// retrieve ingredient quantity by equipement
using (Model.CapacityGrid oModel = new Model.CapacityGrid(Configuration.RemoteDatabase))
oQuantity = oModel.SelectByFormula(strFormulaName, iVersionId);
// code to throw the exception
var o = (oQuantity[0].EquipmentSection.TypeId);
I understand that the using is closing the connection. I thought the ToList() will instantiated the list of objects and the related objects in the include before closing.
Can someone point me out what I do wrong?
Sorry, my question was not clear. I do understand that including the line that throw exception inside the bracket of the using is working, but I do not figure out why does the include does not works?
Thank you!
Try changing
// retrieve ingredient quantity by equipement
using (Model.CapacityGrid oModel = new Model.CapacityGrid(Configuration.RemoteDatabase))
{ // <-- **add these**
oQuantity = oModel.SelectByFormula(strFormulaName, iVersionId);
// code to throw the exception
var o = (oQuantity[0].EquipmentSection.TypeId);
} // <-- **add these**
Ref: http://msdn.microsoft.com/en-us/library/yh598w02.aspx
With no {} to englobe the using, the connection is disposed right after the first line. Because Entity framework uses Expression trees (this means that the request is not executed until it really needs it), your query happens at var o = (oQuantity[0].EquipmentSection.TypeId);.
There are three solutions to your problem. These come from this link here and are the three ways to link to a Related Entity. The first solution is the Lazy loading solution, that you have been using. Just modify your code to this and it will work. The reason why it was throwing an exception is because lazy loading occurs only when you need it. It's a great solution when you need to load the related entities only on a few entities.
// retrieve ingredient quantity by equipement
using (Model.CapacityGrid oModel = new Model.CapacityGrid(Configuration.RemoteDatabase))
{
oQuantity = oModel.SelectByFormula(strFormulaName, iVersionId);
// Lazy-load occurs here, so it needs to have access to the
// context, hence why it is in the using statement.
var o = (oQuantity.First().EquipmentSection.TypeId);
}
The second solution is to use eager-loading (as suggested by #DavidG). Because you only load the related entity of the first entity found, I do not recommend you use this in your case because it will load the EquipmentSection entity of all your oQuantity entities. In your SelectByFormula method, use an Include statement as shown in the related link and it will load it on the first call (it will not duplicate the access to the database, but it will pull more data at once).
The third solution is to avoid relying on Lazy Loading, and can be a good way to go to. It's the Explicit loading technique, which will require you to specify that you want to load the EquipmentSection related entity on the specified entity.
I hope those explanations help you.
Also, you might want to consider returning an IQueryable on your SelectByFormula method. This way, if you have to filter requests, like with the First() method to obtain only the first occurance, you are not pulling everything only for one instance.