This is my first time working with Entity Framework (EF) and I'm trying to learn what exactly executes a query on my database and what doesn't.
This is the code I'm working with. Don't mind the functionality, it isn't important for this question.
using (var db = new Context())
{
//Check if any reviews have been given.
if (combinedReviews.Any())
{
var restaurantsReviewedIds = combinedReviews.Select(rev => rev.RestaurantId);
//(1)
ratedRestaurants = db.Restaurants.Where(rest => restaurantsReviewedIds.Contains(rest.Id))
.DistinctBy(rest => rest.Id)
.ToList();
}
//(2)
var restsClose = db.Restaurants.Where(rest => db.Reviews.Any(rev => rev.RestaurantId == rest.Id))
.OrderBy(rest => rest.Location.Distance(algorithmParams.Location))
.Take(algorithmParams.AmountOfRecommendations);
//(3)
tempList = ratedRestaurants.Union(restsClose).ToList();
var tempListIds = tempList.Select(rest => rest.Id); //Temporary list.
//(4)
restsWithAverage = db.Reviews.Where(rev => tempListIds.Contains(rev.RestaurantId))
.GroupBy(rev => rev.RestaurantId)
.ToList();
}
I have marked each piece of code with numbers, so I'll refer to them with that. Below is what I think is what happens.
This executes a query since I'm calling .ToList() here.
This returns an IQueryable, so this won't execute a query against the database.
This executes the query from (2).
This executes another query since I'm calling .ToList().
How close to the truth am I? Is all of this correct? If this doesn't make sense, could you give an example what executes a query and what doesn't?
I'm sorry for asking so many questions in one question, but I thought I wouldn't need to create so many questions since all of this is about a single topic.
If you don't want to execute a query you can use AsEnumerable.
ToList vs AsEnumerable
ToList – converts an IEnumerable<T> to a List<T>. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
You could also force execution by putting a For Each loop immediately after the query expression, but by calling ToList or ToArray you cache all the data in a single collection object.
ToLookup and ToDictionary also executing the queries.
Here you can find a list of operators and if they are executing query:
https://msdn.microsoft.com/en-us/library/mt693095.aspx.
Linq query execution is different per query. I recommend reading the following page: https://msdn.microsoft.com/en-us/library/bb738633(v=vs.110).aspx
Related
I'm trying to use the .ToLookup() method with an EF Core query and wondering what the best practice is when using it, should I be buffering the query into a list first, or call .ToLookup() directly on the IQueryable?
var lookup = DbContext.Foo.Where(f => f.Id > 1).ToLookup(f => f.Id);
//vs:
var lookup = (await DbContext.Foo.Where(f => f.Id > 1).ToListAsync(cancellation)).ToLookup(f => f.Id);
My main concern is the ToListAsync approach will execute the query asynchronously whereas the direct .ToLookup call looks like it will block until results of the query are returned.
However as #Tim mentioned the ToListAsync approach will end up creating 2 collections in memory.
Thanks
ToLookup creates an in-memory collection similar to a Dictionary<TKey, TValue>, so there is no need to create a list and then call ToLookup, you will just waste CPU and memory for no reason.
So it's similar to ToDictionary, but different to GroupBy. The latter is using "deferred execution", which means you are still in a database context when you consume it, whereas the lookup or dictionary are collections that are already filled.
I'm running into some speed issues in my project and it seems like the primary cause it calls to the database using entity framework. Every time I call the database, it is always done as
database.Include(...).Where(...)
and I'm wondering if that is different than
database.Where(...).Include(...)?
My thinking is that the first way includes everything for all the elements in the target table, then filters out the ones I want, while the second one filters out the ones I want, then only includes everything for those. I don't fully understand entity framework, so is my thinking correct?
Entity Framework delays its querying as long as it can, up until the point where your code start working on the data. Just to prove the example:
var query = db.People
.Include(p => p.Cars)
.Where(p => p.Employer.Name == "Globodyne")
.Select(p => p.Employer.Founder.Cars);
With all these chained calls, EF has not yet called the database. Instead, it has kept track of what you're trying to fetch, and it knows what query to run if you start working with the data. If you never do anything else with query after this point, then you will never hit the database.
However, if you do any of the following:
var result = query.ToList();
var firstCar = query.FirstOrDefault();
var founderHasCars = query.Any();
Now, EF is forced to look at the database because it cannot answer your question unless it actually fetches the data from the database. At this point, not before, does EF actually hit the database.
For reference, this trigger to fetch the data is often referred to as "enumerating the collection", i.e. turning a query into an actual result set.
By deferring the execution of that query for as long as possible, EF is able to wait and see if you're going to filter/order/paginate/transform/... the result set, which could lead to EF needing to return less data than when it executes every command immediately.
This also means that when you call Include, you're not actually hitting the database, so you're not going to be loading data from items that will later be filtered by your Where clause, if you didn't enumerate the collection.
Take these two examples:
var list1 = db.People
.Include(p => p.Cars)
.ToList() // <= enumeration
.Where(p => p.Name == "Bob");
var list2 = db.People
.Include(p => p.Cars)
.Where(p => p.Name == "Bob")
.ToList(); // <= enumeration
These lists will eventually yield the same result. However, the first list will fetch data before you filter it because you called ToList before Where. This means you're going to be loading all people and their cars in memory, only to then filter that list in memory.
The second list, however, will only enumerate the collection when it already knows about the Where clause, and therefore EF will only load people named Bob and their cars into memory. The filtering will happen on the database before it gets sent back to your runtime.
You did not show enough code for me to verify whether you are prematurely enumerating the collection. I hope this answer helps you in determining whether this is the cause of your performance issues.
database.Include(...).Where(...) and I'm wondering if that is different than database.Where(...).Include(...)?
Assuming this code is verbatim (except the missing db set) and there is nothing happening inbetween the Include and Where, the order does not change the execution and therefore it is not the source of your performance issue.
I generally advise you to put your Include statements before anything else (i.e. right after db.MyTable), as a matter of readability. The other operations depends on the specific query you're trying to construct.
Most of the times order of clauses will not make any difference
Include statement tells to SQL Join one table with another
While Where will results in.. yes, SQL Where
When you do something like database.Include(...).Where(...) you are building IQueryable object that will be transleted to direct SQL after you try to access it like with .ToList() or .FirstOrDefault() and those queries are already optimized
So if you still have performance issues - you should use profiler to look for bottlenecks and maybe consider using stored procedures (those could be integrated with EF)
I use lambda expression to query data, whenever i debug and view memory, memory continuous increase.It can reason make exception "System.OutOfMemoryException".How to clear cache memory whenever finish execute lambda expression?
public override IQueryable<TableA> GetList(Func<TableA, bool> conditon, bool isNoTracking = false)
{
var result = Context.TableA
.Join(Context.TableB.AsNoTracking(), x => x.TYU_URICOMCODE, y => y.MACS_KOKYAKU_CODE, CreateOrder)
.Where(conditon).AsQueryable();
return result;
}
I have a few suspicions:
Your CreateOrder side-effect is doing something that is using up memory. Remember it's running per-row returned.
You're inadvertently loading all rows from the database, then using the in-memory LINQ methods to join/filter it.
Note that in the database-specific flavour of LINQ, .Where() returns a WhereIterator, which is already queryable, so building a new queryable is unnecessary, and will also be doing memory allocations. If you're having to call .AsQueryable() to get it to compile, then you're almost certainly falling foul of suspicion #2 above.
Will LINQ use defered execution when we map the result to an object?
var x = from rcrd in DBContext.FooTable
select new Model.BOFoo()
{ Bar = rcrd.Bar };
The above code map rcrd to Model.BOFoo object. Will this mapping cause LINQ to fetch the actual data from the database? Or will it wait until I call x.ToList()?
I'd answer yes. If I don't miss any information about this, LINQ will still use deferred execution even if we map the result to object. The object initialization will also be deferred.
Unless your Linq query includes a method that executes the query, then no, it will not execute and will be deferred.
Examples of methods that execute the query include First(), FirstOrDefault(), ToList(), ToArray, etc..
select is not such a method, not even select new.
I always find it useful to think of the code as it would be in a method chain:
DBContext.FooTable
.Select(rcrd=>new Model.BOFoo{Bar=rcrd.Bar});
Here it is easier to visualize the deferred execution as you are only passing a function into the Select that will be evaluated as needed. The new is merely a part of that function.
So, as har07 and Erik already mentioned, if it is a deferred execution method, then it will remain that way unless forced via another method such as ToList
I'm trying to load a lot of records from the DB and I would like to run them in parallel to speed things up.
Below is some example code which breaks when it tries to access the Applicants property which is null. However in a non-parallel loop, Applicants property is either populated or is an empty list, but is never null. Lazy loading is definitely enabled.
var testList = new List<string>();
Context.Jobs
.AsParallel()
.WithDegreeOfParallelism(5)
.ForAll(
x => testList.Add(x.Applicants.Count().ToString())
);
Can I do something like this? Is it related to the entity framework connection? Can I make it parallel friendly and pass an instance of it into the task or something? I'm just shooting out ideas but really I haven't a clue.
Edit:
Is this post related to mine? My issue sounds kind of similar. Entity Framework lazy loading doesn't work from other thread
PLINQ does not offer a way to parallelize LINQ-to-SQL and LINQ-to-Entities queries. So when you call AsParallel EF should first materialize the query.
Furthermore, it doesn't make any sence to parallelize the query that executes on database, cause database can do that itself.
But if you want to parallelize cliend-side stuff, below code may help:
Context.Jobs
.Select(x => x.Applicants.Count().ToString())
.AsParallel()
.WithDegreeOfParallelism(5)
.ForAll(
x => testList.Add(x)
);
Note that you can access navigation properties only before the query is materialized. (in your case before AsParallel() call). So use Select to get all what you want.
Context.Jobs
.Select(x => new { Job = x, Applicants = x.Applicants })
.AsParallel()
.WithDegreeOfParallelism(5)
.ForAll(
x => testList.Add(x.Applicants.Count().ToString())
);
You also can use Include method to include navigation properties into results of the query...
Context.Jobs
.Include("Applicants")
.AsParallel()
.WithDegreeOfParallelism(5)
.ForAll(
x => testList.Add(x.Applicants.Count().ToString())
);