I use lambda expression to query data, whenever i debug and view memory, memory continuous increase.It can reason make exception "System.OutOfMemoryException".How to clear cache memory whenever finish execute lambda expression?
public override IQueryable<TableA> GetList(Func<TableA, bool> conditon, bool isNoTracking = false)
{
var result = Context.TableA
.Join(Context.TableB.AsNoTracking(), x => x.TYU_URICOMCODE, y => y.MACS_KOKYAKU_CODE, CreateOrder)
.Where(conditon).AsQueryable();
return result;
}
I have a few suspicions:
Your CreateOrder side-effect is doing something that is using up memory. Remember it's running per-row returned.
You're inadvertently loading all rows from the database, then using the in-memory LINQ methods to join/filter it.
Note that in the database-specific flavour of LINQ, .Where() returns a WhereIterator, which is already queryable, so building a new queryable is unnecessary, and will also be doing memory allocations. If you're having to call .AsQueryable() to get it to compile, then you're almost certainly falling foul of suspicion #2 above.
Related
I'm implementing a Unit OF Work pattern (not my choice, I know it's considered by some as an anti-pattern).
I've come across a situation I don't fully understand.
My generic repo constructor:
this.context = context;
this.dbSet = context.Set<T>();
Generic method:
public virtual async Task<IEnumerable<T>> All()
{
return await dbSet.ToListAsync();
}
Using it like this:
var languages = await _unitOfWork.Languages.All();
languages = languages.OrderBy(x => x.Order);
As shown above, I need to go down a line so I can use OrderBy,
I don't understand why.
A second question is, ToListAsync is supposed to return a list, why do I get an IEnumerable?
A second Question is, ToListAsync is suppose to return a list, why do I get an IENUMERABLE?
List is an implementation of IEnumerable. Check What does it mean to "program to an interface"?.
As shown above, I need to go down a line so I can use OrderBy, I don't why.
_unitOfWork.Languages.All() returns a Task, you should get its unwrapped result of IEnumerable to apply OrderBy on it.
To make this work as you expected, you should apply OrderBy on the awaited result:
(await _unitOfWork.Languages.All()).OrderBy(x => x.Order);
Your confusion as far as having them on the same line seems to be on how it's applying the await. This should work fine:
var languages = (await _unitOfWork.Languages.All()).OrderBy(x => x.Order);
You need to do this because your All function is returning a Task while OrderBy is not.
Additionally, you probably don't want to call .ToListAsync() in the first place, that's just a SELECT * without any limit or where clause which may not be the "end of the world" all the time but often will be extremely detrimental to your performance.
As for your second question about why it's returning an IEnumerable<T> it's because List<T> implements IEnumerable<T> and your function signature indicates that it returns IEnumerable<T>
public virtual async Task<IEnumerable<T>> All()
I highly recommend instead of ever dealing with .ToListAsync or IEnumerable<T> you leave it alone and let IQueryable<T> do its proper work against the database.
EDIT: It's been asked in comments why it matters that you'd use IQueryable<T>. The answer is because utilizing .Where, .Any() etc... against IQueryable will mold the underlying query against the DB.
For example, say you want to find a single entity with the Id of 123. If you leave your .ToListAsync() without any other modifications it will pull back every single row in the DB into your program in memory and then iterate every single row looking for that one row.
If, instead, you use IQueryable<T> - at the point where you apply .FirstOrDefault(e => e.Id == 123) it will be applied to the database and it will apply as something like SELECT TOP(1) * FROM MyEntity WHERE [Id] = 123 - pulling back a single row instead of every row in existence.
EDIT2: Note that this also applies to projection. This means something like .Select(e => new { FullName = e.FirstName + " " + e.LastName, State = e.Address.State}) will only pull those 2 columns instead of all columns and any nav properties you include. Doing that after a .ToList() or .ToListAsync will instead pull back all columns/entities and iterate over them to create a whole new set of other entities. This can result in MASSIVE CPU/Memory differences between the 2 approaches.
I need to call the method for each item in the list. So, I have used Where<> query as follows,
List<string> list = new List<string>();
list.Add("Name1");
list.Add("Name2");
list.Add("Name3");
var name = list.Where(n =>
{
return CheckName(n);
});
But in the above case, CheckName() is not hit. The same method is triggered if I use FirstOrDefault<>. I don't know whether it is a framework break or I am going in a wrong way.
As additional info, I am using.NET Framework 4.5.
Has anyone experienced this error? If so, is there any solution to overcome this issue?
You are understanding incorrectly the result of the Where condition. As linq is deffered executed it will only enter the where condition when materialized (by a ToList/FirstOrDefault/Sum and such).
The Where is never actually materialized in your current code (It did as you experienced when using FirstOrDefault) and as such it will never enter the CheckName method. Then, as Where will never return null but "worst case" an empty collection, which is not null, the result is true.
If you debug you will see that name equals true at the end of this. To "overcome" this depends on what is your desired output:
If you want to know if you have any item that matched the predicate then:
var result = list.Any(CheckName);
If you want to retrieve those that match the predicate:
var result = list.Where(CheckName);
If later you want to query and check if results contains anything then:
if(result.Any()) { /* ... */ }
If you only want the results (and thus materializing the query):
list.Where(CheckName).ToList();
Read more about linq being deffered executed here:
Linq and deffered execution
What are the benefits of a Deferred Execution in LINQ?
Just as a side note see how you can change your current code from:
var name = list.Where(n =>
{
return CheckName(n);
})
To:
var name = list.Where(n => CheckName(n));
And eventually to:
var name = list.Where(CheckName);
LINQ has a Deferred Execution principal which means the query will not be executed until and unless you access name variable. If you want to execute it immediately, (just for example) add .ToList() in the end, which is exactly what FirstOrDefault does. It does immediate execution instead of deferred execution.
var name = list.Where(n =>
{
return CheckName(n);
}).ToList() != null;
Also where condition result will never be null. Even if there is no object in list satisfying your condition(s) in CheckName, where will return an empty collection.
The CheckName() method is not executed because of Deferred execution of Linq. The actual statement is not executed till you actually access it. So in your case, for the CheckName(), you should do something like:
var name = list.Where(n =>
{
return CheckName(n);
}).ToList();
When you look at the Where-Method source code you can easily see why:
internal static IEnumerable<T> Where<T>(this IEnumerable<T> enumerable, Func<T, bool> where) {
foreach (T t in enumerable) {
if (where(t)) {
yield return t;
}
}
}
The yield will cause the execution to only happen once the returned IEnumerable<T> is actually accessed. That is what is called deferred execution.
If you need to call a method for each item in a list then you should use a simple for loop:
foreach var name in list
CheckName(name);
Just because LINQ is available, doesn't mean it should be used everywhere there is a collection. It is important to write code that makes sense and is self commenting and using it here has simultaneously introduced a flaw into your logic and made your code harder to read, understand and maintain. It's the wrong tool for the stated purpose
Doubtless you have additional requirements not stated here, like "I want to check every name in a list and make sure that none are null". You can and possibly should use linq for this but it looks more like
bool allNamesOK = list.All(n => n != null);
This code is compact and reads well; we can clearly see the intention (though I wouldn't call the list "list" - "names" would better)
This is my first time working with Entity Framework (EF) and I'm trying to learn what exactly executes a query on my database and what doesn't.
This is the code I'm working with. Don't mind the functionality, it isn't important for this question.
using (var db = new Context())
{
//Check if any reviews have been given.
if (combinedReviews.Any())
{
var restaurantsReviewedIds = combinedReviews.Select(rev => rev.RestaurantId);
//(1)
ratedRestaurants = db.Restaurants.Where(rest => restaurantsReviewedIds.Contains(rest.Id))
.DistinctBy(rest => rest.Id)
.ToList();
}
//(2)
var restsClose = db.Restaurants.Where(rest => db.Reviews.Any(rev => rev.RestaurantId == rest.Id))
.OrderBy(rest => rest.Location.Distance(algorithmParams.Location))
.Take(algorithmParams.AmountOfRecommendations);
//(3)
tempList = ratedRestaurants.Union(restsClose).ToList();
var tempListIds = tempList.Select(rest => rest.Id); //Temporary list.
//(4)
restsWithAverage = db.Reviews.Where(rev => tempListIds.Contains(rev.RestaurantId))
.GroupBy(rev => rev.RestaurantId)
.ToList();
}
I have marked each piece of code with numbers, so I'll refer to them with that. Below is what I think is what happens.
This executes a query since I'm calling .ToList() here.
This returns an IQueryable, so this won't execute a query against the database.
This executes the query from (2).
This executes another query since I'm calling .ToList().
How close to the truth am I? Is all of this correct? If this doesn't make sense, could you give an example what executes a query and what doesn't?
I'm sorry for asking so many questions in one question, but I thought I wouldn't need to create so many questions since all of this is about a single topic.
If you don't want to execute a query you can use AsEnumerable.
ToList vs AsEnumerable
ToList – converts an IEnumerable<T> to a List<T>. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
You could also force execution by putting a For Each loop immediately after the query expression, but by calling ToList or ToArray you cache all the data in a single collection object.
ToLookup and ToDictionary also executing the queries.
Here you can find a list of operators and if they are executing query:
https://msdn.microsoft.com/en-us/library/mt693095.aspx.
Linq query execution is different per query. I recommend reading the following page: https://msdn.microsoft.com/en-us/library/bb738633(v=vs.110).aspx
I was looking into Enumerable.ToLookup API which converts an enumerable sequence into a dictionary type data structure. More details can be found here:
https://msdn.microsoft.com/en-us/library/system.linq.enumerable.tolookup(v=vs.110).aspx
The only difference it carries from ToDictionary API is the fact that it won't give any error if the key selector results in duplicate keys. I need a comparison of deferred execution semantics of these two APIs. As far as I know, ToDictionary API results in immediate execution of the sequence i.e. it doesn't follow deferred execution semantics of LINQ queries. Can anyone help me with the deferred execution behavior of ToLookup API? Is it the same as ToDictionary API or there is some difference?
Easy enough to test...
void Main()
{
var lookup = Inf().ToLookup(i => i / 100);
Console.WriteLine("if you see this, ToLookup is deferred"); //never happens
}
IEnumerable<int> Inf()
{
unchecked
{
for(var i=0;;i++)
{
yield return i;
}
}
}
To recap, ToLookup greedily consumes the source sequence without deferring.
In contrast, the GroupBy operator is deferred, so you can write the following to no ill-effect:
var groups = Inf().GroupBy(i => i / 100); //oops
However, GroupBy is greedy, so when you enumerate, the entire source sequence is consumed.
This means that
groups.SelectMany(g=>g).First();
also fails to complete.
When you think about the problem of grouping, it quickly becomes apparent that when separating a sequence into a sequence of groups, it would be impossible to know if even just one of the groups were complete without completely consuming the entire sequence.
This was sort of covered here, but it was hard to find!
In short -- ToLookup does not defer execution!
ToLookup() -> immediate execution
GroupBy() (and other query methods) -> deferred execution
If you look at the reference implementation source code for both the Enumerable.ToDictionary() and the Enumerable.ToLookup() methods, you will see that both end up executing a foreach loop over the source enumerable. That's one way to confirm that the execution of the source enumerable is not deferred in both cases.
But I mean, the answer is pretty self evident in that if you start off with an enumerable, and the return value of the function is no longer an enumerable, then clearly, it must have been executed (consumed), no?
(That last paragraph was not accurate as pointed out by #spender in the comments)
I have an object IQueryable that represent some query over my db-model.
Now i need to do some validation to that IQeryable and return back the object in that way
private IQuerable<myTableType> ValidateQery ( IQuerable<myTableType> query , string user )
{
return query.Where ( x => db.tblUsers.Where( y => y.User == user && y.Privilege >= x.PrivilegeRequired).Any() )
}
Now what i would like to understand is if the Any() clause is executed immidiatly or if it generate T-SQL that will be merged with previous.
I ask that because i wuold avoid to execute a query against the db in that moment.
Edit :
Thanks for the advice of 'using' clause, it was a fault that i made now (writing this code example) because i am not able to test code now (i have no visual studio on my machine in this moment) but my problem in general was about that situation.
The Any is inside of a lambda in Queryable.Where, and as such is not executed as C# code and is instead translated into an Expression and passed to the query provider when the IQueryable returned from Where is iterated.
On a site node, you're disposing of the context before the IQueryable you return can ever be iterated, so it'll always fail when you do iterate it.
It's part of the query - it's within a Where clause.
So no, it won't be executed immediately. Having said that, you appear to be closing the context that the query uses, so I suspect that by the time it is used, it'll be broken...