The linq query taking too much time. Need to reduce the Time - c#

Here i am using the below query and its taking lots of time around 14 to 15 seconds for retrieving the large amount of data.
In below Query the CreatedDate is of DateTimeOffset data type.
var naId = UnitOfWork.SalesPhases.FirstOrDefault(p => p.PhaseName =="NA").SalesPhaseId;
var rejectedId = UnitOfWork.SalesPhases.FirstOrDefault(p => p.PhaseName =="Rejected").SalesPhaseId;
var data = UnitOfWork.Leads.Query().AsEnumerable()
.Where(p =>(p.SalesPhaseId == naId || p.SalesPhaseId == rejectedId) &&
p.CreatedDate.Date >= fromDate && p.CreatedDate.Date <= toDate).Select(m =>
new
{
m.LeadId,
m.LeadOwnerId,
m.SalesPhaseId,
m.LeadActivities,
m.Employee,
m.SalesPhase,
m.CompanyName,
m.CreatedDate,
m.LeadHistories,
m.LeadAddresses
}).ToList();
I tried using the AsQueryable instead of the AsEnumerable but it gives the below error:
"The specified type member 'Date' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported."
Can you help me out to reduce the execution time of the query?

Your use of AsEnumerable is forcing the filtering to be done locally. It's pulling in all the data, then filtering it in your app. That's clearly very inefficient. Now, it seems that part of your query can't be directly expressed in LINQ to SQL. I see two options here.
Firstly you could do most of your filtering in SQL, but then do the date filtering locally:
var data = UnitOfWork.Leads.Query()
// Do this part of the query in SQL
.Where(p => p.SalesPhaseId == naId ||
p.SalesPhaseId == rejectedId)
.AsEnumerable()
// Do the rest of the query in-process
.Where(p => p.CreatedDate.Date >= fromDate &&
p.CreatedDate.Date <= toDate)
.Select(...)
That's suitable if the first part will filter it down massively, and then you only need to do local processing of a small set of data.
Alternatively, you could work out what your date filtering means in terms of DateTime. It looks like you could do:
// This may not be required, depending on the source.
fromDate = fromDate.Date;
// This will be, although you may be able to get rid of the ".Date" part.
toDate = toDate.Date.AddDays(1);
var data = UnitOfWork.Leads.Query()
// Do this part of the query in SQL
.Where(p => (p.SalesPhaseId == naId ||
p.SalesPhaseId == rejectedId) &&
p.CreatedDate >= fromDate &&
p.CreatedDate < toDate)
.Select(...)
That's created an equivalent query, but without using the Date property in the query itself.

Everything after AsEnumerable() is executed locally rather than on the server. See also
https://stackoverflow.com/a/2013876/141172
This means that all rows in the table are returned from the database, and then filtered in your C# code.
Remove that call so that the filtering happens server-side.
EDIT
Noticed Jon's comment and it reminded me that he reimplemented LINQ to Objects as a learning exercise. His comments about the AsEnumerable() reimplementation are worth reading
I can describe its behaviour pretty easily: it returns source.
That's all it does. There's no argument validation, it doesn't create another iterator. It just returns source.
You may well be wondering what the point is... and it's all about changing the compile-time type of the expression. I'm going to take about IQueryable in another post (although probably not implement anything related to it) but hopefully you're aware that it's usually used for "out of process" queries - most commonly in databases.
Now it's not entirely uncommon to want to perform some aspects of the query in the database, and then a bit more manipulation in .NET - particularly if there are aspects you basically can't implement in LINQ to SQL (or whatever provider you're using). For example, you may want to build a particular in-memory representation which isn't really amenable to the provider's model.
https://msmvps.com/blogs/jon_skeet/archive/2011/01/14/reimplementing-linq-to-objects-part-36-asenumerable.aspx

Your code should like this..
var naId = UnitOfWork.SalesPhases.FirstOrDefault(p => p.PhaseName =="NA").SalesPhaseId;
var rejectedId = UnitOfWork.SalesPhases.FirstOrDefault(p => p.PhaseName =="Rejected").SalesPhaseId;
var data = UnitOfWork.Leads.Query().AsQueryable()
.Where(p =>(p.SalesPhaseId == naId || p.SalesPhaseId == rejectedId) &&
p.CreatedDate>= fromDate.Date && p.CreatedDate <= toDate.Date).Select(m =>
new
{
m.LeadId,
m.LeadOwnerId,
m.SalesPhaseId,
m.LeadActivities,
m.Employee,
m.SalesPhase,
m.CompanyName,
m.CreatedDate,
m.LeadHistories,
m.LeadAddresses
}).ToList();
Firstly, You need to use .ToQueryable instead of .ToIEnumerable().
Secondly, you cannot use .Date to datetime properties inside a entity framework linq query. That only works for in-memory collections like list and arrays.

Related

My ForEach loop with a LINQ expression cannot be translated?

I need to go through all my chemicals and send out a warning to users if the expiration date is either 2 days away or 15 days away.
I have it set like that so the warning is sent out twice. Once 2 weeks before it expires, and once 2 days before it expires.
So I have this loop:
foreach (var chemical in _context.Chemical.Where(c => (c.DateOfExpiration - DateTime.Now).TotalDays == 15 || (c.DateOfExpiration - DateTime.Now).TotalDays == 2))
{
// send out notices
var base = _context.Base.Find(chemical.BaseId);
var catalyst = _context.Catalyst.Find(chemical.CatalystId);
_warningService.SendSafetyWarning(chemical, base, catalyst.Name);
}
But whenever I run it, I get this error:
System.InvalidOperationException: 'The LINQ expression 'DbSet<Chemical>
.Where(p => (p.DateOfExpiration - DateTime.Now).TotalDays == 15 || (p.DateOfExpiration - DateTime.Now).TotalDays == 2)'
could not be translated.
I'm not sure why it's giving me this error when I run it.
Is there a better way of doing what I'm trying to do?
Thanks!
DateTime.TotalDays returns a double.
You might have more luck changing your code to:
foreach (var chemical in _context.Chemical.Where(c => ((int)(c.DateOfExpiration - DateTime.Now).TotalDays == 15) || ((int)(c.DateOfExpiration - DateTime.Now).TotalDays == 2)))
{
// send out notices
var base = _context.Base.Find(chemical.BaseId);
var catalyst = _context.Catalyst.Find(chemical.CatalystId);
_warningService.SendSafetyWarning(chemical, base, catalyst.Name);
}
There are two solutions to this problem.
I'll first explain why it is throwing an error. EF is converting your code to an SQL Expression. When working with Expression you cannot parse in any C# code and expect it to work. EF needs to be able to translate it to SQL. That is why you are getting the exception saying that it can't translate your code.
When working directly in the Where clause on the DbSet or the DbContext you are actualy talking to the Where method for an IQueryable. Which has a parameter of Expression<Func<bool, T>> where T is your DbSet<T> type.
You either need to simplify your query so it can be translated to SQL. Or you can convert your IQueryable to an IEnumerable.
When you want to run your query on the SQL server, you need to use the first approach. But when you can run your query on the client you can use the second approach.
The first approached as mentioned by vivek nuna is using EntityFunctions. It might not be enough for your use case, because it has limited functionality.
The statement made that DateTime.Now is different on each iteratoion is not true. Because the query is converted once, it isn't running the query on every iteration. It simply doesn't know how to translate your query to SQL.
https://learn.microsoft.com/en-us/dotnet/api/system.data.objects.entityfunctions?view=netframework-4.8
The second approach means adding AsEnumerable after your DbSet. This will query all your data from your SQL server, and evaluate in C#. This is usually not recommended if you have a large data set. Example:
var chemicals = _context.Chemical.AsEnumerable(); // this will get the complete collection
chemicals.Where(i => i.Value == true); // everything will work now
And answer to a "better" approach. It's a bit cleaner code:
If you have navigation properties for your Base and Catalyst you can also include this in your query.
Note that you can still query server side before pulling everything client side using a Where on the _context.Chemical and then including and calling AsEnumerable.
var chemicals = _context.Chemical.Include(i => i.Base).Include(i => i.Catalyst).AsEnumerable();
foreach (var chemical in chemicals.Where(i => true)) // set correct where clause
{
_warningService.SendSafetyWarning(chemical, chemical.Base, chemical.Catalyst);
}
You can declare the variable before and then pass it to your expression. Like DateOfExpiration == nextFifteenDays …..
var nextFifteenDays = DateTime.Now.Adddays(15);
var nextTwoDays = DateTime.Now.AddDays(2);
Or you can use EntityFunctions.Diffdays
Reason for the error is well explained by #neil that DateTime.Now will keep on changing, so can’t be used.

Querying external data source with LINQ

I'm storing what basically amounts to log data stored in CSV files. It's of the format <datetime>,<val1>,<val2>, etc. However, the log files are stored by account ID and month, so if you query across months or account IDs you're going to retrieve multiple files.
I'd like to be able to query it with LINQ, so that if I could call logFiles.Where(o => o.Date > 1-1-17 && o.Date < 4-1-17). I suppose I'll need something to examine the date range in that query and notice that it spans 4 months, which then causes it to only examine files in that date range.
Is there any way to do this that does not involve getting my hands very dirty with a custom IQueryable LINQ provider? I can go down that rabbit hole if necessary, but I want to make sure it's the right rabbit hole first.
If you want to filter both on the log file name and on the log file contents in the same Where expression, I don't see a solution without a custom IQueryable LINQ provider, because that's exactly the use case for them: To access data in a smart way based on the expressions used in the LINQ query.
That said, it might be worth to use a multi-step approach as a compromise:
Use LINQ to restrict the log files to be searched,
read the files and
use LINQ for further searching.
Example:
IEnumerable<LogFile> files = LogFiles.Where(f => f.Date > new DateTime(17, 1, 1) && f.AccountID == 4711);
IEnumerable<LogData> data = ParseLogFiles(files);
IEnumerable<LogData> filteredData = data.Where(d => d.val1 == 42 && d.val2 > 17);
LogData firstMatch = filteredData.FirstOrDefault();
If you implement ParseLogFiles (a) with deferred execution and (b) as an extension method on IEnumerable<LogFile>, the resulting code will look-and-feel very similar to pure LINQ:
var filteredData = LogFiles.
Where(f => f.Date > new DateTime(17, 1, 1) && f.AccountID = 4711).
ParseLogFiles().
Where(d => d.val == 42 && d.val2 > 17);
// If ParseLogFiles uses deferred execution, the following line won't read
// more log files than required to get the first matching row:
var firstMatch = filteredData.First();
It's a bit more work than having it all in one single LINQ query, but it saves you from having to implement your own LINQ provider.

Linq to Entities over EF 4 on SQL Server 2005

'Cannot compare elements of type 'System.Collections.Generic.ICollection`1'. Only primitive types (such as Int32, String, and Guid) and entity types are supported.'
Entities & Relationships (Navigation Properties):
Case (1 <-> ∞) MeetingCase (1 <-> ∞) MeetingCaseOutcomes
The offending code is:
IQueryable<Case> cases; // this is obviously attached to a context and able to access data)
var broken = cases
.Where(c => c.MeetingCases
.Where(mc => mc.ExpectedStartDateTime <= DateTime.Now)
.Any(m => m.MeetingCaseOutcomes == null || m.MeetingCaseOutcomes.Count == 0));
I assume the problem is due to lack of support for the 'Any' operator, though I had thought this would work in EF 4 (since support for the related 'Contains' operator has been added).
How should I re-structure this call to produce what I want?
I have been working with EF for several months now and understand many of the runtime gotchas
UPDATE:
The Where clause above does contain a predicate, as follows:
c.MeetingCases.Where(mc => mc.ExpectedStartDateTime <= DateTime.Now)
.Any(m => m.MeetingCaseOutcomes == null || m.MeetingCaseOutcomes.Count == 0)
Because Any returns a boolean, the whole thing produces a predicate expression.
Also, the intent of this logic is to return a set of Case objects which do not have MeetingCaseOutcome records for any MeetingCase records where the meeting has already taken place (hence the comparison to DateTime.Now). This is part of a meeting scheduling system, and this is to check that Outcomes from each meeting are entered into the system following the meeting taking place.
EF 1 and EF 4 both support .Any(). The problem is that you're not using it enough. :)
IQueryable<Case> cases; // this is obviously attached to a context and able to access data)
var broken = cases
.Where(c => c.MeetingCases
.Where(mc => mc.ExpectedStartDateTime <= DateTime.Now)
.Any(m => !m.MeetingCaseOutcomes.Any()));
This part is the problem:
Where(c => c.MeetingCases
.Where..
.Where requires a Expression<Func<T,bool>>, in otherwords a predicate which returns true/false. But your supplying it a predicate which returns a sequence of elements.
I think that second .Where needs to be changed to .Any or .All.
I'm struggling to decipher what query your attempting to execute, but i'm assuming you want a list of Case where at least one associated MeetingCase has a ExpectedStartDateTime before today.
So your query should be:
var cases = cases
.Where(case => case.MeetingCases
.Any(caseMeeting => caseMeeting.ExpectedStartDateTime <= DateTime.Now));
I'm not sure what your trying to do with that last .Any clause.
Also - don't forget about eager-loading/projection problems. You say "it's attached to the context", which is fine - but c.MeetingCases will not return anything unless you have lazy loading on or are eager loading beforehand.

LINQ to Entities: Using DateTime.AddMonth in Lambda Expression

I've been reading some posts but I don't find a solution to a problem that I have with LINQ To Entities, Lambda Expressions and DateTime.AddMonth.
The problem is that I'm trying to use DateTime.AddMonth inside a Lambda Expression and I'm getting this error:
"LINQ to Entities does not recognize the method 'System.DateTime
AddMonths(Int32)' method, and this method cannot be translated into a
store expression"
when I execute this piece of code:
List<Orders> orders = context.Orders
.Where(o => o.IdOrderStatus == 1)
.Where(o => o.PaymentDate.Value.AddMonths(o.Products.ProductCategories.CommissionableMonths) > DateTime.Now)
.ToList();
Is there a way to avoid this exception but mantaining the same behavior?
I don't know a lot about Linq, Lambdas or Entity Framework.
Thank you very much in advance!
Gsus.
You could return the necessary information without filtering on the date, then filter on the date afterward. (Of course, I'm not sure what size your data will be, so this may be inefficient. If so, you'll need a SQL-based solution of some sort -- maybe a stored procedure.)
List<Orders> orders = context.Orders
.Where(o => o.IdOrderStatus == 1)
.ToList();
orders = orders.Where(o.PaymentDate.Value.AddMonths(o.Products.ProductCategories.CommissionableMonths) > DateTime.Now);
This turns the AddMonths portion of the query in to a Linq-to-Objects method instead of Linq-to-Entities call.
try this,
var today =DateTime.Now;
List<Orders> orders = context.Orders
.Where(o => o.IdOrderStatus == 1)
.Where(o => SqlFunctions.DateAdd("month" ,o.Products.ProductCategories.CommissionableMonths,o.PaymentDate) > today)
.ToList();
LINQ to Entities converts your linq expression into SQL code, sometimes this conversion is seamless such as integer addition, selects, groups, etc, but sometimes the abstraction leaks, as in date operations.

How to Use Dates in Where Clause in EF Core?

I need to filter my queries by dates but I don't care in this case about time portion of it that is stored in SQL Database.
I first tried to something like
var now = DateTime.Now.Date;
Where(x => x.CreatedDate.Date.Compare(now) == 0)
but this seems to all get locally checked making the query slow. How can I do this without making it do the check locally?
I am pretty much trying to just find all results that would say have happened today(2020-01-06).
There are a limited number of methods you can use on translatable types when constructing your Lambda / Linq expressions. This is because each method would need additional code so that it could be translated into a sql store expression. It means that you must check that any methods you want to use and expect to be translated into a sql store expression are supported.
In this case the DateTime.Compare is not supported.
The easiest thing to do here is a simple range comparison because the time is included in your persisted value.
var start = DateTime.Now.Date;
var end = start.AddDays(1);
Where(x => x.CreatedDate >= start && x.CreatedDate < end)
This will result in a sargable query.
Use
var now = DateTime.Now.Date
...WHERE(CreatedDate.Date == now)
I just checked that above translates to the following SQL query:
WHERE ((CONVERT(date, [x].[CreatedDate]) = '2019-01-07T00:00:00.000')
I used this (link) method to see what LINQ translates to

Categories