In search page I have some options based on them search query must be different . I have wrote this :
int userId = Convert.ToInt32(HttpContext.User.Identity.GetUserId());
var followings = (from f in _context.Followers
where f.FollowersFollowerId == userId && f.FollowersIsAccept == true
select f.FollowersUserId).ToList();
int value;
if (spto.Page == 0)
{
var post = _context.Posts.AsNoTracking().Where(p => (followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true).Select(p => p).AsEnumerable();
if(spto.MinCost != null)
{
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost).Select(p => p);
}
if (spto.MaxCost != null)
{
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost).Select(p => p);
}
if (spto.TypeId != null)
{
post = post.Where(p => p.PostTypeId == spto.TypeId).Select(p => p);
}
if (spto.CityId != null)
{
post = post.Where(p => p.PostCityId == spto.CityId).Select(p => p);
}
if (spto.IsImmidiate != null)
{
post = post.Where(p => p.PostIsImmediate == true).Select(p => p);
}
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).AsEnumerable().Take(15).Select(p => p).ToList();
if (posts.Count != 0)
return Ok(posts);
return NotFound();
In this case I have 6 Query that take time and the performance is low and code is too long . Is there any better way for writing better code ?
Short answer: if you don't do the ToList and AsEnumerable until the end, then you will only execute one query on your dbContext.
So keep everything IQueryable<...>, until you create List<...> posts:
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages
.Select(i => i.PostImagesImage.ImageAddress)
.FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName,
})
.Take(15)
.ToList();
IQueryable and IEnumerable
For the reason why skipping all the ToList / AsEnumerable would help to improve performance, you need to be aware about the difference between an IEnumerable<...> and an IQueryable<...>.
IEnumerable
A object of a class that implements IEnumerable<...> represents the potential to enumerate over a sequence that the object can produce.
The object holds everything to produce the sequence. Once you ask for the sequence, it is your local process that will execute the code to produce the sequence.
At low level, you produce the sequence by using GetEnumerator and repeatedly call MoveNext. As long as MoveNext returns true, there is a next element in the sequence. You can access this next element using property Current.
Enumerating the sequence is done like this:
IEnumerable<Customer> customers = ...
using (IEnumarator<Customer> customerEnumerator = customers.GetEnumerator())
{
while (customerEnumerator.MoveNext())
{
// there is still a Customer in the sequence, fetch it and process it
Customer customer = customerEnumerator.Current;
ProcessCustomer(customer);
}
}
Well this is a lot of code, so the creators of C# invented foreach, which will do most of the code:
foreach (Customer customer in customers)
ProcessCustomer(customer);
Now that you know what code is behind the foreach, you might understand what happens in the first line of the foreach.
It is important to remember, that an IEnumerable<...> is meant to be processed by your local process. The IEnumerable<...> can call every method that your local process can call.
IQueryable
An object of a class that implements IQueryable<...> seems very much like an IEnumerable<...>, it also represents the potential to produce an enumerable sequence of similar object. The difference however, is that another process is supposed to provide the data.
For this, the IQueryable<...> object holds an Expression and a Provider. The Expression represents a formula to what data must be fetched in some generic format; the Provider knows who must provide the data (usually a database management system), and what language is used to communicate with this DBMS (usually SQL).
As long as you concatenate LINQ methods, or your own methods that only return IQueryable<...>, only the Expression is changed. No query is executed, the database is not contacted. Concatenating such statements is a fast method.
Only when you start enumerating, either at its lowest level using GetEnumerator / MoveNext / Current, or higher level using foreach, the Expression is sent to the Provider, who will translate it to SQL and fetch the data from the database. The returned data is represented as an enumerable sequence to the caller.
Be aware, that there are LINQ methods, that don't return IQueryable<TResult>, but a List<TResult>, TResult, a bool, or int, etc: ToList / FirstOrDefault / Any / Count / etc. Those methods will deep inside call GetEnumerator / MoveNext / Current`; so those are the methods that will fetch data from the database.
Back to your question
Database management systems are extremely optimized for handling data: fetching, ordering, filtering, etc. One of the slower parts of a database query is the transfer of the fetched data to your local process.
Hence it is wise to let the DBMS do as much database handling as possible, and only transfer the data to your local process that you actually plan to use.
So, try to avoid ToList, if your local process doesn't use the fetched data. In your case: you transfer the followings to your local process, only to transfer it back to the database in the IQueryable.Contains method.
Furthermore, (it depends a bit on the framework you are using), the AsEnumerable transfers data to your local process, so your local process has to do the filtering with the Where and the Contains.
Alas you forgot to give us a description of your requirements ("From all Posts, give me only those Posts that ..."), and it is a bit too much for me to analyze all your queries, but you gain most efficiency if you try to keep everything IQueryable<...> as long as possible.
There might be some problems with the Int.TryParse(...). Your provider probably will not know how to translate this into SQL. There are several solutions possible:
Apparently PostCost represents a number. Consider to store it as a number. If it is an amount (price or something, something with a limited number of decimals), consider to store it as a decimal.
If you really can't convince your project leaders that numbers should be stored as decimals, either search for a job where they make proper databases, or consider to create a stored procedure that converts the string that is in PostCost to a decimal / int.
if you will only use fifteen elements, use the IQueryable.Take(15), not the IEnumerable.Take(15).
Further optimizations:
int userId =
var followerUserIds = _context.Followers
.Where(follower => follower.FollowersFollowerId == userId
&& follower.FollowersIsAccept)
.Select(follower => follower.FollowersUserId);
In words: make the following IQueryable, but don't execute it yet: "From all Followers, keep only those Followers that are Accepted and have a FollowersFollowerId equal to userId. From the remaining Followers, take the FollowersUserId".
It seems that you only plan to use it if page is zero. Why create this query also if page not zero?
By the way, never use statements like where a == true, or even worse: if (a == true) then b == true else b == false, this gives readers the impression that you have difficulty to grasp the idea of Booleans, just use: where a and b = a.
Next you decide to create a query that zero or more Posts, and thought it would be a good idea to give it a singular noun as identifier: post.
var post = _context.Posts
.Where(post => (followings.Contains(post.PostsUserId)
|| post.PostsUser.UserIsPublic
|| post.PostsUserId == userId)
&& post.PostIsAccept);
Contains will cause a Join with the Followers table. It will probably be more efficient if you only join Accepted posts with the followers table. So first check on PostIsAccept and the other predicates before you decide to join:
.Where(post => post.PostIsAccept
&& (post.PostsUser.UserIsPublic || post.PostsUserId == userId
|| followings.Contains(post.PostsUserId));
All non-accepted Posts won't have to be joined with the Followings; depending on whether your Provider is smart enough: it won't join all public users, or the one with userId, because it knows that it will already pass the filter.
Consider to use a Contains, instead of Any
It seems to me that you want the following:
I have a UserId; Give me all Accepted Posts, that are either from this user, or that are from a public user, or that have an accepted follower
var posts = dbContext.Posts
.Were(post => post.IsAccepted
&& (post.PostsUser.UserIsPublic || post.PostsUserId == userId
|| dbContext.Followers
.Where(followers => ... // filter the followers as above)
.Any());
Be aware: I still haven't executed the query, I only have changed the Expression!
AFter this first definition of posts, you filter the posts further, depending on various values of spto. You could consider to make this one big query, but I think that won't speed up the process. It will only make it more unreadable.
Finally: why use:
.Select(post => post)
This doesn't do anything to your sequence, it will only make it slower.
Some observations:
.AsEnumerable()
This is meant to hide where operators if you use a custom collection. It should not be needed in this case.
.Select(p => p)
I fail to see any purpose for this, remove it.
int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost
Parsing can be expensive, so you want to do as little as possible, and this does it twice, of four times if you have both min and max. replace with direct compares with value, i.e. int.TryParse(p.PostCost, out value) && value >= spo.MinCost. I would also suggest have an explicit case when there is both a min and max cost to avoid parsing twice.
followings.Contains(p.PostsUserId)
Followings is a list, so it will search thru all items. Use a HashSet to speed up performance. I.e. Replace .ToList() with ToHashSet() when creating the followings list. HashSet uses a hash table to make Contains() a constant time operation rather than a linear operation.
Query order
You would want to order the checks to eliminate as many items as early as possible, and make simple, fast, checks before slower checks.
Merge where operators
A single where operator is in general faster than multiple calls.
Use plain loop
If you really need as high performance as possible it might be better to use regular loops. Linq is great for writing compact code, but performance is usually better with plain loops.
Profile
Whenever you talk about performance it is important to point out the importance of profiling. The comments above are reasonable places to start, but there might be some completely different things that takes time. The only way to know is to profile. That should also give a good indication about the improvements.
I have solved my problem with ternary operator :
var post = _context.Posts.AsNoTracking().Where(p =>
(followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true
&& (spto.MinCost != null ? int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost : 1 == 1)
&& (spto.MaxCost != null ? int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost : 1 == 1)
&& (spto.TypeId != null ? p.PostTypeId == spto.TypeId : 1 == 1)
&& (spto.CityId != null ? p.PostCityId == spto.CityId : 1 == 1)
&& (spto.IsImmidiate != null && spto.IsImmidiate == true ? p.PostIsImmediate == true : 1 == 1)).Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page * 15).Take(15).ToList();
EDIT (Better Code) :
Thanx to #ZoharPeled , #HaraldCoppoolse , #JonasH I have Changed the Code like this :
int value;
var post = _context.Posts.AsNoTracking().Where(p =>
(followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true
&& (spto.MinCost == null || (int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost))
&& (spto.MaxCost == null || (int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost))
&& (spto.TypeId == null || p.PostTypeId == spto.TypeId)
&& (spto.CityId == null || p.PostCityId == spto.CityId)
&& (spto.IsImmidiate == null || p.PostIsImmediate == true)).Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page * 15).Take(15).ToList();
Edit (Best Code) :
int userId = Convert.ToInt32(HttpContext.User.Identity.GetUserId());
var followings = _context.Followers
.Where(follower => follower.FollowersFollowerId == userId
&& follower.FollowersIsAccept)
.Select(follower => follower.FollowersUserId);
int value;
var post = _context.Posts.AsNoTracking().Where(p => p.PostIsAccept
&& (p.PostsUser.UserIsPublic || p.PostsUserId == userId
|| _context.Followers.Where(f => f.FollowersFollowerId == userId
&& f.FollowersIsAccept).Select(f => f.FollowersUserId).Any()));
if (spto.MinCost != null)
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost);
if (spto.MaxCost != null)
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost);
if (spto.TypeId != null)
post = post.Where(p => p.PostTypeId == spto.TypeId);
if (spto.CityId != null)
post = post.Where(p => p.PostCityId == spto.CityId);
if (spto.IsImmidiate != null)
post = post.Where(p => p.PostIsImmediate == true);
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page).Take(15).ToList();
if (posts.Count != 0)
return Ok(posts);
I am using linq and I'm trying to compare two date and I'm getting support error I googled it and used examples but still gets the unsupported error - System.NotSupportedException, What am I doing wrong ?
//vars
int numDaysPassed =Int32.Parse(ConfigurationManager.AppSettings["DaysPassed"]);
DateTime todayPlusConfigDays = DateTime.Today.Date.AddDays(numDaysPassed);
//option 1
List<Customerfiles> Customerfiles = context.Customerfiles.Where(x => x.Status == currentStatus && (x.UpdateDate.Value.Date.CompareTo(todayPlusConfigDays)==0)).ToList<Customerfiles>();
//option 2
List<Customerfiles> Customerfiles = context.Customerfiles.Where(x => x.Status == currentStatus && (x.UpdateDate.Value.Date == todayPlusConfigDays)).ToList<Customerfiles>();
The error you are getting is because while the query is in the database layer, LINQ can't convert every function that you call to a similar call in SQL, because these functions simply don't exist in SQL.
Why don't you do a simple comparison?
List<Customerfiles> Customerfiles = context.Customerfiles.Where(x => x.Status == currentStatus && x.UpdateDate.Value.Date == todayPlusConfigDays).ToList();
If you want to get rid of the problem, without worrying about performance, try this:
List<Customerfiles> Customerfiles = context.Customerfiles.ToList().Where(x => x.Status == currentStatus && x.UpdateDate.Value.Date == todayPlusConfigDays).ToList();
I just added a "ToList()" in the beggining of the query. That will bring the ALL customerFiles to the memory, which usually is a very bad approach. This way, LINQ can do anything you want, because it doesn't need to convert to SQL functions.
Issue: while filtering records within a date range & matching a CityID using LINQ, the query succeeds when written in 2 steps; however, it fails when combined as one query!
How can the LINQ query be rewritten so that -- it can perform both filters (i.e. match the CityId & retrieve records in the date range in the same step to improve performance?
I got it to work in two steps fine,
i.e. do a
var Step1 = db.weekRecord.Where(x => x.CityId == CityRecord.Id).ToList();
and then
Step1.Where(x => x.date.Date >= fromDate.Date
&& x.date.Date <= toDate.Date)
.ToList();
it fails when I combine them!!
// works when done in 2 steps!!
var weeklyWeather = db.weekRecord
.Where(x => x.CityId == CityRecord.Id
&& (x.date >= weekStarting && x.date <= weekEnding))
// - when combined results are NULL!??
var weeklyWeather2 =
db.weekRecord(x => x.date.Date >= fromDate.Date && x.date.Date <= toDate.Date)
.ToList();
After looking up other SO answers, I tried this TruncateTime as well... could not get it to work..
// is this correct, from SO answers, DbFunctions.TruncateTime
var testQueryRecrods = db.weekRecord
.Where(x => x.CityId == CityRecord.Id)
.Where(x =>
DbFunctions.TruncateTime(x.date.Date) >= DbFunctions.TruncateTime(fromDate.Date)
&& DbFunctions.TruncateTime(x.date.Date) <= DbFunctions.TruncateTime(toDate.Date))
.ToList();
ERROR:
[NotSupportedException: The specified type member 'Date' is not
supported in LINQ to Entities. Only initializers, entity members, and
entity navigation properties are supported.]
System.Data.Entity.Core.Objects.ELinq.MemberAccessTranslator.TypedTranslate(ExpressionConverter
parent, MemberExpression linq) +452
System.Data.Entity.Core.Objects.ELinq.TypedTranslator`1.Translate(ExpressionConverter
parent, Expression linq) +49
The question is confused, but I would assume the problem is the .Date. Unlike linq2sql, entity framework can not translate .Date to sql. But you can rewrite it like
var fromDateDate = fromDate.Date;
var toDateDate = toDate.Date;
var testQueryRecrods = db.weekRecord
.Where(x => x.CityId == CityRecord.Id)
.Where(x => DbFunctions.TruncateTime(x.date) >= fromDateDate
&& DbFunctions.TruncateTime(x.date) <= toDateDate)
.ToList();
And it would work. To some point. What EF generates is actually totally stupid in this case. Unlike linq2sql, EF generates query, that is not sargable (in my case*). It can run thousands of times slower than necessary. I would recommend to avoid the conversion to date completely:
var fromDateDate = fromDate.Date;
var toDateDate1 = toDate.Date.AddDays(1);
var testQueryRecrods = db.weekRecord
.Where(x => x.CityId == CityRecord.Id)
.Where(x => x.date >= fromDateDate
&& x.date < toDateDate1)
.ToList();
As #juharr pointed out, when you split the query, you run first half against server and the second half as linq to objects. In that case the .Date works, but you download many more records in the first half than you need.
*the datetime type may be the problem, maybe it would work better with datetime2, I did not test this scenario
An offbest suggestion is to write your own LINQ extensions for this..
public static class ext
{
//This extension compares one date to another... if you can call from Linq
public static bool GreaterThan(this DateTime self, DateTime CompareDate
{
if (self.Year > CompareDate.Year) return true;
else if ((self.Year == CompareDate.Year) && (self.Month > CompareDate.Month)
return true;
else if ((self.Year == CompareDate.Year) && (self.Month == CompareDate.Month) && (self.Day > CompareDate.Day))
return true;
return false;
}
}
I think, there is no option except using DbFunctions.TruncateTime for Linq to Entities. Because, as SQL Server query Linq to Entities should perform convertion datetime to date and the best method which can be used is DbFunctions.TruncateTime. I just debugged the DbFunctions.TruncateTime convertion and the translated query seems like;
WHERE (convert (datetime2, convert(varchar(255), [Extent1].[CreationDate], 102) , 102)) > #p__linq__0
As you see, while performing the conversation, there is a redundant string conversation here. However, the EF would convert the datetime to date in SQL just like this 'cast(CreationDate as date)'. But it is not.
So, there are two options here.
1- If you have very huge table which the performance is affected by redundant string conversations, you should build your query manually in SQL as stored procedure or something like and execute it from context.
2- If you don't have performance considerations like that; just use DbFunctions.TruncateTime(x.date)
var testQueryRecrods = db.weekRecord
.Where(x => x.CityId == CityRecord.Id)
.Where(x => DbFunctions.TruncateTime(x.date) >= fromDate.Date && DbFunctions.TruncateTime(x.date) <= toDate.Date)
.ToList();
I have below LINQ Query:
returnVal = context.ReservationRequests
.Where(s => ((s.RequestStatusId.HasValue) &&
(s.RequestStatusId.Value == ResStatusId)) &&
((string.IsNullOrEmpty(loggedInUserRole)
|| s.SubmitterGroupName == loggedInUserRole)
||(s.CreatedBy == ApplicationSecurityDirector.CurrentUserGuid)))
.Skip(skip)
.Take(take)
.ToList();
What this LINQ query is supposed to do it look into the ReservationRequests table and look for records where RequestStatusId = (suppliedRequestStatusId) and SubmitterGroupName should be equal to the logged in user role (but there are cases where user is not assigned to any role) and it should also return any requests which are created by user.
So basically return all records if assigned to a particular group and also the requests created by logged in person if any.
The above query works fine incases where user is logged in to a group but it does not return correct results when user is not assigned to any groups. In case person is not assigned to any group it should return any records which were CreatedBy the user.
Below is a SQL query that I wrote which returns the correct amount of records for all my cases and I basically need my LINQ to be like this sql but I am not sure what I am missing here.
SELECT *
FROM [MyDB].[dbo].[ReservationRequests]
where
(RequestStatusId = 2)
and
(SubmitterGroupName != null or SubmitterGroupName = null
or createdby = 'C5188D45-TEST-45BE-8C04-123455733A31')
Can someone please look into my LINQ query and see why is it returning incorrect records than my SQL ? Thanks I been looking at this for a while now!
After all the suggestion here is my updated LINQ:
returnVal = context.ReservationRequests
.Where(s => ((s.RequestStatusId.HasValue) &&
(s.RequestStatusId.Value == ResStatusId)) &&
(s.SubmitterGroupName == loggedInUserRole ||
s.CreatedBy == ApplicationSecurityDirector.CurrentUserGuid))
.Skip(skip)
.Take(take)
.ToList();
So question: Will this query work in cases loggedInUserRole is null? In case loggedInUserRole is null then I just want to return records which are CreatedBy the logged in user.
Another Update: (9/22/2017) 9.54 AM
So I ran that statement. It works fine in cases when user is assigned to a group but when user is not assigned to a group instead of showing just the requests opened by the logged in user it returns a lot more records.
incorrect records is relative... the records it's returning are correct I guess. We only can analyse what you're doing here and where the differences are:
In the linq query you search for
s.SubmitterGroupName == loggedInUserRole
in the SQL statement you search for
SubmitterGroupName != null or SubmitterGroupName = null
SubmitterGroupName cant be null and not null at the same time.So I guess, loggedInUserRole is null. This is with or, so this is always true. You only search then for createdby = 'C5188D45-TEST-45BE-8C04-123455733A31'
plus, in the linq-query you
.Skip(skip).Take(take)
this is missing in your sql-statement
so I think what you want is:
returnVal = context.ReservationRequests
.Where(s => ((s.RequestStatusId.HasValue) &&
(s.RequestStatusId.Value == ResStatusId)) &&
(s.SubmitterGroupName == loggedInUserRole || (s.SubmitterGroupName == null && s.CreatedBy == ApplicationSecurityDirector.CurrentUserGuid)))
.Skip(skip)
.Take(take)
.ToList();
I am not entirely sure why you are adding all sorts of checks in there, but this should work:
returnVal = context.ReservationRequests
.Where(s => RequestStatusId == ResStatusId &&
(s.SubmitterGroupName == loggedInUserRole || s.CreatedBy == ApplicationSecurityDirector.CurrentUserGuid)
.Skip(skip)
.Take(take)
.ToList();
So I'm having issues with this LINQ query I had been using for some time, and now it seems to not be working as expected.
messagesWithoutConditional =
await
MobileServiceInstance.GetSyncTable<Messages>()
.OrderByDescending(key => key.SentDate)
.Take(50)
.Where(
p =>
(p.Uuid == myUuid && p.RecipientUuid == otherUuid) ||
(p.Uuid == otherUuid && p.RecipientUuid == myUuid))
.ToListAsync();
So lets say I have this query that simply returns the last 50 messages sent between 2 parties. Now if I want to add an additional condition that the 50 messages should also be before a certain date I would expect to do something like this
messagesWithConditional =
await
MobileServiceInstance.GetSyncTable<Messages>()
.OrderByDescending(key => key.SentDate)
.Take(50)
.Where(
p =>
((p.Uuid == myUuid && p.otherUuid == recipientUuid) ||
(p.Uuid == otherUuid && p.RecipientUuid == myUuid))
&& p.SentDate < 'some date')
.ToListAsync();
Lets suppose I should expect this to return 40 messages, but it returns 0. However, if I alternate the query into this
messagesWithConditional = messagesWithoutConditional.Where(p => p.SentDate < 'some date').ToList();
then I will receive the 40 expected messages by querying the result of my original expression.
How is the second approach any different from the first? Ideally I would like to use the && operator to add a new conditional expression rather than break off into a second where clause
Edit
I should also note that the times are in UTC, and the SentDate attribute is of type DateTimeOffset
As of right now it appears to be a bug, unless some other insight is made. Please refer to the bug report for more information
.Where should be put before the .OrderByDescending and .Take(50)