In search page I have some options based on them search query must be different . I have wrote this :
int userId = Convert.ToInt32(HttpContext.User.Identity.GetUserId());
var followings = (from f in _context.Followers
where f.FollowersFollowerId == userId && f.FollowersIsAccept == true
select f.FollowersUserId).ToList();
int value;
if (spto.Page == 0)
{
var post = _context.Posts.AsNoTracking().Where(p => (followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true).Select(p => p).AsEnumerable();
if(spto.MinCost != null)
{
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost).Select(p => p);
}
if (spto.MaxCost != null)
{
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost).Select(p => p);
}
if (spto.TypeId != null)
{
post = post.Where(p => p.PostTypeId == spto.TypeId).Select(p => p);
}
if (spto.CityId != null)
{
post = post.Where(p => p.PostCityId == spto.CityId).Select(p => p);
}
if (spto.IsImmidiate != null)
{
post = post.Where(p => p.PostIsImmediate == true).Select(p => p);
}
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).AsEnumerable().Take(15).Select(p => p).ToList();
if (posts.Count != 0)
return Ok(posts);
return NotFound();
In this case I have 6 Query that take time and the performance is low and code is too long . Is there any better way for writing better code ?
Short answer: if you don't do the ToList and AsEnumerable until the end, then you will only execute one query on your dbContext.
So keep everything IQueryable<...>, until you create List<...> posts:
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages
.Select(i => i.PostImagesImage.ImageAddress)
.FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName,
})
.Take(15)
.ToList();
IQueryable and IEnumerable
For the reason why skipping all the ToList / AsEnumerable would help to improve performance, you need to be aware about the difference between an IEnumerable<...> and an IQueryable<...>.
IEnumerable
A object of a class that implements IEnumerable<...> represents the potential to enumerate over a sequence that the object can produce.
The object holds everything to produce the sequence. Once you ask for the sequence, it is your local process that will execute the code to produce the sequence.
At low level, you produce the sequence by using GetEnumerator and repeatedly call MoveNext. As long as MoveNext returns true, there is a next element in the sequence. You can access this next element using property Current.
Enumerating the sequence is done like this:
IEnumerable<Customer> customers = ...
using (IEnumarator<Customer> customerEnumerator = customers.GetEnumerator())
{
while (customerEnumerator.MoveNext())
{
// there is still a Customer in the sequence, fetch it and process it
Customer customer = customerEnumerator.Current;
ProcessCustomer(customer);
}
}
Well this is a lot of code, so the creators of C# invented foreach, which will do most of the code:
foreach (Customer customer in customers)
ProcessCustomer(customer);
Now that you know what code is behind the foreach, you might understand what happens in the first line of the foreach.
It is important to remember, that an IEnumerable<...> is meant to be processed by your local process. The IEnumerable<...> can call every method that your local process can call.
IQueryable
An object of a class that implements IQueryable<...> seems very much like an IEnumerable<...>, it also represents the potential to produce an enumerable sequence of similar object. The difference however, is that another process is supposed to provide the data.
For this, the IQueryable<...> object holds an Expression and a Provider. The Expression represents a formula to what data must be fetched in some generic format; the Provider knows who must provide the data (usually a database management system), and what language is used to communicate with this DBMS (usually SQL).
As long as you concatenate LINQ methods, or your own methods that only return IQueryable<...>, only the Expression is changed. No query is executed, the database is not contacted. Concatenating such statements is a fast method.
Only when you start enumerating, either at its lowest level using GetEnumerator / MoveNext / Current, or higher level using foreach, the Expression is sent to the Provider, who will translate it to SQL and fetch the data from the database. The returned data is represented as an enumerable sequence to the caller.
Be aware, that there are LINQ methods, that don't return IQueryable<TResult>, but a List<TResult>, TResult, a bool, or int, etc: ToList / FirstOrDefault / Any / Count / etc. Those methods will deep inside call GetEnumerator / MoveNext / Current`; so those are the methods that will fetch data from the database.
Back to your question
Database management systems are extremely optimized for handling data: fetching, ordering, filtering, etc. One of the slower parts of a database query is the transfer of the fetched data to your local process.
Hence it is wise to let the DBMS do as much database handling as possible, and only transfer the data to your local process that you actually plan to use.
So, try to avoid ToList, if your local process doesn't use the fetched data. In your case: you transfer the followings to your local process, only to transfer it back to the database in the IQueryable.Contains method.
Furthermore, (it depends a bit on the framework you are using), the AsEnumerable transfers data to your local process, so your local process has to do the filtering with the Where and the Contains.
Alas you forgot to give us a description of your requirements ("From all Posts, give me only those Posts that ..."), and it is a bit too much for me to analyze all your queries, but you gain most efficiency if you try to keep everything IQueryable<...> as long as possible.
There might be some problems with the Int.TryParse(...). Your provider probably will not know how to translate this into SQL. There are several solutions possible:
Apparently PostCost represents a number. Consider to store it as a number. If it is an amount (price or something, something with a limited number of decimals), consider to store it as a decimal.
If you really can't convince your project leaders that numbers should be stored as decimals, either search for a job where they make proper databases, or consider to create a stored procedure that converts the string that is in PostCost to a decimal / int.
if you will only use fifteen elements, use the IQueryable.Take(15), not the IEnumerable.Take(15).
Further optimizations:
int userId =
var followerUserIds = _context.Followers
.Where(follower => follower.FollowersFollowerId == userId
&& follower.FollowersIsAccept)
.Select(follower => follower.FollowersUserId);
In words: make the following IQueryable, but don't execute it yet: "From all Followers, keep only those Followers that are Accepted and have a FollowersFollowerId equal to userId. From the remaining Followers, take the FollowersUserId".
It seems that you only plan to use it if page is zero. Why create this query also if page not zero?
By the way, never use statements like where a == true, or even worse: if (a == true) then b == true else b == false, this gives readers the impression that you have difficulty to grasp the idea of Booleans, just use: where a and b = a.
Next you decide to create a query that zero or more Posts, and thought it would be a good idea to give it a singular noun as identifier: post.
var post = _context.Posts
.Where(post => (followings.Contains(post.PostsUserId)
|| post.PostsUser.UserIsPublic
|| post.PostsUserId == userId)
&& post.PostIsAccept);
Contains will cause a Join with the Followers table. It will probably be more efficient if you only join Accepted posts with the followers table. So first check on PostIsAccept and the other predicates before you decide to join:
.Where(post => post.PostIsAccept
&& (post.PostsUser.UserIsPublic || post.PostsUserId == userId
|| followings.Contains(post.PostsUserId));
All non-accepted Posts won't have to be joined with the Followings; depending on whether your Provider is smart enough: it won't join all public users, or the one with userId, because it knows that it will already pass the filter.
Consider to use a Contains, instead of Any
It seems to me that you want the following:
I have a UserId; Give me all Accepted Posts, that are either from this user, or that are from a public user, or that have an accepted follower
var posts = dbContext.Posts
.Were(post => post.IsAccepted
&& (post.PostsUser.UserIsPublic || post.PostsUserId == userId
|| dbContext.Followers
.Where(followers => ... // filter the followers as above)
.Any());
Be aware: I still haven't executed the query, I only have changed the Expression!
AFter this first definition of posts, you filter the posts further, depending on various values of spto. You could consider to make this one big query, but I think that won't speed up the process. It will only make it more unreadable.
Finally: why use:
.Select(post => post)
This doesn't do anything to your sequence, it will only make it slower.
Some observations:
.AsEnumerable()
This is meant to hide where operators if you use a custom collection. It should not be needed in this case.
.Select(p => p)
I fail to see any purpose for this, remove it.
int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost
Parsing can be expensive, so you want to do as little as possible, and this does it twice, of four times if you have both min and max. replace with direct compares with value, i.e. int.TryParse(p.PostCost, out value) && value >= spo.MinCost. I would also suggest have an explicit case when there is both a min and max cost to avoid parsing twice.
followings.Contains(p.PostsUserId)
Followings is a list, so it will search thru all items. Use a HashSet to speed up performance. I.e. Replace .ToList() with ToHashSet() when creating the followings list. HashSet uses a hash table to make Contains() a constant time operation rather than a linear operation.
Query order
You would want to order the checks to eliminate as many items as early as possible, and make simple, fast, checks before slower checks.
Merge where operators
A single where operator is in general faster than multiple calls.
Use plain loop
If you really need as high performance as possible it might be better to use regular loops. Linq is great for writing compact code, but performance is usually better with plain loops.
Profile
Whenever you talk about performance it is important to point out the importance of profiling. The comments above are reasonable places to start, but there might be some completely different things that takes time. The only way to know is to profile. That should also give a good indication about the improvements.
I have solved my problem with ternary operator :
var post = _context.Posts.AsNoTracking().Where(p =>
(followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true
&& (spto.MinCost != null ? int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost : 1 == 1)
&& (spto.MaxCost != null ? int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost : 1 == 1)
&& (spto.TypeId != null ? p.PostTypeId == spto.TypeId : 1 == 1)
&& (spto.CityId != null ? p.PostCityId == spto.CityId : 1 == 1)
&& (spto.IsImmidiate != null && spto.IsImmidiate == true ? p.PostIsImmediate == true : 1 == 1)).Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page * 15).Take(15).ToList();
EDIT (Better Code) :
Thanx to #ZoharPeled , #HaraldCoppoolse , #JonasH I have Changed the Code like this :
int value;
var post = _context.Posts.AsNoTracking().Where(p =>
(followings.Contains(p.PostsUserId) || p.PostsUser.UserIsPublic == true || p.PostsUserId == userId) && p.PostIsAccept == true
&& (spto.MinCost == null || (int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost))
&& (spto.MaxCost == null || (int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost))
&& (spto.TypeId == null || p.PostTypeId == spto.TypeId)
&& (spto.CityId == null || p.PostCityId == spto.CityId)
&& (spto.IsImmidiate == null || p.PostIsImmediate == true)).Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page * 15).Take(15).ToList();
Edit (Best Code) :
int userId = Convert.ToInt32(HttpContext.User.Identity.GetUserId());
var followings = _context.Followers
.Where(follower => follower.FollowersFollowerId == userId
&& follower.FollowersIsAccept)
.Select(follower => follower.FollowersUserId);
int value;
var post = _context.Posts.AsNoTracking().Where(p => p.PostIsAccept
&& (p.PostsUser.UserIsPublic || p.PostsUserId == userId
|| _context.Followers.Where(f => f.FollowersFollowerId == userId
&& f.FollowersIsAccept).Select(f => f.FollowersUserId).Any()));
if (spto.MinCost != null)
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) >= spto.MinCost);
if (spto.MaxCost != null)
post = post.Where(p => int.TryParse(p.PostCost, out value) && Convert.ToInt32(p.PostCost) <= spto.MaxCost);
if (spto.TypeId != null)
post = post.Where(p => p.PostTypeId == spto.TypeId);
if (spto.CityId != null)
post = post.Where(p => p.PostCityId == spto.CityId);
if (spto.IsImmidiate != null)
post = post.Where(p => p.PostIsImmediate == true);
var posts = post.Select(p => new
{
p.Id,
Image = p.PostsImages.Select(i => i.PostImagesImage.ImageAddress).FirstOrDefault(),
p.PostCity.CityName,
p.PostType.TypeName
}).Skip(spto.Page).Take(15).ToList();
if (posts.Count != 0)
return Ok(posts);
In my program I have a search method.
I have two versions of it:
1st version where I search for drawings with a drawing number in my database like normal.
2nd version is the normal search plus the function to find only the newest objects with this drawingnumber in the database.
For both options I´m using the CONTAINS method.
For the normal search it works well but on the newest search, where it is combined with GROUP BY and ORDER BY I feel like there are many objects missing.
I checked when I use the whole drawingnumber, so that the CONTAINS doesn´t need to work.
So for example when I just write 100 for the drawingnumber and using the normal search it finds all objects with the 100 - Thats fine.
However, when I search for the newest objects with only the 100 it finds only a few.
When I use the whole number for the search it finds the newest objects for the right number. So is it bad anyways to use CONTAINS with all the sort- and order stuff or I´m just missing something there?
For better understanding, the drawingnumber there can be different extensions and doktypes. The Dok_Count is the count for the newest document where I´m looking for.
var query =
from z in context.zeichnungs
where (zeichnungsnummer == "" || z.Zeichnungsnummer.Contains(zeichnungsnummer)) &&
(index == "" || index == z.Index) &&
(artikelbezeichnung == "" || artikelbezeichnung == z.Artikelbezeichnung) &&
(status == "" || status == z.Status) && (mmsSachmerkmal == "" || mmsSachmerkmal == z.MMS_Sachmerkmal) &&
(doktyp == "" || doktyp == z.Dokumententyp) && (dateiendung == "" || dateiendung == z.Dateiendung) &&
(z.Datum >= startDate.Date && z.Datum <= endDate.Date) && (status == "" || status == z.Status)
select z;
var sortQuery = query.GroupBy(x => new { x.Dokumententyp, x.Dateiendung }).Select(g => g.OrderByDescending(record => record.Dok_Count).FirstOrDefault());
you are trying to do database normal work with linq this is not it pourpose, use views, thanks for the vote.
I think the data server do this with better performance than in .net classes. Now, I think, you can do a sql profile (or trace) to catch what is the command that the process ask to the database, you will see that the contains its a where in method, in my opinion if you have a alternative way to do this, like join in database, would be better. In other hand, if the number of the records that return this query never will be under 1000, it's no matter, this is a good way.
Sorry for my English :D
I'm trying to work out the most performant way to query a list. I know that there are a ton of examples out there and this has come up loads before, but I'm really new to this and I'm struggling with how to apply some of the concepts to my situation.
private static void KeepMatchesBasedOnRestrictions(ref List<Entity> matches,
List<Entity> preFilteredShifts, List<Entity> locationalInformations)
{
if (matches.Count == 0) return;
matches.RemoveAll(
(match) => ( GeographyHasRestriction(match, preFilteredShifts, locationalInformations) )
);
}
private static bool GeographyHasRestriction(Entity match, List<Entity> preFilteredShifts, List<Entity> locationalInformations)
{
EntityReference fw = match.GetAttributeValue<EntityReference>("crm_fw");
Entity shift = preFilteredShifts.Single<Entity>(
a => match.GetAttributeValue<EntityReference>("crm_shift").Id == a.Id
);
EntityReference trust = shift.GetAttributeValue<EntityReference>("crm_trust");
EntityReference location = shift.GetAttributeValue<EntityReference>("crm_location");
EntityReference ward = shift.GetAttributeValue<EntityReference>("crm_ward");
Dictionary<Guid, Entity> locInfoRecs = locationalInformations.ToDictionary(p => p.Id);
var locationalInformationQuery = from loc in locationalInformations
where (
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& !loc.Contains("crm_trust")
&& !loc.Contains("crm_location")
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& !loc.Contains("crm_location")
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& loc.GetAttributeValue<EntityReference>("crm_location").Id == location.Id
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& loc.GetAttributeValue<EntityReference>("crm_location").Id == location.Id
&& loc.GetAttributeValue<EntityReference>("crm_ward").Id == ward.Id
)
)
select loc;
foreach (Entity loc in locationalInformationQuery)
{
if (loc.GetAttributeValue<bool>("crm_hasrestriction"))
{
return true;
}
}
//return false;
}
So I think my problem is 2-fold;
The locationalInformationQuery query seems to run very slowly... I'm talking something in the region of up to 2 seconds per iteration which is horrible.
I also suspect that the approach of calling matches.RemoveAll() is also somewhat flawed due to the performance issues regarding lists.
So in terms of addressing this, I think that I may be able to get better performance by converting my locationalInformations list to some other type of container such as a Dictionary, HashSet or SortedList. My problem then is that I have no idea how to go about adjusting my query to to take advantage of those more efficient containers.
As far as the second point goes, I'd also be curious to hear about alternatives to using List.RemoveAll(). I have the flexibility to modify my incoming container types within reason to this may be viable.
With regards the list sizes in case its of any use, match contains a few thousand items and preFilteredShifts and locationalInformations each contain > 100,000 items.
As an aside I've tried using Parallel.ForEach instead of foreach, but it made virtually no difference whatsoever.
Edit: Just to clarify some questions, I'm doing all this in memory. I've already completely populated all of my lists so there shouldn't be any additional round trips to the DB. I'm reasonably certain that GetAttributeValue<EntityReference> doesn't initial further DB overhead.
Also, yes this is a local application calling Dynamics CRM Online.
The code -
foreach (Entity loc in locationalInformationQuery)
{
if (loc.GetAttributeValue<bool>("crm_hasrestriction"))
{
return true;
}
}
Can be one reason for slowness. You are fetching more data and then enumerating them in memory. You can perform the check directly before fetch, so you will fetch lesser data and could be faster. Something like this -
return (from loc in locationalInformations
where ((
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& !loc.Contains("crm_trust")
&& !loc.Contains("crm_location")
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& !loc.Contains("crm_location")
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& loc.GetAttributeValue<EntityReference>("crm_location").Id == location.Id
&& !loc.Contains("crm_ward")
)
||
(
loc.GetAttributeValue<EntityReference>("crm_fw").Id == fw.Id
&& loc.GetAttributeValue<EntityReference>("crm_trust").Id == trust.Id
&& loc.GetAttributeValue<EntityReference>("crm_location").Id == location.Id
&& loc.GetAttributeValue<EntityReference>("crm_ward").Id == ward.Id
)
) && loc.GetAttributeValue<bool>("crm_hasrestriction")) // do the check before fetch in here
select loc).Any();
I have on occasion found that querying CRM databases can result in inefficient queries when what the query you are working with is sufficiently complex.
Sometimes this can be due to how the query is generated depending on your method of querying the database, or sometimes it can be that iterating some IEnumerable type collections and checking conditions can result in many SQL queries to the database per iteration. Maybe check what is happening under the covers against your database by using SQL Profiler. It may turn out to be insightful.
One option I've reverted to on occasion where I feel the CRM query limitations simply hamper performance too much is to fall back to straight ADO.NET and SQL against the filtered views where I have access to query plans and have a much better idea and understanding of what is happening. I'm sure many CRM purists are frowning at me right now, but I think it is a fair call in terms of the end users experience and also making your code relatively understandable too. Complex queries can be quite unwieldy in code, and having a SQL query you can refer to, can help immensely in comprehension of your solution. You can also benefit from set based operations and a less "chatty" interface in terms of the number of resultant database calls.
In your question above, if you feel this may be a good option, I'd look at prototyping such a solution by providing a method like ;
private static bool GeographyHasRestrictionBySql(Entity match, List<Entity> preFilteredShifts, List<Entity> locationalInformations)
{
// Query here, and determine your boolean result to return
}
That way you can simply test this quickly and easily by a change in the calling function.
I'm working on EF CTP5 Code First development with an existing database. I need to get the data from two tables by comparing columns of different types.
For example - Here p.ColumnA is varchar as q.ColumnA is int but the values might be the same for few records. So, I'm trying to do Convert.ToInt32 which does not work. I do not have complete control over the database to modify the table.
from p in context.TableA
from q in context.TableB
where p.ColumnZ == "ABC" &&
(p.ColumnA == null || Convert.ToInt32(p.ColumnA) == q.ColumnA) &&
(p.ColumnB == null || p.ColumnB == q.ColumnB)
select p.ColumnC
Can someone suggest a solution? Thanks.
When you write a linq statement that interacts with the entityframework it trys to convert all the commands to SQL. Because there is no sql command for Convert.ToInt32 it is throwing an error.
This post describes a way to cal the sql functions for converting types. It should help you.
As the other posters have explained, LINQ to SQL Entities doesn't know how to translate Convert.ToInt32 into a SQL expression (LINQ to SQL can apparently handle this). From what I can tell, it doesn't support int.Parse, either. But since you're just doing an equality comparison (rather than greater/less than), you should be able to achieve the same result by converting the int to a string, rather than converting the string to an int.
from p in context.TableA
from q in context.TableB
where p.ColumnZ == "ABC" &&
(p.ColumnA == null || p.ColumnA == q.ColumnA.ToString()) &&
(p.ColumnB == null || p.ColumnB == q.ColumnB)
select p.ColumnC
Hey, I am pretty certain I am doing something incorrectly with my lambda expression and that is making it so that I only return a single result. If there is nothing wrong with my query (it has the ability to return multiple results), then I must have an error elsewhere and I should be able to find it. If my results should return more than one result for the third part of the or, only one is being returned. Am I doing something wrong here?
var proQuery = from a in solutionContext.Products
where a.ID == solutionID ||
(a.ParentID == solutionID && a.Versions.All(c => c.VersionNumber == activeNumber)) ||
(a.Product2.ParentID == solutionID &&
a.Versions.All(c => c.VersionNumber == activeNumber))
select a;
If there is anymore information you need, I would be happy to help.
There isn't really enough here for us to go on, but are you sure that the Alls are correct and shouldn't be Anys? Absent additional information, that's one thing to look at.