I had some serious speed issues with the LINQ in this code (variable names have been changed)
var A = _service.GetA(param1, param2); // Returns Enumerable results
var results = (from b in _B.All() // _B.All() returns IQueryable
join c in _C.All() on b.Id equals c.Id // _C.All() returns IQueryable
join a in A on a.Id equals c.Id
where b.someId == id && a.boolVariable // A bool value
select new
{
...
}).ToList();
This LINQ took over 10 seconds to execute even though the number of rows in B and C tables were less than 100k.
I looked into this and by trial and error I managed to get the LINQ execution time to 200ms by changing the code to this:
var A = _service.GetA(param1, param2).Where(a => a.boolVariable); // Returns Enumerable results
var results = (from b in _B.All() // _B.All() returns IQueryable
join c in _C.All() on b.Id equals c.Id // _C.All() returns IQueryable
join a in A on a.Id equals c.Id
where b.someId == id
select new
{
...
}).ToList();
So my question is, why does this simple change have such drastic effects on the LINQ performance? The only change is that I filter the Enumerable list beforehand, the A enumerable has about 30 items before filtering and 15 after filtering.
In your first scenario: first it joins all the records in A which would take long time to join, then filters out for a.boolVariable.
In your second scenario you have a smaller subset of records for A prior to joining - of course this would take less time to join.
Related
I thought that LINQ to objects is great way for differed execution of joined data. In reality, it comes up as bad way to do things...
Here is what we had, having few, few hundred, 3K and 3.5K records in a,b,c,d correspondingly
IEnumerable<MyModel> data =
(from a in AList
from b in BList.Where(r => r.AId == a.Id)
from c in CList.Where(r => r.BId == b.Id)
from d in DList.Where(r => r.SomeId == myId && r.Some2Id == c.Some2Id)
// . . . . . .
Wasn't LINQ supposed to be great about doing it?
In reality, following works much faster, 60 times faster in fact
var dTemp = DList.Where(r => r.SomeId == myId).ToList();
var cTemp = CList.Where(c => dTemp.Any(d => d.Some2Id == c.Some2Id)).ToList();
IEnumerable<MyModel> data =
(from a in AList
from b in BList.Where(r => r.AId == a.Id)
from c in cTemp.Where(r => r.BId == b.Id)
// . . . . . .
And then I came across this article
Q: Is there a way to improve this query without abandoning single LINQ?
Or does this mean that LINQ to objects in form of joins need to be avoided and replaced by some sequential calls if performance is at stake?
Let's analyze the differences.
First query: you are performing a filter on BList, a filter on CListand two filters on DList, all in a deferred-execution manner. You then use a kind of a join.
Second query: you perform a static filter on DList and evaluate it, another static filter on CList based on DList and evaluate it and then a deferred-executed filter on both AList and BList.
The second query is faster because:
DList is not being looked at for useless values (due to previous filters)
CList only contains useful values due to previous filters
Anyway, both queries are wrong. Multiple from is basically a cross-join, as explained here. As #Reddog commented, the best way is to actually use Join:
var data = from a in AList
join b in BList on a.Id equals b.AId
join c in CList on b.Id equals c.BId
join d in DList on c.Some2Id equals d.Some2Id
where d.SomeId == someId;
I'm looking to filter the data in a dataset much like you'd do a where <value> in (select <value> from other_table where year=2016)
So I have a list of the "values":
var BUs = (from b in dc.BusinessUnits
where b.Year == int.Parse(ddlYears.SelectedValue)
orderby b.BuName
select new { b.BUID }).ToList();
So what I need to do is filter this dataset based on the BUID list returned in the BUs var.
IQueryable<Market> markets = (from p in dc.Markets
orderby p.MarketName
select p);
Help? I'm 100% new to linq so I need a concise solution.
Well if you Market entity has a BUID property and this property it's a primitive type (int, string..) or an enum, you can use Contains method:
var BUs = (from b in dc.BusinessUnits
where b.Year == int.Parse(ddlYears.SelectedValue)
orderby b.BuName
select b.BUID );
IQueryable<Market> markets = (from p in dc.Markets
where BUs.Contains(p.BUID)
orderby p.MarketName
select p);
The standard way of filtering by in memory id list is to use Enumerable.Contains method. But you need first to make sure your list contains ids - they way you wrote it it will contain anonymous type with a property called BUID, by changing the first query like this
int year = int.Parse(ddlYears.SelectedValue);
var BUIDs = (from b in dc.BusinessUnits
where b.Year == year
orderby b.BuName
select b.BUID).ToList();
and then use
var markets = (from p in dc.Markets
where BUIDs.Contains(p.BUID)
orderby p.MarketName
select p);
But note that this will be inefficient. Much better option would be to not use the list of BUIDs for filtering, but combining the 2 queries so the whole thing becomes a single query executed in the database, like this
var markets = (from p in dc.Markets
where dc.BusinessUnits.Any(bu => b.Year == year && b.BUID == p.BUID)
orderby p.MarketName
select p);
This is the exact equivalent of, if using your words, much like you'd do a "where in (select from other_table where year=2016)".
I have the following query that gets a list of schools based on the criteria provided. Note: This database is very, very large with 10,000+ records. The end result is a list of 188 schools, which is exactly as we need.
return (from s in Context.Schools
join d in Context.Districts on s.DistrictID equals d.DistrictID
join r in Context.Rosters on s.SchoolID equals r.SchoolID
join te in Context.TestEvents on r.TestEventID equals te.TestEventID
join ta in Context.TestAdministrations on te.TestAdministrationID equals ta.TestAdministrationID
join sr in Context.ScoreResults on r.RosterID equals sr.RosterID into exists
from any in exists.DefaultIfEmpty()
where d.DistrictID == DistrictID
&& ta.SchoolYearID == SchoolYearID.Value
select s)
.Distinct()
.OrderBy(x => x.Name)
.ToList();
The problem is when we call return Json(Schools, JsonRequestBehavior.AllowGet); to send our schools back to the client the operation times out. It appears when stepping thorough the code that for some reason the DbContext is trying to pull in ALL of the properties for this result set, including the ones we don't need. I already have everything I need from the database in this Schools object. Why does it go back and start creating all the associated objects. Is there a way to stop this?.
This is an MVC application using EF 5 Code First.
Instead of selecting the whole entity, select a projection of only what you need:
var results = from s in Context.Schools
...
select new MyClassContainingOnlyAFewProperties {
Prop1 = s.Prop1,
Prop2 = s.Prop2,
//etc.
}
return results;
See also: What does Query Projection mean in Entity Framework?
I have a few Linq queries. Semantically, they are
a join b join c join d where filter1(a) && filter2(c) && filter3(d)
a join b join c where filter1(a) && filter2(c)
a join b join c join e where filter1(a) && filter2(c) && filter4(e)
...
I want to be able to factor out the shared part:
a join b join c where filter1(a) && filter2(c)
and dynamically append join d and filter3(d)
Is there a way to do this? I am already using the Predicate Builder to dynamically build conditionals (filters).
EDIT: I am using Linq-to-SQL.
EDIT: The base query looks like:
from a in As.AsExpandable()
join b in Bs on a.Id equals b.PId
join c in Cs on b.Id equals c.PId
where filter1(a) && filter2(b) && filter3(c)
select new A { ... }
filters are predicates in Predicate Builder. The type of the query is IQueryable<A>.
Next, I'd like to join this with d
from a in BaseQuery()
join d in D on a.Id equals d.PId
Currently join d .. causes a compilation error:
The type of one of the expressions in the join clause is incorrect. Type inference failed in the call to Join
Your example is a bit vague, but it is easy to create a method that returns an IQueryable<T> and reuse that method, if that’s what you mean. Here is an example:
// Reusable method
public IQueryable<SomeObject> GetSomeObjectsByFilter(Context c)
{
return
from someObject in context.SomeObjects
where c.B.A.Amount < 1000
where c.Roles.Contains(r => r.Name == "Admin")
select someObject;
}
You can reuse this method in other places like this:
var q =
from c in GetSomeObjectsByFilter(context)
where !c.D.Contains(d => d.Items.Any(i => i.Value > 100))
select c;
Because the way IQueryable works, only the final query (the collection that you start iterating) will trigger a call to the database, which allows you to build a highly maintainable system by reusing business logic that gets effectively executed inside the database, whiteout the loss of any performance.
I do this all the time and it improves the maintainability of my code big time. It works no matter which O/RM tool you run, because there is no difference in Queryable<T> composition, between writing the query in one peace, or splitting it out to different methods.
Note that you do sometimes need some smart transformations to get the duplicate parts in a single method. Things that might help are returning grouped sets, and returning a set of a different type, than what you think you need. This sounds a bit vaque, but just post a question here at SO when you have problems splitting up a method. There are enough people here that can help you with that.
I can answer half your question easily. Linq makes it simple to append .where clauses to an existing query.
Example:
var x = db.Table1.where(w => w.field1 == nTestValue);
x = x.where(w => w.field2 == nTestValue2);
I believe you can do the joins as well but I have to go find an example in some old code. I'll look if nobody else jumps in with it soon.
I'm wondering if its possible to join together IEnumerable's.
Basically I have a bunch of users and need to get their content from the database so I can search and page through it.
I'm using LINQ to SQL, my code at the moment it:
public IEnumerable<content> allcontent;
//Get users friends
IEnumerable<relationship> friends = from f in db.relationships
where f.userId == int.Parse(userId)
select f;
IEnumerable<relationship> freindData = friends.ToList();
foreach (relationship r in freindData)
{
IEnumerable<content> content = from c in db.contents
where c.userId == r.userId
orderby c.contentDate descending
select c;
// This is where I need to merge everything together
}
I hope that make some sense!
Matt
If I understand correctly what you are trying to do, why don't you try doing:
var result = from r in db.relationships
from c in db.contents
where r.userId == int.Parse(userId)
where c.userId == r.UserId
orderby c.contentDate descending
select new {
Relationship = r,
Content = c
}
This will give you an IEnumerable<T> where T is an anonymous type that has fields Relationship and Content.
If you know your users will have less than 2100 friends, you could send the keys from the data you already loaded back into the database easily:
List<int> friendIds = friendData
.Select(r => r.UserId)
.Distinct()
.ToList();
List<content> result = db.contents
.Where(c => friendIds.Contains(c.userId))
.ToList();
What happens here is that Linq translates each Id into a parameter and then builds an IN clause to do the filtering. 2100 is the maximum number of parameters that SQL server will accept... if you have more than 2100 friends, you'll have to break the ID list up and combine (Concat) the result lists.
Or, if you want a more literal answer to your question - Concat is a method that combines 2 IEnumerables together by creating a new IEnumerable which returns the items from the first and then the items from the second.
IEnumerable<content> results = Enumerable.Empty<content>();
foreach (relationship r in friendData)
{
IEnumerable<content> content = GetData(r);
results = results.Concat(content);
}
If you're doing an INNER join, look at the .Intersect() extension method.
Which things are you merging?
There are two main options you could use: .SelectMany(...) or .Concat(...)