We Keep Coding

Why in LINQ using multiple WHERE predicate is slower the using multiple AND? [duplicate]

This question already has answers here:
Should I use two "where" clauses or "&&" in my LINQ query?
(6 answers)
Closed 8 years ago.
Very often use LINQ to filter object array
I ran a test expression that produces the same result, but as different times, I would like to know the reason for this behavior.
public long testTimeOperetionWHERE()
{
Object[] list = opCoIn.getList();
System.Diagnostics.Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
int i = 0;
while (i<20000)
{
var result = list.Where(o => o.Id>0)
.Where(o => o.Import>0)
.Where(o => o.OrderConfirm==o.NumberConfirm)
.Where(o => o.IdActiveCustomer>100 );
i++;
}
long e = sw.ElapsedMilliseconds;
return e;
}
The time cost result always varies between 90-80
In this case
public long testTimeOperetionAND()
{
Object[] list = opCoIn.getList();
System.Diagnostics.Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
int i = 0;
while (i < 20000)
{
var result = list.Where(o => o.Id > 0
&& o.Import > 0
&& o.OrderConfirm==o.NumberConfirm
&& o.IdActiveCustomer>100);
i++;
}
long e = sw.ElapsedMilliseconds;
return e;
}
The time cost is always less than 5

You are not even executing this LINQ query. You are just defining it.
If you'd for example use foreach, ToList() or Count() you'd get more meaningful results.
while (i < 20000)
{
var result = list.Where(o => o.Id > 0
&& o.Import > 0
&& o.OrderConfirm==o.NumberConfirm
&& o.IdActiveCustomer>100); // not executed
int justToExecuteIt = result.Count(); // executed here
i++;
}
There should not be a great difference between consecutive Wheres and consecutive &&.
I have asked a similar question recently: LINQ: differences between single Where with multiple conditions and consecutive Wheres with single condition
You might also find this question useful to understand the benefits of LINQ's deferred execution: What are the benefits of a Deferred Execution in LINQ?

LINQ Filtering - Optimization

I make one database trip to get a list of entities.
I then would like to separate this list into 2 lists, one for the entities that have not expired (using a start and end) which i call TopListings and another which are regular listings, those that have expired or have start/end date as null (the ones that are not TopListings)
I am not entirely sure which filtering is fasted to separate into 2 lists, should I get the toplist first, then filter second list based on what is NOT in the top list for second?
var listings = ListingAdapter.GetMapListings(criteria);
var topListings = listings.Where(x => x.TopStartDate >= DateTime.Now && x.TopExpireDate >= DateTime.Now);
//I AM NOT SURE WHAT THIS LINE SHOULD BE
var regularListings = listings.Where(x => x.TopStartDate < DateTime.Now || x.TopExpireDate < DateTime.Now || x.TopStartDate == null || x.TopExpireDate == null );
Thank you

You might want to use a LookUp
like this:
var lookup = listings.ToLookup(x => x.TopStartDate >= DateTime.Now && x.TopExpireDate >= DateTime.Now);
var topListings = lookup[true];
var regularListings = lookup[false]; // I assume everything not a topListing is a regular listing.
If this isnt enough, you could create an enum
enum ListingType { Top, Regular, WhatEver };
...
var lookup = listings.ToLookUp(determineListingType); // pass a methoddelegate that determines the listingtype for an element.
...
var topListings = lookup[ListingType.Top];
var regularListings = lookup[ListingType.Regular];
var whateverListings = lookup[ListingType.WhatEver];

In this case, it would probably be easier to use a loop, instead of Linq operators:
var topListings = new List<Listing>();
var regularListings = new List<Listing>();
foreach (var x in listings)
{
if (x.TopStartDate >= DateTime.Now && x.TopExpireDate >= DateTime.Now)
topListings.Add(x);
else
regularListings.Add(x);
}
This is also more efficient, because the list is enumerated only once.

Take a look at the 'Except' operator to make things a little easier. You might have to add a .ToList() on topListings first though.
var regularListings = listings.Except(topListings);
http://blogs.msdn.com/b/charlie/archive/2008/07/12/the-linq-set-operators.aspx

Make use of regular foreach loop that's straight forward. You can iterate through listing with one go and add items to appropriate collections. If you are LINQ kind of guy, ForEach extension is what you are looking for:
var topListings = new List<Listing>();
var regularListings = new List<Listing>();
listing.ForEach(item=>{
if (x.TopStartDate < DateTime.Now
|| // I've inverted the condition, since it is faster-one or two conditions will be checked, instead of always two
x.TopExpireDate < DateTime.Now)
regularListings.Add(x);
else
topListings.Add(x);
});

Linq to Entities performance difference between Expression/Func

I was just testing a simple query that i'm accessing in different ways, but the speed of each can vary by up to 2 seconds. I was hoping someone can clarify why this is the case. My project is in it's very early stages, so I thought I'd make sure I'm doing it right before it gets too big.
Admittedly, my testing style isn't perfect, but i think it's good enough for this.
I'm using a generic Repository and UnitofWork, and I hit the DB (sqlexpress on my local machine) 10,000 times in this while statement. The table only has 64 records. Tests are run in Release mode.
[TestMethod]
public void MyTestMethod()
{
using (var u = new UnitOfWork())
{
TestA(u);
TestB(u);
}
}
TestA (Func):
public void TestA(UnitOfWork u)
{
Stopwatch s = Stopwatch.StartNew();
s.Start();
var x = 0;
var repo = u.Repository<MyEntity>();
var code = "ABCD".First().ToString();
while (x < 10000)
{
var testCase = repo.Single(w => w.Code == code && w.CodeOrder == 0).Name;
x++;
}
s.Stop();
Console.WriteLine("TESTA: " + s.Elapsed);
}
TestB (Expression):
public void TestB(UnitOfWork u)
{
Stopwatch s = Stopwatch.StartNew();
s.Start();
var x = 0;
var repo = u.Repository<MyEntity>();
var code = "ABCD".First().ToString();
while (x < 10000)
{
var testCase = repo.First(w => w.Code == code && w.CodeOrder == 0).Name;
x++;
}
s.Stop();
Console.WriteLine("TESTB: " + s.Elapsed);
}
Even though i'm using the calls First() and Single(), they're not the built-in LINQ calls. They're part of my repository.
First() expression (IQueryable)
public TEntity Single(Func<TEntity, bool> predicate)
{
return dbSet.FirstOrDefault(predicate);
}
Single() func (IEnumerable)
public TEntity First(Expression<Func<TEntity, bool>> predicate)
{
return dbSet.FirstOrDefault(predicate);
}
Output:
Test Name: MyTestMethod
Test Outcome: Passed
Result StandardOutput:
TESTA: 00:00:02.4798818
TESTB: 00:00:03.4212112

First() with Expression<Func<...>> parameter is an extension method on IQueryable<T> and is used by query providers, like LINQ to Entities. Expression tree you provide is transformed into proper SQL query, which is sent to DB and only necessary rows are returned back to your application.
First() with Func<...> parameter is an extension method on IEnumerable<T> and is used by LINQ to Objects, which mean all the records from database will be fetched into application memory, and then element will be search as in-memory query, which is implemented as linear search.
You should definitely use the one from IQueryable<T>, because it will be more efficient (as database is optimized to perform queries).

This is not an answer, but just trying to make sure that the test results are more reliable.
Try writing your tests like this:
public long TestA()
{
using (var u = new UnitOfWork())
{
var s = Stopwatch.StartNew();
var x = 0;
var repo = u.Repository<MyEntity>();
var code = "ABCD".First().ToString();
while (x < 10000)
{
var testCase = repo.Single(w => w.Code == code && w.CodeOrder == 0).Name;
x++;
}
s.Stop();
return s.ElapsedMilliseconds;
}
}
(Obviously TestB is just a minor variant.)
And then your test method becomes:
[TestMethod]
public void MyTestMethod()
{
var dummyA = TestA();
var dummyB = TestB();
var realA = 0L;
var realB = 0L;
for (var i = 0; i < 10; i++)
{
realA += TestA();
realB += TestB();
}
Console.WriteLine("TESTA: " + realA.ToString());
Console.WriteLine("TESTB: " + realA.ToString());
}
Now your results are likely to be more accurate. Let us know the timings now.
Now try changing your tests like this:
public int TestA()
{
var gc0 = GC.CollectionCount(0);
using (var u = new UnitOfWork())
{
var s = Stopwatch.StartNew();
var x = 0;
var repo = u.Repository<MyEntity>();
var code = "ABCD".First().ToString();
while (x < 10000)
{
var testCase = repo.Single(w => w.Code == code && w.CodeOrder == 0).Name;
x++;
}
s.Stop();
}
return GC.CollectionCount(0) - gc0;
}
This should determine how many generation 0 garbage collections are being performed. That might indicate that the performance issues are with your tests and not with the SQL.

I will list some tests you might wanna try to help you narrow the differences between the operations.
Check the actual SQL code
Turn on the debug log for the queries or check it on the SSE logs. It is important since the EF engine should optimize the statements, and you can see what is really beeing sent to the DB.
As you said, the First operation should be faster, since there are optimized SQL operators for that. The Single should be slower since it has to validate all the values, and would scale based on the amount of rows.
Use the real SQL on the database for a reference test
Once you have the real SQL you can also check the differences of time elapsed on the database directly. Implement the same C# test on the DB, a Sotred Procedure maybe, and see what happens.
Try the built-in LINQ for comparison
I dont know if you already did it for the test, but try to use the native LINQ for a comparison.
I made many tests here using LINQ and there were no differences between the two statements you presented, so it actually could be the Expressions. (I used the SS CE btw).
Also, just for the sake of saying it, remmember to create Indexes for columns involved in heavy operations ;)
EF 6.1 has this feature built-in now.
[Index]
public String MyProperty{ get; set; }
Let me know if it was helpful.

How to change var from interface.System.Linq.IQueryable<out T> to class System.Collections.generic.List<T>

I have the following code:
var allWorkorders =
(from wo in context.WORKORDERs
join wot in context.WORKORDERTYPEs on wo.wot_oi equals wot.wotyoi
join pri in context.PRIORITies on wo.prio_oi equals pri.priooi
join s in context.SITEs on wo.BEparn_oi equals s.siteoi
where wo.audt_created_dttm.Value.Year >= now.Year - 3 && wo.audt_created_dttm.Value.Year >= 2006
&& wo.audt_created_dttm < timeframe && (s.id == "NM" || s.id == "TH") &&
(wo.clsdt_date ?? new DateTime(3000, 01, 01)) < DateTime.Now
group pri by new {s.id, pri.prioid, MonthNum = (wo.clsdt_date ?? new DateTime(3000, 01, 01)).Year * 100 +
(wo.clsdt_date ?? new DateTime(3000, 01, 01)).Month} into groupItem
orderby groupItem.Key.MonthNum, groupItem.Key.id
select new {groupItem.Key.id, groupItem.Key.prioid, groupItem.Key.MonthNum, Unit = groupItem.Count()});
allWorkorders.GroupBy(x => new { x.id, x.MonthNum }).Select(x => new {x.Key.id, x.Key.MonthNum,
Denominator = x.Sum(y => y.Unit), Numerator = x.Where(y => SqlMethods.Like(y.prioid, "1%") ||
SqlMethods.Like(y.prioid, "6%")).Sum(y => y.Unit), Data_Indicator = DATA_INDICATOR,
Budgeted = budgetedPlannedOutageHrs, Industry_Benchmark = INDUSTRY_BENCHMARK,
Comments = comments, Executive_Comments = executiveComments,
Fleet_Exec_Comments = fleetExecComments}).ToList();
I want to create a for loop:
for (int counter = 0; counter < allWorkorders.Count; counter++)
{
var item = allWorkorders[counter];
......
However, I get the following error: " '<' cannot be applied to operands of type 'int' and 'method group'"
So even though I have allWorkorders going to ToList() it's not being recognized as a list.
What am I doing wrong? I have done this in the past, the biggest difference being that in the past cases my ToList was at the end of the select statement.

It is trying to use the LINQ extension method method Count() rather than the List<T>.Count
The reason it is doing this is you are not assigning the results of ToList() to anything. This whole statement is basically ignored because you are not using the return value
allWorkorders.GroupBy(x => new { x.id, x.MonthNum }).Select(x => new {x.Key.id, x.Key.MonthNum,
Denominator = x.Sum(y => y.Unit), Numerator = x.Where(y => SqlMethods.Like(y.prioid, "1%") ||
SqlMethods.Like(y.prioid, "6%")).Sum(y => y.Unit), Data_Indicator = DATA_INDICATOR,
Budgeted = budgetedPlannedOutageHrs, Industry_Benchmark = INDUSTRY_BENCHMARK,
Comments = comments, Executive_Comments = executiveComments,
Fleet_Exec_Comments = fleetExecComments}).ToList();

You didn't assign the second line (the one with ToList()) on it to anything. You ended the assignment of allWorkOrders with: "Unit = groupItem.Count()});"
Dropping on ToList() will make it return a list, but since you didn't assign it to anything it immediately goes out of scope and you lose it.

ToList() returns a result. You need something like
var newList = allWorkorders.GroupBy(x => ...).Select(x => ...).ToList();

You can either use a foreach loop instead or add the () to Count, for (int counter = 0; counter < allWorkorders.Count(); counter++). In this case Count is not a property but a Linq extension method that you're calling on an IEnumerable, which is why it complains about a method group when you give it just .Count instead of calling the method .Count().
Also, you have two separate statements there and don't appear to be storing the part which you are performing the .ToList() against anywhere.

Linq to Objects ordering by arbitrary number of parameters

I have a list of Func defining an ordering:
var ordering = new List<Func<Person, IComparable>>
{ x => x.Surname, x => x.FirstName };
I can order the results with something like...
people = people.OrderBy(ordering[0]).ThenBy(ordering[1]);
I'm trying to figure how to do the above when the list can contain any number of sequential orderings. Is it possible?

people = people.OrderBy(ordering[0]).ThenBy(ordering[1]).ThenBy(ordering[2]);
is the same as
var orderedPeople = people.OrderBy(ordering[0]);
orderedPeople = orderedPeople.ThenBy(ordering[1]);
orderedPeople = orderedPeople.ThenBy(ordering[2]);
people = orderedPeople;
so you simply write a loop like this:
if (ordering.Count != 0)
{
var orderedPeople = people.OrderBy(ordering[0]);
for (int i = 1; i < ordering.Count; i++)
{
orderedPeople = orderedPeople.ThenBy(ordering[i]);
}
people = orderedPeople;
}

As others have mentioned, you can use a loop to do this.
If you prefer, you can also use the Aggregate operator:
// Requires a non-empty ordering sequence.
var result2 = ordering.Skip(1)
.Aggregate(people.OrderBy(ordering.First()), Enumerable.ThenBy);
(or)
// Shorter and more "symmetric" but potentially more inefficient.
// x => true should work because OrderBy is a stable sort.
var result = ordering.Aggregate(people.OrderBy(x => true), Enumerable.ThenBy);

You should be able to do something similar to this
people = people.OrderBy(ordering[0])
foreach(var order in ordering.Skip(1))
{
people = people.ThenBy(order);
}
Alternately
for(i = 0; i < ordering.Count; i++)
{
people = i == 0 ? people.OrderBy(ordering[i]) : people.ThenBy(ordering[i]);
}

Remember that LINQ execution is deferred. You can build up the expression sequentially before accessing the results, doing something like:
var ordered = unordered.OrderBy(ordering.First());
foreach (var orderingItem in ordering.Skip(1))
{
ordered = ordered.ThenBy(orderingItem);
}

You might want to do this with dynamically building up you're expression. More info here: Dynamic LINQ and Dynamic Lambda expressions?