We Keep Coding

How to make a linq-query with multiple Contains()/Any() on possibly empty lists?

I am trying to make a query to a database view based on earlier user-choices. The choices are stored in lists of objects.
What I want to achieve is for a record to be added to the reportViewList if the stated value exists in one of the lists, but if for example the clientList is empty the query should overlook this statement and add all clients in the selected date-range. The user-choices are stored in temporary lists of objects.
The first condition is a time-range, this works fine. I understand why my current solution does not work, but I can not seem to wrap my head around how to fix it. This example works when both a client and a product is chosen. When the lists are empty the reportViewList is obviously also empty.
I have played with the idea of adding all the records in the date-range and then removing the ones that does not fit, but this would be a bad solution and not efficient.
Any help or feedback is much appreciated.
List<ReportView> reportViews = new List<ReportView>();
using(var dbc = new localContext())
{
reportViewList = dbc.ReportViews.AsEnumerable()
.Where(x => x.OrderDateTime >= from && x.OrderDateTime <= to)
.Where(y => clientList.Any(x2 => x2.Id == y.ClientId)
.Where(z => productList.Any(x3 => x3.Id == z.ProductId)).ToList();
}

You should not call AsEnumerable() before you have added eeverything to your query. Calling AsEnumerable() here will cause your complete data to be loaded in memory and then be filtered in your application.
Without AsEnumerable() and before calling calling ToList() (Better call ToListAsync()), you are working with an IQueryable<ReportView>. You can easily compose it and just call ToList() on your final query.
Entity Framework will then examinate your IQueryable<ReportView> and generate an SQL expression out of it.
For your problem, you just need to check if the user has selected any filters and only add them to the query if they are present.
using var dbc = new localContext();
var reportViewQuery = dbc.ReportViews.AsQueryable(); // You could also write IQuryable<ReportView> reportViewQuery = dbc.ReportViews; but I prefer it this way as it is a little more save when you are refactoring.
// Assuming from and to are nullable and are null if the user has not selected them.
if (from.HasValue)
reportViewQuery = reportViewQuery.Where(r => r.OrderDateTime >= from);
if (to.HasValue)
reportViewQuery = reportViewQuery.Where(r => r.OrderDateTime <= to);
if(clientList is not null && clientList.Any())
{
var clientIds = clientList.Select(c => c.Id).ToHashSet();
reportViewQuery = reportViewQuery.Where(r => clientIds.Contains(y.ClientId));
}
if(productList is not null && productList.Any())
{
var productIds = productList.Select(p => p.Id).ToHashSet();
reportViewQuery = reportViewQuery.Where(r => productIds .Contains(r.ProductId));
}
var reportViews = await reportViewQuery.ToListAsync(); // You can also use ToList(), if you absolutely must, but I would not recommend it as it will block your current thread.

C#, lambda : How are redundant calls handled?

Im curious about how the compiler handles the following expression:
var collapsed = elements.GroupBy(elm => elm.OrderIdentifier).Select(group => new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = group.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = group.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = group.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = group.First().EfterFladderOpstilTid,
EfterRadanOpstilTid = group.First().EfterRadanOpstilTid,
});
As you can see, I'm using group sum twice, so does anyone know if the "group" list will be iterated twice to get both sums, or will it be optimized so there is actually only 1 complete iteration of the list.

LINQ ist most often not the best way to reach high performance, what you get is productivity in programming, you get a result without much lines of code.
The possibilities to optimize is limited. In case of Querys to SQL, there is one rule of thumb: One Query is better than two queries.
1) there is only one round trip to the SQL_Server
2) SQL Server is made to optimize those queries, and optimization is getting better if the server knows, what you want to do in the next step. Optimization is done per query, not over multiple queries.
In case of Linq to Objects, there is absolutely no gain in building huge queries.
As your example shows, it will probably cause multiple iterations. You keep your code simpler and easier to read - but you give up control and therefore performance.

The compiler certainly won't optimize any of that.
If this is using LINQ to Objects, and therefore delegates, the delegate will iterate over each group 5 times, for the 5 properties.
If this is using LINQ to SQL, Entity Framework or something similar, and therefore expression trees, then it's basically up to the query provider to optimize this appropriately.

You can optimise your request by adding two field in the grouping key
var collapsed = elements.GroupBy(elm => new{
OrderIdentifier=elm.OrderIdentifier,
EfterFladderOpstilTid=elm.EfterFladderOpstilTid,
EfterRadanOpstilTid=elm.EfterRadanOpstilTid
})
.Select(group => new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = group.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = group.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = group.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = group.Key.EfterFladderOpstilTid,
EfterRadanOpstilTid = group.Key.EfterRadanOpstilTid,
});
Or by using LET statement
var collapsed = from groupedElement in
(from element in elements
group element by element.OrderIdentifier into g
select g)
let First = groupedElement.First()
select new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = groupedElement.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = groupedElement.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = groupedElement.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = First.EfterFladderOpstilTid,
EfterRadanOpstilTid = First.EfterRadanOpstilTid
};

Refactor GroupBy to avoid slowing down operation on big dataset

I have a big collection where i need to get the newest item based on two properties.
The first step is ordering the list based on the date prop. This is all fine and pretty quick.
Then I group the newlist by two properties, and take the first item from each.
var one = Fisks.Where(s=>s.Havn.Id == 1).OrderByDescending(s=>s.Date);
var two = one.GroupBy(s=>new {s.Arter.Name, s.Sort});
var three = two.Select(s=>s.FirstOrDefault());
This works, but it is really slow when using it on the large collection. How can I avoid using the groupBy but still get the same result?
Thanks!

Using LINQ only for the first step and then taking the first ones in a loop gives you more control over the process and avoids grouping altogether:
var query = Fisks
.Where(f => f.Havn.Id == 1)
.OrderByDescending(f => f.Date)
.ThenBy(f => f.Arter.Name)
.ThenBy(f => f.Sort);
var list = new List<Fisk>();
foreach (Fisk fisk in query) {
if (list.Count == 0) {
list.Add(fisk);
} else {
Fisk last = list[list.Count - 1];
if (fisk.Sort != last.Sort || fisk.Arter.Name != last.Arter.Name) {
list.Add(fisk);
}
}
}

Generally I advise against ordering before doing something that possibly destroys that order (such as GroupBy can do in SQL as generated by LINQ2SQL). Also try ordering only the stuff you are going to use. You can improve your query performance, if you limit the selection only the required fields/properties. You can fiddle around with this sample and use your real backend instead:
var Fisks=new[]{
new {Havn=new{Id=1},Date=DateTime.MinValue,Arter=new{Name="A"},Sort=1,Title="A1"},
new {Havn=new{Id=1},Date=DateTime.MinValue.AddDays(1),Arter=new{Name="A"},Sort=1,Title="A2"},
new {Havn=new{Id=1},Date=DateTime.MinValue,Arter=new{Name="B"},Sort=1,Title="B1",},
new {Havn=new{Id=1},Date=DateTime.MinValue.AddDays(2),Arter=new{Name="B"},Sort=1,Title="B2",},
new {Havn=new{Id=1},Date=DateTime.MinValue.AddDays(2),Arter=new{Name="B"},Sort=1,Title="B3",},
};
var stopwatch=Stopwatch.StartNew();
var one = Fisks.Where(s=>s.Havn.Id == 1).OrderByDescending(s=>s.Date);
var two = one.GroupBy(s=>new {s.Arter.Name, s.Sort});
var three = two.Select(s=>s.FirstOrDefault());
var answer=three.ToArray();
stopwatch.Stop();
stopwatch.ElapsedTicks.Dump("elapsed Ticks");
answer.Dump();
stopwatch.Restart();
answer=Fisks
.Where(f=>f.Havn.Id.Equals(1))
.GroupBy(s=>new {s.Arter.Name, s.Sort},(k,g)=>new{
s=g.OrderByDescending(s=>s.Date).First()//TOP 1 -> quite fast
})
.Select(g=>g.s)
.OrderByDescending(s=>s.Date) // only fully order results
.ToArray();
stopwatch.Stop();
stopwatch.ElapsedTicks.Dump("elapsed Ticks");
answer.Dump();
If you're working against any SQL Server you should check the generated SQL in LINQPad. You don't want to end up with a n+1 Query. Having an index on Havn.Id and Fisks.Date might also help.

Looping through fields to get sum

I know there are probably 100 much easier ways to do this but until I see it I can't comprehend how to go about it. I'm using linqpad for this. Before I go to phase 2 I need to get this part to work!
I've connected to an SQL database.
I'm running a query to retrieve some desired records.
var DesiredSym = (from r in Symptoms
where r.Status.Equals(1) && r.Create_Date < TimespanSecs
select r).Take(5);
So, in this example, I retrieve 5 'records' essentially in my DesiredSym variable as iQueryable (linqpad tells me this)
The DesiredSym contains a large number of fields including a number feilds that hold a int of Month1_Used, Month2_Used, Month3_Used .... Month12_Use.
So I want to loop through the DesiredSym and basically get the sum of all the Monthx_Used fields.
foreach (var MonthUse in DesiredSym)
{
// get sum of all fields where they start with MonthX_Used;
}
This is where I'm not clear on how to proceed or even if I'm on the right track. Thanks for getting me on the right track.

Since you've got a static number of fields, I'd recommend this:
var DesiredSym =
(from r in Symptoms
where r.Status.Equals(1) && r.Create_Date < TimespanSecs
select retireMe)
.Take(5);
var sum = DesiredSym.Sum(s => s.Month1_Use + s.Month2_Use + ... + s.Month12_Use);
You could use reflection, but that would be significantly slower and require more resources, since you'd need to pull the whole result set into memory first. But just for the sake of argument, it would look something like this:
var t = DesiredSym.GetType().GenericTypeArguments[0];
var props = t.GetProperties().Where(p => p.Name.StartsWith("Month"));
var sum = DesiredSym.AsEnumerable()
.Sum(s => props.Sum(p => (int)p.GetValue(s, null)));
Or this, which is a more complicated use of reflection, but it has the benefit of still being executed on the database:
var t = DesiredSym.GetType().GenericTypeArguments[0];
var param = Expression.Parameter(t);
var exp = t.GetProperties()
.Where(p => p.Name.StartsWith("Month"))
.Select(p => (Expression)Expression.Property(param, p))
.Aggregate((x, y) => Expression.Add(x, y));
var lambda = Expression.Lambda(exp, param);
var sum = DesiredSym.Sum(lambda);
Now, to these methods (except the third) to calculate the sum in batches of 5, you can use MoreLINQ's Batch method (also available on NuGet):
var DesiredSym =
from r in Symptoms
where r.Status.Equals(1) && r.Create_Date < TimespanSecs
select retireMe;
// first method
var batchSums = DesiredSym.Batch(5, b => b.Sum(s => s.Month1_Use ...));
// second method
var t = DesiredSym.GetType().GenericTypeArguments[0];
var props = t.GetProperties().Where(p => p.Name.StartsWith("Month"));
var batchSums = DesiredSym.Batch(5, b => b.Sum(s => props.Sum(p => (int)p.GetValue(s, null))));
Both these methods will be a bit slower and use more resources since all the processing has to be don in memory. For the same reason the third method will not work, since MoreLinq does not support the IQueryable interface.

Identify items in one list not in another of a different type

I need to identify items from one list that are not present in another list. The two lists are of different entities (ToDo and WorkshopItem). I consider a workshop item to be in the todo list if the Name is matched in any of the todo list items.
The following does what I'm after but find it awkward and hard to understand each time I revisit it. I use NHibernate QueryOver syntax to get the two lists and then a LINQ statement to filter down to just the Workshop items that meet the requirement (DateDue is in the next two weeks and the Name is not present in the list of ToDo items.
var allTodos = Session.QueryOver<ToDo>().List();
var twoWeeksTime = DateTime.Now.AddDays(14);
var workshopItemsDueSoon = Session.QueryOver<WorkshopItem>()
.Where(w => w.DateDue <= twoWeeksTime).List();
var matches = from wsi in workshopItemsDueSoon
where !(from todo in allTodos
select todo.TaskName)
.Contains(wsi.Name)
select wsi;
Ideally I'd like to have just one NHibernate query that returns a list of WorkshopItems that match my requirement.

I think I've managed to put together a Linq version of the answer put forward by #CSL and will mark that as the accepted answer as it put me in the direction of the following.
var twoWeeksTime = DateTime.Now.AddDays(14);
var subquery = NHibernate.Criterion.QueryOver.Of<ToDo>().Select(t => t.TaskName);
var matchingItems = Session.QueryOver<WorkshopItem>()
.Where(w => w.DateDue <= twoWeeksTime &&
w.IsWorkshopItemInProgress == true)
.WithSubquery.WhereProperty(x => x.Name).NotIn(subquery)
.Future<WorkshopItem>();
It returns the results I'm expecting and doesn't rely on magic strings. I'm hesitant because I don't fully understand the WithSubquery (and whether inlining it would be a good thing). It seems to equate to
WHERE WorkshopItem.Name IS NOT IN (subquery)
Also I don't understand the Future instead of List. If anyone would shed some light on those that would help.

I am not 100% sure how to achieve what you need using LINQ so to give you an option I am just putting up an alternative solution using nHibernate Criteria (this will execute in one database hit):
// Create a query
ICriteria query = Session.CreateCriteria<WorkShopItem>("wsi");
// Restrict to items due within the next 14 days
query.Add(Restrictions.Le("DateDue", DateTime.Now.AddDays(14));
// Return all TaskNames from Todo's
DetachedCriteria allTodos = DetachedCriteria.For(typeof(Todo)).SetProjection(Projections.Property("TaskName"));
// Filter Work Shop Items for any that do not have a To-do item
query.Add(SubQueries.PropertyNotIn("Name", allTodos);
// Return results
var matchingItems = query.Future<WorkShopItem>().ToList()

I'd recommend
var workshopItemsDueSoon = Session.QueryOver<WorkshopItem>()
.Where(w => w.DateDue <= twoWeeksTime)
var allTodos = Session.QueryOver<ToDo>();
Instead of
var allTodos = Session.QueryOver<ToDo>().List();
var workshopItemsDueSoon = Session.QueryOver<WorkshopItem>()
.Where(w => w.DateDue <= twoWeeksTime).List();
So that the collection isn't iterated until you need it to be.
I've found that it's helpfull to use linq extension methods to make subqueries more readable and less awkward.
For example:
var matches = from wsi in workshopItemsDueSoon
where !allTodos.Select(it=>it.TaskName).Contains(wsi.Name)
select wsi
Personally, since the query is fairly simple, I'd prefer to do it like so:
var matches = workshopItemsDueSoon.Where(wsi => !allTodos.Select(it => it.TaskName).Contains(wsi.Name))
The latter seems less verbose to me.