We Keep Coding

C# linq operator as variable

(In theory) We have the following query:
var variableDate = DateTime.Parse("Nov 2, 2021")
var results = (from x in db.FooBar
where x.Date == variableDate
select x).ToList();
We can simply modify the variableDate to 11/3/21, 11/4/21, etc. to return results for different dates.
The question is can we use this same query to return all results after variableDate with some modification to the operator(s)/variable(s)? For example including the operator (> or ==) within variableDate (let's call it variableDateFormula):
> 11/2/2021
or
== 11/2/2021
This way we can simply call the same query with the same variable to return results for different operators:
var results = (from x in db.FooBar
where x.Date variableDateFormula
select x).ToList();
I know the suggestion above will not work, it is just for visualization. I have a query that takes up 9 lines and has 8 different iterations depending on 3 values passed into the function. I would love to avoid writing basically the same query 8 times and taking up ~70 lines, and instead dynamically change the operators/criteria to have only 1 query.

Yes, the operator is just syntactic sugar for a function call and you can change that function, for example with a lambda:
var equalFunc = ((d1,d2) => d1 == d2);
var greaterFunc = ((d1,d2) => DateTime.Compare(d1, d2) > 0);
var actualFunc = greaterFunc; // Insert logic here to choose the appropriate function
Then your select becomes
from x in db.FooBar
where actualFunc(x.Date, variableDate)
select x

Yes. The LINQ is just building an ExpressionTree that Entity Framework translates in to SQL, so you can use other expressions or lambdas as parameters that get inserted into the tree, like in this LINQpad example:
void Main()
{
var aDate = new DateTime(2002, 1, 1);
GetResults(d => d.TheDate == aDate).Dump();
GetResults(d => d.TheDate <= aDate).Dump();
GetResults(d => d.TheDate > aDate).Dump();
}
IEnumerable<Data> GetResults(Func<Data, bool> op)
{
var data = new List<Data>
{
new Data{TheDate = new DateTime(2000,1,1)},
new Data{TheDate = new DateTime(2001,1,1)},
new Data{TheDate = new DateTime(2002,1,1)},
new Data{TheDate = new DateTime(2003,1,1)},
};
return data.Where(d => op(d));
}
public class Data
{
public DateTime TheDate { get; set; }
}
Which produces the following results:
If you are doing more complex logic in your expressions, the EF Database Provider may not be able to translate the expression to SQL though. There will be limitations. You couldn't for example, do this:
GetResults(d => d.TheDate.ToString().Reverse() == "1234");
because it wouldn't understand the custom Reverse extension method.

I would suggest to use LINQKit for such task. EF Core cannot translate local variable as expression function.
var variableDate = DateTime.Parse("Nov 2, 2021");
Expression<Func<DateTime, DateTime, bool>> compareFunc = (d1, d2) => d1 > d2;
var results = (from x in db.FooBar
where compareFunc.Invoke(x.Date, variableDate)
select x).ToList();
For enabling LINQKit for EF Core add the following to the options:
builder
.UseSqlServer(connectionString)
.WithExpressionExpanding(); // enabling LINQKit extension
Or if you use other LINQ provider, add AsExpandable() at the top of the query:
var results = (from x in db.FooBar.AsExpandable()
where compareFunc.Invoke(x.Date, variableDate)
select x).ToList();

finding sequential patterns of objects in a list with particular properties

I have a class like this:
public class TestResults
{
public String TestName {get;set;}
public Int32 StudentID {get;set;}
public Decimal Score {get;set;}
public Date TestTaken {get;set;}
}
So some objects mike look like this:
test.TestName = "Big Important Test";
test.StudentID = 17;
test.Score = 0.75M;
test.TestTaken = "1/1/2015";
tests.add(test);
test.TestName = "Big Important Test";
test.StudentID = 12;
test.Score = 0.89M;
test.TestTaken = "1/1/2015";
tests.add(test);
test.TestName = "Sneaky Pop Quiz in Chemistry";
test.StudentID = 17;
test.Score = 0.97M;
test.TestTaken = "2/1/2015";
tests.add(test);
test.TestName = "Sneaky Pop Quiz in Chemistry";
test.StudentID = 17;
test.Score = 0.97M;
test.TestTaken = "2/1/2015";
tests.add(test);
What I'm trying to determine is something like "For every student, show me students with large jumps in their scores?" I asked a similar question a while back in the dba.stackexchange.com world and have used the LEAD function, but now I'd like to move the logic into C#.
So a concrete question I'd want to code for would be (as an example):
Show me students who've jumped from the 60 and 70 percent range to the
90 range.
I know I can write a rat's nest of loops and branching logic, but was wondering if there are any more elegant and more comprehensive ways of identifying sequences of patterns in LINQ / C# land.
I've heard people talk about F#, but have no practical experience with that. Additionally, I think the "pattern matching" I'm talking about is a bit more involved than some of the simple string-pattern-matching I keep running across.

You could use LINQ to get the answer. Here is an example of a way you could do it:
var scores = tests.GroupBy(t => t.StudentID)
.Select(g => new { StudentID = g.Key, Min = g.Min(i => i.Score), Max = g.Max(i => i.Score) })
.Where(s => s.Max - s.Min > .20M);
foreach(var score in scores)
Console.WriteLine("Student: {0} Jump: {1}", score.StudentID, score.Max - score.Min);
The LINQ statement first groups by StudentID. Next it projects the StudentID and Min and Max scores from each group to a new anonymous type. Finally, apply a where condition that only returns items with a "large jump in score". I define "large jump in score" as the difference between max score and min score is greater than .20.
Note: this code will work even when a student has more than 2 scores in the list.
UPDATE:
Since you have updated your post I understand your question better. Here is an updated answer:
var scores = tests.GroupBy(t => t.StudentID)
.Select(g => new { StudentID = g.Key, Min = g.OrderBy(i => i.Score).First(), Max = g.OrderByDescending(i => i.Score).First() })
.Where(s => (s.Min.Score >= .60M & s.Min.Score < .80M) & s.Max.Score >= .90M & s.Min.TestTaken < s.Max.TestTaken);
foreach(var score in scores)
Console.WriteLine("Student: {0} Jump: {1}", score.StudentID, score.Max.Score - score.Min.Score);
This uses a similar approach, but instead of recording the min and max scores in the anonymous type, I record the TestResults instance having the min score and max score. In ther where clause we check that the TestResults having the min score is in the 60-80 range. We check that the TestResults having the max score is in the 90+ range. Finally, we check that the min score occurred on a date before the max one occurred.

You can do something like this:
const decimal differenceLimit = 0.05M;
var studentIdsWithJump = tests.GroupBy (g => g.StudentID)
.Where(g => g.OrderBy(c => c.Score)
.GroupAdjacentBy((first, second) =>
first.Score + differenceLimit < second.Score
).Count() > 1
)
.Select(g => g.Key);
With the helper method taken from here:
public static class LinqExtensions
{
public static IEnumerable<IEnumerable<T>> GroupAdjacentBy<T>(this IEnumerable<T> source, Func<T, T, bool> predicate)
{
using (var e = source.GetEnumerator())
{
if (e.MoveNext())
{
var list = new List<T> { e.Current };
var pred = e.Current;
while (e.MoveNext())
{
if (predicate(pred, e.Current))
{
list.Add(e.Current);
}
else
{
yield return list;
list = new List<T> { e.Current };
}
pred = e.Current;
}
yield return list;
}
}
}
}
This gives you the jumps for all ranges. If you want to narrow it down, you could add a further .Where() for scores > 60, and adjust the differenceLimit accordingly

How to calculate a running total using linq

I have a linq query result as shown in the image. In the final query (not shown) I am grouping by Year by LeaveType. However I want to calculate a running total for the leaveCarriedOver per type over years. That is, sick LeaveCarriedOver in 2010 becomes "opening" balance for sick leave in 2011 plus the one for 2011.
I have done another query on the shown result list that looks like:
var leaveDetails1 = (from l in leaveDetails
select new
{
l.Year,
l.LeaveType,
l.LeaveTaken,
l.LeaveAllocation,
l.LeaveCarriedOver,
RunningTotal = leaveDetails.Where(x => x.LeaveType == l.LeaveType).Sum(x => x.LeaveCarriedOver)
});
where leaveDetails is the result from the image.
The resulting RunningTotal is not cumulative as expected. How can I achieve my initial goal. Open to any ideas - my last option will be to do it in javascript in the front-end. Thanks in advance

The simple implementation is to get the list of possible totals first then get the sum from the details for each of these categories.
getting the distinct list of Year and LeaveType is a group by and select first of each group. we return a List<Tuple<int, string>> where Int is the year and string is the LeaveType
var distinctList = leaveDetails1.GroupBy(data => new Tuple<int, string>(data.Year, data.LeaveType)).Select(data => data.FirstOrDefault()).ToList();
then we want total for each of these elements so you want a select of that list to return the id (Year and LeaveType) plus the total so an extra value to the Tuple<int, string, int>.
var totals = distinctList.Select(data => new Tuple<int, string, int>(data.Year, data.LeaveType, leaveDetails1.Where(detail => detail.Year == data.Year && detail.LeaveType == data.LeaveType).Sum(detail => detail.LeaveCarriedOver))).ToList();
reading the line above you can see it take the distinct totals we want to list, create a new object, store the Year and LeaveType for reference then set the last Int with the Sum<> of the filtered details for that Year and LeaveType.

If I completely understand what you are trying to do then I don't think I would rely on the built in LINQ operators exclusively. I think (emphasis on think) that any combination of the built in LINQ operators is going to solve this problem in O(n^2) run-time.
If I were going to implement this in LINQ then I would create an extension method for IEnumerable that is similar to the Scan function in reactive extensions (or find a library out there that has already implemented it):
public static class EnumerableExtensions
{
public static IEnumerable<TAccumulate> Scan<TSource, TAccumulate>(
this IEnumerable<TSource> source,
TAccumulate seed,
Func<TAccumulate, TSource, TAccumulate> accumulator)
{
// Validation omitted for clarity.
foreach(TSource value in source)
{
seed = accumulator.Invoke(seed, value);
yield return seed;
}
}
}
Then this should do it around O(n log n) (because of the order by operations):
leaveDetails
.OrderBy(x => x.LeaveType)
.ThenBy(x => x.Year)
.Scan(new {
Year = 0,
LeaveType = "Seed",
LeaveTaken = 0,
LeaveAllocation = 0.0,
LeaveCarriedOver = 0.0,
RunningTotal = 0.0
},
(acc, x) => new {
x.Year,
x.LeaveType,
x.LeaveTaken,
x.LeaveAllocation,
x.LeaveCarriedOver,
RunningTotal = x.LeaveCarriedOver + (acc.LeaveType != x.LeaveType ? 0 : acc.RunningTotal)
});
You don't say, but I assume the data is coming from a database; if that is the case then you could get leaveDetails back already sorted and skip the sorting here. That would get you down to O(n).
If you don't want to create an extension method (or go find one) then this will achieve the same thing (just in an uglier way).
var temp = new
{
Year = 0,
LeaveType = "Who Cares",
LeaveTaken = 3,
LeaveAllocation = 0.0,
LeaveCarriedOver = 0.0,
RunningTotal = 0.0
};
var runningTotals = (new[] { temp }).ToList();
runningTotals.RemoveAt(0);
foreach(var l in leaveDetails.OrderBy(x => x.LeaveType).ThenBy(x => x.Year))
{
var s = runningTotals.LastOrDefault();
runningTotals.Add(new
{
l.Year,
l.LeaveType,
l.LeaveTaken,
l.LeaveAllocation,
l.LeaveCarriedOver,
RunningTotal = l.LeaveCarriedOver + (s == null || s.LeaveType != l.LeaveType ? 0 : s.RunningTotal)
});
}
This should also be O(n log n) or O(n) if you can pre-sort leaveDetails.

If I understand the question you want something like
decimal RunningTotal = 0;
var results = leaveDetails
.GroupBy(r=>r.LeaveType)
.Select(r=> new
{
Dummy = RunningTotal = 0 ,
results = r.OrderBy(o=>o.Year)
.Select(l => new
{
l.Year,
l.LeaveType ,
l.LeaveAllocation,
l.LeaveCarriedOver,
RunningTotal = (RunningTotal = RunningTotal + l.LeaveCarriedOver )
})
})
.SelectMany(a=>a.results).ToList();
This is basically using the Select<TSource, TResult> overload to calculate the running balance, but first grouped by LeaveType so we can reset the RunningTotal for every LeaveType, and then ungrouped at the end.

You have to use Window Function Sum here. Which is not supported by EF Core and earlier versions of EF. So, just write SQL and run it via Dapper
SELECT
l.Year,
l.LeaveType,
l.LeaveTaken,
l.LeaveAllocation,
l.LeaveCarriedOver,
SUM(l.LeaveCarriedOver) OVER (PARTITION BY l.Year, l.LeaveType) AS RunningTotal
FROM leaveDetails l
Or, if you are using EF Core, use package linq2db.EntityFrameworkCore
var leaveDetails1 = from l in leaveDetails
select new
{
l.Year,
l.LeaveType,
l.LeaveTaken,
l.LeaveAllocation,
l.LeaveCarriedOver,
RunningTotal = Sql.Ext.Sum(l.LeaveCarriedOver).Over().PartitionBy(l.Year, l.LeaveType).ToValue()
};
// switch to alternative LINQ translator
leaveDetails1 = leaveDetails1.ToLinqToDB();

Use Group By in order to remove duplicates

I am looking for a simple way of removing duplicates without having to implement the class IComparable, having to override GetHashCode etc..
I think this can be achieved with linq. I have the class:
class Person
{
public string Name;
public ing Age;
}
I have a list of about 500 People List<Person> someList = new List<Person()
now I want to remove people with the same name and if there is a duplicate I want to keep the person that had the greater age. In other words if I have the list:
Name----Age---
Tom, 24 |
Alicia, 22 |
Alicia, 12 |
I will like to end up with:
Name----Age---
Tom, 24 |
Alicia, 22 |
How can I do this with a query? My list is not that long so I don't want to create a hash set nor implement the IComparable interface. It will be nice if I can do this with a linq query.
I think this can be done with the groupBy extension method by doing something like:
var people = // the list of Person
person.GroupBy(x=>x.Name).Where(x=>x.Count()>1)
... // select the person that has the greatest age...

people
.GroupBy(p => p.Name)
.Select(g => g.OrderByDescending(p => p.Age).First())
This will work across different Linq providers. If this is just Linq2Objects, and speed is important (usually, it isn't) consider using one of the many MaxBy extensions found on the web (here's Skeet's) and replacing
g.OrderByDescending(p => p.Age).First()
with
g.MaxBy(p => p.Age)

This can be trivially easy so long as you first create a helper function MaxBy that is capable of selecting the item from a sequence who's selector is largest. Unfortunately the Max function in LINQ won't work as we want to select the item from the sequence, not the selected value.
var distinctPeople = people.GroupBy(person => person.Name)
.Select(group => group.MaxBy(person => person.Age));
And then the implementation of MaxBy:
public static TSource MaxBy<TSource, TKey>(this IEnumerable<TSource> source,
Func<TSource, TKey> keySelector, IComparer<TKey> comparer = null)
{
comparer = comparer ?? Comparer<TKey>.Default;
using (var iterator = source.GetEnumerator())
{
if (!iterator.MoveNext())
throw new ArgumentException("Source must have at least one item");
var maxItem = iterator.Current;
var maxKey = keySelector(maxItem);
while (iterator.MoveNext())
{
var nextKey = keySelector(iterator.Current);
if (comparer.Compare(nextKey, maxKey) > 0)
{
maxItem = iterator.Current;
maxKey = nextKey;
}
}
return maxItem;
}
}
Note that while you can achieve the same result by sorting the sequence and then taking the first item, doing so is less efficient in general than doing just one pass with a max function.

I prefer to be simple:
var retPeople = new List<Person>;
foreach (var p in person)
{
if(!retPeople.Contains(p))
{
retPeople.Add(p);
}
}
Making Person to implement IComparable

I got rid of my last answer because I realized it was too slow and was too complicated. Here is the solution that makes a little more sense
var peoplewithLargestAgeByName =
from p in people
orderby p.Name
group p by p.Name into peopleByName
select peopleByName.First ( );
This is the same solution as the solution #spender contributed, but with the linq syntax.

How can I conditionally apply a Linq operator?

We're working on a Log Viewer. The use will have the option to filter by user, severity, etc. In the Sql days I'd add to the query string, but I want to do it with Linq. How can I conditionally add where-clauses?

if you want to only filter if certain criteria is passed, do something like this
var logs = from log in context.Logs
select log;
if (filterBySeverity)
logs = logs.Where(p => p.Severity == severity);
if (filterByUser)
logs = logs.Where(p => p.User == user);
Doing so this way will allow your Expression tree to be exactly what you want. That way the SQL created will be exactly what you need and nothing less.

If you need to filter base on a List / Array use the following:
public List<Data> GetData(List<string> Numbers, List<string> Letters)
{
if (Numbers == null)
Numbers = new List<string>();
if (Letters == null)
Letters = new List<string>();
var q = from d in database.table
where (Numbers.Count == 0 || Numbers.Contains(d.Number))
where (Letters.Count == 0 || Letters.Contains(d.Letter))
select new Data
{
Number = d.Number,
Letter = d.Letter,
};
return q.ToList();
}

I ended using an answer similar to Daren's, but with an IQueryable interface:
IQueryable<Log> matches = m_Locator.Logs;
// Users filter
if (usersFilter)
matches = matches.Where(l => l.UserName == comboBoxUsers.Text);
// Severity filter
if (severityFilter)
matches = matches.Where(l => l.Severity == comboBoxSeverity.Text);
Logs = (from log in matches
orderby log.EventTime descending
select log).ToList();
That builds up the query before hitting the database. The command won't run until .ToList() at the end.

I solved this with an extension method to allow LINQ to be conditionally enabled in the middle of a fluent expression. This removes the need to break up the expression with if statements.
.If() extension method:
public static IQueryable<TSource> If<TSource>(
this IQueryable<TSource> source,
bool condition,
Func<IQueryable<TSource>, IQueryable<TSource>> branch)
{
return condition ? branch(source) : source;
}
This allows you to do this:
return context.Logs
.If(filterBySeverity, q => q.Where(p => p.Severity == severity))
.If(filterByUser, q => q.Where(p => p.User == user))
.ToList();
Here's also an IEnumerable<T> version which will handle most other LINQ expressions:
public static IEnumerable<TSource> If<TSource>(
this IEnumerable<TSource> source,
bool condition,
Func<IEnumerable<TSource>, IEnumerable<TSource>> branch)
{
return condition ? branch(source) : source;
}

When it comes to conditional linq, I am very fond of the filters and pipes pattern.
http://blog.wekeroad.com/mvc-storefront/mvcstore-part-3/
Basically you create an extension method for each filter case that takes in the IQueryable and a parameter.
public static IQueryable<Type> HasID(this IQueryable<Type> query, long? id)
{
return id.HasValue ? query.Where(o => i.ID.Equals(id.Value)) : query;
}

Doing this:
bool lastNameSearch = true/false; // depending if they want to search by last name,
having this in the where statement:
where (lastNameSearch && name.LastNameSearch == "smith")
means that when the final query is created, if lastNameSearch is false the query will completely omit any SQL for the last name search.

Another option would be to use something like the PredicateBuilder discussed here.
It allows you to write code like the following:
var newKids = Product.ContainsInDescription ("BlackBerry", "iPhone");
var classics = Product.ContainsInDescription ("Nokia", "Ericsson")
.And (Product.IsSelling());
var query = from p in Data.Products.Where (newKids.Or (classics))
select p;
Note that I've only got this to work with Linq 2 SQL. EntityFramework does not implement Expression.Invoke, which is required for this method to work. I have a question regarding this issue here.

It isn't the prettiest thing but you can use a lambda expression and pass your conditions optionally. In TSQL I do a lot of the following to make parameters optional:
WHERE Field = #FieldVar OR #FieldVar IS NULL
You could duplicate the same style with a the following lambda (an example of checking authentication):
MyDataContext db = new MyDataContext();
void RunQuery(string param1, string param2, int? param3){
Func checkUser = user =>
((param1.Length > 0)? user.Param1 == param1 : 1 == 1) &&
((param2.Length > 0)? user.Param2 == param2 : 1 == 1) &&
((param3 != null)? user.Param3 == param3 : 1 == 1);
User foundUser = db.Users.SingleOrDefault(checkUser);
}

I had a similar requirement recently and eventually found this in he MSDN.
CSharp Samples for Visual Studio 2008
The classes included in the DynamicQuery sample of the download allow you to create dynamic queries at runtime in the following format:
var query =
db.Customers.
Where("City = #0 and Orders.Count >= #1", "London", 10).
OrderBy("CompanyName").
Select("new(CompanyName as Name, Phone)");
Using this you can build a query string dynamically at runtime and pass it into the Where() method:
string dynamicQueryString = "City = \"London\" and Order.Count >= 10";
var q = from c in db.Customers.Where(queryString, null)
orderby c.CompanyName
select c;

You can create and use this extension method
public static IQueryable<TSource> WhereIf<TSource>(this IQueryable<TSource> source, bool isToExecute, Expression<Func<TSource, bool>> predicate)
{
return isToExecute ? source.Where(predicate) : source;
}

Just use C#'s && operator:
var items = dc.Users.Where(l => l.Date == DateTime.Today && l.Severity == "Critical")
Edit: Ah, need to read more carefully. You wanted to know how to conditionally add additional clauses. In that case, I have no idea. :) What I'd probably do is just prepare several queries, and execute the right one, depending on what I ended up needing.

You could use an external method:
var results =
from rec in GetSomeRecs()
where ConditionalCheck(rec)
select rec;
...
bool ConditionalCheck( typeofRec input ) {
...
}
This would work, but can't be broken down into expression trees, which means Linq to SQL would run the check code against every record.
Alternatively:
var results =
from rec in GetSomeRecs()
where
(!filterBySeverity || rec.Severity == severity) &&
(!filterByUser|| rec.User == user)
select rec;
That might work in expression trees, meaning Linq to SQL would be optimised.

Well, what I thought was you could put the filter conditions into a generic list of Predicates:
var list = new List<string> { "me", "you", "meyou", "mow" };
var predicates = new List<Predicate<string>>();
predicates.Add(i => i.Contains("me"));
predicates.Add(i => i.EndsWith("w"));
var results = new List<string>();
foreach (var p in predicates)
results.AddRange(from i in list where p.Invoke(i) select i);
That results in a list containing "me", "meyou", and "mow".
You could optimize that by doing the foreach with the predicates in a totally different function that ORs all the predicates.