We Keep Coding

Dynamically construct where clause with Func<T, string> lambda - linq to entities

(this is for .Net Framework 4.7)
I'm trying to write up some extension methods to aid in creating dynamic where clauses for various entities. I started a few days ago, so there's likely a lot that I don't know and some that I probably misunderstood.
I managed to create one extension method already for filtering by 1 property which works as I expect it to (I did use reflection to get the property, couldn't get it working with an interface - well, without it executing the sql that is).
I can't seem to be able to get this one working for a lambda expression though.
Note, that the solution must not trigger sql execution. Because I was able to write up some variants that "worK', but they will trigger sql execution.
The way I work with this is that once I have the code ready, I start debugging and have the "query" in the watch. And it looks like this (notice the sql code)
Once I step over my FilterString method call, it either turns into a sql result, or I get an exception (with current code), which it shouldn't:
So here's my current code that throws the exception (currently not dealing with the "match" parameter, I am implementing an "equals" call. There will be others like, starts With, like, etc)
The exception is just one of those "type mismatch" having function cannot be passed as param to string Equals or what not.
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
var selector = Expression.Lambda<Func<T, string>>(getItemString, param);
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
prototype.Body.ReplaceParameter(prototype.Parameters[0], selector.Body),
selector.Parameters[0]);
return query.Where(predicate);
}
and the one that executes the sql instead of just generating it
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
//var value = Expression.Constant(getItemString);
var equals = typeof(string).GetMethod("Equals", new Type[] { typeof(string) });
var item = Expression.Invoke(getItemString, param);
var body = Expression.Call(Expression.Constant(criteriaItem),
equals,
item);
return query.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
calling these is done like so
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.MaybeSomeOtherProp.SomeString);
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.Name);
This same extension method will be called on any number of different entities, with nay number of different properties and prop names. I guess I could make use of the reflection version I got working and passing in all the property names in some array of some sort, but that is just plain ugly.
So long story short, how can I get this working in the way I explained above, taht is: having the sql generated instead of executed?
Thank you,
Note, the "ReplaceParameter" extension method is the one from here: https://stackoverflow.com/a/39206392/630515

So, you're trying to merge your prototype item => item == criteriaItem. With a passed in string property expression, like (r) => r.SomeProperty.Name to create (r) => r.SomeProperty.Name == criteriaItem.
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
ReplacingExpressionVisitor.Replace(
prototype.Parameters[0],
getItemString.Body,
prototype.Body),
getItemString.Parameters[0]);
And I think you're trying to do it this way so that criteriaItem is bound to an sql parameter, rather than being inlined as a string constant. But your question was a little hard to follow.

Why does Entity Framework create a different sql queries for very similar code

I've been recently playing around with SQL server profiler and noticed weird behaviour of generating two different queries for a code which in my opinion should works the same. Obviously I'm wrong hence the question.
Let's start from the top. I've a very simple repository class which consists of these methods:
public virtual TEntity GetSingle(Func<TEntity, bool> where, bool asNoTracking = true, params Expression<Func<TEntity, object>>[] includedNavigationProperties)
{
IQueryable<TEntity> dbQuery = this.ResolveIQueryableForType<TEntity>(asNoTracking, includedNavigationProperties);
return dbQuery.Where(where).FirstOrDefault();
}
public virtual IQueryable<TEntity> AsQueryable(bool asNoTracking = true, params Expression<Func<TEntity, object>>[] includedNavigationProperties)
{
IQueryable<TEntity> dbQuery = this.ResolveIQueryableForType<TEntity>(asNoTracking, includedNavigationProperties);
return dbQuery;
}
private IQueryable<TEntityType> ResolveIQueryableForType<TEntityType>(bool asNoTracking, params Expression<Func<TEntityType, object>>[] includedNavigationProperties)
where TEntityType : class
{
IQueryable<TEntityType> dbQuery = _context.Set<TEntityType>();
// Apply eager loading
if (includedNavigationProperties != null)
{
foreach (Expression<Func<TEntityType, object>> navigationProperty in includedNavigationProperties)
{
dbQuery = dbQuery.Include<TEntityType, object>(navigationProperty);
}
}
if (asNoTracking)
{
return dbQuery.AsNoTracking();
}
else
{
return dbQuery;
}
}
Later in the application I do this call (where AccessTokenRepository is an object of my repository type):
accessToken = _repository.AccessTokenRepository.AsQueryable().Where(x => x.AccessTokenID == accessTokenId).FirstOrDefault();
which results in this query:
exec sp_executesql N'SELECT TOP (1)
[Extent1].[AccessTokenID] AS [AccessTokenID],
[Extent1].[IssuedUtc] AS [IssuedUtc],
[Extent1].[ExpiresUtc] AS [ExpiresUtc],
[Extent1].[ValidForTimeSpan] AS [ValidForTimeSpan],
[Extent1].[CreatedDateTime] AS [CreatedDateTime]
FROM [dbo].[AccessToken] AS [Extent1]
WHERE [Extent1].[AccessTokenID] = #p__linq__0',N'#p__linq__0 uniqueidentifier',#p__linq__0='62A1BE60-3569-4E80-BC8E-FC01B0FFC266'
But similar call (which I would say should result in same SQL):
accessToken = _repository.AccessTokenRepository.GetSingle(x => x.AccessTokenID == accessTokenId);
results in:
SELECT
[Extent1].[AccessTokenID] AS [AccessTokenID],
[Extent1].[IssuedUtc] AS [IssuedUtc],
[Extent1].[ExpiresUtc] AS [ExpiresUtc],
[Extent1].[ValidForTimeSpan] AS [ValidForTimeSpan],
[Extent1].[CreatedDateTime] AS [CreatedDateTime]
FROM [dbo].[AccessToken] AS [Extent1]
and that looks like entire table load. Could someone please explain this subtle differences in the load behaviour?
Thank you

That's because the first parameter of your GetSingle method is defined as
Func<TEntity, bool> where
Instead of
Expression<Func<TEntity, bool>> where
And when you're passing that Func<TEntity, bool> (which is simply a generic delegate) to the Where() method, you're calling Enumerable.Where() (instead of Queryable.Where()) hence loading the entire DbSet to memory - and the SQL would not include the WHERE clause.
See What is the difference between IQueryable<T> and IEnumerable<T>?

Your GetSingle method takes Func<> as an argument, which forces the IQueryable<> to be cast to IEnumerable<>, causing the query to be "finished" (query executed will have a form dictated by an expression created until the collection was down-cast) and every subsequent operation to be performed in-memory. You have to use Expression<Func<>> so the engine can properly analyze the expression tree and generate correct query.

Passing compiled expressions to where() clause

I'm using calculated properties on my EF object which can't be passed directly to a where() clause:
{"The specified type member 'SomeProp' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported."}
Based on what I read on SO, this could be avoided by passing a Expression<Func<T, bool>> as the argument to where().
Compiling the expression to a Func works fine:
Expression<Func<Foo, bool>> expr = e => f => f.SomeCalculatedProperty == 1;
Func<Foo, bool> compiled = expr.Compile();
Foo result = dbContext.Foo.Where(compiled);
But passing the expression gives me the error above, i.e:
Expression<Func<Foo, bool>> expr = e => f => f.SomeCalculatedProperty == 1;
Foo result = dbContext.Foo.Where(e);
Based on this, shouldn't the last example work fine?

In example one you are calling the extension method Queryable.Where<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate) this converts the query to sql and applies the filter server side.
In example two you are calling the extension method Enumerable.Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate), this returns all rows from the database then in memory in the program it applies the filter.
For your third example, I will just quote Servy's comment
Your first and third snippets are functionally identical. There must
be some difference that you haven't shown for one to work and another
to not.

Logical Inverse of a Func<T, bool>

I have some fairly complex Entity Framework queries throughout my codebase and I decided to centralize the logic into the models. Basically, picture a bunch of controllers with large queries and lots of repeated code in their expression trees. So I took out parts of those expression trees and moved them to the models, allowing for less repetition.
For example, let's say I often need to fetch models called Entity which are in a state of Not Deleted. On my Entity model I have:
public static Func<Entity, bool> IsNotDeleted = e =>
e.Versions != null ?
e.Versions.OrderByDescending(v => v.VersionDate).FirstOrDefault() != null ?
e.Versions.OrderByDescending(v => v.VersionDate).First().ActivityType != VersionActivityType.Delete :
false :
false;
(This is one of the smaller examples, mostly just checking for valid data before trying to examine that data.)
And using it would look like:
var entities = EntityRepository.Entities.Where(Entity.IsNotDeleted).Where(...
I'm finding, however, that while sometimes I want records which are not deleted, other times I want records which are deleted. To do that, is there a way to invert the logic from the consuming code? Something conceptually akin to this (which obviously doesn't work):
var entities = EntityRepository.Entities.Where(!Entity.IsDeleted).Where(...
I'd prefer not to have two Func<>s on the object, one for IsDeleted and one for IsNotDeleted which are nearly identical. The Func<> returns a bool, is there a syntax to call the inverse of it when putting it in a .Where() clause?

Consider the following extension methods.
public static class Functional
{
public static Func<T, bool> Not<T>(this Func<T, bool> f)
{
return x => !f(x);
}
public static Expression<Func<T, bool>> Not<T>(
this Expression<Func<T, bool>> f)
{
// 1. Break the lambda f apart into its parameters and body.
// 2. Wrap the body expression with a unary not expression (!).
// 3. Construct a new lambda with the modified body.
return Expression.Lambda<Func<T, bool>>(
Expression.Not(f.Body), f.Parameters);
}
}
Entity.IsDeleted.Not() is the same as Entity.IsNotDeleted().
Note you probably want to be using Expression<Func<T, bool>> - not Func<T, bool> - so that your lambda logic can be used database-side rather than client-side.
You can use it like this:
Expression<Func<int, bool>> isNegative = x => x < 0;
Expression<Func<int, bool>> isNonNegative = isNegative.Not();

You don't need full blown functional lambda decalration. Go the Church-Turing way - reccursion for the IsNotDeleted definition:
public static Func<Entity, bool> IsNotDeleted = e => !IsDeleted(e);
Answer above is even more "Church-Turing-y" :)

Complex LINQ sorting using Lambda Expressions

Does anyone have/know of an IQueryable.OrderBy extension that takes an Expression (retrieved, for example, by Reflection)? I believe the function would look something like this:
public static IQueryable<TEntity> OrderBy<TEntity>
(this IQueryable<TEntity> source, Expression sortExpression)
Expression would be assumed to be an Expression<Func<TEntity, T>> where TEntity is the same object being sorted on, and T is a type that needs to be determined in order to create the new IQueryable.
I've found many examples of extensions that take a string, including Dynamic Linq, like this:
public static IQueryable<TEntity> OrderBy<TEntity>(
this IQueryable<TEntity> source, string sortExpression)
If it's possible to take the string and use Reflection to look up the type from the object in question, it should also be possible to take the Expression, and get the value type which is right there IN the Expression.
Following is a detailed explanation of why I'd like to have this, which you may or may not need.
I have a rather large list of complex records to sort. Because the list is so long, I prefer to have the sorting done on the database side. To handle more complex properties, I've created Expressions that provide the sorting functionality, like so:
if (model.sortExpression == "PlannedValue")
{
Expression<Func<BDopp, decimal>> sorter = BDopp.PlannedValueSorter;
if (model.sortDirection == "DESC")
opps = opps.OrderByDescending(sorter).AsQueryable();
else
opps = opps.OrderBy(sorter).AsQueryable();
}
BDOpp.PlannedValueSorter retrieves a static expression from the object which allows sorting to be done without opps are still of type IQueryable:
public static Expression<Func<BDopp, decimal>> PlannedValueSorter
{
get
{
return z => z.BudgetSchedules
.Where(s => s.Type == 1)
.Sum(s => s.Value * s.Workshare * z.valueFactor / 100 / 100);
}
}
Sorting for simple properties is done with Extension methods that use Reflection to build an expression based on the name of the property passed as a string.
This works well, but for the complex types, I still need branching logic, and I'd rather not do that. What I'd rather do is check for a static property containing the expression, and then simply apply it. I can get the expression like this:
PropertyInfo info = typeof(BDopp).GetProperty(model.sortExpression + "Sorter",
BindingFlags.Static | BindingFlags.Public);
Expression expr = (Expression)info.GetValue(null, null);
For the PlannedValue property, this gets me the expression sorted in PlannedValueSorter, which I already know works.
Update:
Various digging around has gotten me what I think might be some progress:
public static IQueryable<TEntity> OrderBy<TEntity>(this IQueryable<TEntity> source,
Expression<Func<TEntity, dynamic>> sortExpression)
{
var unary = sortExpression.Body as UnaryExpression;
Type actualExpressionType = unary.Operand.Type;
actualExpressionType is in fact the return type of the Expression (for this particular property, it's decimal).
Unfortunately I'm mostly just working by trial and error, since I haven't yet wrapped my brain around how all this works, so my attempt to update the query like so is not working:
MethodCallExpression resultExp = Expression.Call(typeof(Queryable),
"OrderBy",
new Type[] { typeof(TEntity), actualExpressionType },
source.Expression, sortExpression);
return source.Provider.CreateQuery<TEntity>(resultExp);
It compiles okay, but the following error is thrown at runtime:
No generic method 'OrderBy' on type 'System.Linq.Queryable' is
compatible with the supplied type arguments and arguments. No type
arguments should be provided if the method is non-generic.

As I understand it you have an Expression and want to order/filter by it. It might surprise you how simple it is:
Expression<Func<TEntity, T>> sortExpr = ...; //you already have this
var ordered = myQuery.OrderBy(sortExpr);
OrderBy always uses an Expression so you can directly use it. If you only have a variable of type Expression (which points to an object of type Expression<Func<TEntity, T>>) and don't know statically what T is, you can do this:
Expression sortExpr = ...; //you already have this
var ordered = myQuery.OrderBy((dynamic)sortExpr);
dynamic will figure this out at runtime using reflection. No need to do reflection yourself. All that is needed here is overload resolution (performed by the C# runtime binder).

Okay, I've got a solution:
public static IQueryable<TEntity> OrderBy<TEntity>(
this IQueryable<TEntity> source,
Expression<Func<TEntity, dynamic>> sortExpression,
bool descending)
{
var unary = sortExpression.Body as UnaryExpression;
var operand = unary.Operand;
Type actualExpressionType = operand.Type;
MethodCallExpression resultExp =
Expression.Call(typeof(Queryable),
descending? "OrderByDescending" : "OrderBy",
new Type[] { typeof(TEntity), actualExpressionType },
source.Expression,
Expression.Lambda(operand, sortExpression.Parameters));
return source.Provider.CreateQuery<TEntity>(resultExp);
}
The bool descending is to allow for the standard OrderBy and OrderByDescending overloads. Two major breakthroughs here, at least for me:
Getting the Operand out of the expression.
Using Expression.Call and Expression.Lambda to create the new expression - this allows me to use an actual "Type" variable, whereas the Expression<Func<TEntity, T>> syntax requires you to use a type that's known at compile time.

Try this:
public static IEnumerable<T> OrderBy<T>(
this IEnumerable<T> collection,
string columnName
//, SortDirection direction
)
{
ParameterExpression param = Expression.Parameter(typeof(T), "x"); // x
Expression property = Expression.Property(param, columnName); // x.ColumnName
Func<T, object> lambda = Expression.Lambda<Func<T, object>>( // x => x.ColumnName
Expression.Convert(property, typeof(object)),
param)
.Compile();
Func<IEnumerable<T>, Func<T, object>, IEnumerable<T>> expression = (c, f) => c.OrderBy(f); // here you can use OrderByDescending basing on SortDirection
IEnumerable<T> sorted = expression(collection, lambda);
return sorted;
}
Usage:
IEnumerable<T> collection = ...
IEnumerable<T> ordered = collection.OrderBy("PropName");
See Code Project and sandbox.