We Keep Coding

Dynamically construct where clause with Func<T, string> lambda - linq to entities

(this is for .Net Framework 4.7)
I'm trying to write up some extension methods to aid in creating dynamic where clauses for various entities. I started a few days ago, so there's likely a lot that I don't know and some that I probably misunderstood.
I managed to create one extension method already for filtering by 1 property which works as I expect it to (I did use reflection to get the property, couldn't get it working with an interface - well, without it executing the sql that is).
I can't seem to be able to get this one working for a lambda expression though.
Note, that the solution must not trigger sql execution. Because I was able to write up some variants that "worK', but they will trigger sql execution.
The way I work with this is that once I have the code ready, I start debugging and have the "query" in the watch. And it looks like this (notice the sql code)
Once I step over my FilterString method call, it either turns into a sql result, or I get an exception (with current code), which it shouldn't:
So here's my current code that throws the exception (currently not dealing with the "match" parameter, I am implementing an "equals" call. There will be others like, starts With, like, etc)
The exception is just one of those "type mismatch" having function cannot be passed as param to string Equals or what not.
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
var selector = Expression.Lambda<Func<T, string>>(getItemString, param);
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
prototype.Body.ReplaceParameter(prototype.Parameters[0], selector.Body),
selector.Parameters[0]);
return query.Where(predicate);
}
and the one that executes the sql instead of just generating it
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
//var value = Expression.Constant(getItemString);
var equals = typeof(string).GetMethod("Equals", new Type[] { typeof(string) });
var item = Expression.Invoke(getItemString, param);
var body = Expression.Call(Expression.Constant(criteriaItem),
equals,
item);
return query.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
calling these is done like so
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.MaybeSomeOtherProp.SomeString);
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.Name);
This same extension method will be called on any number of different entities, with nay number of different properties and prop names. I guess I could make use of the reflection version I got working and passing in all the property names in some array of some sort, but that is just plain ugly.
So long story short, how can I get this working in the way I explained above, taht is: having the sql generated instead of executed?
Thank you,
Note, the "ReplaceParameter" extension method is the one from here: https://stackoverflow.com/a/39206392/630515

So, you're trying to merge your prototype item => item == criteriaItem. With a passed in string property expression, like (r) => r.SomeProperty.Name to create (r) => r.SomeProperty.Name == criteriaItem.
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
ReplacingExpressionVisitor.Replace(
prototype.Parameters[0],
getItemString.Body,
prototype.Body),
getItemString.Parameters[0]);
And I think you're trying to do it this way so that criteriaItem is bound to an sql parameter, rather than being inlined as a string constant. But your question was a little hard to follow.

Logical Inverse of a Func<T, bool>

I have some fairly complex Entity Framework queries throughout my codebase and I decided to centralize the logic into the models. Basically, picture a bunch of controllers with large queries and lots of repeated code in their expression trees. So I took out parts of those expression trees and moved them to the models, allowing for less repetition.
For example, let's say I often need to fetch models called Entity which are in a state of Not Deleted. On my Entity model I have:
public static Func<Entity, bool> IsNotDeleted = e =>
e.Versions != null ?
e.Versions.OrderByDescending(v => v.VersionDate).FirstOrDefault() != null ?
e.Versions.OrderByDescending(v => v.VersionDate).First().ActivityType != VersionActivityType.Delete :
false :
false;
(This is one of the smaller examples, mostly just checking for valid data before trying to examine that data.)
And using it would look like:
var entities = EntityRepository.Entities.Where(Entity.IsNotDeleted).Where(...
I'm finding, however, that while sometimes I want records which are not deleted, other times I want records which are deleted. To do that, is there a way to invert the logic from the consuming code? Something conceptually akin to this (which obviously doesn't work):
var entities = EntityRepository.Entities.Where(!Entity.IsDeleted).Where(...
I'd prefer not to have two Func<>s on the object, one for IsDeleted and one for IsNotDeleted which are nearly identical. The Func<> returns a bool, is there a syntax to call the inverse of it when putting it in a .Where() clause?

Consider the following extension methods.
public static class Functional
{
public static Func<T, bool> Not<T>(this Func<T, bool> f)
{
return x => !f(x);
}
public static Expression<Func<T, bool>> Not<T>(
this Expression<Func<T, bool>> f)
{
// 1. Break the lambda f apart into its parameters and body.
// 2. Wrap the body expression with a unary not expression (!).
// 3. Construct a new lambda with the modified body.
return Expression.Lambda<Func<T, bool>>(
Expression.Not(f.Body), f.Parameters);
}
}
Entity.IsDeleted.Not() is the same as Entity.IsNotDeleted().
Note you probably want to be using Expression<Func<T, bool>> - not Func<T, bool> - so that your lambda logic can be used database-side rather than client-side.
You can use it like this:
Expression<Func<int, bool>> isNegative = x => x < 0;
Expression<Func<int, bool>> isNonNegative = isNegative.Not();

You don't need full blown functional lambda decalration. Go the Church-Turing way - reccursion for the IsNotDeleted definition:
public static Func<Entity, bool> IsNotDeleted = e => !IsDeleted(e);
Answer above is even more "Church-Turing-y" :)

Building a custom predicate to act as a filter using a foreach loop

I need to filter a list of documents by passing them to a custom filter that I'm struggling to build dynamically using a foreach loop :
var mainPredicate = PredicateBuilder.True<Document>();
// mainPredicate is combined to other filters successfully here ...
var innerPredicate = PredicateBuilder.False<Document>();
foreach (var period in periods)
{
var p = period;
Expression<Func<Document, bool>> inPeriod =
d => d.Date >= p.DateFrom && d.Date <= p.DateTo;
innerPredicate = innerPredicate.Or(d => inPeriod.Invoke(d));
}
mainPredicate = mainPredicate.And(innerPredicate);
This last line :
documents = this.ObjectSet.AsExpandable().Where(mainPredicate).ToList();
Throws this exception :
The parameter 'd' was not bound in the specified LINQ to Entities
query expression.
Anyone knows why I'm getting this exception ? I don't understand where the 'd' parameter I am passing to the InPeriod method gets lost. I don't know what is missing for this to work. My code is the same as many other examples that work perfectly. Any additionnal theoric theoric information about invoking expressions and how it works behind the scenes are welcome.

I don't understand why you do this:
innerPredicate = innerPredicate.Or(d => inPeriod.Invoke(d));
When you could just avoid the Invoke completely, like this:
innerPredicate = innerPredicate.Or(inPeriod);
This should work perfectly fine.
BTW, I have a feeling there's a bug with LINQKit here (unless there's some documentation that suggests that it doesn't support this scenario).
When I tried this similar code:
Expression<Func<int, bool>> first = p1 => p1 > 4;
Expression<Func<int, bool>> second = p2 => p2 < 2;
// Expand is similar to AsExpandable, except it works on
// expressions, not queryables.
var composite = first.Or(d => second.Invoke(d))
.Expand();
...LINQKit generated the following composite expression:
p1 => ((p1 > 4) OrElse (d < 2)) // what on earth is d?
... which indeed has the unbound parameter d (NodeType = Parameter, Name = 'd').
Dodging the Invoke with first.Or(second).Expand() generates the perfectly sensible:
p1 => ((p1 > 4) OrElse (p1 < 2)) // much better now...

Finally, I have found a way to avoid combining multiple predicates to the main expression tree.
Given that each predicate represents a different filter and I want the final, combined filter to be a series of must-be-respected conditions, we can say that each of the predicates has to return true for the final predicate to return true.
For that to work, the predicates has to be combined with AND. So, the resulting SQL query must look like this :
predicate1 AND predicate2 AND predicate3 ...
A better way to combine these predicates with AND is to chain Where query operators to the final query, like this :
var documents = this.ObjectSet.AsExpandable()
.Where(mainPredicate)
.Where(otherPredicate)
.Where(yetAnotherPredicate)
.ToList();
The resulting SQL query will combine each of these predicates with AND. That is just what I wanted to do.
It is easier than hacking out an expression tree by myself.

I use these extension methods:
public static class Extensions
{
public static Expression<Func<T, bool>> OrElse<T>(this Expression<Func<T, bool>> source, Expression<Func<T, bool>> predicate)
{
InvocationExpression invokedExpression = Expression.Invoke(predicate, source.Parameters.Cast<Expression>());
return Expression.Lambda<Func<T, bool>>(Expression.OrElse(source.Body, invokedExpression), source.Parameters);
}
public static Expression<Func<T, bool>> AndAlso<T>(this Expression<Func<T, bool>> source, Expression<Func<T, bool>> predicate)
{
InvocationExpression invokedExpression = Expression.Invoke(predicate, source.Parameters.Cast<Expression>());
return Expression.Lambda<Func<T, bool>>(Expression.AndAlso(source.Body, invokedExpression), source.Parameters);
}
}

Complex LINQ sorting using Lambda Expressions

Does anyone have/know of an IQueryable.OrderBy extension that takes an Expression (retrieved, for example, by Reflection)? I believe the function would look something like this:
public static IQueryable<TEntity> OrderBy<TEntity>
(this IQueryable<TEntity> source, Expression sortExpression)
Expression would be assumed to be an Expression<Func<TEntity, T>> where TEntity is the same object being sorted on, and T is a type that needs to be determined in order to create the new IQueryable.
I've found many examples of extensions that take a string, including Dynamic Linq, like this:
public static IQueryable<TEntity> OrderBy<TEntity>(
this IQueryable<TEntity> source, string sortExpression)
If it's possible to take the string and use Reflection to look up the type from the object in question, it should also be possible to take the Expression, and get the value type which is right there IN the Expression.
Following is a detailed explanation of why I'd like to have this, which you may or may not need.
I have a rather large list of complex records to sort. Because the list is so long, I prefer to have the sorting done on the database side. To handle more complex properties, I've created Expressions that provide the sorting functionality, like so:
if (model.sortExpression == "PlannedValue")
{
Expression<Func<BDopp, decimal>> sorter = BDopp.PlannedValueSorter;
if (model.sortDirection == "DESC")
opps = opps.OrderByDescending(sorter).AsQueryable();
else
opps = opps.OrderBy(sorter).AsQueryable();
}
BDOpp.PlannedValueSorter retrieves a static expression from the object which allows sorting to be done without opps are still of type IQueryable:
public static Expression<Func<BDopp, decimal>> PlannedValueSorter
{
get
{
return z => z.BudgetSchedules
.Where(s => s.Type == 1)
.Sum(s => s.Value * s.Workshare * z.valueFactor / 100 / 100);
}
}
Sorting for simple properties is done with Extension methods that use Reflection to build an expression based on the name of the property passed as a string.
This works well, but for the complex types, I still need branching logic, and I'd rather not do that. What I'd rather do is check for a static property containing the expression, and then simply apply it. I can get the expression like this:
PropertyInfo info = typeof(BDopp).GetProperty(model.sortExpression + "Sorter",
BindingFlags.Static | BindingFlags.Public);
Expression expr = (Expression)info.GetValue(null, null);
For the PlannedValue property, this gets me the expression sorted in PlannedValueSorter, which I already know works.
Update:
Various digging around has gotten me what I think might be some progress:
public static IQueryable<TEntity> OrderBy<TEntity>(this IQueryable<TEntity> source,
Expression<Func<TEntity, dynamic>> sortExpression)
{
var unary = sortExpression.Body as UnaryExpression;
Type actualExpressionType = unary.Operand.Type;
actualExpressionType is in fact the return type of the Expression (for this particular property, it's decimal).
Unfortunately I'm mostly just working by trial and error, since I haven't yet wrapped my brain around how all this works, so my attempt to update the query like so is not working:
MethodCallExpression resultExp = Expression.Call(typeof(Queryable),
"OrderBy",
new Type[] { typeof(TEntity), actualExpressionType },
source.Expression, sortExpression);
return source.Provider.CreateQuery<TEntity>(resultExp);
It compiles okay, but the following error is thrown at runtime:
No generic method 'OrderBy' on type 'System.Linq.Queryable' is
compatible with the supplied type arguments and arguments. No type
arguments should be provided if the method is non-generic.

As I understand it you have an Expression and want to order/filter by it. It might surprise you how simple it is:
Expression<Func<TEntity, T>> sortExpr = ...; //you already have this
var ordered = myQuery.OrderBy(sortExpr);
OrderBy always uses an Expression so you can directly use it. If you only have a variable of type Expression (which points to an object of type Expression<Func<TEntity, T>>) and don't know statically what T is, you can do this:
Expression sortExpr = ...; //you already have this
var ordered = myQuery.OrderBy((dynamic)sortExpr);
dynamic will figure this out at runtime using reflection. No need to do reflection yourself. All that is needed here is overload resolution (performed by the C# runtime binder).

Okay, I've got a solution:
public static IQueryable<TEntity> OrderBy<TEntity>(
this IQueryable<TEntity> source,
Expression<Func<TEntity, dynamic>> sortExpression,
bool descending)
{
var unary = sortExpression.Body as UnaryExpression;
var operand = unary.Operand;
Type actualExpressionType = operand.Type;
MethodCallExpression resultExp =
Expression.Call(typeof(Queryable),
descending? "OrderByDescending" : "OrderBy",
new Type[] { typeof(TEntity), actualExpressionType },
source.Expression,
Expression.Lambda(operand, sortExpression.Parameters));
return source.Provider.CreateQuery<TEntity>(resultExp);
}
The bool descending is to allow for the standard OrderBy and OrderByDescending overloads. Two major breakthroughs here, at least for me:
Getting the Operand out of the expression.
Using Expression.Call and Expression.Lambda to create the new expression - this allows me to use an actual "Type" variable, whereas the Expression<Func<TEntity, T>> syntax requires you to use a type that's known at compile time.

Try this:
public static IEnumerable<T> OrderBy<T>(
this IEnumerable<T> collection,
string columnName
//, SortDirection direction
)
{
ParameterExpression param = Expression.Parameter(typeof(T), "x"); // x
Expression property = Expression.Property(param, columnName); // x.ColumnName
Func<T, object> lambda = Expression.Lambda<Func<T, object>>( // x => x.ColumnName
Expression.Convert(property, typeof(object)),
param)
.Compile();
Func<IEnumerable<T>, Func<T, object>, IEnumerable<T>> expression = (c, f) => c.OrderBy(f); // here you can use OrderByDescending basing on SortDirection
IEnumerable<T> sorted = expression(collection, lambda);
return sorted;
}
Usage:
IEnumerable<T> collection = ...
IEnumerable<T> ordered = collection.OrderBy("PropName");
See Code Project and sandbox.

LINQ to entities - Building where clauses to test collections within a many to many relationship

So, I am using the Linq entity framework. I have 2 entities: Content and Tag. They are in a many-to-many relationship with one another. Content can have many Tags and Tag can have many Contents. So I am trying to write a query to select all contents where any tags names are equal to blah
The entities both have a collection of the other entity as a property(but no IDs). This is where I am struggling. I do have a custom expression for Contains (so, whoever may help me, you can assume that I can do a "contains" for a collection). I got this expression from: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2670710&SiteID=1
Edit 1
I ended up finding my own answer.

After reading about the PredicateBuilder, reading all of the wonderful posts that people sent to me, posting on other sites, and then reading more on Combining Predicates and Canonical Function Mapping.. oh and I picked up a bit from Calling functions in LINQ queries (some of these classes were taken from these pages).
I FINALLY have a solution!!! Though there is a piece that is a bit hacked...
Let's get the hacked piece over with :(
I had to use reflector and copy the ExpressionVisitor class that is marked as internal. I then had to make some minor changes to it, to get it to work. I had to create two exceptions (because it was newing internal exceptions. I also had to change the ReadOnlyCollection() method's return from:
return sequence.ToReadOnlyCollection<Expression>();
To:
return sequence.AsReadOnly();
I would post the class, but it is quite large and I don't want to clutter this post any more than it's already going to be. I hope that in the future that class can be removed from my library and that Microsoft will make it public. Moving on...
I added a ParameterRebinder class:
public class ParameterRebinder : ExpressionVisitor {
private readonly Dictionary<ParameterExpression, ParameterExpression> map;
public ParameterRebinder(Dictionary<ParameterExpression, ParameterExpression> map) {
this.map = map ?? new Dictionary<ParameterExpression, ParameterExpression>();
}
public static Expression ReplaceParameters(Dictionary<ParameterExpression, ParameterExpression> map, Expression exp) {
return new ParameterRebinder(map).Visit(exp);
}
internal override Expression VisitParameter(ParameterExpression p) {
ParameterExpression replacement;
if (map.TryGetValue(p, out replacement)) {
p = replacement;
}
return base.VisitParameter(p);
}
}
Then I added a ExpressionExtensions class:
public static class ExpressionExtensions {
public static Expression<T> Compose<T>(this Expression<T> first, Expression<T> second, Func<Expression, Expression, Expression> merge) {
// build parameter map (from parameters of second to parameters of first)
var map = first.Parameters.Select((f, i) => new { f, s = second.Parameters[i] }).ToDictionary(p => p.s, p => p.f);
// replace parameters in the second lambda expression with parameters from the first
var secondBody = ParameterRebinder.ReplaceParameters(map, second.Body);
// apply composition of lambda expression bodies to parameters from the first expression
return Expression.Lambda<T>(merge(first.Body, secondBody), first.Parameters);
}
public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> first, Expression<Func<T, bool>> second) {
return first.Compose(second, Expression.And);
}
public static Expression<Func<T, bool>> Or<T>(this Expression<Func<T, bool>> first, Expression<Func<T, bool>> second) {
return first.Compose(second, Expression.Or);
}
}
And the last class I added was PredicateBuilder:
public static class PredicateBuilder {
public static Expression<Func<T, bool>> True<T>() { return f => true; }
public static Expression<Func<T, bool>> False<T>() { return f => false; }
}
This is my result... I was able to execute this code and get back the resulting "content" entities that have matching "tag" entities from the tags that I was searching for!
public static IList<Content> GetAllContentByTags(IList<Tag> tags) {
IQueryable<Content> contentQuery = ...
Expression<Func<Content, bool>> predicate = PredicateBuilder.False<Content>();
foreach (Tag individualTag in tags) {
Tag tagParameter = individualTag;
predicate = predicate.Or(p => p.Tags.Any(tag => tag.Name.Equals(tagParameter.Name)));
}
IQueryable<Content> resultExpressions = contentQuery.Where(predicate);
return resultExpressions.ToList();
}
Please let me know if anyone needs help with this same thing, if you would like me to send you files for this, or just need more info.

Summing it up...
contentQuery.Where(
content => content.Tags.Any(tag => tags.Any(t => t.Name == tag.Name))
);
So is that what you're expecting?
I'm a little confused.

This is what the question itself asks for:
contentQuery.Where(
content => content.Tags.Any(tag => tag.Name == "blah")
);
I'm not sure what the thought process was to get to the questioner's code, really, and I'm not entirely sure exactly what its really doing. The one thing I'm really sure of is that .AsQueryable() call is completely unnecessary -- either .Tags is already an IQueryable, or the .AsQueryable() is just going to fake it for you -- adding extra calls in where there doesn't need to be any.

The error is related to the 'tags' variable. LINQ to Entities does not support a parameter that is a collection of values. Simply calling tags.AsQueryable() -- as suggested in an ealier answer -- will not work either because the default in-memory LINQ query provider is not compatible with LINQ to Entities (or other relational providers).
As a workaround, you can manually build up the filter using the expression API (see this forum post) and apply it as follows:
var filter = BuildContainsExpression<Element, string>(e => e.Name, tags.Select(t => t.Name));
var query = source.Where(e => e.NestedValues.Any(filter));

tags.Select(testTag => testTag.Name)
Where does the tags variable gets initialized from? What is it?

NOTE: please edit the question itself, rather than replying with an answer -- this is not a discussion thread, and they can re-order themselves at any time
If you're searching for all Contents that are marked with any one of a set of tags:
IEnumerable<Tag> otherTags;
...
var query = from content in contentQuery
where content.Tags.Intersection(otherTags).Any()
select content;
It looks like you might be using LINQ To SQL, in which case it might be better if you write a stored procedure to do this one: using LINQ to do this will probably not run on SQL Server -- it's very likely it will try to pull down everything from contentQuery and fetch all the .Tags collections. I'd have to actually set up a server to check that, though.