We Keep Coding

How to create a reusable 'Contains' expression for EF Core

Problem
I need to execute a partial text search, alongside other filters, via a generic repository using expressions.
State of current code
I have a generic method that returns paged results from my database (via a common repository layer).
In the following working example;
PagedRequest contains the current pagesize and page number, and is used during respective Skip / Take operations.
PagedResult contains a collection of the results, along with the total number of records.
public Task<PagedResult<Person>> GetPeopleAsync(PersonSearchParams searchParams,
PagedRequest pagedRequest = null)
{
ParameterExpression argParam = Expression.Parameter(typeof(Locum), "locum");
// start with a "true" expression so we have an expression to "AndAlso" with
var alwaysTrue = Expression.Constant(true);
var expr = Expression.Equal(alwaysTrue, alwaysTrue);
if (searchParams != null)
{
BinaryExpression propExpr;
if (searchParams.DateOfBirth.HasValue)
{
propExpr = GetExpression(searchParams.DateStart,
nameof(Incident.IncidentDate),
argParam,
ExpressionType.GreaterThanOrEqual);
expr = Expression.AndAlso(expr, propExpr);
}
if (searchParams.DateOfDeath.HasValue)
{
propExpr = GetExpression(searchParams.DateEnd,
nameof(Incident.IncidentDate),
argParam,
ExpressionType.LessThanOrEqual);
expr = Expression.AndAlso(expr, propExpr);
}
if (searchParams.BranchId.HasValue && searchParams.BranchId.Value != 0)
{
propExpr = GetExpression(searchParams.BranchId,
nameof(Incident.BranchId), argParam);
expr = Expression.AndAlso(expr, propExpr);
}
}
var lambda = Expression.Lambda<Func<Locum, bool>>(expr, argParam);
return _unitOfWork.Repository.GetAsync(filter: lambda, pagedRequest: pagedRequest);
}
This is using my static GetExpression method for Expression.Equal, Expression.GreaterThanOrEqual and Expression.LessThanOrEqual queries as follows;
private static BinaryExpression GetExpression<TValue>(TValue value,
string propName, ParameterExpression argParam, ExpressionType? exprType = null)
{
BinaryExpression propExpr;
var prop = Expression.Property(argParam, propName);
var valueConst = Expression.Constant(value, typeof(TValue));
switch (exprType)
{
case ExpressionType.GreaterThanOrEqual:
propExpr = Expression.GreaterThanOrEqual(prop, valueConst);
break;
case ExpressionType.LessThanOrEqual:
propExpr = Expression.LessThanOrEqual(prop, valueConst);
break;
case ExpressionType.Equal:
default:// assume equality
propExpr = Expression.Equal(prop, valueConst);
break;
}
return propExpr;
}
NOTE: this code is working correctly.
Problem
Using example from other SO answers I have tried the following;
Expressions
I have tried getting the contains via an Expression;
static Expression<Func<bool>> GetContainsExpression<T>(string propertyName,
string propertyValue)
{
var parameterExp = Expression.Parameter(typeof(T), "type");
var propertyExp = Expression.Property(parameterExp, propertyName);
MethodInfo method = typeof(string).GetMethod("Contains", new[] { typeof(string) });
var someValue = Expression.Constant(propertyValue, typeof(string));
var containsMethodExp = Expression.Call(propertyExp, method, someValue);
return Expression.Lambda<Func<bool>>(containsMethodExp);
}
This has to be converted to a BinaryExpression so it can be added to the expression tree using AndAlso. I've tried to compare the Expression with a true value, but this isn't working
if (searchParams.FirstName.IsNotNullOrWhiteSpace())
{
var propExpr = GetContainsExpression<Locum>(nameof(Locum.Firstname),
searchParams.FirstName);
var binExpr = Expression.MakeBinary(ExpressionType.Equal, propExpr, propExpr);
expr = Expression.AndAlso(expr, binExpr);
}
MethodCallExpression
I also tried returning the MethodCallExpression (instead of the Lambda above), using the following;
static MethodCallExpression GetContainsMethodCallExpression<T>(string propertyName,
string propertyValue)
{
var parameterExp = Expression.Parameter(typeof(T), "type");
var propertyExp = Expression.Property(parameterExp, propertyName);
MethodInfo method = typeof(string).GetMethod("Contains", new[] { typeof(string) });
var someValue = Expression.Constant(propertyValue, typeof(string));
var containsMethodExp = Expression.Call(propertyExp, method, someValue);
return containsMethodExp;
}
I used this as follows;
if (searchParams.FirstName.IsNotNullOrWhiteSpace())
{
var propExpr = GetContainsMethodCallExpression<Person>(nameof(Person.FirstName),
searchParams.FirstName);
var binExpr = Expression.MakeBinary(ExpressionType.Equal, propExpr, alwaysTrue);
expr = Expression.AndAlso(expr, binExpr);
}
Exceptions
These expression are passed to a generic method that pages information out of the database, and the exceptions are thrown during the first execution of the query when I Count the total matching number of record on the constructed query.
System.InvalidOperationException: 'The LINQ expression 'DbSet()
.Where(p => True && p.FirstName.Contains("123") == True)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable ', 'AsAsyncEnumerable ', 'ToList ', or 'ToListAsync '. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.'
This exception is thrown on a Count method I am using in my paging code. This code is already working without any filters, and with the ExpressionType filters described at the top, so I haven't included this code as I don't believe it is relevant.
pagedResult.RowCount = query.Count();

This has to be converted to a BinaryExpression so it can be added to the expression tree using AndAlso
Negative. There is no requirement Expression.AndAlso (or Expression.OrElse) operands to be binary expressions (it would have been strange like requiring left or right operand of && or || to be always comparison operators). The only requirement is them to be bool returning expressions, hence call to string Contains is a perfectly valid operand expression.
So start by changing the type of the inner local variable from BinaryExpression to Expression:
if (searchParams != null)
{
Expression propExpr;
// ...
}
The same btw applies for the initial expression - you don't need true == true, simple
Expression expr = Expression.Constant(true); would do the same.
Now you could emit method call to string.Contains in a separate method similar to the other that you've posted (passing the ParameterExpression and building property selector expression) or inline similar to:
if (searchParams.FirstName.IsNotNullOrWhiteSpace())
{
var propExpr = Expression.Property(argParam, nameof(Person.FirstName));
var valueExpr = Expression.Constant(searchParams.FirstName);
var containsExpr = Expression.Call(
propExpr, nameof(string.Contains), Type.EmptyTypes, valueExpr);
expr = Expression.AndAlso(expr, containsExpr);
}

building OR expression from List of strings [duplicate]

This question already has answers here:
Combining two expressions (Expression<Func<T, bool>>)
(10 answers)
Closed 9 months ago.
In the code below :
Expression<Func<WorkflowTask, bool>> filterBefore = wt => true;
filterBefore = filterBefore.And(wt => wt.code == "XK");
List<string> sourceLanguages = new List<string>() { "FR", "DE", "NL" };
//HOW TO BUILD OR CONDITIONS DYNAMICALLY BASED ON SOURCE LANGUAGES LIST ?
filterBefore = filterBefore.And(wt => wt.SourceLanguages.Contains("FR") || wt.WorkflowTaskContextualInfo.SourceLanguages.Contains("DE"));
I don't know how to build dynamically the OR condition on the SourceLanguages List. That list could contain any number of values (I've hardcoded it here for the sake of example).
wt.WorkflowTaskContextualInfo.SourceLanguages is a string with comma-separated values ("FR, EN" for instance)
The And expression is defined as below :
public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> expr1, Expression<Func<T, bool>> expr2)
{
var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
return Expression.Lambda<Func<T, bool>>(Expression.AndAlso(expr1.Body, invokedExpr), expr1.Parameters);
}

LINQKit's PredicateBuilder is designed specifically to address this kind of need. But if you feel that's too much overhead, you can craft your own Expression tree with a few simple utilities, as I've described in this answer
First, a general-purpose Expression Replacer:
public class ExpressionReplacer : ExpressionVisitor
{
private readonly Func<Expression, Expression> replacer;
public ExpressionReplacer(Func<Expression, Expression> replacer)
{
this.replacer = replacer;
}
public override Expression Visit(Expression node)
{
return base.Visit(replacer(node));
}
}
Next, a simple utility method to replace one parameter's usage with another parameter in a given expression:
public static T ReplaceParameter<T>(T expr, ParameterExpression toReplace, ParameterExpression replacement)
where T : Expression
{
var replacer = new ExpressionReplacer(e => e == toReplace ? replacement : e);
return (T)replacer.Visit(expr);
}
This is necessary because the lambda parameters in two different expressions are actually different parameters, even when they have the same name. For example, if you want to end up with q => q.first.Contains(first) || q.last.Contains(last), then the q in q.last.Contains(last) must be the exact same q that's provided at the beginning of the lambda expression.
Next we need a general-purpose Join method that's capable of joining Func<T, TReturn>-style Lambda Expressions together with a given Binary Expression generator.
public static Expression<Func<T, TReturn>> Join<T, TReturn>(Func<Expression, Expression, BinaryExpression> joiner, IReadOnlyCollection<Expression<Func<T, TReturn>>> expressions)
{
if (!expressions.Any())
{
throw new ArgumentException("No expressions were provided");
}
var firstExpression = expressions.First();
var otherExpressions = expressions.Skip(1);
var firstParameter = firstExpression.Parameters.Single();
var otherExpressionsWithParameterReplaced = otherExpressions.Select(e => ReplaceParameter(e.Body, e.Parameters.Single(), firstParameter));
var bodies = new[] { firstExpression.Body }.Concat(otherExpressionsWithParameterReplaced);
var joinedBodies = bodies.Aggregate(joiner);
return Expression.Lambda<Func<T, TReturn>>(joinedBodies, firstParameter);
}
Now, applying that to your example:
Expression<Func<WorkflowTask, bool>> codeCriteria = wt => wt.code == "XK";
var langCriteria = new List<string>() { "FR", "DE", "NL" }
.Select(lang => (Expression<Func<WorkflowTask, bool>>)(wt => wt.SourceLanguages.Contains(lang)))
.ToList();
var filter = Join(Expression.And, new[] { codeCriteria, Join(Expression.Or, langCriteria)});
filter will now have the equivalent of wt => wt.code == "XK" && (wt.SourceLanguages.Contains("FR") || wt.SourceLanguages.Contains("DE") || wt.SourceLanguages.Contains("NL"))

I would put the required languages in an array or list.
var required = new string[]{ "FR", "DE" };
Then you can query with
wt => required.Any(r => wt.SourceLanguages.Contains(r))
or, the other way round
wt => wt.SourceLanguages.Any(sl => required.Contains(sl))

I did not feel like importing a whole library, and the sample given seemed a bit of a stretch so I think i found an easier solution using BinaryExpression Update(Expression left, LambdaExpression? conversion, Expression right).
The samples below accepts a list of string, and constructs a chain of OR expressions. Each OR expressions calls a entity framework's LIKE method. In the end the whole ordeal gets translated nicely to SQL - so if you're on the job of making dynamic filters like i did - this should help you out.
private static Expression<Func<TMeta, bool>> GetMetaKeyFilterPredicateExpression<TMeta>(List<string> metaKeyNames)
where TMeta : IMetaKeyValuePair
{
var parameter = Expression.Parameter(typeof(TMeta));
var property = Expression.Property(Expression.Convert(parameter, typeof(IMetaKeyValuePair)),
propertyName: nameof(IMetaKeyValuePair.MetaKey));
Expression body = null!;
BinaryExpression? predicateExpression = null;
foreach (var metaKeyName in metaKeyNames)
{
var likeExpression = Expression.Call(typeof(DbFunctionsExtensions),
nameof(DbFunctionsExtensions.Like),
null,
Expression.Constant(EF.Functions),
property,
Expression.Constant(metaKeyName)
);
predicateExpression =
predicateExpression == null
? Expression.Or(likeExpression, Expression.Constant(false))
: predicateExpression.Update(predicateExpression, null, likeExpression);
}
body = (Expression?)predicateExpression ?? Expression.Constant(true);
var expr = Expression.Lambda<Func<TMeta, bool>>(body: body, parameter);
return expr;
}
var expr = GetMetaKeyFilterPredicateExpression<TMeta>(metaKeyNames);
qMetaKeyValuePairs = qMetaKeyValuePairs.Where(expr);

Building a Linq to EF query to a variable list of keywords in a variable number of columns?

I am trying to come up with a utility method to build a Linq Query or Linq Predicate to add to an Linq to EF query to do search for all terms in a list of terms in a variable number of columns.
I am trying to use PredicateBuilder to build the where clause. With one search term and a fixed list of columns it is relatively easy.
The pseudo code that I am trying to work up looks like this so far:
private static Predicate<Project> CreateDynamicSearch(IEnumerable<strings> searchableColumns, string[] searchTerms)
{
var predicate = PredicateBuilder.True<Project>();
foreach (var columnName in searchableColumns)
{
foreach (var term in searchTerms)
{
predicate = predicate.And(a => a.**columnName**.Contains(term));
}
predicate = predicate.Or(predicate);
}
return predicate;
}
The biggest issue I have is handling the expression for the columnName. Previous advice was to use an expression tree but I do not understand how that works into this scenario.
** Update **
I've taken the code as you have it after the update. It builds but when I actually make the call it errors on the Extension.Property(param,columnName); line, with the error Instance property 'Name' is not defined for type 'System.Func`2[Myclass,System.Boolean]' message. The columnName = "Name"
** Update 2 **
The way it's called:
var test = CreateDynamicSearch<Func<Project, bool>>(searchCols, searchTerms);

You can build expression for predicate yourself, in this case it's relatively easy:
private static Expression<Func<T, bool>> CreateDynamicSearch<T>(IEnumerable<string> searchableColumns, string[] searchTerms) {
// start with true, since we combine with AND
// and true AND anything is the same as just anything
var predicate = PredicateBuilder.True<T>();
foreach (var columnName in searchableColumns) {
// start with false, because we combine with OR
// and false OR anything is the same as just anything
var columnFilter = PredicateBuilder.False<T>();
foreach (var term in searchTerms) {
// a =>
var param = Expression.Parameter(typeof(T), "a");
// a => a.ColumnName
var prop = Expression.Property(param, columnName);
// a => a.ColumnName.Contains(term)
var call = Expression.Call(prop, "Contains", new Type[0], Expression.Constant(term));
columnFilter = columnFilter.Or(Expression.Lambda<Func<T, bool>>(call, param));
}
predicate = predicate.And(columnFilter);
}
return predicate;
}
In response to comment
I was just curious if there was some way you could combine the
expression created by Expression.Property(param, columnName) with the
one the compiler generates for (string s) -> s.Contains(term)
You can do that with (for example) like this:
// a =>
var param = Expression.Parameter(typeof(T), "a");
// a => a.ColumnName
var prop = Expression.Property(param, columnName);
// s => s.Contains(term)
Expression<Func<string, bool>> contains = (string s) => s.Contains(term);
// extract body - s.Contains(term)
var containsBody = (MethodCallExpression)contains.Body;
// replace "s" parameter with our property - a.ColumnName.Contains(term)
// Update accepts new target as first parameter (old target in this case is
// "s" parameter and new target is "a.ColumnName")
// and list of arguments (in this case it's "term" - we don't need to update that).
//
var call = containsBody.Update(prop, containsBody.Arguments);
columnFilter = columnFilter.Or(Expression.Lambda<Func<T, bool>>(call, param));

How to Create an Expression tree for .Where(x => x.<deep property>.Select(y => y.id).Intersect(List<int>).Any())

I'm creating a method that receives a Queryable<T> source, a string with a property name/path (could be a deep property for example "TrParent.DataTypes" to achieve this x => x.TrParent.DataTypes) and Enumerable<int> which holds the values I need to intersect.
Basically I come from the need to create the following query dynamically (I mean <DT_Det_Tr> and TrParent.DataTypes being know only at runtime, in the example DT_Det_Tr is not a type it is a class):
var _vals = new List<int>();
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any()
);
Please keep in mind that the preceding query is just an example of what I need to achieve dynamically, what I really need is an expression tree that creates a predicate like the one shown above but using a dynamic type and with the deep navigation property specified within a string.
So, I'm using this function to create the expression for the deep property:
private static LambdaExpression CreateDelegateExpression<T>(out Type resultingtype, string property, string parameterName = "x")
{
var type = typeof(T);
ParameterExpression param = Expression.Parameter(type, parameterName);
Expression expr = param;
foreach (string prop in property.Split('.'))
{
PropertyInfo pi = type.GetProperty(prop);
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
Type delegateType = typeof(Func<,>).MakeGenericType(typeof(T), type);
LambdaExpression lambda = Expression.Lambda(delegateType, expr, param);
resultingtype = type;
return lambda;
}
And here is what I have so far for my function:
public static IQueryable<T> Intersect<T>(this IQueryable<T> source, string property, IEnumerable<int> value)
{
//List of ids
var _value = Expression.Constant(value);
//Get delegate expression to the deep property and it's inner type
Type type = null;
var lambda = CreateDelegateExpression<T>(out type, property, "x");
var enumtype = type.GetGenericArguments()[0];
ParameterExpression tpe = Expression.Parameter(enumtype, "y");
Expression propExp = Expression.Property(tpe, enumtype.GetProperty("Id"));
MethodInfo innermethod = typeof(Queryable).GetMethods().Where(x => x.Name == "Select").First();
//Error on next line...
var selectCall = Expression.Call(typeof(Queryable),
"Select",
new Type[] { enumtype, typeof(long) },
lambda,
propExp);
//TODO: Add rest of logic and actually filter the source
return source;
}
In the var selectCall = line I'm getting error:
No generic method 'Select' on type 'System.Linq.Queryable' is compatible with the supplied type arguments and arguments. No type arguments should be provided if the method is non-generic.
I've read a lot here on SO and other sites but I can't get past this part, I feel I'm going to bump into more trouble when I get to the .Intersect(List<int>).Any() part so any help on that also would be grand, thanks.

After a lot of thought, investigation and attempts I came up with a solution.
First, I made a simpler version of my goal query (the static example I used in my question), so instead of:
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any()
);
I made this:
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Any(y => _vals.Contains(y.Id))
);
Which is a lot easier to translate to expressions (or at least it was for me) because it omits the Select call.
I got rid of the method I was using to create the deep navigation property expression and streamlined it in my Intersect function, this was because it was doing some work I don't really need here plus I needed access to some of the variables I use inside it, then I made this:
public static IQueryable<T> Intersect<T>(this IQueryable<T> source, string property, IEnumerable<int> value)
{
var type = typeof(T);
var _value = Expression.Constant(value); //List of ids
//Declare parameter for outer lambda
ParameterExpression param = Expression.Parameter(type, "x");
//Outer Lambda
Expression expr = param;
foreach (string prop in property.Split('.')) //Dig for deep property
{
PropertyInfo pi = type.GetProperty(prop);
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
//Get deep property's type
var enumtype = type.GetGenericArguments()[0];
//Declare parameter for inner lambda
ParameterExpression tpe = Expression.Parameter(enumtype, "y");
//Inner Collection lambda logic
//Property for inner lambda
Expression propExp = Expression.Property(tpe, enumtype.GetProperty("Id"));
//Contains method call .Contains(y.Id)
var containsMethodExp = Expression.Call(typeof(Enumerable), "Contains", new[] { propExp.Type }, _value, propExp);
//Create Expression<Func<enumtype, bool>>
var innerDelegateType = typeof(Func<,>).MakeGenericType(enumtype, typeof(bool));
//Create Inner lambda y => _vals.Contains(y.Id)
var innerFunction = Expression.Lambda(innerDelegateType, containsMethodExp, tpe);
//Get Any method info
var anyMethod = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(enumtype);
//Call Any with inner function .Any(y => _vals.Contains(y.Id))
var outerFunction = Expression.Call(anyMethod, expr, innerFunction);
//Call Where
MethodCallExpression whereCallExpression = Expression.Call
(
typeof(Queryable),
"Where",
new Type[] { source.ElementType },
source.Expression,
Expression.Lambda<Func<T, bool>>(outerFunction, new ParameterExpression[] { param })
);
//Create and return query
return source.Provider.CreateQuery<T>(whereCallExpression);
}
I hope this helps anyone trying to develop a similar solution.
Working with expression trees can be very hard and frustrating at first, but it's a really powerful tool once you get the hold of it.

If you have access to the dynamic keyword from c# 4.0, you might be able to work around the problem like this:
var _vals = new List<int>();
var res = dbContext.Set<DT_Det_Tr>()
.Where(obj => { dynamic x = obj;
return x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any();
}
);
But I don't know enough about the details of the problem you want to solve to say for sure.

Building Expression Tree Using a Parameter's Indexer

Given a class that has a property that is a Dictionary
public class Product
{
public Dictionary<string, string> Attributes { get { return attributes; } }
private Dictionary<string, string> attributes = new Dictionary<string, string>();
}
I want to be able to match products in a list of products based on criteria that are retrieved from a data store that are in the format of
Brand == Tyco
Color != Blue
My current approach is to construct an expression from the filter, and then pass that expression as the parameter to a LINQ Where method call like so
products = products.Where(myConstructedExpression);
where myConstructedExpression would normally be a lamda expression that looks like
p => p.Attributes[attribute] == value
I have assembled the following code for testing purposes, but it always fails the call to lambda.Compile() regardless of what I have tried for he left expression.
Dictionary<string, ExpressionType> expressionType = new Dictionary<string, ExpressionType>();
expressionType.Add("==", ExpressionType.Equal);
expressionType.Add("!=", ExpressionType.NotEqual);
string filter = "Brand == Tyco";
string[] fields = filter.Split(' ');
string attribute = fields[0];
string op = fields[1];
string value = fields[2];
Product product = new Product();
product.Attributes.Add("Brand", "Tyco");
var parameter = Expression.Parameter(typeof(Product), "p");
var left = /***** THIS IS WHAT I AM FAILING TO CONSTRUCT PROPERLY ********/
var right = Expression.Constant(value);
var operation = Expression.MakeBinary(expressionType[op], left, right);
var lambda = Expression.Lambda<Func<Product, bool>>(operation, parameter);
var result = lambda.Compile()(product);
Questions
Is this even a reasonable approach, and, if so,
How do I construct the left expression?

So to get p => p.Attributes["Brand"] <someoperator> "Tyco", you can do this.
The "trick", to work with indexed types, is to use their Item property (you could also work with the get_item method)
var parameter = Expression.Parameter(typeof(Product), "p");
Expression left = Expression.Property(parameter, "Attributes");
left = Expression.Property(left, "Item", new Expression[] { Expression.Constant(attribute) });
EDIT
the version with the IDictionary.ContainsKey(<value>) test
really step by step, but I think this makes things clearer at first.
//left part of lambda, p
var parameter = Expression.Parameter(typeof(Product), "p");
//right part
//p.Attributes
Expression left = Expression.Property(parameter, "Attributes");
var method = typeof(IDictionary<string, string>).GetMethod("ContainsKey");
//p.Attributes.ContainsKey("Brand");
Expression containsExpression = Expression.Call(left, method, Expression.Constant(attribute));
//p.Attributes.Item["Brand"]
Expression keyExpression= Expression.Property(left, "Item", new Expression[] { Expression.Constant(attribute) });
//"Tyco"
var right = Expression.Constant(value);
//{p => IIF(p.Attributes.ContainsKey("Brand"), (p.Attributes.Item["Brand"] == "Tyco"), False)}
Expression operation = Expression.Condition(
containsExpression,
Expression.MakeBinary(expressionType[op], keyExpression, right),
Expression.Constant(false));
var lambda = Expression.Lambda<Func<Product, bool>>(operation, parameter);