We Keep Coding

EF Core 5 - In memory object list filtering not working against DBset

I need to filter against a DBset using a list of objects passed in as a variable to my repo.
The following worked in EF6
List<MyFilterObj> incomingList = [{A: 1, B: 2}, {A: 3, B: 4}]
return await context.MyDBset
.Where(x => incomingList.Any(v =>
v.A == x.A &&
v.B == x.B)).ToListAsync();
If I try this same approach in EF Core 5 I get the following error:
The LINQ expression (MyDBset) could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I can't seperate the incoming list into seperate A and B lists as the combination is essential to the filter accuracy. Otherwise I could have used this approach
return await context.MyDBset
.Where(x => incomingListA.Contains(x.A) &&
incomingListB.Contains(x.B)).ToListAsync();
This approach does work in EF Core 5
Basically I want EF to translate my query into something like:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) OR (m.A = 3 AND m.B = 4)
If I loop through my incomingList adding "where" filters to a query the resultant SQL uses "AND" instead of "OR" like so:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) AND (m.A = 3 AND m.B = 4)
Right now my work around involves binging the result set into memory with some basic filters then running an in memory filter using the original syntax. This is obviously not ideal.
Any suggestions? Thanks

Since EF Core is not willing to translate such expression (it's a well known limitation that Contains with primitive value is the only supported operation on in-memory collection in L2E query since it directly translates to SQL IN operator), you have to do it yourself. Any is basically equivalent of Or as you mentioned, so for not so big list and top level query it could easily be done by using some predicate builder implementation to dynamically build || predicate. Or directly using the the Expression class methods as in this custom extension method from my answer to How to simplify repetitive OR condition in Where(e => e.prop1.contains() || e.prop2.contains() || ...):
public static partial class QueryableExtensions
{
public static IQueryable<T> WhereAnyMatch<T, V>(this IQueryable<T> source, IEnumerable<V> values, Expression<Func<T, V, bool>> match)
{
var parameter = match.Parameters[0];
var body = values
// the easiest way to let EF Core use parameter in the SQL query rather than literal value
.Select(value => ((Expression<Func<V>>)(() => value)).Body)
.Select(value => Expression.Invoke(match, parameter, value))
.Aggregate<Expression>(Expression.OrElse);
var predicate = Expression.Lambda<Func<T, bool>>(body, parameter);
return source.Where(predicate);
}
}
with sample usage for your scenario:
context.MyDBset
.WhereAnyMatch(incomingList, (x, v) => v.A == x.A && v.B == x.B)
Normally instead of Expression.Invoke helper utilities like this would use custom ExpressionVisitor to replace the parameters of the source expression (thus simulation a "call"), which produces more naturally looking query expression, but for EF Core it doesn't really matter since it recognizes and correctly translates invocation expressions inside the expression tree.
But just in case Expression.Invoke doesn't work, here is the parameter replacing version - it should work with all query providers. First, the helpers
public static partial class ExpressionBuilder
{
public static Expression ReplaceParameter(this Expression source, ParameterExpression parameter, Expression value)
=> new ParameterReplacer { Parameter = parameter, Value = value }.Visit(source);
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Parameter;
public Expression Value;
protected override Expression VisitParameter(ParameterExpression node)
=> node == Parameter ? Value : node;
}
}
and then simply replace the invocation
.Select(value => Expression.Invoke(match, parameter, value))
with
.Select(value => match.Body.ReplaceParameter(match.Parameters[1], value))
because specifically here we are reusing the first parameter (var parameter = match.Parameters[0];), otherwise it should be replaced as well.

Dynamically construct where clause with Func<T, string> lambda - linq to entities

(this is for .Net Framework 4.7)
I'm trying to write up some extension methods to aid in creating dynamic where clauses for various entities. I started a few days ago, so there's likely a lot that I don't know and some that I probably misunderstood.
I managed to create one extension method already for filtering by 1 property which works as I expect it to (I did use reflection to get the property, couldn't get it working with an interface - well, without it executing the sql that is).
I can't seem to be able to get this one working for a lambda expression though.
Note, that the solution must not trigger sql execution. Because I was able to write up some variants that "worK', but they will trigger sql execution.
The way I work with this is that once I have the code ready, I start debugging and have the "query" in the watch. And it looks like this (notice the sql code)
Once I step over my FilterString method call, it either turns into a sql result, or I get an exception (with current code), which it shouldn't:
So here's my current code that throws the exception (currently not dealing with the "match" parameter, I am implementing an "equals" call. There will be others like, starts With, like, etc)
The exception is just one of those "type mismatch" having function cannot be passed as param to string Equals or what not.
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
var selector = Expression.Lambda<Func<T, string>>(getItemString, param);
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
prototype.Body.ReplaceParameter(prototype.Parameters[0], selector.Body),
selector.Parameters[0]);
return query.Where(predicate);
}
and the one that executes the sql instead of just generating it
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
//var value = Expression.Constant(getItemString);
var equals = typeof(string).GetMethod("Equals", new Type[] { typeof(string) });
var item = Expression.Invoke(getItemString, param);
var body = Expression.Call(Expression.Constant(criteriaItem),
equals,
item);
return query.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
calling these is done like so
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.MaybeSomeOtherProp.SomeString);
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.Name);
This same extension method will be called on any number of different entities, with nay number of different properties and prop names. I guess I could make use of the reflection version I got working and passing in all the property names in some array of some sort, but that is just plain ugly.
So long story short, how can I get this working in the way I explained above, taht is: having the sql generated instead of executed?
Thank you,
Note, the "ReplaceParameter" extension method is the one from here: https://stackoverflow.com/a/39206392/630515

So, you're trying to merge your prototype item => item == criteriaItem. With a passed in string property expression, like (r) => r.SomeProperty.Name to create (r) => r.SomeProperty.Name == criteriaItem.
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
ReplacingExpressionVisitor.Replace(
prototype.Parameters[0],
getItemString.Body,
prototype.Body),
getItemString.Parameters[0]);
And I think you're trying to do it this way so that criteriaItem is bound to an sql parameter, rather than being inlined as a string constant. But your question was a little hard to follow.

Move a lambda expression to a static extension method in Entity Framework

I am trying to refactor the following line of an Entity Framework query into a generalized static extension method:
dbContext.Employees
.Where(e => permissionResolver.AuthorizedUsers.Select(p => p.Id).Contains(e.Id))
.OrderBy(...)
PermissionResolver is just an instance I receive a list of IDs from to compare against a user ID stored in the current record. It compiles perfectly to a SQL statement WHERE Id IN (....).
Now I am trying to create an extension method for IQueryable<T> that I can use for any type of record, I just want to pass in a property where the owner's ID is stored in.
So that is what I came up with:
public static IQueryable<T> AuthorizedRecords<T>(this IQueryable<T> query, Expression<Func<T, Int32>> property, IPermissionResolver permissionResolver)
{
Expression<Func<T, Boolean>> idIsAuthorized = entity => permissionResolver.AuthorizedUsers.Select(e => e.Id).ToList().Contains(property.Compile()(entity));
return query.Where(idIsAuthorized);
}
I'm getting a runtime error that this expression cannot be translated into SQL.
How can I combine the property expression to the main query expression that it can be translated into SQL correctly? Is there a better way to rewrite the query expression?

property.Compile() converts the expression tree into a delegate, this delegate cannot be properly translated back to an expression tree/SQL.
You need to construct expression tree like that:
var ids = permissionResolver.AuthorizedUsers.Select(e => e.Id).ToList().AsEnumerable();
// method Enumerable.Contains<int>()
var methodContains = typeof(System.Linq.Enumerable).GetMethods()
.Where(m => m.Name == "Contains" && m.GetParameters().Length == 2)
.First()
.MakeGenericMethod(typeof(int));
var lambdaParam = property.Parameters.Single();
var lambda = Expression.Lambda(
Expression.Call(
methodContains,
Expression.Constant(ids),
property.Body),
lambdaParam
);
var predicate = (Expression<Func<T, bool>>)lambda;
return query.Where(predicate);

The short answer is you cannot use custom extension methods in entity framework queries. Under the hood entity framework parses expression tree Expression<Func<>> into sql query. It seems kind of impossible to translate any possible extension method to sql so they support limited set of Linq methods to properly translate it. Some useful information about expression trees: https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/expression-trees.
Some refactoring you can do to simplify your query is to use Any method like this:
dbContext.Employees
.Where(e => permissionResolver.AuthorizedUsers.Any(u => u.Id == e.Id))
.OrderBy(...

LINQ to Entities: queryable extension method not reconized inside where condition

i am using asp.net 4.5 mvc 5 with code first entity framework and have the following issue:
i have two models, "PostBody" and "PostHeader". 1 PostHeader has 1-n PostBodies.
A PostBody can be "Deleted" (Property as flag).
My extension method should give me every PostBody from an IQueryable-Object which is not deleted as an IQueryable:
public static IQueryable<TSource> GetActiveVersions<TSource>(this IQueryable<TSource> source)
where TSource : PostBody
{
return source.Where(x => x.Deleted == false);
}
this is working fine when i do somethin like this
var x = db.Bodies.GetActiveVersions().ToList();
or this
var y = db.Headers.FirstOrDefault().Bodies.AsQueryable().GetActiveVersions().ToList();
etc. - But as soon as i use my extension method as part of an expression paramater within e.g. the where method, i'm running into a NotSupportedException:
var z = db.Headers.Where(h => h.Bodies.AsQueryable().GetActiveVersions().Count() > 0).ToList();
System.NotSupportedException: LINQ to Entities does not recognize the method 'System.Linq.IQueryable`1[WebApplication5.Models.PostBody] GetActiveVersions[PostBody](System.Linq.IQueryable`1[WebApplication5.Models.PostBody])' method, and this method cannot be translated into a store expression.
what am i doing wrong? or - how can i use my extension method within the where condition?

Maybe the parser cannot resolve the extension method in the linq expression. You could try to centralize your condition and use it as a expression tree, for sample:
public static Expression<Func<TSource, bool>> GetActiveVersions()
where TSource : PostBody
{
return x => x.Deleted == false;
}
And apply this expressin in your subquery (with linq) for sample:
var z = db.Headers.Where(h => h.Bodies.AsQueryable().Any(GetActiveVersions())).ToList();
Instead using the Count() > 0, prefer using .Any() to avoid to access all records of the table.

Linq, Expressions, NHibernate and Like comparison

I am trying to do a like comparison based on an outside parameter (passed by a search form) that determines type of comparison ("%string" or "string%" or "%string%")
I was thinking in the following direction:
query = query.Where(
Entity.StringProperty.Like("SearchString", SelectedComparsionType)
)
Like method would than based on selected type return
.StartsWith() or .EndsWith() or .SubString()
My knowledge of expressions is apparently far from great, since i haven't been able to construct a method that could yield the right result (server side comparison in SQL just like with StartsWith method).

The easy way
Just use
if (comparison == ComparisonType.StartsWith)
query = query.Where(e => e.StringProperty.StartsWith("SearchString"));
else if ...
The hard way
If you want to do something like this, either make sure your LINQ provider can be told of this new method somehow, and how it translates to SQL (unlikely), or prevent your method from ever reaching the LINQ provider, and provide the provider something it understands (hard). For example, instead of
query.Where(e => CompMethod(e.StringProperty, "SearchString", comparsionType))
you can create something like
var query = source.WhereLike(e => e.StringProperty, "SearchString", comparsionType)
with the following code
public enum ComparisonType { StartsWith, EndsWith, Contains }
public static class QueryableExtensions
{
public static IQueryable<T> WhereLike<T>(
this IQueryable<T> source,
Expression<Func<T, string>> field,
string value,
SelectedComparisonType comparisonType)
{
ParameterExpression p = field.Parameters[0];
return source.Where(
Expression.Lambda<Func<T, bool>>(
Expression.Call(
field.Body,
comparisonType.ToString(),
null,
Expression.Constant(value)),
p));
}
}
You can even add additional criteria this way
var query = from e in source.WhereLike(
e => e.StringProperty, "SearchString", comparsionType)
where e.OtherProperty == 123
orderby e.StringProperty
select e;
The very, very hard way
It would (technically) be possible to rewrite the expression tree before the provider sees it, so you can use the query you had in mind in the first place, but you'd have to
create a Where(this IQueryable<EntityType> source, Expression<Func<EntityType, bool>> predicate) to intercept the Queryable.Where,
rewrite the expression tree, replacing your CompMethod, wherever it is, with one of the String methods,
call the original Queryable.Where with the rewritten expression,
and first of all, be able to follow the extension method above in the first place!
But that's probably way too complicated for what you had in mind.

Sounds like you should be wanting to use:
query = query.Where(
Entity.StringProperty.Contains("SearchString")
)
This should map to:
WHERE StringProperty LIKE '%SearchString%'
This should also work for more advanced search masks such as "Mr? Sm%th", but I haven't had to test any search strings like that myself yet.
UPDATE: Based on OPs edit
It sounds like what you are asking for is something like the following:
public enum SelectedComparsionType
{
StartsWith,
EndsWith,
Contains
}
public static bool Like(this string searchString, string searchPattern, SelectedComparsionType searchType)
{
switch (searchType)
{
case SelectedComparsionType.StartsWith:
return searchString.StartsWith(searchPattern);
case SelectedComparsionType.EndsWith:
return searchString.EndsWith(searchPattern);
case SelectedComparsionType.Contains:
default:
return searchString.Contains(searchPattern);
}
}
This would allow you to write code as you require, i.e:
query = query.Where(
Entity.StringProperty.Like("SearchString", SelectedComparsionType.StartsWith)
)
However, personally, I would replace any use of SelectedComparsionType, with a direct call to the required string function. I.e
query = query.Where(
Entity.StringProperty.StartsWith("SearchString")
)
As this will still map to a SQL 'LIKE' query.

This is exactly what I had in mind, thank you. I had something similar already written, but it didn't translate to SQL. For example, it worked if I did this directly:
Entity.StringProperty.EndsWith("SearchString");
It didn't work if I used a dedicated method:
CompMethod("BaseString","SearchString",SelectedComparsionType.EndsWith)
I think it probably has something to do with expression evaluation, i'm just not sure what.

You will be better off using Regex to solve this problem.