With the recent release of Entity Framework Core 3.0, LINQ queries are no longer evaluated on the client by default. I'm a big fan of this change, as it revealed some potentially dangerous client-side evaluation in my project that I thought was translated to SQL; however, it also made some of the helper methods that I was using to avoid crazy chains of ternaries unusable.
Has anyone manged to nest LINQ expressions for use with Entity Framework Core 3.0? Here's an example of what I'm hoping to achieve:
[Fact]
public async Task Can_use_custom_expression()
{
var dbContext = new ApplicationDbContext(new DbContextOptionsBuilder<ApplicationDbContext>().UseInMemoryDatabase("Test").Options);
dbContext.Users.Add(new ApplicationUser { FirstName = "Foo", LastName = "Bar" });
dbContext.SaveChanges();
string query = "Foo";
Expression<Func<string, string, bool>> valueCheck = (value, expected) => !string.IsNullOrEmpty(value) && value.Contains(expected);
var valueCheckFunc = valueCheck.Compile();
Expression<Func<ApplicationUser, bool>> whereExpression = (u) => valueCheckFunc(u.FirstName, query);
var user = await dbContext.Users
.Where(whereExpression)
.FirstOrDefaultAsync();
Assert.NotNull(user);
}
When I run this example, I get the following exception:
Message:
System.InvalidOperationException : The LINQ expression 'Where<ApplicationUser>(
source: DbSet<ApplicationUser>,
predicate: (a) => Invoke(__valueCheckFunc_0, a.FirstName, __query_1)
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
Like I said, I do not want to evaluate this expression client-side, but I would like to avoid having to chain a dozen or so !string.IsNullOrEmpty(x) && x.Contains(y)'s in a single expression. I'd love some tips on how to achieve this.
If you want your expressions to be translatable to Sql by EF you need to avoid calling delegates or methods (with some exceptions, of course). But what you want to achieve is doable by replacing the delegate invocation with its defining expression. For that you need a specialized ExpressionVisitor.
The following visitor will traverse expressions replacing delegate references within its wrapping invocations by a lambda expression body:
public class DelegateByLambda: ExpressionVisitor
{
LambdaExpression delegateReferenceExpression;
LambdaExpression lambdaExpression;
Stack<InvocationExpression> invocations;
public DelegateByLambda(LambdaExpression delegateReferenceExpression, LambdaExpression lambdaExpression)
{
this.delegateReferenceExpression = delegateReferenceExpression;
this.lambdaExpression = lambdaExpression;
this.invocations = new Stack<InvocationExpression>();
}
protected override Expression VisitParameter(ParameterExpression node)
{
var paramIndex = lambdaExpression.Parameters.IndexOf(node);
if (paramIndex >= 0)
{
InvocationExpression call = invocations.Peek();
return base.Visit(call.Arguments[paramIndex]);
}
return base.VisitParameter(node);
}
protected override Expression VisitInvocation(InvocationExpression node)
{
if (node.Expression.ToString() == delegateReferenceExpression.Body.ToString())
{
invocations.Push(node);
var result = base.Visit(lambdaExpression.Body);
invocations.Pop();
return result;
}
return base.VisitInvocation(node);
}
}
This class has no protection against attempting to replace delegate invocations by lambdas with mismatching arguments (number and types) however, the following extension method will do the trick:
public static class DelegateByLambdaExtensions
{
public static Expression<T> Replace<T, X>(this Expression<T> source, Expression<Func<X>> delegateReference, Expression<X> lambdaReference)
{
return new DelegateByLambda(delegateReference, lambdaReference).Visit(source) as Expression<T>;
}
}
So, all you need to do in your code is to call the replace extension method on the expression you want to translate passing an expression returning the delegate and desired lambda expression for expansion. Your sample should look like this:
Expression<Func<string, string, bool>> valueCheck = (value, expected) => !string.IsNullOrEmpty(value) && value.Contains(expected);
var valueCheckFunc = valueCheck.Compile();
Expression<Func<ApplicationUser, bool>> whereExpression = (u) => valueCheckFunc(u.FirstName, query);
whereExpression = whereExpression.Replace(() => valueCheckFunc, valueCheck);
var user = dbContext.Users
.Where(whereExpression)
.FirstOrDefault();
Console.WriteLine(user != null ? $"Found {user.FirstName} {user.LastName}!" : "User not found!");
A working sample can be found here. https://dotnetfiddle.net/Lun3LA
Related
I know that if (includeFilter) query = query.Where (...) gives the same result as query = query.Where (n => !includeFilter || <expression>).
But what if I have <expression1> and <expression2> and I want query.Where (n => <expression1> || <expression2>)? Except that there are includeFilter1 and includeFilter2 too.
Simply query.Where (n => (!includeFilter1 || <expression1>) || (!includeFilter2 || <expression2>)) won't work, because in case includeFilter1 is false, <expression2> will have no effect.
Update:
I just got to this solution:
```query.Where(n => !includeFilter1 || ).Union(query.Where(n => !includeFilter2 || ))```
I'm going to use it with IQueryable; is it OK? Is there a better solution?
Update 2:
I was wrong. What I needed was this:
var query1 = query.Where(n => !includeFilter1 || <expression1>)
var query2 = query.Where(n => !includeFilter2 || <expression2>)
if (includeFilter1 && includeFilter2)
query = query1.Union(query2);
else if (includeFilter1)
query = query1;
else if (includeFilter2)
query = query2;
I couldn't come up with a more elegant way.
In general .Union() is best reserved for cases where you can't represent the logic of the filter in a single query, or when the data is being drawn from different sources in each part of the query. Each .Union() call most likely results in a separate query execution, which means the query optimizer has little to no chance to produce a reasonable execution plan.
Using .Where(n => !includeFilter || <expression>) is also less that optimal on the face of it. The expression tree will close over the includeFilter variable and may not be optimized particularly well depending on the ORM you're using. Best to avoid that where possible.
There are several options for this kind of query composition. The simplest, which works fine for a couple of entries, is this:
if (includeFilter1)
{
if (includeFilter2)
query = query.Where(q => <expression1> || <expression2>);
else
query = query.Where(q => <expression1>);
}
else if (includeFilter2)
query = query.Where(q => <expression2>);
Not very elegant perhaps, but it works just fine and only takes a few lines of code to implement... if <expression1> and <expression2> are known at compile time and aren't parameters to the method.
What I think you're looking for is a more generic way to handle filter expression composition. LINQ has some rather interesting expression handling and while expressions can be scary at first, once you get the hang of them you can do some quite cool things.
For instance, here's an extension (and a supporting class) that will take two predicates on the same type and join them with an Or:
using System.Linq.Expressions
public static class ExpressionExtensions
{
public static Expression<Func<T, bool>> Or<T>(this Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
// Useful for chained or conditional composition:
if (left is null)
return right;
if (right is null)
return left;
// Just in case:
if (ReferenceEquals(left, right))
return left;
// change right parameter references to match left
var rightBody = ReplaceVisitor.Replace(right.Body, right.Parameters[0], left.Parameters[0]);
// build 'or' expression
var body = Expression.OrElse(left.Body, rightBody);
// Create lambda (function) for predicate
var result = Expression.Lambda<Func<T, bool>>(body, left.Parameters[0]);
return result;
}
}
// Helper class, replaces expression instances
// Used to get the right parameters in composed expressions.
internal class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression _from;
private readonly Expression _to;
private ReplaceVisitor(Expression from, Expression to)
{
_from = from;
_to = to;
}
public override Expression Visit(Expression e)
=> ReferenceEquals(e, _from) ? _to : base.Visit(e);
public static T Replace<T>(T expression, Expression from, Expression to)
where T : Expression
{
var visitor = new ReplaceVisitor(from, to);
return (T)visitor.Visit(expression);
}
}
(You can create a similar .And() extension using the AndAlso expression.)
Using that you can directly merge your filter expresions with an Or like so:
public IQueryable<T> ApplyFilters<T>(IQueryable<T> query,
bool useFilter1, Expression<Func<T, bool>> predicate1,
bool useFilter2, Expression<Func<T, bool>> predicate2
)
{
Expression<Func<T, bool>> predicate = null;
if (useFilter1)
predicate = predicate.Or(predicate1);
if (useFilter2)
predicate = predicate.Or(predicate2);
return predicate is null ? query : query.Where(predicate);
}
Or perhaps this:
public IQueryable<T> AnyOf<T>(IQueryable<T> query, params Expression<Func<T, bool>>[] filters)
{
Expression<Func<T, bool>> predicate = null;
foreach (var filter in filters)
predicate = predicate.Or(filter);
return predicate is null ? query : query.Where(predicate);
}
Try it out, see if it fits your use case.
I think you need
query.Where(n => (!includeFilter1 || <expression1>) &&
(!includeFilter2 || <expression2>))
I need to filter against a DBset using a list of objects passed in as a variable to my repo.
The following worked in EF6
List<MyFilterObj> incomingList = [{A: 1, B: 2}, {A: 3, B: 4}]
return await context.MyDBset
.Where(x => incomingList.Any(v =>
v.A == x.A &&
v.B == x.B)).ToListAsync();
If I try this same approach in EF Core 5 I get the following error:
The LINQ expression (MyDBset) could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I can't seperate the incoming list into seperate A and B lists as the combination is essential to the filter accuracy. Otherwise I could have used this approach
return await context.MyDBset
.Where(x => incomingListA.Contains(x.A) &&
incomingListB.Contains(x.B)).ToListAsync();
This approach does work in EF Core 5
Basically I want EF to translate my query into something like:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) OR (m.A = 3 AND m.B = 4)
If I loop through my incomingList adding "where" filters to a query the resultant SQL uses "AND" instead of "OR" like so:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) AND (m.A = 3 AND m.B = 4)
Right now my work around involves binging the result set into memory with some basic filters then running an in memory filter using the original syntax. This is obviously not ideal.
Any suggestions? Thanks
Since EF Core is not willing to translate such expression (it's a well known limitation that Contains with primitive value is the only supported operation on in-memory collection in L2E query since it directly translates to SQL IN operator), you have to do it yourself. Any is basically equivalent of Or as you mentioned, so for not so big list and top level query it could easily be done by using some predicate builder implementation to dynamically build || predicate. Or directly using the the Expression class methods as in this custom extension method from my answer to How to simplify repetitive OR condition in Where(e => e.prop1.contains() || e.prop2.contains() || ...):
public static partial class QueryableExtensions
{
public static IQueryable<T> WhereAnyMatch<T, V>(this IQueryable<T> source, IEnumerable<V> values, Expression<Func<T, V, bool>> match)
{
var parameter = match.Parameters[0];
var body = values
// the easiest way to let EF Core use parameter in the SQL query rather than literal value
.Select(value => ((Expression<Func<V>>)(() => value)).Body)
.Select(value => Expression.Invoke(match, parameter, value))
.Aggregate<Expression>(Expression.OrElse);
var predicate = Expression.Lambda<Func<T, bool>>(body, parameter);
return source.Where(predicate);
}
}
with sample usage for your scenario:
context.MyDBset
.WhereAnyMatch(incomingList, (x, v) => v.A == x.A && v.B == x.B)
Normally instead of Expression.Invoke helper utilities like this would use custom ExpressionVisitor to replace the parameters of the source expression (thus simulation a "call"), which produces more naturally looking query expression, but for EF Core it doesn't really matter since it recognizes and correctly translates invocation expressions inside the expression tree.
But just in case Expression.Invoke doesn't work, here is the parameter replacing version - it should work with all query providers. First, the helpers
public static partial class ExpressionBuilder
{
public static Expression ReplaceParameter(this Expression source, ParameterExpression parameter, Expression value)
=> new ParameterReplacer { Parameter = parameter, Value = value }.Visit(source);
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Parameter;
public Expression Value;
protected override Expression VisitParameter(ParameterExpression node)
=> node == Parameter ? Value : node;
}
}
and then simply replace the invocation
.Select(value => Expression.Invoke(match, parameter, value))
with
.Select(value => match.Body.ReplaceParameter(match.Parameters[1], value))
because specifically here we are reusing the first parameter (var parameter = match.Parameters[0];), otherwise it should be replaced as well.
(this is for .Net Framework 4.7)
I'm trying to write up some extension methods to aid in creating dynamic where clauses for various entities. I started a few days ago, so there's likely a lot that I don't know and some that I probably misunderstood.
I managed to create one extension method already for filtering by 1 property which works as I expect it to (I did use reflection to get the property, couldn't get it working with an interface - well, without it executing the sql that is).
I can't seem to be able to get this one working for a lambda expression though.
Note, that the solution must not trigger sql execution. Because I was able to write up some variants that "worK', but they will trigger sql execution.
The way I work with this is that once I have the code ready, I start debugging and have the "query" in the watch. And it looks like this (notice the sql code)
Once I step over my FilterString method call, it either turns into a sql result, or I get an exception (with current code), which it shouldn't:
So here's my current code that throws the exception (currently not dealing with the "match" parameter, I am implementing an "equals" call. There will be others like, starts With, like, etc)
The exception is just one of those "type mismatch" having function cannot be passed as param to string Equals or what not.
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
var selector = Expression.Lambda<Func<T, string>>(getItemString, param);
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
prototype.Body.ReplaceParameter(prototype.Parameters[0], selector.Body),
selector.Parameters[0]);
return query.Where(predicate);
}
and the one that executes the sql instead of just generating it
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
//var value = Expression.Constant(getItemString);
var equals = typeof(string).GetMethod("Equals", new Type[] { typeof(string) });
var item = Expression.Invoke(getItemString, param);
var body = Expression.Call(Expression.Constant(criteriaItem),
equals,
item);
return query.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
calling these is done like so
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.MaybeSomeOtherProp.SomeString);
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.Name);
This same extension method will be called on any number of different entities, with nay number of different properties and prop names. I guess I could make use of the reflection version I got working and passing in all the property names in some array of some sort, but that is just plain ugly.
So long story short, how can I get this working in the way I explained above, taht is: having the sql generated instead of executed?
Thank you,
Note, the "ReplaceParameter" extension method is the one from here: https://stackoverflow.com/a/39206392/630515
So, you're trying to merge your prototype item => item == criteriaItem. With a passed in string property expression, like (r) => r.SomeProperty.Name to create (r) => r.SomeProperty.Name == criteriaItem.
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
ReplacingExpressionVisitor.Replace(
prototype.Parameters[0],
getItemString.Body,
prototype.Body),
getItemString.Parameters[0]);
And I think you're trying to do it this way so that criteriaItem is bound to an sql parameter, rather than being inlined as a string constant. But your question was a little hard to follow.
I've inherited some code from a former employee that queries a mongo DB using compiled Linq expressions and the MongoRepository library (which sits on top of the MongoDB C# driver).
These were taking a long time to run - usually around 6 minutes(!) and causing problems with controller methods that used them.
So I've simplified the code and removed the call to .Compile() the lambda expression and this has seemed to solve the issue (takes <10s to run now).
My questions is: why does compiling this expression cause problems when querying mongo?
This was the gist of the original code (hacked out, so out of context sorry):
public class BaseMongoRepository<T> : MongoRepository<T, Guid> where T : IEntity<Guid> {
protected BaseMongoRepository(string connectionString) : base(connectionString) { }
protected bool IsSatisfiedBy(T entity) {
Expression<Func<T, bool>> func = x => x != null && x.ToString() == "foo"; // query was passed in, but you get the idea
var predicate = func.Compile(); // THIS LINE??
return predicate(entity);
}
public IEnumerable<T> Find() {
return base.collection.AsQueryable().Where(IsSatisfiedBy);
}
}
And I simplified it to something that just uses a regular predicate Func:
public IEnumerable<T> Find() {
return base.collection.AsQueryable().Where(x => x != null && x.ToString() == "foo");
}
Any thoughts most appreciated!
The expression can be converted by a provider to a real sql query, but a delegate can not be interpreted.
The predicate variable in this code:
Expression<Func<T, bool>> func = x => x != null && x.ToString() == "foo";
var predicate = func.Compile();
is essentially the same as:
Func<T, bool> predicate = x => x != null && x.ToString() == "foo";
When you use such a delegate, all data from the database is transferred into memory and after that the predicate is applied.
Pseudo code example:
// Using the delegate:
var data = dbContext.Users.Where(usr => IsSatisfiedBy(usr)).ToList();
// This will result in the following steps:
var userList = ExecuteQuery("SELECT * FROM Users"); // all users are fetched.
var satisfied = userList.Where(usr => IsSatisfiedBy(usr))
// Using an expression:
var data = dbContext.Users.Where(usr => usr.Name == "foo");
// This will result in the following step:
var satisfied = ExecuteQuery("SELECT * FROM Users WHERE Name = 'foo'"); // Filtered before returned to caller.
The reason for the performance problem is to record all the records of the related object and then filter it out. First you need to create the query and register from mongo db.
review for soruce code
https://github.com/fsefacan/MongoDbRepository
This question already has answers here:
Chaining / building LINQ query with an OR instead of AND
(2 answers)
Closed 6 years ago.
Appologies if this is a simple question; suppose I have an EF Query object with 2 methods:
public static IQueryable<Animal> FourLegged(this IQueryable<Animal> query)
{
return query.Where(r => r.NumberOfLegs == 4);
}
public static IQueryable<Animal> WithoutTail(this IQueryable<Animal> query)
{
return query.Where(r => !r.HasTail);
}
Now, in my service layer, to get animals that are four-legged AND without a tail, I can do:
_animalService.Query()
.FourLegged()
.WithoutTail();
That will result in an sql query like so:
select * from Animal where NumberOfLegs = 4 AND HasTail = 0
How do I use the 2 query methods with an OR instead? I want animals that are either 4 legged OR without a tail
select * from Animal where NumberOfLegs = 4 OR HasTail = 0
In Nhibernate I would have used a simple disjunction, but I can't find that in EF.
Thanks
Solution: I ended up using LinqKit predicates mentioned on this answer. It works quite well and I can reuse predicates too.
You can’t really do this when you already called query.Where(). The predicates there are already collected in the IQueryable and they are all combined by AND.
In order to get an OR you will have to make a single query.Where() call and pass a single expression that covers your various disjunctive predicates.
In your case, the combined predicate would look like this:
query.Where(r => (r.NumberOfLegs == 4) || (!r.HasTail))
To make that more dynamic, you essentially need to build a custom expression composition function that works like this:
Expression<Func<Animal, bool>> fourLegged = r => r.NumberOfLegs == 4;
Expression<Func<Animal, bool>> withoutTail = r => !r.HasTail;
query = query.Where(CombineDisjunctivePredicates(fourLegged, withoutTail));
So let’s write that CombineDisjunctivePredicates function:
public Expression<Func<T, bool>> CombineDisjunctivePredicates<T>(params Expression<Func<T, bool>>[] predicates)
{
Expression current = Expression.Constant(false);
ParameterExpression param = Expression.Parameter(typeof(T), "obj");
foreach (var predicate in predicates)
{
var visitor = new ReplaceExpressionVisitor(predicate.Parameters[0], param);
current = Expression.Or(current, visitor.Visit(predicate.Body));
}
return Expression.Lambda<Func<T, bool>>(current, param);
}
This basically takes a number of predicates and combines them by combining the expression bodies using the boolean OR. Since we are combining different expressions which may have different expression parameters, we also need to make sure to replace all expression parameter references in the expression bodies using a common parameter. We do this using a simple ReplaceExpressionVisitor, easily implemented like this:
public class ReplaceExpressionVisitor : ExpressionVisitor
{
private readonly Expression _original;
private readonly Expression _replacement;
public ReplaceExpressionVisitor(Expression original, Expression replacement)
{
_original = original;
_replacement = replacement;
}
public override Expression Visit(Expression node)
{
return node == _original ? _replacement : base.Visit(node);
}
}
And that’s all you need to combine the predicates. You just need to make sure to change your methods now so they don’t call query.Where themselves but return a Expression<Func<Animal, bool>> instead.