We Keep Coding

EF Core 5 - In memory object list filtering not working against DBset

I need to filter against a DBset using a list of objects passed in as a variable to my repo.
The following worked in EF6
List<MyFilterObj> incomingList = [{A: 1, B: 2}, {A: 3, B: 4}]
return await context.MyDBset
.Where(x => incomingList.Any(v =>
v.A == x.A &&
v.B == x.B)).ToListAsync();
If I try this same approach in EF Core 5 I get the following error:
The LINQ expression (MyDBset) could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I can't seperate the incoming list into seperate A and B lists as the combination is essential to the filter accuracy. Otherwise I could have used this approach
return await context.MyDBset
.Where(x => incomingListA.Contains(x.A) &&
incomingListB.Contains(x.B)).ToListAsync();
This approach does work in EF Core 5
Basically I want EF to translate my query into something like:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) OR (m.A = 3 AND m.B = 4)
If I loop through my incomingList adding "where" filters to a query the resultant SQL uses "AND" instead of "OR" like so:
SELECT * FROM MyTable AS m
Where (m.A = 1 AND m.B = 2) AND (m.A = 3 AND m.B = 4)
Right now my work around involves binging the result set into memory with some basic filters then running an in memory filter using the original syntax. This is obviously not ideal.
Any suggestions? Thanks

Since EF Core is not willing to translate such expression (it's a well known limitation that Contains with primitive value is the only supported operation on in-memory collection in L2E query since it directly translates to SQL IN operator), you have to do it yourself. Any is basically equivalent of Or as you mentioned, so for not so big list and top level query it could easily be done by using some predicate builder implementation to dynamically build || predicate. Or directly using the the Expression class methods as in this custom extension method from my answer to How to simplify repetitive OR condition in Where(e => e.prop1.contains() || e.prop2.contains() || ...):
public static partial class QueryableExtensions
{
public static IQueryable<T> WhereAnyMatch<T, V>(this IQueryable<T> source, IEnumerable<V> values, Expression<Func<T, V, bool>> match)
{
var parameter = match.Parameters[0];
var body = values
// the easiest way to let EF Core use parameter in the SQL query rather than literal value
.Select(value => ((Expression<Func<V>>)(() => value)).Body)
.Select(value => Expression.Invoke(match, parameter, value))
.Aggregate<Expression>(Expression.OrElse);
var predicate = Expression.Lambda<Func<T, bool>>(body, parameter);
return source.Where(predicate);
}
}
with sample usage for your scenario:
context.MyDBset
.WhereAnyMatch(incomingList, (x, v) => v.A == x.A && v.B == x.B)
Normally instead of Expression.Invoke helper utilities like this would use custom ExpressionVisitor to replace the parameters of the source expression (thus simulation a "call"), which produces more naturally looking query expression, but for EF Core it doesn't really matter since it recognizes and correctly translates invocation expressions inside the expression tree.
But just in case Expression.Invoke doesn't work, here is the parameter replacing version - it should work with all query providers. First, the helpers
public static partial class ExpressionBuilder
{
public static Expression ReplaceParameter(this Expression source, ParameterExpression parameter, Expression value)
=> new ParameterReplacer { Parameter = parameter, Value = value }.Visit(source);
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Parameter;
public Expression Value;
protected override Expression VisitParameter(ParameterExpression node)
=> node == Parameter ? Value : node;
}
}
and then simply replace the invocation
.Select(value => Expression.Invoke(match, parameter, value))
with
.Select(value => match.Body.ReplaceParameter(match.Parameters[1], value))
because specifically here we are reusing the first parameter (var parameter = match.Parameters[0];), otherwise it should be replaced as well.

Get value of expression as a string

I'm working on a generic repository for Entity Framework. To search for values I have a Find method on my repository that takes an expression. During this process I want to log the expression. I do this by getting expression.Body.
Here is my code
public async Task<TEntity> FindFirstOrDefault<TEntity>(Expression<Func<TEntity, bool>> expression) where TEntity : class
{
string expBody = expression.Body.ToString();
// Log expBody here
return await _context.Set<TEntity>().Where(expression).FirstOrDefaultAsync().ConfigureAwait(false);
}
The above code works fine when testing with a hard coded condition like this.
var blogs = await uow.Repository.FindFirstOrDefault<Blog>(x => x.BlogId == 1).ConfigureAwait(false);
When I debug that above line has expBody with a value of (x.BlogId == 1)
However if I do the comparison against a variable like this:
int id = 1;
var blogs = await uow.Repository.FindFirstOrDefault<Blog>(x => x.BlogId == id).ConfigureAwait(false);
I end up getting this for expBody:
(x.BlogId == value(TestProject.ValueService+<>c__DisplayClass4_0).intId)
How can I get the actual expression body like when I'm using a hard coded value when I'm actually referencing a variable?

Entity Framework Core nested expressions

With the recent release of Entity Framework Core 3.0, LINQ queries are no longer evaluated on the client by default. I'm a big fan of this change, as it revealed some potentially dangerous client-side evaluation in my project that I thought was translated to SQL; however, it also made some of the helper methods that I was using to avoid crazy chains of ternaries unusable.
Has anyone manged to nest LINQ expressions for use with Entity Framework Core 3.0? Here's an example of what I'm hoping to achieve:
[Fact]
public async Task Can_use_custom_expression()
{
var dbContext = new ApplicationDbContext(new DbContextOptionsBuilder<ApplicationDbContext>().UseInMemoryDatabase("Test").Options);
dbContext.Users.Add(new ApplicationUser { FirstName = "Foo", LastName = "Bar" });
dbContext.SaveChanges();
string query = "Foo";
Expression<Func<string, string, bool>> valueCheck = (value, expected) => !string.IsNullOrEmpty(value) && value.Contains(expected);
var valueCheckFunc = valueCheck.Compile();
Expression<Func<ApplicationUser, bool>> whereExpression = (u) => valueCheckFunc(u.FirstName, query);
var user = await dbContext.Users
.Where(whereExpression)
.FirstOrDefaultAsync();
Assert.NotNull(user);
}
When I run this example, I get the following exception:
Message:
System.InvalidOperationException : The LINQ expression 'Where<ApplicationUser>(
source: DbSet<ApplicationUser>,
predicate: (a) => Invoke(__valueCheckFunc_0, a.FirstName, __query_1)
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
Like I said, I do not want to evaluate this expression client-side, but I would like to avoid having to chain a dozen or so !string.IsNullOrEmpty(x) && x.Contains(y)'s in a single expression. I'd love some tips on how to achieve this.

If you want your expressions to be translatable to Sql by EF you need to avoid calling delegates or methods (with some exceptions, of course). But what you want to achieve is doable by replacing the delegate invocation with its defining expression. For that you need a specialized ExpressionVisitor.
The following visitor will traverse expressions replacing delegate references within its wrapping invocations by a lambda expression body:
public class DelegateByLambda: ExpressionVisitor
{
LambdaExpression delegateReferenceExpression;
LambdaExpression lambdaExpression;
Stack<InvocationExpression> invocations;
public DelegateByLambda(LambdaExpression delegateReferenceExpression, LambdaExpression lambdaExpression)
{
this.delegateReferenceExpression = delegateReferenceExpression;
this.lambdaExpression = lambdaExpression;
this.invocations = new Stack<InvocationExpression>();
}
protected override Expression VisitParameter(ParameterExpression node)
{
var paramIndex = lambdaExpression.Parameters.IndexOf(node);
if (paramIndex >= 0)
{
InvocationExpression call = invocations.Peek();
return base.Visit(call.Arguments[paramIndex]);
}
return base.VisitParameter(node);
}
protected override Expression VisitInvocation(InvocationExpression node)
{
if (node.Expression.ToString() == delegateReferenceExpression.Body.ToString())
{
invocations.Push(node);
var result = base.Visit(lambdaExpression.Body);
invocations.Pop();
return result;
}
return base.VisitInvocation(node);
}
}
This class has no protection against attempting to replace delegate invocations by lambdas with mismatching arguments (number and types) however, the following extension method will do the trick:
public static class DelegateByLambdaExtensions
{
public static Expression<T> Replace<T, X>(this Expression<T> source, Expression<Func<X>> delegateReference, Expression<X> lambdaReference)
{
return new DelegateByLambda(delegateReference, lambdaReference).Visit(source) as Expression<T>;
}
}
So, all you need to do in your code is to call the replace extension method on the expression you want to translate passing an expression returning the delegate and desired lambda expression for expansion. Your sample should look like this:
Expression<Func<string, string, bool>> valueCheck = (value, expected) => !string.IsNullOrEmpty(value) && value.Contains(expected);
var valueCheckFunc = valueCheck.Compile();
Expression<Func<ApplicationUser, bool>> whereExpression = (u) => valueCheckFunc(u.FirstName, query);
whereExpression = whereExpression.Replace(() => valueCheckFunc, valueCheck);
var user = dbContext.Users
.Where(whereExpression)
.FirstOrDefault();
Console.WriteLine(user != null ? $"Found {user.FirstName} {user.LastName}!" : "User not found!");
A working sample can be found here. https://dotnetfiddle.net/Lun3LA

Cannot implicity convert type System.Linq.Expression<System.Func<Object, bool>> to bool

actually I have an Expression like this that work very well in the case of Linq to Entity
public static Expression<Func<Tender, bool>> GetPublic()
{
var now = DateTime.Now.GetEasternStandardDateTime();
return tender => tender.EndDate > DateTime.Now &&
tender.IsClosed == false &&
tender.IsCancel == false &&
tender.PrivacyLevelId == 1;
}
I can use the expression like this : _repositoryObject.Where(GetPublic()) and I will get results.
But I cannot use the same expression when I have to do Linq to SQL pattern like this
var results = from t in Tenders
where Tender.GetPublic()
select t;
I have this error
Cannot implicity convert type
System.Linq.Expression> to bool for Linq to
SQL
and I don't know why...
Karine

Firstly, I suspect you actually mean you can call it as:
_repositoryObject.Where(GetPublic())
It's important that you call the method to get the expression tree.
Your query expression is being converted into:
var results = Tenders.Where(t => Tender.GetPublic());
... which isn't what you're looking for. Basically, query expressions are translated into method calls using lambda expressions. You don't want that in this case, so you shouldn't use a query expression.
If you want to do other things with a query expression, you can always mix and match, e.g.
var results = from t in Tenders.Where(Tender.GetPublic())
where t.SomeOtherProperty
select t.SomethingElse;

If you change your method to take an IQueryable<Tender>, and return another IQueryable<Tender> where your criteria are added, then it might work.
public static IQueryable<Tender> AddPublicCriteria(this IQueryable<Tender> tenderQuery) {
// I've omitted your call to DateTime.Now.GetEasternStandardDateTime()
// since you do not use it.
return tenderQuery
.Where(tender => tender.EndDate > DateTime.Now &&
tender.IsClosed == false &&
tender.IsCancel == false &&
tender.PrivacyLevelId == 1
);
}
var query = Tenders; // Original table or data-set
query = query.AddPublicCriteria(); // Add criteria
var results = query.ToList(); // Run query and put the results in a list

Verify method call with Lambda expression - Moq

I have a Unit of Work implementation with, among others, the following method:
T Single<T>(Expression<Func<T, bool>> expression) where T : class, new();
and I call it, for instance, like this:
var person = _uow.Single<Person>(p => p.FirstName == "Sergi");
How can I verify that the Single method has been called with an argument of FirstName == "Sergi"?
I've tried the following, but to no avail:
// direct approach
session.Verify(x => x.Single<Person>(p => p.FirstName == "Sergi"));
// comparing expressions
Expression<Func<Person, bool>> expression = p => p.FirstName == "Sergi");
session.Verify(x => x
.Single(It.Is<Expression<Func<Person, bool>>>(e => e == expression));
They all result in the folowing error:
Expected invocation on the mock at least once, but was never performed
Any ideas on how that can be done?
I'm using the latest Moq from NuGet, version 4.0.10827.0
UPDATE: A Specific example
What I'm seeing is that whenever I use string literals inside the lambda, Verify works. As soon as I'm comparing variables it fails. Case in point:
// the verify
someService.GetFromType(QuestionnaireType.Objective)
session.Verify(x => x.Single<Questionnaire>(q =>
q.Type == QuestionnaireType.Objective));
// QuestionnaireType.Objective is just a constant:
const string Objective = "objective";
// the method where it's called (FAILS):
public Questionnaire GetFromType(string type)
{
// this will fail the Verify
var questionnaire = _session
.Single<Questionnaire>(q => q.Type == type);
}
// the method where it's called (PASSES):
public Questionnaire GetFromType(string type)
{
// this will pass the Verify
var questionnaire = _session
.Single<Questionnaire>(q => q.Type == QuestionnaireType.Objective);
}
How come the Verify fails as soon as I use the method parameter in the lambda expression?
What would be the proper way to write this test?

The direct approach works just fine for me:
// direct approach
session.Verify(x => x.Single<Person>(p => p.FirstName == "Sergi"));
The expression object doesn't return true for equivalent expressions so this will fail:
// comparing expressions
Expression<Func<Person, bool>> expression = p => p.FirstName == "Sergi");
session.Verify(x => x
.Single(It.Is<Expression<Func<Person, bool>>>(e => e == expression));
To understand why, run the following NUnit test:
[Test]
public void OperatorEqualEqualVerification()
{
Expression<Func<Person, bool>> expr1 = p => p.FirstName == "Sergi";
Expression<Func<Person, bool>> expr2 = p => p.FirstName == "Sergi";
Assert.IsTrue(expr1.ToString() == expr2.ToString());
Assert.IsFalse(expr1.Equals(expr2));
Assert.IsFalse(expr1 == expr2);
Assert.IsFalse(expr1.Body == expr2.Body);
Assert.IsFalse(expr1.Body.Equals(expr2.Body));
}
And as the test above indicates, comparing by the expression body will also fail, but string comparison works, so this works as well:
// even their string representations!
session.Verify(x => x
.Single(It.Is<Expression<Func<Person, bool>>>(e =>
e.ToString() == expression.ToString()));
And here's one more style of test you can add to the arsenal that also works:
[Test]
public void CallbackVerification()
{
Expression<Func<Person, bool>> actualExpression = null;
var mockUow = new Mock<IUnitOfWork>();
mockUow
.Setup(u => u.Single<Person>(It.IsAny<Expression<Func<Person, bool>>>()))
.Callback( (Expression<Func<Person,bool>> x) => actualExpression = x);
var uow = mockUow.Object;
uow.Single<Person>(p => p.FirstName == "Sergi");
Expression<Func<Person, bool>> expectedExpression = p => p.FirstName == "Sergi";
Assert.AreEqual(expectedExpression.ToString(), actualExpression.ToString());
}
As you have a number of test cases that fail that shouldn't, you likely have a different problem.
UPDATE: Per your update, consider the following setup and expressions:
string normal_type = "NORMAL";
// PersonConstants is a static class with NORMAL_TYPE defined as follows:
// public const string NORMAL_TYPE = "NORMAL";
Expression<Func<Person, bool>> expr1 = p => p.Type == normal_type;
Expression<Func<Person, bool>> expr2 = p => p.Type == PersonConstants.NORMAL_TYPE;
One expression references an instance variable of the containing method. The other represents an expression that references a const member of a static class. The two are different expressions, regardless of the values that may be assigned to the variables at runtime. If however, string normal_type is changed to const string normal_type then the expressions are again the same as each reference a const on the right hand side of the expression.

I would also like to share another approach to comparing the parameter expression to the expected expression. I searched StackOverflow for "how to compare expressions," and I was led to these articles:
how-to-test-expressions-equality
most-efficient-way-to-test-equality-of-lambda-expressions
I was then led to this Subversion repository for db4o.net. In one of their projects, namespace Db4objects.Db4o.Linq.Expressions, they include a class named ExpressionEqualityComparer. I was able to checkout this project from the repository, compile, build, and create a DLL to use in my own project.
With the ExpressionEqualityComparer, you can modify the Verify call to something like the following:
session.Verify(x => x
.Single(It.Is<Expression<Func<Person, bool>>>(e =>
new ExpressionEqualityComparer().Equals(e, expression))));
Ultimately, the ExpressionEqualityComparer and the ToString() techniques both return true in this case (with the ToString most likely being faster - speed not tested). Personally, I prefer the comparer approach since I feel it is more self-documenting and better reflects your design intent (comparing the expression objects rather a string comparison of their ToString outputs).
Note: I'm still looking for a db4o.net license file in this project, but I've not modified the code in anyway, included the copyright notice, and (since the page is publicly available) I'm assuming that's enough for now... ;-)