We Keep Coding

How could I generate a c# .NET core function that can mimic the Include().ThenInclude.(Where) behaviour?

I'm trying to create a generic blazor component that and has a config object.
So far I've been able to successfully mimic .Where(), .Include().ThenInclude() separately, but now I need to add condition to the include.
My config file has the following structure
class Config
{
public List<string> includes {get;set;}
public List<Expression<Func<T,bool>>> conditions {get;set;}
}
database structure
For the example lets assume the following structure.
Dataset (1)<--(n) DatasetItem
DatasetItem (1)-->(1) Image
DatasetItem (1)-->(1) Label
Image (1)<--(n) Labels
Label(1)<--(n) Tags
where an image can be a part of more than one dataset and it's tags might or might not be included in the same dataset. Imagine an picture of a city park with dogs, people, and cars. Someone would want a dataset of vehicles and consider only the cars and ignore the labels about animals and people.
A Label include the coordinates and one or more TagNames (vehicle, car, red, fourWheeler, etc)
what works
And I can give conditions and includes this way:
var config = new Config<DatasetItem>()
{
includes = new() {"Image", "Image.Labels", "Image.Labels.Tags"}
conditions = new() {di => di.datasetId == 1, di => di.Label.width > 100}
}
the problems and limitations of this approach
For this to work I use the .Include(str) overload, but I don't like that approach since it is prone to fail.
The other problem with this approach is a limitation on the conditions.
For example, I can not add a condition to get only the Tags that belong to a specific category.
It seems that is not possible to add a condition that would give me this filter. Best I got was with the use of All or Any, but those do not give the desire result.
config.conditions.Add(
di => di.Images.Any(i => Labels.Any(l => l.Tags.Any(t => t.Name == "vehicle"))
)
what I'm looking for
Whith regular use of Include, obtaining the desire result would be like this.
DatasetItems.Include(di => di.Images)
.ThenInclude(i => i.Labels)
.ThenInclude(l => l.Tags.Where(t => t.Name == "vehicle"))
So, is there any way to achieve this?
I thing the solution would require to use Expression Trees, Reflection, and Recursive functions using MakeGenericMethod, but I havent found a way to achive this.
the ideal solution would be to find a structure like the one for conditions
List<Expression<Func<T,something>>> that lets me add the condition in this way
includes.add(di => di.Images.Labels.Tags.Where(t => t.Name == "vehicle"))
a failed attempt
I'm including a failed attempt I took to solve this, just in case this helps to narrow down the intention and for if someone can use it for solving the problem. This attempt is far from ideal.
property was a dot-separated string, like "Image.Labels.Tags" and filter was suppoused to be the Where expression.
This function does not compile.
public static IQueryable<T> FilteredInclude<T>(this IQueryable<T> query, string property, Func<dynamic, bool> filter) where T : class
{
var type = typeof(T);
var properties = property.Split('.');
var propertyInfo = type.GetProperty(properties[0]);
var parameter = Expression.Parameter(type, "x");
var propertyExpression = Expression.Property(parameter, propertyInfo);
var lambda = Expression.Lambda(propertyExpression, parameter);
var filteredQuery = query.Where(x => filter(propertyInfo.GetValue(x)));
if (properties.Length == 1)
{
var methodInfo = typeof(QueryableExtensions).GetMethod(nameof(Include));
methodInfo = methodInfo.MakeGenericMethod(type, propertyInfo.PropertyType);
var result = methodInfo.Invoke(null, new object[] { filteredQuery, lambda });
return (IQueryable<T>)result;
}
else
{
var nextProperty = string.Join(".", properties.Skip(1));
var result = filteredQuery.FilteredInclude(nextProperty, filter);
var thenIncludeMethod = typeof(Microsoft.EntityFrameworkCore.Query.Internal.IncludeExpression)
.GetMethod("ThenInclude")
.MakeGenericMethod(propertyInfo.PropertyType);
var resultWithThenInclude = thenIncludeMethod.Invoke(result, new object[] { lambda });
return (IQueryable<T>)resultWithThenInclude;
}
}

Add a where clause to IQueryable without using generics

I'm trying to write a piece of code that is going to be used to select a value from a database. As a test I have already created this test version of the method that "works" but isn't great.
private object? FindObjectByProperty(IQueryable query, PropertyInfo primaryKeyProperty, object lookupValue)
{
object? result = null;
foreach (var value in query)
{
if (!primaryKeyProperty.GetValue(value)!.Equals(lookupValue))
continue;
result = value;
break;
}
return result;
}
The code will iterate through the query and find the first entry that matches the Equals. However, if the query contains millions of rows, as it's likely to when the IQueryable is really a DbSet (Which in this use case it is. Only I don't know what T, as it could be any of the DbSets on my EF Core data context.
What I'd like to end up with is something that looks something along these lines....
private object? FindObjectByProperty(IQueryable query, PropertyInfo primaryKeyProperty, object lookupValue)
{
return query.Where( x => x.*primaryKeyProperty* == lookupValue).FirstOrDefault();
}
The above code is pseudo code, the problems that I have is that query does not have a .Where available. Also primaryKeyProperty will need to be translated to the actual property.
The idea is that when this code is executed, ultimately executing the query, will generate a sql statement which selects a single item and returns it.
Can anyone help with solving this?
Update:
I'm working on a solution to this, so far this is what I've come up with
//using System.Linq.Dynamic.Core;
private object? FindObjectByProperty(IQueryable query, PropertyInfo primaryKeyProperty, object lookupValue)
{
var parameter = Expression.Parameter(query.ElementType);
var e1 = Expression.Equal(Expression.Property(parameter, primaryKeyProperty.Name), Expression.Constant(lookupValue));
var lambda = Expression.Lambda<Func<object, bool>>(e1, parameter);
return query.Where(lambda).FirstOrDefault();
}
This is failing on the because the func uses Object, when it really needs the real type. Trying to figure that bit out. This is getting closer.
Update #2:
Here's the answer that I needed
private object? FindObjectByProperty(IQueryable query, PropertyInfo primaryKeyProperty, object lookupValue)
{
var parameter = Expression.Parameter(query.ElementType);
var propExpr = Expression.Property(parameter, primaryKeyProperty);
var lambdaBody = Expression.Equal(propExpr, Expression.Constant(lookupValue, primaryKeyProperty.PropertyType));
var filterEFn = Expression.Lambda(lambdaBody, parameter);
return query.Where(filterEFn).FirstOrDefault();
}
The difference is that the Expression.Lambda no longer tries to define the func using generics. This is the only change that I needed to do to make the code function as I wanted.
In the test case that I was using, the T-SQL produced to lookup the value looks like this...
SELECT TOP(1) [g].[Id], [g].[Deleted], [g].[Guid], [g].[Name], [g].[ParentId]
FROM [Glossaries].[Glossaries] AS [g]
WHERE [g].[Id] = CAST(3 AS bigint)
The table [Glossaries].[Glossaries] is provided by the input query. The column name Id is provided by the primaryKeyProperty, and the number 3 is provided by the lookupValue.
This is perfect for my needs as I simply needed to select that one row and nothing else, so that I can effectively lazy load the my object property when I need it, and not before.
Also this code will be reused for many different tables.

In order to build add Where to an IQueryable where you don't have access to the actual IQueryable<T>, you need to take a step back (or up?) from calling Where at compile-time, and build the Where call at runtime, as well as the predicate lambda:
public static class DBExt {
public static IQueryable WherePropertyIs<T2>(this IQueryable src, PropertyInfo propInfo, T2 propValue) {
// return src.Where(s => s.{propInfo} == propValue)
// (T s)
var sParam = Expression.Parameter(src.ElementType, "s");
// s.propInfo
var propExpr = Expression.Property(sParam, propInfo);
// s.{propInfo} == propValue
var lambdaBody = Expression.Equal(propExpr, Expression.Constant(propValue));
// (T s) => s.{PropInfo} == propValue
var filterEFn = Expression.Lambda(lambdaBody, sParam);
var origQuery = src.Expression;
// IQueryable<Tx>.Where<Tx, Expression<Func<Tx, bool>>>()
var whereGenericMI = typeof(Queryable).GetMethods("Where", 2).Where(mi => mi.GetParameters()[1].ParameterType.GenericTypeArguments[0].GenericTypeArguments.Length == 2).First();
// IQueryable<T>.Where<T, Expression<Func<T, bool>>>()
var whereMI = whereGenericMI.MakeGenericMethod(src.ElementType);
// src.Where(s => s.{propertyInfo} == propValue)
var newQuery = Expression.Call(whereMI, origQuery, filterEFn);
return src.Provider.CreateQuery(newQuery);
}
}

Linq extending Expressions

I'm trying to write a generic wildcard Search for the ServiceStack.OrmLite.SqlExpressionVisitor that has the following signature:
public static SqlExpressionVisitor<T> WhereWildcardSearch<T> (this SqlExpressionVisitor<T> ev, Expression<Func<T,string>> field, string search)
where ev is the rest of the filter, field is the getter for the field to search by and search is the entered term.
Normally (non-generic) I would write the following:
if(search.StartsWith('*') && search.EndsWith('*'))
ev = ev.Where(x => x.foo.Contains(search.Trim('*')));
and of course also variants for x.foo.StartsWith or EndsWith.
Now I am searching for something like (pseudocode:)
ev = ev.Where(x => field(x).Contains(search.Trim('*')));
Of course I can't compile and call the expression directly, as this should be translated to Sql using Linq2Sql.
This is my code so far:
var getFieldExpression = Expression.Invoke (field, Expression.Parameter (typeof (T), "getFieldParam"));
var searchConstant = Expression.Constant (search.Trim('*'));
var inExp = Expression.Call (getFieldExpression, typeof(String).GetMethod("Contains"), searchConstant);
var param = Expression.Parameter (typeof (T), "object");
var exp = Expression.Lambda<Func<T, bool>> (inExp, param);
ev = ev.Where (exp);
Please don't tell me that I should directly write SQL with $"LIKE %search%" or something - I know that there are other ways, but solving this would help my understanding of Linq and Expressions in general and it bugs me when I can't solve it.

Here is how it can be done (I think it will be clear for you without much additional explanations what you did wrong, but if not - feel free to request a clarification):
// extract property name from passed expression
var propertyName = ((MemberExpression)field.Body).Member.Name;
var param = Expression.Parameter(typeof(T), "object");
var searchConstant = Expression.Constant(search.Trim('*'));
var contains = typeof(String).GetMethod("Contains");
// object.FieldName.Contains(searchConstant)
var inExp = Expression.Call(Expression.PropertyOrField(param, propertyName), contains, searchConstant);
// object => object.FieldName.Contains(searchConstant)
var exp = Expression.Lambda<Func<T, bool>>(inExp, param);
In response to comment. You have two expression trees: one is being passed to you and another one which you are building (exp). In this simple case they both use the same number of parameters and those parameters are of the same type (T). In this case you can reuse parameter from field expression tree, like this:
// use the same parameter
var param = field.Parameters[0];
var searchConstant = Expression.Constant(search.Trim('*'));
var contains = typeof(String).GetMethod("Contains");
// note field.Body here. Your `field` expression is "parameter => parameter.Something"
// but we need just "parameter.Something" expression here
var inExp = Expression.Call(field.Body, contains, searchConstant);
// pass the same parameter to new tree
var exp = Expression.Lambda<Func<T, bool>>(inExp, param);
In more complicated cases you might need to use ExpressionVisitor to replace parameters in one expression tree to reference to parameters from another (final) expression tree.

Combine Expressions instead of using multiple queries in Entity Framework

I have following generic queryable (which may already have selections applied):
IQueryable<TEntity> queryable = DBSet<TEntity>.AsQueryable();
Then there is the Provider class that looks like this:
public class Provider<TEntity>
{
public Expression<Func<TEntity, bool>> Condition { get; set; }
[...]
}
The Condition could be defined per instance in the following fashion:
Condition = entity => entity.Id == 3;
Now I want to select all Provider instances which have a Condition that is met at least by one entity of the DBSet:
List<Provider> providers = [...];
var matchingProviders = providers.Where(provider => queryable.Any(provider.Condition))
The problem with this: I'm starting a query for each Provider instance in the list. I'd rather use a single query to achieve the same result. This topic is especially important because of questionable performance. How can I achieve the same results with a single query and improve performance using Linq statements or Expression Trees?

Interesting challenge. The only way I see is to build dynamically UNION ALL query like this:
SELECT TOP 1 0 FROM Table WHERE Condition[0]
UNION ALL
SELECT TOP 1 1 FROM Table WHERE Condition[1]
...
UNION ALL
SELECT TOP 1 N-1 FROM Table WHERE Condition[N-1]
and then use the returned numbers as index to get the matching providers.
Something like this:
var parameter = Expression.Parameter(typeof(TEntity), "e");
var indexQuery = providers
.Select((provider, index) => queryable
.Where(provider.Condition)
.Take(1)
.Select(Expression.Lambda<Func<TEntity, int>>(Expression.Constant(index), parameter)))
.Aggregate(Queryable.Concat);
var indexes = indexQuery.ToList();
var matchingProviders = indexes.Select(index => providers[index]);
Note that I could have built the query without using Expression class by replacing the above Select with
.Select(_ => index)
but that would introduce unnecessary SQL query parameter for each index.

Here is another (crazy) idea that came in my mind. Please note that similar to my previous answer, it doesn't guarantee better performance (in fact it could be worse). It just presents a way to do what you are asking with a single SQL query.
Here we are going to create a query that returns a single string with length N consisting of '0' and '1' characters with '1' denoting a match (something like string bit array). The query will use my favorite group by constant technique to build dynamically something like this:
var matchInfo = queryable
.GroupBy(e => 1)
.Select(g =>
(g.Max(Condition[0] ? "1" : "0")) +
(g.Max(Condition[1] ? "1" : "0")) +
...
(g.Max(Condition[N-1] ? "1" : "0")))
.FirstOrDefault() ?? "";
And here is the code:
var group = Expression.Parameter(typeof(IGrouping<int, TEntity>), "g");
var concatArgs = providers.Select(provider => Expression.Call(
typeof(Enumerable), "Max", new[] { typeof(TEntity), typeof(string) },
group, Expression.Lambda(
Expression.Condition(
provider.Condition.Body, Expression.Constant("1"), Expression.Constant("0")),
provider.Condition.Parameters)));
var concatCall = Expression.Call(
typeof(string).GetMethod("Concat", new[] { typeof(string[]) }),
Expression.NewArrayInit(typeof(string), concatArgs));
var selector = Expression.Lambda<Func<IGrouping<int, TEntity>, string>>(concatCall, group);
var matchInfo = queryable
.GroupBy(e => 1)
.Select(selector)
.FirstOrDefault() ?? "";
var matchingProviders = matchInfo.Zip(providers,
(match, provider) => match == '1' ? provider : null)
.Where(provider => provider != null)
.ToList();
Enjoy:)
P.S. In my opinion, this query will run with constant speed (regarding number and type of the conditions, i.e. can be considered O(N) in the best, worst and average cases, where N is the number of the records in the table) because the database has to perform always a full table scan. Still it will be interesting to know what's the actual performance, but most likely doing something like this just doesn't worth the efforts.
Update: Regarding the bounty and the updated requirement:
Find a fast query that only reads a record of the table once and ends the query if already all conditions are met
There is no standard SQL construct (not even speaking about LINQ query translation) that satisfies both conditions. The constructs that allow early end like EXISTS can be used for a single condition, thus when executed for multiple conditions will violate the first rule of reading the table record only once. While the constructs that use aggregates like in this answer satisfy the first rule, but in order to produce the aggregate value they have to read all the records, thus cannot exit earlier.
Shortly, there is no query that can satisfy both requirements. What about the fast part, it really depends of the size of the data and the number and type of the conditions, table indexes etc., so again there is simply no "best" general solution for all cases.

Based on this Post by #Ivan I created an expression that is slightly faster in some cases.
It uses Any instead of Max to get the desired results.
var group = Expression.Parameter(typeof(IGrouping<int, TEntity>), "g");
var anyMethod = typeof(Enumerable)
.GetMethods()
.First(m => m.Name == "Any" && m.GetParameters()
.Count() == 2)
.MakeGenericMethod(typeof(TEntity));
var concatArgs = Providers.Select(provider =>
Expression.Call(anyMethod, group,
Expression.Lambda(provider.Condition.Body, provider.Condition.Parameters)));
var convertExpression = concatArgs.Select(concat =>
Expression.Condition(concat, Expression.Constant("1"), Expression.Constant("0")));
var concatCall = Expression.Call(
typeof(string).GetMethod("Concat", new[] { typeof(string[]) }),
Expression.NewArrayInit(typeof(string), convertExpression));
var selector = Expression.Lambda<Func<IGrouping<int, TEntity>, string>>(concatCall, group);
var matchInfo = queryable
.GroupBy(e => 1)
.Select(selector)
.First();
var MatchingProviders = matchInfo.Zip(Providers,
(match, provider) => match == '1' ? provider : null)
.Where(provider => provider != null)
.ToList();

The approach I tried here was to create Conditions and nest them into one Expression. If one of the Conditions is met, we get the index of the Provider for it.
private static Expression NestedExpression(
IEnumerable<Expression<Func<TEntity, bool>>> expressions,
int startIndex = 0)
{
var range = expressions.ToList();
range.RemoveRange(0, startIndex);
if (range.Count == 0)
return Expression.Constant(-1);
return Expression.Condition(
range[0].Body,
Expression.Constant(startIndex),
NestedExpression(expressions, ++startIndex));
}
Because the Expressions still may use different ParameterExpressions, we need an ExpressionVisitor to rewrite those:
private class PredicateRewriterVisitor : ExpressionVisitor
{
private readonly ParameterExpression _parameterExpression;
public PredicateRewriterVisitor(ParameterExpression parameterExpression)
{
_parameterExpression = parameterExpression;
}
protected override Expression VisitParameter(ParameterExpression node)
{
return _parameterExpression;
}
}
For the rewrite we only need to call this method:
private static Expression<Func<T, bool>> Rewrite<T>(
Expression<Func<T, bool>> exp,
ParameterExpression parameterExpression)
{
var newExpression = new PredicateRewriterVisitor(parameterExpression).Visit(exp);
return (Expression<Func<T, bool>>)newExpression;
}
The query itself and the selection of the Provider instances works like this:
var parameterExpression = Expression.Parameter(typeof(TEntity), "src");
var conditions = Providers.Select(provider =>
Rewrite(provider.Condition, parameterExpression)
);
var nestedExpression = NestedExpression(conditions);
var lambda = Expression.Lambda<Func<TEntity, int>>(nestedExpression, parameterExpression);
var matchInfo = queryable.Select(lambda).Distinct();
var MatchingProviders = Providers.Where((provider, index) => matchInfo.Contains(index));
Note: Another option which isn't really fast as well

Here is another view of the problem that has nothing to do with expressions.
Since the main goal is to improve the performance, if the attempts to produce the result with single query don't help, we could try improving the speed by parallelizing the execution of the original multi query solution.
Since it's really a LINQ to Objects query (which internally executes multiple EF queries), theoretically it should be a simple matter of turning it into a PLINQ query by inserting AsParallel like this (non working):
var matchingProviders = providers
.AsParallel()
.Where(provider => queryable.Any(provider.Condition))
.ToList();
However, it turns out that EF DbContext is not well suited for multi thread access, and the above simply generates runtime errors. So I had to resort to TPL using one of the Parallel.ForEach overloads that allows us to supply local state, which I used to allocate several DbContext instances during the execution.
The final working code looks like this:
var matchingProviders = new List<Provider<TEntity>>();
Parallel.ForEach(providers,
() => new
{
context = new MyDbContext(),
matchingProviders = new List<Provider<TEntity>>()
},
(provider, state, data) =>
{
if (data.context.Set<TEntity>().Any(provider.Condition))
data.matchingProviders.Add(provider);
return data;
},
data =>
{
data.context.Dispose();
if (data.matchingProviders.Count > 0)
{
lock (matchingProviders)
matchingProviders.AddRange(data.matchingProviders);
}
}
);
If you have a multi core CPU (which is normal nowadays) and a good database server, this should give you the improvement you are seeking for.

Use Optional OR Clause in Linq.Table.Where()

Is there a way to make the ProjectID check below part of an optional block? I'm a recent .Net convert from Java EE and I'm looking for something similar to the Hibernate Criteria API. I'd like to simplify the block below and only have to call Where() once. I'm also not sure of the performance implications of doing a Where() with lambdas as I just started working with .Net a week ago.
public IQueryable<Project> FindByIdOrDescription(string search)
{
int projectID;
bool isID = int.TryParse(search, out projectID);
IQueryable<Project> projects;
if (isID)
{
projects = dataContext.Projects.Where(p => p.ProjectDescription.Contains(search) || p.ProjectID == projectID);
}
else
{
projects = dataContext.Projects.Where(p => p.ProjectDescription.Contains(search));
}
return projects;
}

If you're looking for optionally adding AND xyz, you could simply chain Where calls:
var projects = dataContext.Projects.Where(p => p.ProjectDescription.Contains(search));
if (isID)
projects = projects.Where(p => p.ProjectID == projectID);
Unfortunately things are not so easy when you'd like to do an OR xyz. For this to work, you'll need to build a predicate expression by hand. It's not pretty. One way to do this is
Expression<Func<Project, bool>> predicate = p => p.ProjectDescription.Contains(search);
if (isID)
{
ParameterExpression param = expr.Body.Parameters[0];
predicate = Expression.Lambda<Func<Project, bool>>(
Expression.Or(
expr.Body,
Expression.Equal(
Expression.Property(param, "ProjectID"),
Expression.Constant(projectID))),
param);
}
var projects = dataContext.Projects.Where(predicate);
Note that adding a condition to the existing predicate is a lot more work than creating the initial expression, because we need to completely rebuild the expression. (A predicate must always use a single parameter; declaring two expressions using lambda syntax will create two separate parameter expression objects, one for each predicate.) Note that the C# compiler does roughly the same behind the scenes when you use lambda syntax for the initial predicate.
Note that this might look familiar if you're used to the criteria API for Hibernate, just a little more verbose.
Note, however, that some LINQ implementations are pretty smart, so the following may also work:
var projects = dataContext.Projects.Where(
p => p.ProjectDescription.Contains(search)
|| (isID && p.ProjectID == projectID));
YMMV, though, so do check the generated SQL.

The query isn't parsed and executed until the first time you actually access the elements of the IQueryable. Therefore, no matter how many where's you keep appending (which you're not even doing here anyway) you don't need to worry about hitting the DB too many times.

Delegates / expressions can be "chained", like this (untested pseudo-code):
Expression<Predicate<Project>> predicate = p => p.ProjectDescription.Contains(search);
if ( isID )
predicate = p => predicate(p) || p.ProjectID == projectId;
return dataContext.Where(predicate);