I'm trying to write a generic extension method for ordering an IQueryable database collection. (There is more to it, but this is the part I'm s tuck on.)
I want to provide:
the database class (e.g. Customer,Employer)
the LINQ expression for the column
the type of the column (int,string,etc.)
I started with a class that should contain the information but I'm getting stuck how to define and consume the expression.
For the usage I am thinking something like this, but not sure where to provide the type for each column
var sortedList = queryable.MyCustomSortMethod<User>( item => new List<SortItem>
{ item.Column_Name, "ASC" }, //string
{ item.Column_BirthYear "DESC" }, //int
{ item.Column_BirthDate, "ASC" } //date
}
Then I have a class to specify the selector and its direction
public class SortItem<TEntity>
{
public Expression<Func<T> SortFieldSelector<T> { get; set; }
public SortDirectionType SortDirectionType { get; set; }
//but how to provide column type
}
Then the extension method something like this
public static IQueryable<TEntity> MyCustomSortMethod(
this IQueryable<TEntity> queryable,
List<SortItem<TEntity> selectors)
{
foreach (var selector in selectors) {
if(selector.Direction == "asc")
queryable = queryable.OrderBy(selector.SortFieldSelector);
else
queryable = queryable.OrderByDescending(selector.SortFieldSelector);
}
//I need to specify the type of the OrderBy column somehow
// otherwise compiler errors with "arguments cannot be inferred..."
return queryable;
}
The rabbit holes I am exploring are getting pretty ugly and I'm sure there is a simple way to do it.
The fact that I need to use an extension method is a firm requirement, but the rest is still flexible if it simplifies the solution.
Any help is much appreciated.
You can do something like this:
public class SortItem<TEntity>
{
public Expression<Func<TEntity, object>> Selector { get; }
public ListSortDirection Direction { get; }
public SortItem(Expression<Func<TEntity, object>> selector, ListSortDirection direction)
{
Selector = selector;
Direction = direction;
}
}
public static IQueryable<TEntity> MyCustomSortMethod<TEntity>(
this IQueryable<TEntity> queryable,
IEnumerable<SortItem<TEntity>> selectors)
{
foreach (var selector in selectors)
{
IOrderedQueryable<TEntity> ordered = queryable as IOrderedQueryable<TEntity>;
if (ordered == null)
{
if (selector.Direction == ListSortDirection.Ascending)
queryable = queryable.OrderBy(selector.Selector);
else
queryable = queryable.OrderByDescending(selector.Selector);
}
else
{
if (selector.Direction == ListSortDirection.Ascending)
queryable = ordered.ThenBy(selector.Selector);
else
queryable = ordered.ThenByDescending(selector.Selector);
}
}
return queryable;
}
(note the use of ThenBy instead of OrderedBy; if you use OrderBy every time, it just overrides the previous order, instead of completing it)
Disclaimer: I didn't try it, so I'm not sure it actually works with Func<TEntity, object>.
Related
This question already has answers here:
Combining two expressions (Expression<Func<T, bool>>)
(10 answers)
Closed 1 year ago.
I want to create EFCore query that will return all entities that meet some conditions for their related entities.
For example entities look like this (that's pretty simplified example):
public class MyEntity
{
public int Id { get; set; }
public List<MyOtherEntity> OtherEntities { get; set; }
}
public class MyOtherEntity
{
public int Id { get; set; }
public int SomeProperty1 { get; set; }
public int SomeProperty2 { get; set; }
}
And I have a method that takes array of simplified MyOtherEntity objects:
public class MySimpleOtherEntity
{
public int SomeProperty1 { get; set; }
public int SomeProperty2 { get; set; }
}
Now I have some method that takes IEnumerable of these simplified objects, and I want to return all of the MyEntity objects that have in their relations MyOtherEntities that match all of the required conditions:
public IEnumerable<MyEntity> GetMyEntitiesByMyOtherEntities(IEnumerable<MySimpleOtherEntity> entities)
{
// example with some static values
// we want to find all MyEntities that have MyOtherEntity with value 1,2 AND MyOtherEntity with value 2,2
_dataContext
.Where(x => x.OtherEntities.Any(y => y.SomeProperty1 == 1 && y.SomeProperty2 == 2)
&&
x.OtherEntities.Any(y => y.SomeProperty1 == 2 && y.SomeProperty2 == 2)
&&
.
. // and so on
.)
.ToList();
The query above is translated correctly to SQL. I already created a solution with glueing some raw SQL parts that gives correct results, because it's just attaching AND EXISTS parts with proper subqueries.
That being said I would (if possible) rather like to have it as some dynamic LINQ Where expression. SQL parser creates pretty much as good SQL as I would do for this example, but with raw SQL queries I lose some of the control that EFCore gives me.
I created some list of predicates that I would like to chain together and inject into .Where:
public IEnumerable<MyEntity> GetMyEntitiesByMyOtherEntities(IEnumerable<MySimpleOtherEntity> entities)
{
var predicates = new List<Expression<Func<MyEntity, bool>>>();
foreach(var entity in entities)
{
predicates.Add(x => x.OtherEntities.Any(y => y.SomeProperty1 == entity.SomeProperty1
&& y.SomeProperty2 == entity.SomeProperty2);
}
}
Unfortunately I don't know how to chain them properly. I tried to use
var combinedPredicate = predicates.Aggregate((l, r) => Expression.AndAlso(l, r));
But it has some casting issues (probably related to AndAlso returning BinaryExpression?) that won't allow me to do it in such simple way.
How can I achieve that so it's not overly complicated?
Since it's a "And" that should be applied between each condition why you do not use "Where" multiple time ?
var predicates = ...
var myElements = ...
foreach(var predicate in predicate)
{
myElements = myElements.Where(predicate);
}
The aggregatation you tried to do with expression could work but will be a bit more complicated.
EDIT here is how you can do it by aggregating expressions :
var param = predicates.First().Parameters.First();
var body = predicates.Select(s => s.Body).Aggregate(Expression.AndAlso);
var lambda = (Expression<Func<Temp, bool>>)Expression.Lambda(body, param);
So the first part of the code is not so difficult. Let's say you have two predicates :
t => t.Value < 10;
t => t.Value > 5;
The first parameter will be kept (t, I'll explain why later).
Then we extract the body of the expression so we get :
t.Value < 10;
t.Value > 5;
Then we aggregate them with an "And" :
t.Value < 10 && t.Value > 5
Then we create a lambda again :
t => t.Value < 10 && t.Value > 5
So everything seems fine but if you try to compile it you will get an error.
Why? Everything seems OK visually.
It's because the "t" at the beginning and the "t" in the second condition are not the same... They have the same name but they come from different expressions (so different objects were created and the name is not enough to same they are the same...).
In order to solve that you need to check every time the parameter is used to replace it by the same value
You need to implement a "visitor" (from the Visitor pattern) that will inspect the whole expression to replace usage of the parameter :
public static class ExpressionHelper
{
public static Expression<Func<T, bool>> ReplaceParameters<T>(this Expression<Func<T, bool>> expression, ParameterExpression param)
{
return (Expression<Func<T, bool>>)new ReplaceVisitor<T>(param).Modify(expression);
}
private class ReplaceVisitor<T> : ExpressionVisitor
{
private readonly ParameterExpression _param;
public ReplaceVisitor(ParameterExpression param)
{
_param = param;
}
public Expression Modify(Expression expression)
{
return Visit(expression);
}
protected override Expression VisitParameter(ParameterExpression node)
{
return node.Type == typeof(T) ? _param : node;
}
}
}
This implementation is naive and have surely a lot of flaws but in basic cases like this it will be enough I think.
Then you can use it by adding this line to the first block of code :
lambda = lambda.ReplaceParameters(param);
And you can now use it with EF... Or even for an in memory object:
var result = lambda.Compile()(new Temp() {Value = 5});
I'm having product entity:
public class Product : DomainBase
{
public virtual string Name { get; set; }
}
And there should be option to select products by filter, which contains an array of names, like:
public static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var session = Database.OpenSession())
{
var products = session.Query<Product>();
var result = products.Where(product => names.Any(name => product.Name.Contains(name)));
return result.ToList();
}
}
but it throws
System.NotSupportedException: Specified method is not supported.
What is right approach, to accomplish such filtering?
Without knowing more about what database you're connecting to or what library (is it RavenDB.. having done a quick Google?) then it's hard to be completely sure what the problem is.
However, what I think is happening is that you are giving an expression to the IQueryable "Where" extension method and the library is trying to turn that into search criteria to run against the db.. and failing because "Any" is not supported in nested criteria like that (again, I'm guessing).
The LINQ expressions that may or may not be translated into the database language (eg. SQL) vary by the library that performs the translation and vary by the database being talked to.
For example, the following (which is basically what you want to do) works fine with Entity Framework:
private static void Test(IEnumerable<string> names)
{
using (var context = new NORTHWNDEntities())
{
foreach (var product in context.Products.Where(product => names.Any(name => product.ProductName.Contains(name))))
{
Console.WriteLine(product.ProductName);
}
}
Console.ReadLine();
}
One easy option for you is to change your code to
public static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var session = Database.OpenSession())
{
var products = session.Query<Product>();
return result = products.ToList().Where(product => names.Any(name => product.Name.Contains(name)));
}
}
This should work.. however, it will get all Products from the database and perform the filtering in-memory. This is less efficient than getting the database to perform the search.
An alternative would be to generate an "Expression<Func<Product, bool>>" filter yourself that is easier for the library that you're using to translate. If, instead, of a nested "Any" criteria, you could generate a simple set of "OR" name checks then there is a better change of it working. The following will achieve that - but it's quite a lot of code. If this is something that you need to do in several places then some of the code could be made more general and reused.
private static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var context = new NORTHWNDEntities())
{
return context.Products.Where(GetCombinedOrFilter(names)).ToList();
}
}
private static Expression<Func<Product, bool>> GetCombinedOrFilter(IEnumerable<string> names)
{
var filter = GetNameFilter(names.First());
foreach (var name in names.Skip(1))
filter = CombineFiltersAsOr(filter, GetNameFilter(name));
return filter;
}
private static Expression<Func<Product, bool>> GetNameFilter(string name)
{
return product => product.ProductName.Contains(name);
}
private static Expression<Func<Product, bool>> CombineFiltersAsOr(Expression<Func<Product, bool>> x, Expression<Func<Product, bool>> y)
{
// Combine two separate expressions into one by combining as "Or". In order for this to work, instead of there being a parameter
// for each expression, the parameter from the first expression must be shared between them both (otherwise things will go awry
// when this is translated into a database query) - this is why ParameterRebinder.ReplaceParameters is required.
var expressionParameter = x.Parameters.Single();
return Expression.Lambda<Func<Product, bool>>(
Expression.Or(x.Body, ParameterRebinder.ReplaceParameters(y.Body, toReplace: y.Parameters.Single(), replaceWith: expressionParameter)),
expressionParameter
);
}
// Borrowed and tweaked from https://blogs.msdn.microsoft.com/meek/2008/05/02/linq-to-entities-combining-predicates/
public sealed class ParameterRebinder : ExpressionVisitor
{
public static Expression ReplaceParameters(Expression expression, ParameterExpression toReplace, ParameterExpression replaceWith)
{
return new ParameterRebinder(toReplace, replaceWith).Visit(expression);
}
private readonly ParameterExpression _toReplace, _replaceWith;
private ParameterRebinder(ParameterExpression toReplace, ParameterExpression replaceWith)
{
_toReplace = toReplace;
_replaceWith = replaceWith;
}
protected override Expression VisitParameter(ParameterExpression p)
{
if (p == _toReplace)
p = _replaceWith;
return base.VisitParameter(p);
}
}
Update: I didn't notice your nhibernate tag - whoops! Using the criteria combining methods that nhibernate has is probably easier than all this.. :) I would have commented on your answer rather than updating my own but I haven't got the requisite 50 rep yet..
You are trying to mix both kinds of conditions and applying IEnumerable methods on string properties.
Your query should look like this:
var result = products.Where(product => names.Contains(product.Name));
to find exact matches.
For a combination of exact matches and StartsWith it should look like this:
var results = products.Where(product => (names.Contains(product.Name) || names.Any(name => name.StartsWith(product.Name))));
As I did a dive into NHibenrate documentation, it contains CriteriaAPI, so I came up to this
using (var session = Database.OpenSession())
{
var products = session.CreateCriteria<Product>();
if (names == null)
{
return products.List<Product>();
}
var orClause = Expression.Disjunction();
foreach (var name in names)
{
orClause.Add(Expression.Like(nameof(Product.Name), name, MatchMode.Start));
}
products.Add(orClause);
return products.List<Product>();
}
My (EF db first) type sits behind interface IPolicyNumber. I get IQueryable<T> and want to check what I got is correct type (does this table is searchable by that column which is determined by having that interface). Currently I am using typeof(IPolicyNumber).IsAssignableFrom(typeof(T)) which is a bit old school, I was wondering if there was a way to use something like:
IQueryable<T>.ElementType is IPolicyNumber
Full method is below:
public static IQueryable<T> ApplySearch<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
if (search != null && search.PolicyNumber.HasValue && typeof(IPolicyNumber).IsAssignableFrom(typeof(T)))
{
queryable = queryable.SearchByPolicyNumber(search);
}
return queryable;
}
public static IQueryable<IPolicyNumber> SearchByPolicyNumber<IPolicyNumber>(this IQueryable<IPolicyNumber> queryable, SearchModel search)
{
var policyNumberParameterLambda = Expression.Parameter((typeof(IPolicyNumber)));
var policyNumberColumnLambda = Expression.Property(policyNumberParameterLambda, "POLICY_NO");
var lambda = Expression.Lambda<Func<IPolicyNumber, bool>>(
Expression.Equal(policyNumberColumnLambda,
Expression.Convert(Expression.Constant(search.PolicyNumber), policyNumberColumnLambda.Type)
), policyNumberParameterLambda);
return queryable.Where(lambda);
}
Syntax to match interface to type while using IQueryable<T>.ElementType would be following:
typeof(IPolicyNumber).IsAssignableFrom(queryable.ElementType)
I am working with EF6 and am using db first generated models for MSSQL and Oracle. In few places I am searching by multiple search criteria which results in UNION ALL sql generated where each query is being in it's own sub-select.
One of columns in Oracle table is CLOB and linq to sql after it wraps all selects with UNION ALL at the top of all UNIONS it calls SELECT DISTINCT
"UnionAll1"."UNIQUE_ID" AS "C1", ... which requires to compare CLOBs and fails on Oracle side.
ORA-00932: inconsistent datatypes: expected - got CLOB
Description: An unhandled exception occurred during the execution of
the current web request. Please review the stack trace for more
information about the error and where it originated in the code.
Exception Details: Oracle.ManagedDataAccess.Client.OracleException:
ORA-00932: inconsistent datatypes: expected - got CLOB
Is there a way to remove that DISTINCT statement? How can I make this work?
UPDATE
Mechanism that generates LINQ looks like this:
public static IQueryable<T> ApplySearch<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
var subQueries = new List<IQueryable<T>>();
if (search != null)
{
if (search.PolicyNumber.HasValue && typeof (IPolicyNumber).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByPolicyNumber(search));
}
if (search.UniqueId.HasValue && typeof (IUniqueId).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByUniqueId(search));
}
if (!string.IsNullOrWhiteSpace(search.PostCode) && typeof(IPostCode).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByPostCode(search));
}
}
return subQueries.DefaultIfEmpty(queryable)
.Aggregate((a, b) => a.Union(b));
}
Example of specific search method:
public static IQueryable<IRequestId> SearchByRequestId<IRequestId>(this IQueryable<IRequestId> queryable, SearchModel search)
{
var interfacesToColumnNames = new Dictionary<Type, string>
{
{typeof (IRequestId<>), "requestid"},
{typeof (IRequest_Id<>), "request_id"},
};
var paramLambda = Expression.Parameter(typeof (IRequestId));
var columnLambda = Expression.Property(paramLambda, interfacesToColumnNames.Single(o => queryable.ElementType.GetInterfaces().Any(oo => oo.Name == o.Key.Name)).Value);
var lambda = Expression.Lambda<Func<IRequestId, bool>>(
Expression.Equal(columnLambda, Expression.Convert(Expression.Constant(search.RequestId), columnLambda.Type)), paramLambda);
queryable = queryable.Where(lambda);
return queryable;
}
Example where it gets called in controller:
public ActionResult QUOTE_HOUSE()
{
var onlineDocs =
this.DatabaseManager.GetEntities<QUOTE_HOUSE>().ApplySearch(Search)
.Take(10);
return View("QUOTE_HOUSE", onlineDocs.ToList());
}
Based on the additional information from the comments, the problematic queries are produced by the following procedure:
public static IQueryable<T> ApplySearch<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
var subQueries = new List<IQueryable<T>>();
if (search != null)
{
if (search.PolicyNumber.HasValue && typeof (IPolicyNumber).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByPolicyNumber(search));
}
if (search.UniqueId.HasValue && typeof (IUniqueId).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByUniqueId(search));
}
if (!string.IsNullOrWhiteSpace(search.PostCode) && typeof(IPostCode).IsAssignableFrom(queryable.ElementType))
{
subQueries.Add(queryable.SearchByPostCode(search));
}
}
return subQueries.DefaultIfEmpty(queryable)
.Aggregate((a, b) => a.Union(b));
}
where I assume the supporting methods are something like this
public static IQueryable<T> SearchByPolicyNumber<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
return queryable.Where(x => predicate_using_PolicyNumber(x, search));
}
public static IQueryable<T> SearchByUniqueId<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
return queryable.Where(x => predicate_using_UniqueId(x, search));
}
public static IQueryable<T> SearchByPostCode<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
return queryable.Where(x => predicate_using_PostCode(x, search));
}
The problem is that EF translates the LINQ Union operator to SQL UNION ALL subquery with applied DISTINCT SELECT ... as you already found. I have no idea why it does it this way instead of simply translating it to SQL UNION, but actually there is no guarantee that it would work with such type of columns either.
The only way to solve the issue I see is to eliminate the Union operator by replacing it with a single Where with Or conditions. In order to do that, you have to slightly change your design.
First, extract the predicate part from the supporting methods:
public static class SearchPredicates
{
public static Expression<Func<T, bool>> ByPolicyNumber<T>(SearchModel search) where T : class
{
return x => predicate_using_PolicyNumber(x, search);
}
public static Expression<Func<T, bool>> ByUniqueId<T>(SearchModel search) where T : class
{
return x => predicate_using_UniqueId(x, search);
}
public static Expression<Func<T, bool>> ByPostCode<T>(SearchModel search) where T : class
{
return x => predicate_using_PostCode(x, search);
}
}
Then modify the main method like this:
public static IQueryable<T> ApplySearch<T>(this IQueryable<T> queryable, SearchModel search) where T : class
{
var predicates = new List<Expression<<Func<T, bool>>>();
if (search != null)
{
if (search.PolicyNumber.HasValue && typeof (IPolicyNumber).IsAssignableFrom(queryable.ElementType))
predicates.Add(SearchPredicates.ByPolicyNumber(search));
if (search.UniqueId.HasValue && typeof (IUniqueId).IsAssignableFrom(queryable.ElementType))
predicates.Add(SearchPredicates.ByUniqueId(search));
if (!string.IsNullOrWhiteSpace(search.PostCode) && typeof(IPostCode).IsAssignableFrom(queryable.ElementType))
predicates.Add(SearchPredicates.ByPostCode(search));
}
if (predicates.Count == 0)
return queryable;
var parameter = predicates[0].Parameters[0];
var condition = predicates[0].Body;
for (int i = 1; i < predicates.Count; i++)
condition = Expression.Or(condition, predicates[i].Body.ReplaceParameter(predicates[i].Parameters[0], parameter));
var predicate = Expression.Lambda<Func<T, bool>>(condition, parameter);
return queryable.Where(predicate);
}
You can use any EF compatible predicate builder, here I'm building the predicate manually. The helper method used is:
public static class ExpressionUtils
{
public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
{
return new ParameterReplacer { Source = source, Target = target }.Visit(expression);
}
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Source;
public Expression Target;
protected override Expression VisitParameter(ParameterExpression node)
{
return node == Source ? Target : base.VisitParameter(node);
}
}
}
With all that applied, hopefully the issue will be resolved.
Just figured an alternative.
return subQueries.DefaultIfEmpty(queryable)
.Aggregate((a, b) => a.Concat(b));
then after ToList() where you are actually using output .Distinct(new YourEqualityComparer())). To filter out possible duplicates.
This isn't ideal solution since requires you to manually implement IEquitable on structures, but if you are after performance it is likely to be faster. OR will require composite indexes on database while querying by two separate single column indexes then combining the data will be not require to have all possible column-index combinations on table. Another downside you are likely to use Take() before ToList() and then after filtering you might get to few records (and have to re-query).
I have a set of POCOs, all of which implement the following simple interface:
interface IIdObject
{
int Id { get; set; }
}
A subset of these POCOs implement this additional interface:
interface IDeletableObject : IIdObject
{
bool IsDeleted { get; set; }
}
I have a repository hierarchy that looks something like this:
IRepository<T> <: BasicRepository<T> <: ValidatingRepository<T> (where T is IIdObject)
I'm trying to add a FilteringRepository to the hierarchy such that all of the POCOs that implement IDeletableObject have a Where(p => p.IsDeleted == false) filter applied before any other queries take place. My goal is to avoid duplicating the hierarchy solely for IDeletableObjects.
My first attempt looked like this:
public override IQueryable<T> Query()
{
return base.Query().Where(t => ((IDeletableObject)t).IsDeleted == false);
}
This works well with LINQ to Objects, but when I switch to an EF backend I get: "LINQ to Entities only supports casting Entity Data Model primitive types."
I went on to try some fancier parameterized solutions, but they ultimately failed because I couldn't make T covariant in the following case for some reason I don't quite understand:
interface IQueryFilter<out T> // error
{
Expression<Func<T, bool>> GetFilter();
}
I'd be happy to go into more detail on my more complicated solutions if it would help, but I think I'll stop here for now in hope that someone might have an idea for me to try.
Thanks very much in advance!
This is too big for comment, so...
You can create expressions dynamically. I've created helper methods:
public static class ExpressionHelper
{
public static MemberExpression PropertyExpression(this Expression expr,string propertyName)
{
var properties = propertyName.Split('.');
MemberExpression expression = null;
foreach (var property in properties)
{
if (expression == null)
expression = Expression.Property(expr, property);
else
expression = Expression.Property(expression, property);
}
return expression;
}
public static BinaryExpression EqualExpression<T>(this Expression expr, string propertyName, T value)
{
return Expression.Equal(expr.PropertyExpression(propertyName), Expression.Constant(value, typeof(T)));
}
}
Then you can use:
//Checking if T implements IDeletableObject
if (typeof(IDeletableObject).IsAssignableFrom(typeof(T)))
{
//a
var parameter = Expression.Parameter(typeof(T), "a");
//a.IsDeleted == false
var where = parameter.EqualExpression("IsDeleted", false);
//a => a.IsDeleted == false
var condition = Expression.Lambda<Func<T, bool>>(where, parameter);
list = list.Where(condition);
}
EDIT
You can also use Dynamic Linq Library. It uses expressions too, but doesn't force you to think about how it all works, just write simple conditions as string. I don't know how it handles bool values.