We Keep Coding

Search a term in multiple columns

I'd like to create an extension to search terms in multiple columns.
Terms are separated with space, each term must appears to at least one given column.
Here what I've done so far:
public static IQueryable<TSource> SearchIn<TSource>(this IQueryable<TSource> query,
string searchText,
Expression<Func<TSource, string>> expression,
params Expression<Func<TSource, string>>[] expressions)
{
if (string.IsNullOrWhiteSpace(searchText))
{
return query;
}
// Concat expressions
expressions = new[] { expression }.Concat(expressions).ToArray();
// Format search text
var formattedSearchText = searchText.FormatForSearch();
var searchParts = formattedSearchText.Replace('\'', ' ').Split(' ');
// Initialize expression
var pe = Expression.Parameter(typeof(TSource), "entity");
var predicateBody = default(Expression);
// Search in each expressions, put OR in between
foreach (var expr in expressions)
{
var exprBody = default(Expression);
// Search for each words, put AND in between
foreach (var searchPart in searchParts)
{
// Create property or field expression
var left = Expression.PropertyOrField(pe, ((MemberExpression)expr.Body).Member.Name);
// Create the constant expression with current word
var search = Expression.Constant(searchPart, typeof(string));
// Create the contains function
var contains = Expression.Call(left, typeof(string).GetMethod(nameof(string.Contains), new Type[] { typeof(string) }), search);
// Check if there already a predicate body
if (exprBody == null)
{
exprBody = contains;
}
else
{
exprBody = Expression.And(exprBody, contains);
}
}
if (predicateBody == null)
{
predicateBody = exprBody;
}
else
{
predicateBody = Expression.OrElse(predicateBody, exprBody);
}
}
// Build the where method expression
var whereCallExpression = Expression.Call(
typeof(Queryable),
nameof(Queryable.Where),
new Type[] { query.ElementType },
query.Expression,
Expression.Lambda<Func<TSource, bool>>(predicateBody, new ParameterExpression[] { pe }));
// Apply the condition to the query and return it
return query.Provider.CreateQuery<TSource>(whereCallExpression);
}
It works well as long as given expressions are simple:
// It works well
query.SearchIn("foo", x => x.Column1, x => x.Column2);
But it does not work when trying to navigate through navigation properties:
// Not working
query.SearchIn("foo", x => x.Nav1.Column1);
It gives me an exception.
'Column1' is not a member of type 'Nav1'.
I understand the problem but I can't find the solution to pass through Nav1.
I need help with this one.

Instead of parsing lambda expression body just call it with given parameter:
var left = Expression.Invoke(expr, pe);
However it works only in EF Core.
In EF6 you would need to get property or field of each nested member like this:
var left = expr.Body.ToString()
.Split('.')
.Skip(1) //skip the original parameter name
.Aggregate((Expression)pe, (a, c) => Expression.PropertyOrField(a, c));
It will work only for simple lambdas like:
x => x.Prop1.Nav1
If that's not enough you would need some more advanced parsing algorithm with ExpressionVisitor for example.

How to Create an Expression tree for .Where(x => x.<deep property>.Select(y => y.id).Intersect(List<int>).Any())

I'm creating a method that receives a Queryable<T> source, a string with a property name/path (could be a deep property for example "TrParent.DataTypes" to achieve this x => x.TrParent.DataTypes) and Enumerable<int> which holds the values I need to intersect.
Basically I come from the need to create the following query dynamically (I mean <DT_Det_Tr> and TrParent.DataTypes being know only at runtime, in the example DT_Det_Tr is not a type it is a class):
var _vals = new List<int>();
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any()
);
Please keep in mind that the preceding query is just an example of what I need to achieve dynamically, what I really need is an expression tree that creates a predicate like the one shown above but using a dynamic type and with the deep navigation property specified within a string.
So, I'm using this function to create the expression for the deep property:
private static LambdaExpression CreateDelegateExpression<T>(out Type resultingtype, string property, string parameterName = "x")
{
var type = typeof(T);
ParameterExpression param = Expression.Parameter(type, parameterName);
Expression expr = param;
foreach (string prop in property.Split('.'))
{
PropertyInfo pi = type.GetProperty(prop);
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
Type delegateType = typeof(Func<,>).MakeGenericType(typeof(T), type);
LambdaExpression lambda = Expression.Lambda(delegateType, expr, param);
resultingtype = type;
return lambda;
}
And here is what I have so far for my function:
public static IQueryable<T> Intersect<T>(this IQueryable<T> source, string property, IEnumerable<int> value)
{
//List of ids
var _value = Expression.Constant(value);
//Get delegate expression to the deep property and it's inner type
Type type = null;
var lambda = CreateDelegateExpression<T>(out type, property, "x");
var enumtype = type.GetGenericArguments()[0];
ParameterExpression tpe = Expression.Parameter(enumtype, "y");
Expression propExp = Expression.Property(tpe, enumtype.GetProperty("Id"));
MethodInfo innermethod = typeof(Queryable).GetMethods().Where(x => x.Name == "Select").First();
//Error on next line...
var selectCall = Expression.Call(typeof(Queryable),
"Select",
new Type[] { enumtype, typeof(long) },
lambda,
propExp);
//TODO: Add rest of logic and actually filter the source
return source;
}
In the var selectCall = line I'm getting error:
No generic method 'Select' on type 'System.Linq.Queryable' is compatible with the supplied type arguments and arguments. No type arguments should be provided if the method is non-generic.
I've read a lot here on SO and other sites but I can't get past this part, I feel I'm going to bump into more trouble when I get to the .Intersect(List<int>).Any() part so any help on that also would be grand, thanks.

After a lot of thought, investigation and attempts I came up with a solution.
First, I made a simpler version of my goal query (the static example I used in my question), so instead of:
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any()
);
I made this:
var res = dbContext.Set<DT_Det_Tr>()
.Where
(x => x.TrParent.DataTypes
.Any(y => _vals.Contains(y.Id))
);
Which is a lot easier to translate to expressions (or at least it was for me) because it omits the Select call.
I got rid of the method I was using to create the deep navigation property expression and streamlined it in my Intersect function, this was because it was doing some work I don't really need here plus I needed access to some of the variables I use inside it, then I made this:
public static IQueryable<T> Intersect<T>(this IQueryable<T> source, string property, IEnumerable<int> value)
{
var type = typeof(T);
var _value = Expression.Constant(value); //List of ids
//Declare parameter for outer lambda
ParameterExpression param = Expression.Parameter(type, "x");
//Outer Lambda
Expression expr = param;
foreach (string prop in property.Split('.')) //Dig for deep property
{
PropertyInfo pi = type.GetProperty(prop);
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
//Get deep property's type
var enumtype = type.GetGenericArguments()[0];
//Declare parameter for inner lambda
ParameterExpression tpe = Expression.Parameter(enumtype, "y");
//Inner Collection lambda logic
//Property for inner lambda
Expression propExp = Expression.Property(tpe, enumtype.GetProperty("Id"));
//Contains method call .Contains(y.Id)
var containsMethodExp = Expression.Call(typeof(Enumerable), "Contains", new[] { propExp.Type }, _value, propExp);
//Create Expression<Func<enumtype, bool>>
var innerDelegateType = typeof(Func<,>).MakeGenericType(enumtype, typeof(bool));
//Create Inner lambda y => _vals.Contains(y.Id)
var innerFunction = Expression.Lambda(innerDelegateType, containsMethodExp, tpe);
//Get Any method info
var anyMethod = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(enumtype);
//Call Any with inner function .Any(y => _vals.Contains(y.Id))
var outerFunction = Expression.Call(anyMethod, expr, innerFunction);
//Call Where
MethodCallExpression whereCallExpression = Expression.Call
(
typeof(Queryable),
"Where",
new Type[] { source.ElementType },
source.Expression,
Expression.Lambda<Func<T, bool>>(outerFunction, new ParameterExpression[] { param })
);
//Create and return query
return source.Provider.CreateQuery<T>(whereCallExpression);
}
I hope this helps anyone trying to develop a similar solution.
Working with expression trees can be very hard and frustrating at first, but it's a really powerful tool once you get the hold of it.

If you have access to the dynamic keyword from c# 4.0, you might be able to work around the problem like this:
var _vals = new List<int>();
var res = dbContext.Set<DT_Det_Tr>()
.Where(obj => { dynamic x = obj;
return x.TrParent.DataTypes
.Select(t => t.Id)
.Intersect(_vals)
.Any();
}
);
But I don't know enough about the details of the problem you want to solve to say for sure.

Building Expression Tree Using a Parameter's Indexer

Given a class that has a property that is a Dictionary
public class Product
{
public Dictionary<string, string> Attributes { get { return attributes; } }
private Dictionary<string, string> attributes = new Dictionary<string, string>();
}
I want to be able to match products in a list of products based on criteria that are retrieved from a data store that are in the format of
Brand == Tyco
Color != Blue
My current approach is to construct an expression from the filter, and then pass that expression as the parameter to a LINQ Where method call like so
products = products.Where(myConstructedExpression);
where myConstructedExpression would normally be a lamda expression that looks like
p => p.Attributes[attribute] == value
I have assembled the following code for testing purposes, but it always fails the call to lambda.Compile() regardless of what I have tried for he left expression.
Dictionary<string, ExpressionType> expressionType = new Dictionary<string, ExpressionType>();
expressionType.Add("==", ExpressionType.Equal);
expressionType.Add("!=", ExpressionType.NotEqual);
string filter = "Brand == Tyco";
string[] fields = filter.Split(' ');
string attribute = fields[0];
string op = fields[1];
string value = fields[2];
Product product = new Product();
product.Attributes.Add("Brand", "Tyco");
var parameter = Expression.Parameter(typeof(Product), "p");
var left = /***** THIS IS WHAT I AM FAILING TO CONSTRUCT PROPERLY ********/
var right = Expression.Constant(value);
var operation = Expression.MakeBinary(expressionType[op], left, right);
var lambda = Expression.Lambda<Func<Product, bool>>(operation, parameter);
var result = lambda.Compile()(product);
Questions
Is this even a reasonable approach, and, if so,
How do I construct the left expression?

So to get p => p.Attributes["Brand"] <someoperator> "Tyco", you can do this.
The "trick", to work with indexed types, is to use their Item property (you could also work with the get_item method)
var parameter = Expression.Parameter(typeof(Product), "p");
Expression left = Expression.Property(parameter, "Attributes");
left = Expression.Property(left, "Item", new Expression[] { Expression.Constant(attribute) });
EDIT
the version with the IDictionary.ContainsKey(<value>) test
really step by step, but I think this makes things clearer at first.
//left part of lambda, p
var parameter = Expression.Parameter(typeof(Product), "p");
//right part
//p.Attributes
Expression left = Expression.Property(parameter, "Attributes");
var method = typeof(IDictionary<string, string>).GetMethod("ContainsKey");
//p.Attributes.ContainsKey("Brand");
Expression containsExpression = Expression.Call(left, method, Expression.Constant(attribute));
//p.Attributes.Item["Brand"]
Expression keyExpression= Expression.Property(left, "Item", new Expression[] { Expression.Constant(attribute) });
//"Tyco"
var right = Expression.Constant(value);
//{p => IIF(p.Attributes.ContainsKey("Brand"), (p.Attributes.Item["Brand"] == "Tyco"), False)}
Expression operation = Expression.Condition(
containsExpression,
Expression.MakeBinary(expressionType[op], keyExpression, right),
Expression.Constant(false));
var lambda = Expression.Lambda<Func<Product, bool>>(operation, parameter);

How to add an AND/OR expression to the following dynamic linq expression

T is a type that may or may not have a specific property, lets say 'City'. For the types that have a property named 'City', I would like to restrict the records such that only residents of the Gotham are returned and they are sorted.
public static IQueryable<T> OrderBy<T>(this IQueryable<T> source, string ordering, params object[] values)
{
var type = typeof(T);
var property = type.GetProperty(ordering);
var parameter = Expression.Parameter(type, "p");
var propertyAccess = Expression.MakeMemberAccess(parameter, property);
var orderByExp = Expression.Lambda(propertyAccess, parameter);
MethodCallExpression resultExp = Expression.Call(
typeof(Queryable),
"OrderBy",
new Type[] { type, property.PropertyType },
source.Expression,
Expression.Quote(orderByExp));
string propertyToRestrictOn = "City";
string restrictedValue = "Gotham";
var restrictedProperty = type.GetProperty(propertyToRestrictOn);
if(null ! = restrictedProperty )
{
// TODO: What to add here so than only those records are returned that have a
// property named City and the value is 'Gotham'???
}
return source.Provider.CreateQuery<T>(resultExp);
}
if possible please name/link some helpful literature here as well just in case I have to create more complex queries i.e. mix And/OR
The code was borrowed from
How do I apply OrderBy on an IQueryable using a string column name within a generic extension method?

I'm not quite sure if I have understood you correctly, but I think I was in the same situation a few months ago.
The posted code here was my solution.
I think you should be especially interested in line 24.
Edit:
PropertyInfo p = ... // I used reflection in my examply to get properties with a certain Attribute
var expressionParameter = Expression.Parameter(typeof(SomeClass), "lambda");
var parameter = new [] { expressionParameter };
var propertyAccess = Expression.Property(expressionParameter, p);
var nullCheck = Expression.NotEqual(propertyAccess, Expression.Constant(null, p.PropertyType));
var nullCheckLambda = Expression.Lambda<Func<SomeClass, Boolean>>(nullCheck, parameter);
var containsMethodInfo = typeof(String).GetMethod("Contains", new[] { typeof(String) });
var contains = Expression.Call(propertyAccess, containsMethodInfo, Expression.Constant("ell"));
var containsLambda = Expression.Lambda<Func<SomeClass, Boolean>>(contains, new[] { expressionParameter });
var predicate = Expression.Lambda<Func<SomeClass, Boolean>>(
// line 24
Expression.AndAlso(nullCheckLambda.Body, containsLambda.Body), parameter);
Console.WriteLine(predicate.ToString());

I think you're making this harder than you have to. In the first part of your code (the OrderBy()), you don't actually need to generate the whole query expression, just the lambda inside it.
And in the second part (the optional Where()) you can do pretty much the same thing, just add Expression.Equal() and Expression.Constant():
public static IQueryable<T> OrderBy<T>(this IQueryable<T> source, string ordering)
{
var type = typeof(T);
var property = type.GetProperty(ordering);
var parameter = Expression.Parameter(type, "p");
var propertyAccess = Expression.MakeMemberAccess(parameter, property);
// necessary for value types to work
var cast = Expression.Convert(propertyAccess, typeof(object));
var orderByExp = Expression.Lambda<Func<T, object>>(cast, parameter);
IQueryable<T> result = source.OrderBy(orderByExp);
string propertyToRestrictOn = "City";
string restrictedValue = "Gotham";
var restrictedProperty = type.GetProperty(propertyToRestrictOn);
if (restrictedProperty != null)
{
var restrictionParameter = Expression.Parameter(type, "p");
var restrictionPropertyAccess =
Expression.MakeMemberAccess(restrictionParameter, restrictedProperty);
var restrictionEquality =
Expression.Equal(restrictionPropertyAccess,
Expression.Constant(restrictedValue));
var whereExp =
Expression.Lambda<Func<T, bool>>(restrictionEquality, restrictionParameter);
result = result.Where(whereExp);
}
return result;
}
Also, if your method is going to do more than just ordering, I think you shouldn't call it OrderBy().

You're half-way there already. You have the ordered expression already, so you're just going to use the Queryable.OrderBy expression call in a Queryable.Where expression (or vice-versa, doesn't really matter).
if(null != restrictedProperty )
{
var notEqualExp = Expression.NotEqual(parameter,
Expression.Constant(restrictedValue, typeof(string)));
resultExp = Expression.Call(
typeof(Queryable),
"Where",
new Type[] { type },
resultExp,
Expression.Lambda(notEqualExp, parameter));
}
Haven't worked with building Expressions by hand in a while, so this is purely being done from memory. However, it should at least get you started and give you something to work with.
P.S. I would actually perform this check BEFORE the OrderBy method call. That way you end up with Queryably.Where(...).OrderBy(...) instead. But I guess if this is getting translated by the provider anyway, then shouldn't matter. However, just something I'd do to reduce any ambiguity of the generated query.

EntitySet<T>.Where(myPredicate) throws NotSupportedException

EDIT: Let's try this again. This time I've used the AdventureWorks sample database so you can all play along. This will rule out anything crazy I've done in my own database. Here's a new example demonstrating what works and what I would expect to work (but doesn't). Can anyone explain why it doesn't work or suggest a different way of achieving my goal (refactoring out the common expression so it can be reused elsewhere)?
using (AdventureWorksDataContext db = new AdventureWorksDataContext())
{
// For simplicity's sake we'll just grab the first result.
// The result should have the name of the SubCategory and an array of Products with ListPrice greater than zero.
var result = db.ProductSubcategories.Select(subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products.Where(product => product.ListPrice > 0).ToArray()
}).First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0.", result.ProductArray.Length, result.Name);
// Output should say: There are 3 products in SubCategory Bib-Shorts with ListPrice > 0.
// This won't work. I want to pull the expression out so that I can reuse it in several other places.
Expression<Func<Product, bool>> expression = product => product.ListPrice > 0;
result = db.ProductSubcategories.Select(subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products.Where(expression).ToArray() // This won't compile because Products is an EntitySet<Product> and that doesn't have an overload of Where that accepts an Expression.
}).First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0.", result.ProductArray.Length, result.Name);
}
</Edit>
The following LINQ to SQL works fine:
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(plan => plan.CancelDate == null && plan.UpgradeDate == null).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
However, I want to break out the predicate passed to the Where since it's used throughout the code. But if I try and pass a defined predicate into the Where it fails, such as:
Func<DatabaseAccess.Maintenance, bool> activePlanPredicate = plan => plan.CancelDate == null && plan.UpgradeDate == null;
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(activePlanPredicate).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
This makes no sense to me. Can anyone explain what's going on? Maintenances is of type EntitySet<DatabaseAccess.Maintenance>. The error I get is:
System.NotSupportedException:
Unsupported overload used for query
operator 'Where'..
EDIT: For those interested, here's what Reflector has for the first (working) example with Optimization set to .NET 2.0:
using (BugsDatabaseDataContext db = new BugsDatabaseDataContext())
{
ParameterExpression CS$0$0001;
ParameterExpression CS$0$0006;
ParameterExpression CS$0$0010;
return db.SubAccounts.Select<SubAccount, ServiceTicket>(Expression.Lambda<Func<SubAccount, ServiceTicket>>(
Expression.MemberInit(
Expression.New(
(ConstructorInfo) methodof(ServiceTicket..ctor),
new Expression[0]),
new MemberBinding[]
{
Expression.Bind(
(MethodInfo) methodof(ServiceTicket.set_MaintenancePlans),
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.ToArray),
new Expression[]
{
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.Select),
new Expression[]
{
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.Where),
new Expression[]
{
Expression.Property(CS$0$0001 = Expression.Parameter(typeof(SubAccount), "subAccount"), (MethodInfo) methodof(SubAccount.get_Maintenances)),
Expression.Lambda<Func<Maintenance, bool>>(
Expression.AndAlso(
Expression.Equal(
Expression.Property(CS$0$0006 = Expression.Parameter(typeof(Maintenance), "plan"), (MethodInfo) methodof(Maintenance.get_CancelDate)),
Expression.Convert(Expression.Constant(null, typeof(DateTime?)), typeof(DateTime?)), false, (MethodInfo) methodof(DateTime.op_Equality)
),
Expression.Equal(
Expression.Property(CS$0$0006, (MethodInfo) methodof(Maintenance.get_UpgradeDate)),
Expression.Convert(Expression.Constant(null, typeof(DateTime?)), typeof(DateTime?)), false, (MethodInfo) methodof(DateTime.op_Equality)
)
),
new ParameterExpression[] { CS$0$0006 }
)
}
),
Expression.Lambda<Func<Maintenance, string>>(
Expression.Call(
CS$0$0010 = Expression.Parameter(typeof(Maintenance), "plan"),
(MethodInfo) methodof(object.ToString),
new Expression[0]
),
new ParameterExpression[] { CS$0$0010 }
)
}
)
}
)
)
}
),
new ParameterExpression[] { CS$0$0001 }
)
).ToList<ServiceTicket>();
}
EDIT: The Reflector output for the second example (using a predicate) is mostly similar. The biggest difference being that, in the call to Enumerable.Where, rather than passing an Expression.Lambda it passes Expression.Constant(activePlanPredicate).

I don't fully understand the guts of Linq to Entities, but there is an Open Source (usable in proprietary software) toolkit specifically designed to help solve this problem, called LinqKit, linked off this O'Reilly-related article:
http://www.albahari.com/nutshell/predicatebuilder.aspx
Since I don't fully understand the guts, I'll just quote them:
Entity Framework's query processing pipeline cannot handle invocation expressions, which is why you need to call AsExpandable on the first object in the query. By calling AsExpandable, you activate LINQKit's expression visitor class which substitutes invocation expressions with simpler constructs that Entity Framework can understand.
Here is a direct link to LinqKit.
And here is the type of code that this project enables:
using LinqKit;
// ...
Expression<Func<Product, bool>> expression = product => product.ListPrice > 0;
var result = db.ProductSubcategories
.AsExpandable() // This is the magic that makes it all work
.Select(
subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products
// Products isn't IQueryable, so we must call expression.Compile
.Where(expression.Compile())
})
.First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0."
, result.ProductArray.Count()
, result.Name
);
The result is:
There are 3 products in SubCategory Bib-Shorts with ListPrice > 0.
Yay, no exception, and we can extract the predicate!

I'd refactor the original like this
private bool IsYourPredicateSatisfied(Maintenance plan)
{
return plan.CancelDate == null && plan.UpgradeDate == null;
}
Then your Where clause is Where(m => IsYourPredicateSatisfied(m))

Try this:
Expression<Func<DatabaseAccess.Maintenance, bool>> activePlanPredicate = plan => plan.CancelDate == null && plan.UpgradeDate == null;
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(activePlanPredicate).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
I don't have VisualStudio in front of me, so that may require some tweaking. The issue you're running into is that you want to access the IQueryable extension of Where, but just having a Func<T,bool> gives you the IEnumerable extension.