I'm having a go at making a flexible exporter to export info from a SQL db accessed via LINQ to SQL, whereby the program can dynamically choose which fields to select and do all the processing server side.
The final aim is to have a simple statement like:
var result = db.Product.Select(p => selector(p));
Where selector is a dynamically created Expression Tree describing the fields to select. Currently, I have each of the db fields assigned to it's own individual selector like:
Expression<Func<Product, string>> name = p => p.Name;
Expression<Func<Product, string>> createdAt = p => p.createdAt.ToString();
At the moment, I'm taking the chosen fields and trying to make one concatenated expression out of them that returns a comma delimited string result from the select statement, something like:
Expression<
Func<
Func<Product, string>,
Func<Product, string>,
Func<Product, string>>> combinator = (a, b) => (p) => a(p) + "," + b(p);
// So I can do something like...
Expression<Func<Product, string>> aggregate = p => "";
foreach (var field in fieldsToSelect)
aggregate = combinator(aggregate, field);
// This would create...
Expression<Func<Products, string>> aggregate = p => a(p) + "," + b(p);
Once I've built up my selector with however many fields, I can execute it in the statement and all the processing is done on the server. However, I've been unable to properly create an expression to concatenate the two child functions in such a manner that the result isn't a Func that's simply executed after the round trip to fetch the results:
var joinedSelector = combinator.Compile();
Func<Products, string> result = joinedSelector(firstSelector.Compile(), secondSelector.Compile());
var query = db.Product.Select(p => result(p)).ToList();
From my limited understanding of Expression Trees, this doesn't actually result in one as the statements are compiled to normal Funcs. I've looked at Expression.Coalesce() but am not sure if it's what I'm after (think it just does "??").
I'm pretty new to this sort of stuff, and would appreciate any help.
Even if people can think of better ways to attack the original problem, I'd still quite like an explanation of how to achieve what I'm trying to do just for the sake of learning how to use Expression Trees.
So you're looking to create a Combine method that can combine the results of two selectors. To do this we'll need a method that accepts two functions with the same input and same output, and then a function accepting two instances of that output type and returning a new value.
The function will need to replace all instances of the parameters of the selectors' body with a common parameter. It will then replace the two parameters of the result function with the corresponding bodies of the different selectors. Then we just wrap all of that up in a lambda.
public static Expression<Func<T, TResult>> Combine
<T, TIntermediate1, TIntermediate2, TResult>(
this Expression<Func<T, TIntermediate1>> first,
Expression<Func<T, TIntermediate2>> second,
Expression<Func<TIntermediate1, TIntermediate2, TResult>> resultSelector)
{
var param = Expression.Parameter(typeof(T));
var body = resultSelector.Body.Replace(
resultSelector.Parameters[0],
first.Body.Replace(first.Parameters[0], param))
.Replace(
resultSelector.Parameters[1],
second.Body.Replace(second.Parameters[0], param));
return Expression.Lambda<Func<T, TResult>>(body, param);
}
This uses the following method to replace all instances of one expression with another:
public static Expression Replace(this Expression expression,
Expression searchEx, Expression replaceEx)
{
return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
}
internal class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression from, to;
public ReplaceVisitor(Expression from, Expression to)
{
this.from = from;
this.to = to;
}
public override Expression Visit(Expression node)
{
return node == from ? to : base.Visit(node);
}
}
Now we can write the following:
Expression<Func<Product, string>> name = p => p.Name;
Expression<Func<Product, string>> createdAt = p => p.createdAt.ToString();
Expression<Func<Product, string>> aggregate =
Combine(name, createdAt, (a, b) => a + "," + b);
It actually comes out a bit simpler than in your mockup, as the result selector doesn't need to know anything about how its inputs are generated, or that they're actually selectors. This solution also allows for each selector to select out different types, and for them to differ from the result, simply because there's no real cost to adding all of this when the generic arguments are just going to be inferred.
And with this in place you can even easily aggregate an arbitrary number of selectors, given the type restrictions you've put in place:
IEnumerable<Expression<Func<Product, string>>> selectors = new[]{
name,
createdAt,
name,
};
var finalSelector = selectors.Aggregate((first, second) =>
Combine(first, second, (a, b) => a + "," + b));
This would let you, for example, have a params method accepting any number of selectors (of a common input and output type) and be able to aggregate all of their results together.
Related
I want to dynamically build a LINQ query so I can do something like
var list = n.Elements().Where(getQuery("a", "b"));
instead of
var list = n.Elements().Where(e => e.Name = new "a" || e.Name == "c");
(Most of the time, I need to pass XNames with namespaces, not just localnames...)
My problem is in accessing the array elements:
private static Func<XElement, bool> getQuery(XName[] names)
{
var param = Expression.Parameter(typeof(XElement), "e");
Expression exp = Expression.Constant(false);
for (int i = 0; i < names.Length; i++)
{
Expression eq = Expression.Equal(
Expression.Property(param, typeof(XElement).GetProperty("Name")!.Name),
/*--->*/ Expression.Variable(names[i].GetType(), "names[i]")
);
}
var lambda = Expression.Lambda<Func<XElement, bool>>(exp, param);
return lambda.Compile();
}
Obviously the Variable expression is wrong, but I'm having difficulty building an expression capable of accessing the array values.
Do you need to create an expression and compile it? Unless I'm missing some nuance to this, all you need is a function that returns a Func<XElement, bool>.
private Func<XElement, bool> GetQuery(params string[] names)
{
return element => names.Any(n => element.Name == n);
}
This takes an array of strings and returns a Func<XElement>. That function returns true if the element name matches any of the arguments.
You can then use that as you described:
var list = n.Elements.Where(GetQuery("a", "b"));
There are plenty of ways to do something like this. For increased readability an extension like this might be better:
public static class XElementExtensions
{
public static IEnumerable<XElement> WhereNamesMatch(
this IEnumerable<XElement> elements,
params string[] names)
{
return elements.Where(element =>
names.Any(n => element.Name == n));
}
}
Then the code that uses it becomes
var list = n.Elements.WhereNamesMatch("a", "b");
That's especially helpful when we have other filters in our LINQ query. All the Where and other methods can become hard to read. But if we isolate them into their own functions with clear names then the usage is easier to read, and we can re-use the extension in different queries.
If you want to write it as Expression you can do it like so:
public static Expression<Func<Person, bool>> GetQuery(Person[] names)
{
var parameter = Expression.Parameter(typeof(Person), "e");
var propertyInfo = typeof(Person).GetProperty("Name");
var expression = names.Aggregate(
(Expression)Expression.Constant(false),
(acc, next) => Expression.MakeBinary(
ExpressionType.Or,
acc,
Expression.Equal(
Expression.Constant(propertyInfo.GetValue(next)),
Expression.Property(parameter, propertyInfo))));
return Expression.Lambda<Func<Person, bool>>(expression, parameter);
}
Whether or not you compile the expression is determined by the means you want to achieve. If you want to pass the expression to a query provider (cf. Queryable.Where) and have e.g. the database filter your values, then you may not compile the expression.
If you want to filter a collection in memory, i.e. you enumerate all elements (cf. Enumerable.Where) and apply the predicate to all the elements, then you have to compile the expression. In this case you should probably not use the Expression api as this adds complexity to your code and you are then more vulnerable to runtime errors.
(this is for .Net Framework 4.7)
I'm trying to write up some extension methods to aid in creating dynamic where clauses for various entities. I started a few days ago, so there's likely a lot that I don't know and some that I probably misunderstood.
I managed to create one extension method already for filtering by 1 property which works as I expect it to (I did use reflection to get the property, couldn't get it working with an interface - well, without it executing the sql that is).
I can't seem to be able to get this one working for a lambda expression though.
Note, that the solution must not trigger sql execution. Because I was able to write up some variants that "worK', but they will trigger sql execution.
The way I work with this is that once I have the code ready, I start debugging and have the "query" in the watch. And it looks like this (notice the sql code)
Once I step over my FilterString method call, it either turns into a sql result, or I get an exception (with current code), which it shouldn't:
So here's my current code that throws the exception (currently not dealing with the "match" parameter, I am implementing an "equals" call. There will be others like, starts With, like, etc)
The exception is just one of those "type mismatch" having function cannot be passed as param to string Equals or what not.
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
var selector = Expression.Lambda<Func<T, string>>(getItemString, param);
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
prototype.Body.ReplaceParameter(prototype.Parameters[0], selector.Body),
selector.Parameters[0]);
return query.Where(predicate);
}
and the one that executes the sql instead of just generating it
public static IQueryable<T> FilterString<T>(this IQueryable<T> query, Match match,
string criteriaItem, Expression<Func<T, string>> getItemString)
where T : class
{
if (string.IsNullOrEmpty(criteriaItem))
{
return query;
}
var param = Expression.Parameter(typeof(T), "r");
//var value = Expression.Constant(getItemString);
var equals = typeof(string).GetMethod("Equals", new Type[] { typeof(string) });
var item = Expression.Invoke(getItemString, param);
var body = Expression.Call(Expression.Constant(criteriaItem),
equals,
item);
return query.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
calling these is done like so
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.MaybeSomeOtherProp.SomeString);
query = query.FilterString(match, criteria_value, (r) => r.SomeProperty.Name);
This same extension method will be called on any number of different entities, with nay number of different properties and prop names. I guess I could make use of the reflection version I got working and passing in all the property names in some array of some sort, but that is just plain ugly.
So long story short, how can I get this working in the way I explained above, taht is: having the sql generated instead of executed?
Thank you,
Note, the "ReplaceParameter" extension method is the one from here: https://stackoverflow.com/a/39206392/630515
So, you're trying to merge your prototype item => item == criteriaItem. With a passed in string property expression, like (r) => r.SomeProperty.Name to create (r) => r.SomeProperty.Name == criteriaItem.
Expression<Func<string, bool>> prototype = item => item == criteriaItem;
var predicate = Expression.Lambda<Func<T, bool>>(
ReplacingExpressionVisitor.Replace(
prototype.Parameters[0],
getItemString.Body,
prototype.Body),
getItemString.Parameters[0]);
And I think you're trying to do it this way so that criteriaItem is bound to an sql parameter, rather than being inlined as a string constant. But your question was a little hard to follow.
Here is my code:
IQueryable<DAL.TradeCard> data = dc.TradeCard.Include("Address").Include("Vehicle");
string orderNumber = "ORD_NR_2";
Expression<Func<DAL.TradeCard, bool>> whereClause = a => a.orderNumber == orderNumber;
// Expression<Func<DAL.TradeCard, bool>> whereClause = a => a.orderNumber == "ORD_NR_2";
List<DAL.TradeCard> dataAsList = data.Where(whereClause).ToList();
If I use the commented line, then the value of whereClause will look like this:
{a => (a.orderNumber == "ORD_NR_2")}
If instead of the commented line I use the other definiton then the value of whereClause will look like this:
{a => (a.orderNumber ==
value(app_Employee.UI.UserFunctions.LejelentettTetelek+<>c__DisplayClass0).orderNumber)}
This is a problem, because I want to save the whereClause and use it in other places where the orderNumber variable does not exist.
So how to use the value of the orderNumber variable instead of a reference to it in a Linq expression. I want to make "value(app_Employee.UI.UserFunctions.LejelentettTetelek+<>c__DisplayClass0).orderNumber" into "ORD_NR_2".
The code in this blog post (Link is dead, Archived version: https://web.archive.org/web/20160122054419/http://blogs.msdn.com/b/mattwar/archive/2007/08/01/linq-building-an-iqueryable-provider-part-iii.aspx) provides a way to evaluate all sections of an expression into values, in all of the places that it can be done.
First it walks through the expression tree from the bottom up, indicating which objects don't have an parameter objects as any of their children. Then it walk through the tree from the top down, evaluating all expressions to a value that don't have a parameter in them.
We can also create an additional method specifically for an expression representing a Func with one parameter so that you don't need to do the cast when you call it:
public static Expression<Func<TIn, TOut>> Simplify<TIn, TOut>(
this Expression<Func<TIn, TOut>> expression)
{
return (Expression<Func<TIn, TOut>>)Evaluator.PartialEval(expression);
}
This allows you to write:
string orderNumber = "ORD_NR_2";
Expression<Func<DAL.TradeCard, bool>> whereClause = a => a.orderNumber == orderNumber;
string foo = whereClause.Simplify().ToString();
//foo will be "{a => (a.orderNumber == "ORD_NR_2")}"
I'm using Entity Framework Code First with SQL Server, with a domain entity that is similar to this:
public class Item
{
public ICollection<ItemLocation> ItemLocations { get; set; }
}
An item can be assigned to many locations throughout it's life, but only one can be active at any time, and we use this to get the actual location of the item:
public Location
{
get
{
return ItemLocations.Where(x => x.IsActive).Select(x => x.Location).FirstOrDefault()
}
}
This property works as expected if I load the entire item object:
var item = (from i in db.Items select i).FirstOrDefault();
Console.WriteLine(item.Location.Name);
However, I can't use this in my LINQ queries where I need to return an anonymous type, like this:
var items = from i in db.Items
select new
{
ItemId = i.ItemId,
LocationName = i.Location.Name
};
Instead, I have to use the full query every time:
var items = from i in db.Items
select new
{
ItemId = i.ItemId,
LocationName = i.ItemLocations.Where(x => x.IsActive).Select(x => x.Location).FirstOrDefault().Name
};
Ideally, I'd like to keep the logic for retrieving an item location in one place (like a property), rather than having to scatter these all over the place.
What is the best way to achieve this?
So to start with, if we want to be able to combine this sub-query with another query then we need to define it as an Expression object, rather than as C# code. If it has already been compiled into IL code then the query provider cannot inspect it to look at what operations are performed and translate that into SQL code. Creating an Expression representing this operation is easy enough:
public static readonly Expression<Func<Item, ItemLocation>> LocationSelector =
item => item.ItemLocations.Where(x => x.IsActive)
.Select(x => x.Location)
.FirstOrDefault();
Now that we have an expression to get a location from an item, we need to combine that with your custom expression for selecting out an anonymous object from an item, using this location. To do this we'll need a Combine method that can take one expression selecting an object into another object, as well as another expression that takes the original object, the result of the first expression, and computes a new result:
public static Expression<Func<TFirstParam, TResult>>
Combine<TFirstParam, TIntermediate, TResult>(
this Expression<Func<TFirstParam, TIntermediate>> first,
Expression<Func<TFirstParam, TIntermediate, TResult>> second)
{
var param = Expression.Parameter(typeof(TFirstParam), "param");
var newFirst = first.Body.Replace(first.Parameters[0], param);
var newSecond = second.Body.Replace(second.Parameters[0], param)
.Replace(second.Parameters[1], newFirst);
return Expression.Lambda<Func<TFirstParam, TResult>>(newSecond, param);
}
Internally, this simply replaces all instances of the parameter of the second expression with the body of the first; the rest of the code is simply ensuring a single parameter throughout and wrapping the result back into a new lambda. This code depends on the ability to replace all instances of one expression with another, which we can do using:
public static Expression Replace(this Expression expression,
Expression searchEx, Expression replaceEx)
{
return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
}
internal class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression from, to;
public ReplaceVisitor(Expression from, Expression to)
{
this.from = from;
this.to = to;
}
public override Expression Visit(Expression node)
{
return node == from ? to : base.Visit(node);
}
}
Now that we have our Combine method all we need to do is call it:
db.Items.Select(Item.LocationSelector.Combine((item, location) => new
{
ItemId = item.ItemId,
LocationName = location.Name
}));
And voila.
If we wanted, we could print out the expression generated by the call to Combine instead of passing it to Select. Doing that, it prints out:
param => new <>f__AnonymousType3`2(ItemId = param.ItemId,
LocationName = param.ItemLocations.Where(x => x.IsActive)
.Select(x => x.Location).FirstOrDefault().Name)
(whitespace added by myself)
That is exactly the query that you had specified out manually, however here we're re-using the existing sub-query without needing to type it out every single time.
How can I build an expression tree when parts of the expression are passed as arguments?
E.g. what if I wanted to create expression trees like these:
IQueryable<LxUser> test1(IQueryable<LxUser> query, string foo, string bar)
{
query=query.Where(x => x.Foo.StartsWith(foo));
return query.Where(x => x.Bar.StartsWith(bar));
}
but by creating them indirectly:
IQueryable<LxUser> test2(IQueryable<LxUser> query, string foo, string bar)
{
query=testAdd(query, x => x.Foo, foo);
return testAdd(query, x => x.Bar, bar);
}
IQueryable<T> testAdd<T>(IQueryable<T> query,
Expression<Func<T, string>> select, string find)
{
// how can I combine the select expression with StartsWith?
return query.Where(x => select(x) .. y => y.StartsWith(find));
}
Result:
While the samples didn't make much sense (sorry but I was trying to keep it simple), here's the result (thanks Quartermeister).
It can be used with Linq-to-Sql to search for a string that starts-with or is equal to the findText.
public static IQueryable<T> WhereLikeOrExact<T>(IQueryable<T> query,
Expression<Func<T, string>> selectField, string findText)
{
Expression<Func<string, bool>> find;
if (string.IsNullOrEmpty(findText) || findText=="*") return query;
if (findText.EndsWith("*"))
find=x => x.StartsWith(findText.Substring(0, findText.Length-1));
else
find=x => x==findText;
var p=Expression.Parameter(typeof(T), null);
var xpr=Expression.Invoke(find, Expression.Invoke(selectField, p));
return query.Where(Expression.Lambda<Func<T, bool>>(xpr, p));
}
e.g.
var query=context.User;
query=WhereLikeOrExact(query, x => x.FirstName, find.FirstName);
query=WhereLikeOrExact(query, x => x.LastName, find.LastName);
You can use Expression.Invoke to create an expression that represents applying one expression to another, and Expression.Lambda to create a new lambda expression for the combined expression. Something like this:
IQueryable<T> testAdd<T>(IQueryable<T> query,
Expression<Func<T, string>> select, string find)
{
Expression<Func<string, bool>> startsWith = y => y.StartsWith(find);
var parameter = Expression.Parameter(typeof(T), null);
return query.Where(
Expression.Lambda<Func<T, bool>>(
Expression.Invoke(
startsWith,
Expression.Invoke(select, parameter)),
parameter));
}
The inner Expression.Invoke represents the expression select(x) and the outer one represents calling y => y.StartsWith(find) on the value returned by select(x).
You could also use Expression.Call to represent the call to StartsWith without using a second lambda:
IQueryable<T> testAdd<T>(IQueryable<T> query,
Expression<Func<T, string>> select, string find)
{
var parameter = Expression.Parameter(typeof(T), null);
return query.Where(
Expression.Lambda<Func<T, bool>>(
Expression.Call(
Expression.Invoke(select, parameter),
"StartsWith",
null,
Expression.Constant(find)),
parameter));
}
This Works:
public IQueryable<T> Add<T>(IQueryable<T> query, Expression<Func<T, string>> Selector1,
Expression<Func<T, string>> Selector2, string data1, string data2)
{
return Add(Add(query, Selector1, data1), Selector2, data2);
}
public IQueryable<T> Add<T>(IQueryable<T> query, Expression<Func<T, string>> Selector, string data)
{
var row = Expression.Parameter(typeof(T), "row");
var expression =
Expression.Call(
Expression.Invoke(Selector, row),
"StartsWith", null, Expression.Constant(data, typeof(string))
);
var lambda = Expression.Lambda<Func<T, bool>>(expression, row);
return query.Where(lambda);
}
You use it like:
IQueryable<XlUser> query = SomehowInitializeIt();
query = Add(query, x => x.Foo, y => y.Bar, "Foo", "Bar");
Usually you don't do that in the way you descirbed (using the IQueryable Interface) but you rather use Expressions like Expression<Func<TResult, T>>. Having said that, you compose higher order functions (such as where or select) into a query and pass in expressions that will "fill in" the desired functionality.
For example, consider the signature of the Enumerable.Where method:
Where<TSource>(IEnumerable<TSource>, Func<TSource, Boolean>)
The function takes a delegate as its second argument that is called on each element. The value you return from that delegate indicates to the higher order function if it shall yield the current element (include it in the result or not).
Now, let's take a look at Queryable.Where:
Queryable.Where<TSource>-Methode (IQueryable<TSource>, Expression<Func<TSource, Boolean>>)
We can observe the same pattern of a higher order function, but instead of an Func<> delegate it takes an Expression. An expression is basically a data representation of your code. Compiling that expression will give you a real (executable) delegate. The compiler does a lot of heavy lifting to build expression trees from lambdas you assign to Expression<...>. Expression trees make it possible to compile the described code against different data sources, such as a SQL Server Database.
To come back to your example, what I think you're looking for is a selector. A selector takes each input element and returns a projection of it. It's signature looks like this: Expression<Func<TResult, T>>. For example you could specify this one:
Expression<Func<int, string>> numberFormatter = (i) => i.ToString(); // projects an int into a string
To pass in a selector, your code would need to look like this:
IQueryable<T> testAdd<T>(IQueryable<T> query, Expression<Func<string, T>> selector, string find)
{
// how can I combine the select expression with StartsWith?
return query.Select(selector) // IQueryable<string> now
.Where(x => x.StartsWith(find));
}
This selector would allow you to project the input string to the desired type. I hope I got your intention corrrectly, it's hard to see what you're trying to achieve.