I've read this answer and understood from it the specific case it highlights, which is when you have a lambda inside another lambda and you don't want to accidentally have the inner lambda also compile with the outer one. When the outer one is compiled, you want the inner lambda expression to remain an expression tree. There, yes, it makes sense quoting the inner lambda expression.
But that's about it, I believe. Is there any other use case for quoting a lambda expression?
And if there isn't, why do all the LINQ operators, i.e. the extensions on IQueryable<T> that are declared in the Queryable class quote the predicates or lambdas they receive as arguments when they package that information in the MethodCallExpression.
I tried an example (and a few others over the last couple of days) and it doesn't seem to make any sense to quote a lambda in this case.
Here's a method call expression to a method that expects a lambda expression (and not a delegate instance) as its only parameter.
I then compile the MethodCallExpression by wrapping it inside a lambda.
But that doesn't compile the inner LambdaExpression (the argument to the GimmeExpression method) as well. It leaves the inner lambda expression as an expression tree and does not make a delegate instance of it.
In fact, it works well without quoting it.
And if I do quote the argument, it breaks and gives me an error indicating that I am passing in the wrong type of argument to the GimmeExpression method.
What's the deal? What's this quoting all about?
private static void TestMethodCallCompilation()
{
var methodInfo = typeof(Program).GetMethod("GimmeExpression",
BindingFlags.NonPublic | BindingFlags.Static);
var lambdaExpression = Expression.Lambda<Func<bool>>(Expression.Constant(true));
var methodCallExpression = Expression.Call(null, methodInfo, lambdaExpression);
var wrapperLambda = Expression.Lambda(methodCallExpression);
wrapperLambda.Compile().DynamicInvoke();
}
private static void GimmeExpression(Expression<Func<bool>> exp)
{
Console.WriteLine(exp.GetType());
Console.WriteLine("Compiling and executing expression...");
Console.WriteLine(exp.Compile().Invoke());
}
You have to pass the argument as a ConstantExpression:
private static void TestMethodCallCompilation()
{
var methodInfo = typeof(Program).GetMethod("GimmeExpression",
BindingFlags.NonPublic | BindingFlags.Static);
var lambdaExpression = Expression.Lambda<Func<bool>>(Expression.Constant(true));
var methodCallExpression =
Expression.Call(null, methodInfo, Expression.Constant(lambdaExpression));
var wrapperLambda = Expression.Lambda(methodCallExpression);
wrapperLambda.Compile().DynamicInvoke();
}
private static void GimmeExpression(Expression<Func<bool>> exp)
{
Console.WriteLine(exp.GetType());
Console.WriteLine("Compiling and executing expression...");
Console.WriteLine(exp.Compile().Invoke());
}
The reason should be pretty obvious - you're passing a constant value, so it has to be a ConstantExpression. By passing the expression directly, you're explicitly saying "and get the value of exp from this complicated expression tree". And since that expression tree doesn't actually return a value of Expression<Func<bool>>, you get an error.
The way IQueryable works doesn't really have much to do with this. The extension methods on IQueryable have to preserve all information about the expressions - including the types and references of the ParameterExpressions and similar. This is because they don't actually do anything - they just build the expression tree. The real work happens when you call queryable.Provider.Execute(expression). Basically, this is how the polymorphism is preserved even though we're doing composition, rather than inheritance (/interface implementation). But it does mean that the IQueryable extension methods themselves cannot do any shortcuts - they don't know anything about the way the IQueryProvider is actually going to interpret the query, so they can't throw anything away.
The most important benefit you get from this, though, is that you can compose the queries and subqueries. Consider a query like this:
from item in dataSource
where item.SomeRelatedItem.Where(subItem => subItem.SomeValue == 42).Count() > 2
select item;
Now, this is translated to something like this:
dataSource.Where(item => item.SomeRelatedItem.Where(subItem => subItem.SomeValue == 42).Count() > 2);
The outer query is pretty obvious - we'll get a Where with the given predicate. The inner query, however, is actually going to be a Call to Where, taking the actual predicate as an argument.
By making sure that actual invocations of the Where method are actually translated into a Call of the Where method, both of these cases become the same, and your LINQProvider is that one bit simpler :)
I've actually written LINQ providers that don't implement IQueryable, and which actually have some useful logic in the methods like Where. It's a lot simpler and more efficient, but has the drawback described above - the only way to handle subqueries would be to manually Invoke the Call expressions to get the "real" predicate expression. Yikes - that's quite an overhead for a simple LINQ query!
And of course, it helps you compose different queryable providers, although I haven't actually seen (m)any examples of using two completely different providers in a single query.
As for the difference between Expression.Constant and Expression.Quote themselves, they seem rather similar. The crucial difference is that Expression.Constant will treat any closures as actual constants, rather than closures. Expression.Quote on the other hand, will preserve the "closure-ness" of the closures. Why? Because the closure objects themselves are also passed as Expression.Constant :) And since IQueryable trees are doing lambdas of lambdas of lambdas of [...], you really don't want to lose the closure semantics at any point.
Related
In one of my projects I have an ExpressionVisitor to translate provided expression into some query string. But before translating it I need to evaluate all refferences in the expression to real values. To do that I use Evaluator.PartialEval method from EntityFramework Project.
Assuming I have this query:
var page = 100;
var query = myService.AsQueryable<Product>()
//.Where(x=>x.ProductId.StartsWith(p.ProductId))
.Skip(page)
.Take(page);
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression);
As you can see I have commented Where method. In this case evaluatedQueryExpr will not contain the methods Take and Skip.
However, if I use any other method with Expression before Take or Skip everything works, Evaluator evaluates an expression correctly and return it fully.
I found out that the problem occurs in the line 80 of the Evaluator class:
return Expression.Constant(fn.DynamicInvoke(null), e.Type);
Could you explain why this happens and suggest a workaround?
Update
here is a project on github
LinqToSolrQueriable inherited from IOrderedQueryable
LinqToSolrProvider inherited from IQueryProvider including line range causing the issue
The good news are that the expression is not really reduced (Skip and Take are still there :), but is simply converted from MethodCallExpression to ConstantExpression containing the original expression:
query.Expression:
.Call System.Linq.Queryable.Take(
.Call System.Linq.Queryable.Skip(
.Constant<LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]),
100),
100)
evaluatedQueryExpr:
.Constant<System.Linq.IQueryable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product])
Here the debug display is giving you a wrong impression. If you take the ConstaintExpression.Value, you'll see that it's a IQueryable<Product> with Expression property being exactly the same as the original query.Expression.
The bad news are that this is not what you expect from PartialEval - in fact it doesn't do anything useful in this case (except potentially breaking your query translation logic).
So why is this happening?
The method you are using from EntityFramework.Extended library is in turn taken (as indicated in the comments) from MSDN Sample Walkthrough: Creating an IQueryable LINQ Provider. It can be noticed that the PartialEval method has two overloads - one with Func<Expression, bool> fnCanBeEvaluated parameter used to identify whether a given expression node can be part of the local function (in other words, to be partially evaluated or not), and one without such parameter (used by you) which simply calls the first passing the following predicate:
private static bool CanBeEvaluatedLocally(Expression expression)
{
return expression.NodeType != ExpressionType.Parameter;
}
The effect is that it stops evaluation of ParameterExpression type expressions and any expressions containing directly or indirectly ParameterExpression. The last should explain the behavior you are observing. When the query contains Where (and basically any LINQ operator) with parametrized lambda expression (hence parameter) before the Skip / Take calls, it would stop evaluation of the containing methods (which you can see from the above query.Expression debug view - the Where call will be inside the Skip).
Now, this overload is used by the MSDN example to evaluate a concrete nested Where method lambda expression and is not generally applicable for any type of expression like IQueryable.Expression. In fact the linked project is using the PartialEval method in a single place inside QueryCache class, and also calling the other overload passing a different predicate which in addition to ParameterExpressions stops the evaluation of any expression with result type of IQueryable.
Which I think is the solution of your problem as well:
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression,
// can't evaluate parameters or queries
e => e.NodeType != ExpressionType.Parameter &&
!typeof(IQueryable).IsAssignableFrom(e.Type)
);
I'm starting to look under the LINQ hood a little more, and I'm having trouble understanding some of the LINQ extension method overloads.
For example, lets say I'm querying a DbContext using .Where(). I'm always passing a standard Func<>, as opposed to an Expression<> of said Func<>. Example query below:
var db = new MyContext();
var foo = db.products.Where(p => p.Category == "books");
Here is where I'm confused. When I look at the available method signatures, I would assume the overload I'm using above would be returning me an IEnumerable...but it's actually returning an IQueryable. How is this possible, if the IQueryable overload is expecting an Expression as opposed to just a Func? It feels like the compiler is somehow helping me out by (in this case) building the Expression for me, but I can't find a resource that explains if this is the case. Thanks!
It feels like the compiler is somehow helping me out by (in this case) building the Expression for me
That is exactly what is happening. The compiler can interpret lambdas (as long as they don't have a "statement body") as either delegates or expression trees. The Queryable.Where<T>(this IQuerayble<T>, ...) extension method takes precedence over the Enumerable.Where<T>(this IEnumerable<T>, ...) extension method, so the compiler chooses to interpret your predicate as an expression tree.
It feels like the compiler is somehow helping me out by (in this case) building the Expression for me
Correct, the compiler is compiling the lambda into an Expression rather than a Func.
I can't find a resource that explains if this is the case
From MSDN:
When a lambda expression is assigned to a variable of type Expression<TDelegate>, the compiler emits code to build an expression tree that represents the lambda expression.
You're not passing a Func in that example. You're passing an Expression. A lambda can compile down into either a delegate or an expression, based on the context around it. In this case the context around it is expecting an expression, so that is what the lambda compiles into. If you actually did pass in a Func and not a lambda (which could be either) then you would get an IEnumerable for your result, not an IQueryable.
The compiler can automatically build an expression tree for you from a lamba expression, and that is what's happening here.
http://msdn.microsoft.com/en-gb/library/bb397951.aspx
A lambda expression is an anonymous method, which under the covers is a delegate so I can do something like this:
delegate bool Foo(int x);
Foo bar = x => x == 1;
Passing this delegate to an Enumerable extension method makes perfect sense, as the typical expected argument is a Func, which is shorthand for a delegate:
public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);
However, I am unclear about how it is possible to pass in the delegate to a Queryable extension method like this one:
public static IQueryable<TSource> Where<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);
This method expects an Expression<TDelegate> argument, but it is perfectly legal to pass in a lambda expression. What is the mechanism that coerces the lambda expression into Expression<TDelegate> so that it may be consumed?
I am familiar with the fact that Queryable methods build out expression trees for parsing by providers, I'm just curious about this one aspect that isn't immediately obvious to me.
UPDATE
I'm becoming less ignorant about my ignorance. Lambda expressions aren't delegates, but can be used to create either delegates or expressions:
Expression<Func<int, bool>> foo = c => c == 1;
Does the compiler infer the type based on the context? I'm guessing that must be the case, as this isn't legal:
var foo = c => c == 1;
This is described in the specification:
4.6 Expression Tree Types
If a conversion exists from a lambda expression to a delegate type D,
a conversion also exists to the expression tree type Expression<D>.
Whereas the conversion of a lambda expression to a delegate type
generates a delegate that references executable code for the lambda
expression, conversion to an expression tree type creates an
expression tree representation of the lambda expression. Expression
trees are efficient in-memory data representations of lambda
expressionsand make the structure of the lambda expressiontransparent
and explicit
So there is a conversion from a lambda to a compatible expression tree type, and the compiler emits the equivalent expression tree instead of creating a delegate.
Quite simply you can't.
However, to make IQueryable methods useful, VS2008 and above include a clever compiler trick. That a lambda expression that is a single statement may be assignable to both a delegate, and an Expression<TDelegate>. The compiler normally will hoist the expression and make a method.
But for an assignment to an Expression<TDelegate> it breaks down the statements into their syntactic meaning and turns that into an expression tree.
e.g.
Func<int,int> func = x=>x*x;
Expression<Func<int,int>> expression = x=>x*x;
The first one will probably be turned into a static method with a garbled name something like::
private static int <B012>SomeMethod(int x){
return x*x;
}
Where as the second statement will be transformed into something like::
ParameterExpression paramX = Expression.Parameter(typeof(int));
Expression<Func<int,int>> expression = Expression.Lambda<Func<int,int>>(
Expression.Multiply(paramX,paramX),paramX);
But you can not do::
expression = func;
This is not valid, as func is a delegate. You can do this though::
func=expression.Compile()
Which compiles the expressions into a func.
**Note the suggested transformations may not be a 100% correct.
The reason they did this was to allow LINQ-to-Objects (Basically Map/Reduce from other language) to share the same friendly syntax as LINQ-To-Providers. So you can write a statement that means the same thing but can change where the filtering and transformation happens.
GetEmployees().Where(e=>e.LastName=="Smith")
Can read the same, but could in theory be describing doing the filtering on this box, or an the database, or parsing an xml file or any number of various things.
I believe this has to do with how the query is built on an IQueryable. The method requires an expression tree because it can be looked inside and structured to match a more optimized (potentially) query or map more closely to the underlying data source. So simply passing in a Func would allow only for execution, whereas an Expression<Func> allows for expression trees which can be observed and executed.
Also, probably more closely answering your exact question, check this SO post out.
I have the following code to get a collection of Types from an Assembly where the Type is a Class:
Assembly assembly = Assembly.LoadFile(DLLFile);
var types = assembly.GetTypes().AsEnumerable().Where(x => x.IsClass);
This works fine and as expected. However I wanted to pull out the lambda expression to a Linq Expression variable (as later on it will be used in a parameter of this method). So I did the following:
private Expression<Func<Type, bool>> _standardFilter = (x => x.IsClass);
Assembly assembly = Assembly.LoadFile(DLLFile);
var types = assembly.GetTypes().AsEnumerable().Where(_standardFilter);
However this won't compile with the error:
System.Collections.Generic.IEnumerable<System.Type>' does not contain a
definition for 'Where' and the best extension method overload
'System.Linq.Enumerable.Where<TSource>(System.Collections.Generic.IEnumerable<TSource>, System.Func<TSource,int,bool>)' has some invalid arguments
I understand that my expression doesn't conform to the predicate System.Func<TSource,int,bool>, however the Where function has an overload that takes a predicate of System.Func<TSource,bool>, which as far as I can tell should work.
I have tried converting the result of assembly.GetTypes() (which is an array) to a List in several ways without it helping the issue.
I have also made sure that I have got all the correct using statements for this class as that seems to be an issue several people run into during my Googling.
In the past I have managed to use this same technique on a IQueryable collection, but I don't understand why this won't work when the Where function is available on the IEnumerable collection and should accept the predicate I am providing.
Thanks very much for any assistance.
Compile your expression into executable code (delegate):
var types = assembly.GetTypes().AsEnumerable().Where(_standardFilter.Compile());
As #Kirk stated, it's better not to use expression tree, if you are not going to analyze it. Simply use filter of type Func<Type, bool>.
Note that the Enumerable extension method takes a Func, not an Expression. So your argument is incorrectly typed.
This is in contrast with Entity Framework, which takes Expression<Func> rather than plain Func.
So be attentive to method signatures, they are similar, can both be called with the same lambda, but are actually different!
The reason behind that is that EF inspects the Expression and converts it to SQL code, whereas Linq to Object simply executes the predicate in memory.
You could try this instead:
private Func<Type, bool> standardFilter;
standardFilter = (type) => {
return type.IsClass;
};
var types = assembly.GetTypes().AsEnumerable().Any(x => standardFilter(x));
Then you're free to change your implementation of standardFilter. As long as it takes in a Type and returns bool for whether or not that should be included it will remain modular and accomplish what I think you're going for.
I know what a Lambda Expression is.
But I am not sure if that is the same thing as an Expression. There seems to be more to know here than I know.
I am looking at wrapping IQueryable and that uses Expressions a lot. So, for example, is there more to the 'Expression' parameter here than can be thought of in a Lambda?
public InterceptedQuery(InterceptingProvider provider, Expression expression)
{
this._provider = provider;
this._expression = expression;
}
A lambda expression is a compiler feature that is compiled, depending on context, into one of two things:
A (hidden) function and a delegate to it
An Expression
Once the application has been compiled, the concept of a lambda expression doesn't exist, as it's been turned into one of the above two options.
I'm not sure what you mean by
is there more to the 'Expression' parameter here than can be thought of in a Lambda
An expression encapsulates and expressed application logic in an inspectable form (in other words, it lets you see what the developer wrote in terms of properties and functions invoked, constants included, comparisons, etc.). This is how query providers (for the most part, object-relational mappers like the Entity Framework) take code and turn it into SQL.
Lambdas are generally compiled code, where Expression represents "abstract syntax trees" (AST), ie. a data structure that represents code, and can be compiled to code. IQueryable generally operates on Expression because it's supposed to compile the AST to code that runs in different environments, like an SQL server, instead of just the host machine. There are IQueryableProviders that compile to SQL (Linq2Sql), to JavaScript, to OpenGL shaders (Bling), and more.
The C# compiler can sometimes turn a lambda into an expression, if the method parameter expects an expression of the right type:
void Foo(Expression<Func<int>>) { ... }
...
Foo(() => 3);