I've written some code that takes an Expression and acts as a proxy to make certain calls. The main bit of code that makes this work is such
private static IEnumerable<object> GetArguments(MethodCallExpression body)
{
IEnumerable<object> arguments =
body.Arguments.Select(
expression =>
{
MemberExpression member = expression as MemberExpression;
return ((dynamic) member.Member).GetValue(((ConstantExpression) member.Expression).Value);
});
return arguments;
}
This gets the values of the parameters present in the Expression and returns them in an IEnumerable. That's the behavior I get when I wrote this in the initially consuming project and I can see the resulting argument values being output correctly. I even moved this to a test bed project to test the code and it worked fine there as well. However, when I put the .cs files in another project, updating the namespaces, and reference it to be able to use it widely, I get exceptions such as
System.InvalidCastException: Unable to cast object of type 'System.Linq.Expressions.PropertyExpression' to type 'System.Linq.Expressions.ConstantExpression'.
and I can't even use my application.
I've rolled my extraction back to the "it works locally" state to ensure it, in fact, worked, and it did. However, I have noticed I still get some exceptions in my logging even though it didn't throw up on me like it does when it's external to the project. I can still use my application in spite of this.
Repeating the move to an external project and using it by way of a reference reintroduces those errors and I can't even use my app.
New question: how do I handle PropertyExpressions? They don't expose a Value property and are internal so I there's no cast and check. MemberExpression also doesn't have a Value property.
The simplest solution would be not to rely on the shape of the expressions at all and let the Expression Tree library evaluate each subexpression for you by building a LambdaExpression and then using Compile() on it and executing the returned delegate:
private static IEnumerable<object> GetArguments(MethodCallExpression body)
{
return body.Arguments.Select(
expression => Expression.Lambda<Func<object>>(
Expression.Convert(expression, typeof(object))).Compile()());
}
Note that compiling each subexpression will take some time, so this approach is going to be relatively slow.
You should evaluate the expression and not rely on it's shape. The easiest way is to call .Compile().Invoke(), but that is slow and creates a memory-leak. You can avoid that by interpreting the expression (stepping recursively through it with a visitor and perform the operations in it manually) or cache your compilation result. I wrote a library that can do both, is fast and used in production for quite some time now: https://github.com/Miaplaza/expression-utils
Related
In one of my projects I have an ExpressionVisitor to translate provided expression into some query string. But before translating it I need to evaluate all refferences in the expression to real values. To do that I use Evaluator.PartialEval method from EntityFramework Project.
Assuming I have this query:
var page = 100;
var query = myService.AsQueryable<Product>()
//.Where(x=>x.ProductId.StartsWith(p.ProductId))
.Skip(page)
.Take(page);
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression);
As you can see I have commented Where method. In this case evaluatedQueryExpr will not contain the methods Take and Skip.
However, if I use any other method with Expression before Take or Skip everything works, Evaluator evaluates an expression correctly and return it fully.
I found out that the problem occurs in the line 80 of the Evaluator class:
return Expression.Constant(fn.DynamicInvoke(null), e.Type);
Could you explain why this happens and suggest a workaround?
Update
here is a project on github
LinqToSolrQueriable inherited from IOrderedQueryable
LinqToSolrProvider inherited from IQueryProvider including line range causing the issue
The good news are that the expression is not really reduced (Skip and Take are still there :), but is simply converted from MethodCallExpression to ConstantExpression containing the original expression:
query.Expression:
.Call System.Linq.Queryable.Take(
.Call System.Linq.Queryable.Skip(
.Constant<LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]),
100),
100)
evaluatedQueryExpr:
.Constant<System.Linq.IQueryable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product])
Here the debug display is giving you a wrong impression. If you take the ConstaintExpression.Value, you'll see that it's a IQueryable<Product> with Expression property being exactly the same as the original query.Expression.
The bad news are that this is not what you expect from PartialEval - in fact it doesn't do anything useful in this case (except potentially breaking your query translation logic).
So why is this happening?
The method you are using from EntityFramework.Extended library is in turn taken (as indicated in the comments) from MSDN Sample Walkthrough: Creating an IQueryable LINQ Provider. It can be noticed that the PartialEval method has two overloads - one with Func<Expression, bool> fnCanBeEvaluated parameter used to identify whether a given expression node can be part of the local function (in other words, to be partially evaluated or not), and one without such parameter (used by you) which simply calls the first passing the following predicate:
private static bool CanBeEvaluatedLocally(Expression expression)
{
return expression.NodeType != ExpressionType.Parameter;
}
The effect is that it stops evaluation of ParameterExpression type expressions and any expressions containing directly or indirectly ParameterExpression. The last should explain the behavior you are observing. When the query contains Where (and basically any LINQ operator) with parametrized lambda expression (hence parameter) before the Skip / Take calls, it would stop evaluation of the containing methods (which you can see from the above query.Expression debug view - the Where call will be inside the Skip).
Now, this overload is used by the MSDN example to evaluate a concrete nested Where method lambda expression and is not generally applicable for any type of expression like IQueryable.Expression. In fact the linked project is using the PartialEval method in a single place inside QueryCache class, and also calling the other overload passing a different predicate which in addition to ParameterExpressions stops the evaluation of any expression with result type of IQueryable.
Which I think is the solution of your problem as well:
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression,
// can't evaluate parameters or queries
e => e.NodeType != ExpressionType.Parameter &&
!typeof(IQueryable).IsAssignableFrom(e.Type)
);
LINQPad example:
void Main()
{
One(i => PrintInteger(i));
One(PrintInteger);
Two(i => PrintInteger(i));
// Two(PrintInteger); - won't compile
}
static void One(Action<int> a)
{
a(1);
}
static void Two(Expression<Action<int>> e)
{
e.Compile()(2);
}
static void PrintInteger(int i)
{
Console.WriteLine(i);
}
Uncommenting the Two(PrintInteger); line results in an error:
cannot convert from 'method group' to
'System.Linq.Expressions.Expression<System.Action<int>>'
This is similar to Convert Method Group to Expression, but I'm interested in the "why." I understand that Features cost money, time and effort; I'm wondering if there's a more interesting explanation.
Because, in order to get the expression tree, we need a representation of the method in (uncompiled) source form. Lambda expressions are locally available in the source code and therefore are always available uncompiled. But methods may not be from inside the current assembly, and may thus be available only in compiled form.
Granted, the C# compiler could decompile the assembly’s IL code to retrieve an expression tree but as you mentioned, implementing feature costs money, this particular feature isn’t trivial, and the benefits are unclear.
There is no reason in principle. It could be done this way. The compiler could just create the lambda by itself before converting it (this is obviously always possible - it knows the exact method being called so it can just create a lambda from its parameters).
There is one catch, though. The name of the parameter of your lambda is normally hard-coded into the IL being generated. If there is no lambda, there is no name. But the compiler could either create a dummy name or reuse the names of the method being called (they are always available in the .NET assembly format).
Why didn't the C# team decide to enable this? The only reason that comes to mind is that they wanted to spend their time elsewhere. I applaud them for that decision. I'd rather have LINQ or async than this obscure feature.
In the One example, you are implicitly creating an Action<int> delegate. It's the same as:
One( new Action<int>( PrintInteger ) );
I believe this is in the language to improve the syntax for subscribing to events.
The same thing doesn't happen for Expression<T>, which is why your second example doesn't compile.
EDIT :
It's called a "method group conversion". It's in the C# spec - section 6.6
An implicit conversion (§6.1) exists from a method group (§7.1) to a compatible delegate type
What happens behind the curtains when I include a function into my compiled query, like I do with DataConvert.ToThema() here to convert a table object into my custom business object:
public static class Queries
{
public static Func<MyDataContext, string, Thema> GetThemaByTitle
{
get
{
var func = CompiledQuery.Compile(
(MyDataContext db, string title) =>
(from th in elan.tbl_Thema
where th.Titel == title
select DataConvert.ToThema(th)).Single()
);
return func;
}
}
}
public static class DataConvert
{
public static Thema ToThema(tbl_Thema tblThema)
{
Thema thema = new Thema();
thema.ID = tblThema.ThemaID;
thema.Titel = tblThema.Titel;
// and some other stuff
return thema;
}
}
and call it like this
Thema th = Queries.GetThemaByTitle.Invoke(db, "someTitle");
Apparently the function is not translated in to SQL or something (how could it), but it also does not hold when I set a breakpoint there in VS2010.
It works without problems, but I don't understand how or why. What exactly happens there?
Your DataConvert.ToThema() static method is simply creating an instance of a type which has a default constructor, and setting various properties, is that correct? If so, it's not terribly different from:
(from th in elan.tbl_Thema
where th.Titel == title
select new Thema{ID=th.ThemaID, Titel=th.Titel, etc...}
).Single());
When you call Queries.GetThemaByTitle, a query is being compiled. (The way you are calling this, by the way, may or may not actually be giving you any benefits from pre-compiling). That 'Query' is actually a code expression tree, only part of which is intended to generate SQL code that is sent to the database.
Other parts of it will generate IL code which is grabbing what is returned from the database and putting it into some form for your consumption. LINQ (EF or L2S) is smart enough to be able to take your static method call and generate the IL from it to do what you want - and maybe it's doing so with an internal delegate or some such. But ultimately, it doesn't need to be (much) different from what would be generated from I substituted above.
But note that this happens regardless what the type is that you get back; somewhere, IL code is being generated that puts DB values into a CLR object. That is the other part of those expression trees.
If you want a more detailed look at those expression trees and what they involved, I'd have to dig for ya, but I'm not sure from your question if that's what you are looking for.
Let me start by pointing out, that whether you compile your query or not does not matter. You would observe the very same results even if you did not pre-compile.
Technically, as Andrew has pointed out, making this work is not that complicated. When your LINQ expression is evaluated an expression tree is constructed internally. Your function appears as a node in this expression tree. No magic here. You'll be able to write this expression both in L2S and L2E and it will compile and run fine. That is until you try to actually execute the actual SQL query against the database. This is where difference begins. L2S seems to happily execute this task, whereas L2E fails with NotSupportedException, and reporting that it does not know how to convert ToThema into store query.
So what's happening inside? In L2S, as Andrew has explained, the query compiler understands that your function can be run separately from the store query has been executed. So it emits calls to your function into the object reading pipeline (where data read from SQL is transformed to the objects that are returned as the result of your call).
Once thing Andrew was not quite right, is that it matters what's inside your static method. I don't think it does.
If you put a break point in the debugger to your function, you will see that it's called once per returned row. In the stack trace you will see "Lightweight Function", which, in reality, means that the method was emitted at run time. So this is how it works for Linq to Sql.
Linq to Entity team seemed to go different route. I do not know, what was the reasoning, why they decided to ban all InvocationExpressions from L2E queries. Perhaps these were performance reason, or may be the fact that they need to support all kind of providers, not SQL Server only, so that data readers might behave differently. Or they simply thought that most people wouldn't realize that some of those are executed per returned row and preferred to keep this option closed.
Just my thoughts. If anyone has any more insight, please chime in!
I have a method that alters an "Account" object based on the action delegate passed into it:
public static void AlterAccount(string AccountID, Action<Account> AccountAction) {
Account someAccount = accountRepository.GetAccount(AccountID);
AccountAction.Invoke(someAccount);
someAccount.Save();
}
This works as intended...
AlterAccount("Account1234", a => a.Enabled = false);
...but now what I'd like to try and do is have a method like this:
public static void AlterAccount(string AccountID, string AccountActionText) {
Account someAccount = accountRepository.GetAccount(AccountID);
Action<Account> AccountAction = MagicLibrary.ConvertMagically<Action<Account>>(AccountActionText);
AccountAction.Invoke(someAccount);
someAccount.Save();
}
It can then be used like:
AlterAccount("Account1234", "a => a.Enabled = false");
to disable account "Account1234".
I've had a look at the linq dynamic query library, which seems to do more or less what I want but for Func type delegates, and my knowledge of Expression trees etc isn't quite good enough to work out how to achieve what I want.
Is there an easy way to do what I want, or do I need to learn expressions properly and write a load of code?
(The reason I want to do this is to allow an easy way of bulk updating account objects from a powershell script where the user can specify a lambda expression to perform the changes.)
The Dynamic LINQ library is a fine choice, as it'll generate expressions you can compile to code in a lightweight fashion.
The example you provided actually produces a boolean -- so you should be able to ask for a Func and it might sort it out.
Edit: This of course is wrong, as Expressions don't have assignment in them at all.
So, another potential way is to take two lambdas. One to find the property you want, one to provide a value:
(a => a.AccountId), (a => true)
Then use reflection to set the property referenced in the first lambda with the result of the second one. Hackish, but it's still probably lightweight compared to invoking the C# compiler.
This way you don't have to do much codegen yourself - the expressions you get will contain most everything you need.
You may try this: Dynamic Lambda Expressions Using An Isolated AppDomain
It compiles a lambda expression using CodeDOM compiler. In order to dispose the in-memory assembly that gets created, the compiler runs on an isolated AppDomain. For the passing the expression through the domain boundary, it has to be serialized. Alas, Expression<> is not Serializable. So, a trick has to be used. All the details are explained in the post.
I'm the author of that component, by the way. I would like very much to hear your feedback from it.
There is no general way to parse a string into a lambda expression without a full compilation, because lambda expressions can reference things that are defined outside the lambda expression. I know of no library that handles the specific case you want. There's a long discussion of this on a thread on a C# discussion group.
The easiest way to get what you want is to compile a method at runtime. You can write a function that takes in the string "a.Enabled = true; return a;" and sticks that in the middle of a function that takes an Account as a parameter. I would use this library as a starting point, but you can also use the function mentioned on another thread.
That's easy:
Use CodeDom to generate the module containing the "surrounding class" you'll use to build the expression; this class must implement the interface known to your application
Use CodeSnippedExpression to inject the expression into its member.
Use Activator type to create the instance of this class in runtime.
Basically, you need to build the following class with CodeDom:
using System;
using MyNamespace1;
using ...
using MyNamespace[N];
namespace MyNamespace.GeneratedTypes
{
public class ExpressionContainer[M] : IHasAccountAction
{
public Action<Account> AccountAction {
get {
return [CodeSnippedExpression must be used here];
}
}
}
}
Assuming that IHasAccountAction is:
public IHasAccountAction {
public Action<Account> AccountAction { get; }
}
If this is done, you can get the expression compiled from string with ease. If you need its expression tree representation, use Expression<Action<Account>> instead of Action<Account> in generated type.
In a previous question I asked how to make "Computed properties" in a linq to sql object. The answer supplied there was sufficient for that specific case but now I've hit a similar snag in another case.
I have a database with Items that have to pass through a number of Steps. I want to have a function in my database that retrieves the Current step of the item that I can then build on. For example:
var x = db.Items.Where(item => item.Steps.CurrentStep().Completed == null);
The code to get the current step is:
Steps.OrderByDescending(step => step.Created).First();
So I tried to add an extension method to the EntitySet<Step> that returned a single Step like so:
public static OrderFlowItemStep CurrentStep(this EntitySet<OrderFlowItemStep> steps)
{
return steps.OrderByDescending(o => o.Created).First();
}
But when I try to execute the query at the top I get an error saying that the CurrentStep() function has no translation to SQL. Is there a way to add this functionality to Linq-to-SQL in any way or do I have to manually write the query every time? I tried to write the entire query out first but it's very long and if I ever change the way to get the active step of an item I have to go over all the code again.
I'm guessing that the CurrentStep() method has to return a Linq expression of some kind but I'm stuck as to how to implement it.
The problem is that CurrentStep is a normal method. Hence, the Expression contains a call to that method, and naturally SQL cannot execute arbitrary .NET methods.
You will need to represent the code as an Expression. I have one in depth example here: http://www.atrevido.net/blog/2007/09/06/Complicated+Functions+In+LINQ+To+SQL.aspx
Unfortunately, the C# 3.0 compiler has a huge omission and you cannot generate calls to Expressions. (i.e., you can't write "x => MyExpression(x)"). Working around it either requires you to write the Expression manually, or to use a delegate as a placeholder. Jomo Fisher has an interesting post about manipulating Expression trees in general.
Without actually having done it, the way I'd probably approach it is by making the CurrentStep function take the predicate you want to add ("Completed == null"). Then you can create a full Expression> predicate to hand off to Where. I'm lazy, so I'm going to do an example using String and Char (String contains Chars, just like Item contains Steps):
using System;
using System.Linq;
using System.Linq.Expressions;
class Program {
static void Main(string[] args) {
Console.WriteLine(StringPredicate(c => Char.IsDigit(c)));
var func = StringPredicate(c => Char.IsDigit(c)).Compile();
Console.WriteLine(func("h2ello"));
Console.WriteLine(func("2ello"));
}
public static Expression<Func<string,bool>> StringPredicate(Expression<Func<char,bool>> pred) {
Expression<Func<string, char>> get = s => s.First();
var p = Expression.Parameter(typeof(string), "s");
return Expression.Lambda<Func<string, bool>>(
Expression.Invoke(pred, Expression.Invoke(get, p)),
p);
}
}
So "func" is created by using StringPredicate to create an Expression. For the example, we compile it to execute it locally. In your case, you'd pass the whole predicate to "Where" so it gets translated to SQL.
The "get" expression is where you put your "extension" stuff (OrderByWhatever, First, etc.). This is then passed in to the predicate that's given to you.
Don't worry if it looks complicated; it sorta is at first. If you haven't done this kinda stuff before, it'll take a bit of time (the first time I did this kinda stuff, it took hours to get it right :|.. now it comes slightly easier). Also, as I mentioned, you can write a helper method to do this re-writing for you (so you don't directly need to use the Expression.Whatever methods), but I haven't seen any examples and haven't really needed it yet.
Check out my answer to "switch statement in linq" and see if that points you in the right direction...
The technique i demonstrate there is the one that got me past the scary "no translation to SQL" error.