I've been troubleshooting an unusual issue with some EF 6 code where queries running with the Oracle.ManagedDataAccess.Client driver are sometimes taking several minutes to return results even when the underlying query executes within 2ms. An example query would be as follows:
var result = users.Where(u => u.username == varUserName).FirstOrDefault();
This query might take several minutes to return, however if I replace the query with the same thing with a constant in the lambda function, it runs instantaneously:
var result = users.Where(u => u.username == "testUsername").FirstOrDefault();
To work around this issue, I can either write parameterised SQL queries, or I can manually generate an appropriate lambda expression tree:
var userParam = Expression.Parameter(typeof(Entity.User), "user");
var userNameField = Expression.Property(userParam, "username");
var userNameConstant = Expression.Constant(varUserName, typeof(string));
var equalUserName = Expression.Equal(userNameField, userNameConstant);
var lambda = Expression.Lambda<Func<Entity.User, bool>>(equalUserName, new ParameterExpression[] { userParam });
var result = users.Where(lambda).FirstOrDefault();
Because this works, it begs the question: is there a way to easily generate lambda expression trees which result in variables being directly included as constants, instead of references to variables?
For example, something like this would be ideal:
var lambdaExpression = (u => u.username == varUserName).ReplaceVariablesWithConstants();
It can be done relatively easy with ExpressionVisitor which evaluates the ConstantExpression members like this:
public static class ExpressionUtils
{
public static Expression<TDelegate> ReplaceVariablesWithConstants<TDelegate>(this Expression<TDelegate> source)
{
return source.Update(
new ReplaceVariablesWithConstantsVisitor().Visit(source.Body),
source.Parameters);
}
class ReplaceVariablesWithConstantsVisitor : ExpressionVisitor
{
protected override Expression VisitMember(MemberExpression node)
{
var expression = Visit(node.Expression);
if (expression is ConstantExpression)
{
var variable = ((ConstantExpression)expression).Value;
var value = node.Member is FieldInfo ?
((FieldInfo)node.Member).GetValue(variable) :
((PropertyInfo)node.Member).GetValue(variable);
return Expression.Constant(value, node.Type);
}
return node.Update(expression);
}
}
}
It's kinda hard. You need to modify the ExpressionsTree with a ExpressionsVisitor. It would probably become something like:
var lambdaExpression = ReplaceVariablesWithConstants(u => u.username == varUserName);
Related
I have an IQueryable<T> from my DbSet in Entity Framework. I am provided a "Fuzzy Search String", named searchText, like so:
public List<T> Search<T>(string searchText)
{
using (var context = ...)
{
var baseQuery = context.Set<T>().AsQueryable();
baseQuery = baseQuery.Where(x =>
DbFunctions.Like(x.PropertyName, searchText)
|| DbFunctions.Like(x.PropertyTwo, searchText)
|| DbFunctions.Like(x.PropertyThree, searchText)
|| DbFunctio..... etc
);
return baseQuery.ToList();
}
}
But given the generic nature, I don't know what properties there are on the type. I can provide an abstract method to somebody implementing this which allows them to give me a List of Properties (or even PropertyInfo or whatever else, I can figure that out). But I don't know how to dynamically create the expression. This is what I have so far:
var baseQuery = context.Set<T>().AsQueryable();
var expression = baseQuery.Expression;
var colName = "colName"; // Or names, I can iterate.
var parameter = Expression.Parameter(typeof(T), "x");
var selector = Expression.PropertyOrField(parameter, colName);
expression = Expression.Call(typeof(DbFunctions), nameof(DbFunctions.Like),
new Type[] { baseQuery.ElementType, selector.Type },
expression, Expression.Quote(Expression.Lambda(selector, parameter)));
The problem here is... well, it doesn't work to begin with. But mainly that I'm not using the searchText anywhere in it, and don't know how to plug it in. I THINK I'm close... but have spent an inordinate amount of time on it.
Hopefully I'm getting your query logic right: if you want to build a set of LIKE conditions based on known type and list of column names, you could try something like this:
static private MethodInfo dbLikeMethod = typeof(DbFunctions).GetMethod(nameof(DbFunctions.Like), BindingFlags.Public | BindingFlags.Static, null, new Type[] {typeof(string), typeof(string)}, null); // I am targeting DbFunctions.Like(string, string). You might want another overload (or even mix them up depending on your inputs)
public List<T> Search<T>(string searchText) where T: class
{
using (var context = new ...)
{
var baseQuery = context.Set<T>().AsQueryable().Where(CreateExpression<T>(searchText));// you could probably find a more elegant way of plugging it into your query
return baseQuery.ToList();
}
}
Expression<Func<T, bool>> CreateExpression<T>(string searchText) where T : class
{
var cols = new List<string> {
"PropertyName",
"PropertyTwo" // i understand you've got a way to figure out which strings you need here
};
var parameter = Expression.Parameter(typeof(T), "x");
var dbLikeCalls = cols.Select(colName => Expression.Call(dbLikeMethod, Expression.PropertyOrField(parameter, colName), Expression.Constant(searchText))); // for convenience, generate list of DbFunctions.Like(x.<Property>, searchText) expressions here
var aggregatedCalls = dbLikeCalls.Skip(1).Aggregate((Expression)dbLikeCalls.First(), (accumulate, call) => Expression.OrElse(accumulate, call)); // aggregate the list using || operators: use first item as a seed and keep adding onto it
return Expression.Lambda<Func<T, bool>>(aggregatedCalls, parameter);
}
Consider this code:
public List<Clients> GetFilteredClients(DateTime? FromDate = null,
DateTime? ToDate = null,
int? fromLocationType = null,
int? toLocationType = null)
{
Func<Clients, bool> fromDateFilter = f => true;
if (FromDate.HasValue)
{
fromDateFilter = z => z.Insert_Date.Value.Date >= FromDate.Value.Date;
}
Func<Clients, bool> toDateFilter = f => true;
if (ToDate.HasValue)
{
toDateFilter = z => z.Insert_Date.Value.Date <= ToDate.Value.Date;
}
Func<Clients, bool> fromLocationTypeFilter = f => true;
if (fromLocationType.HasValue)
{
fromOrgFilter = z => z.LocationTypeId >= fromLocationType.Value;
}
Func<Clients, bool> toLocationTypeFilter = f => true;
if (toLocationType.HasValue)
{
toLocationTypeFilter = z => z.LocationTypeId <= toLocationType.Value;
}
var filtered = DB_Context.Clients
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();
return filtered;
}
I have something like 100K records in the DB, I need only the top 1000 that answer to the requirements of:
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
However the execution time still takes something like 10 seconds.
Any idea why?
You must use Expression<Func<...>> rather than Func<...>. When you use Func, only the enumerable methods can be used on the queryable, which in this case means you first download everything to memory, and then do the filtering. If you switch over to Expression<...>, the O/RM will do the filtering on the DB server, rather than in your application.
Also, there's better ways to do what you're doing. For example, you can build the conditions like so:
var query = DB_Context.Clients.AsQueryable();
if (FromDate.HasValue) query = query.Where(...);
if (ToDate.HasValue) query = query.Where(...);
...
return query.OrderByDescending(k => k.Id).Take(1000).ToList();
Of course, this means that whatever DB provider you're using must be able to support the kind of filtering you're trying to do - you'll need to consult the documentation.
You are using delegates instead LINQ expressions. That leads to processing a data by your application and not by SQL Server.
LINQ expressions look like lambda expressions thanks for the syntax, but they are not same thing. The compiler takes a decision what to create (delegates or LINQ expressions) depending on the situation.
If an object implements the IQueriable interface, then the compiler uses the Queryable class and generates LINQ expression trees, which later can be translated into a SQL query or other form by the specific IQueryProvider.
Otherwise, the compiler uses extensions from the Enumerable class, which create iterators over source collection (all records from the table in your case).
As an example. The code bellow will be compilled into LINQ expressions.
// Source code
IQueryable<Clients> source = null;
IQueryable<Clients> result = source.Where(c => c.LocationTypeId >= 1);
// Compiller generated code
IQueryable<Clients> source = null;
Expression parameterC = Expression.Parameter(typeof(Clients), "c");
IQueryable<Clients> result = Queryable.Where<Clients>(
source,
Expression.Lambda<Func<Clients, bool>>(
Expression.LessThanOrEqual(
Expression.Property(
parameterC ,
typeof(Clients).GetProperty("LocationTypeId").GetGetMethod()
),
Expression.Constant(1, typeof(int))
),
new ParameterExpression[]
{
parameterC
}
);
And this code uses delegates:
// Source code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = source.Where(filter );
// Compiller generated code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = Enumerable.Where(source, filter);
So, to solve you problem use Expression<Func<Clients, bool>> instead of Func<Clients, bool>:
IQueryable<Clients> result = DB_Context.Clients;
if (someFilter.HasValue)
result = result.Where(c => c.SomeProperty == someFilter.Value);
// other filters
return query
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();
So I had written this LINQ query using reflection, and later found out it isn't supported. What would be the best way to get the same functionality from this code?
List<Profile> profilesFromUUID = await MobileService.GetTable<Profile>().Where(p => typeof(Profile)
.GetProperty(handler.Name + "UUID").GetValue(p) == obj.uuid).ToListAsync();
Use the reflection to create the query, not in the query. Consider:
public static IQueryable<Profile> Filter(
this IQueryable<Profile> source, string name, Guid uuid)
{
// .<name>UUID
var property = typeof(Profile).GetProperty(name + "UUID");
// p
var parExp = Expression.Parameter(typeof(Profile));
// p.<name>UUID
var methodExp = Expression.Property(parExp, property);
// uuid
var constExp = Expression.Constant(uuid, typeof(Guid));
// p.<name>UUID == uuid
var binExp = Expression.Equal(methodExp, constExp);
// p => p.<name>UUID == uuid
var lambda = Expression.Lambda<Func<Profile, bool>>(binExp, parExp);
// source.Where(p => p.<name>UUID == uuid)
return source.Where(lambda);
}
This builds up the expression first (so if name was "Test" it would create the expression corresponding with p => p.TestUUID == uuid and then uses that in the call to Where.
Because this step is done first, rather than within the expression itself, there's no need for the query engine to try to translate typeof or GetProperty() into SQL (which it of course, couldn't do).
So:
var filtered = MobileService.GetTable<Profile>().Filter(handler.Name, obj.uuid);
Returns an IQueryable<Profile> with the appropriate Where attached. And so:
var profilesFromUUID = await MobileService.GetTable<Profile>().Filter(handler.Name, obj.uuid).ToListAsync();
Will as a whole first use reflection to build the query, then apply the query, then produce a list from it asynchrously and then wait for its results.
It's worth noting that since Filter() will accept any IQueryable<Profile> they can be either chained or unioned. So:
MobileService.GetTable<Profile>().Filter("A", uuid0).Filter("B", uuid1);
Is equivalent to:
from p in MobileService.GetTable<Profile>() where p.AUUID = uuid0 && p.BUUID == uuid1
And:
MobileService.GetTable<Profile>().Filter("A", uuid0).Union(
MobileService.GetTable<Profile>().Filter("B", uuid1))
Is equivalent to:
from p in MobileService.GetTable<Profile>() where p.AUUID = uuid0 || p.BUUID == uuid1
A more generalised version would be:
public static IQueryable<TSource> FilterByNamedProperty<TSource, TValue>(this IQueryable<TSource> source, string propertyName, TValue value)
{
var property = typeof(TSource).GetProperty(propertyName);
var parExp = Expression.Parameter(typeof(TSource));
var methodExp = Expression.Property(parExp, property);
var constExp = Expression.Constant(value, typeof(TValue));
var binExp = Expression.Equal(methodExp, constExp);
var lambda = Expression.Lambda<Func<TSource, bool>>(binExp, parExp);
return source.Where(lambda);
}
Then while you have to do the + "UUID" in the calling code, you can use this to do analogous queries with any IQueryable<> of any element type.
How about just compare all property name? By definition UUID would not have collision anyway. Since Profile is just a data class, the # of the property for UUID is fixed.
List<Profile> profilesFromUUID = await MobileService.GetTable<Profile>()
.Where(p =>
p.A_UUID == obj.uuid ||
p.B_UUID == obj.uuid ||
p.C_UUID == obj.uuid)
.ToListAsync();
Or add a method (extension method) for Profile like:
public static Guid GetUUIDByTableName(this Profile value, string tableName)
{
switch (tableName)
{
case "A_": return value.A_UUID;
case "B_": return value.B_UUID;
default: return Guid.Empty;
}
}
List<Profile> profilesFromUUID = await MobileService.GetTable<Profile>()
.Where(p => p.GetUUIDByTableName(handler.Name) == obj.uuid)
.ToListAsync();
I have a collection List<List<object>>, which I need to filter out based on wheater the List<object> collection contains given element. I was able to build the where clause, but i get the following Exception:
An exception of type 'System.InvalidOperationException' occurred in System.Core.dll but was not handled in user code
Additional information: variable 'x' of type 'System.Collections.Generic.List`1[System.Object]' referenced from scope '', but it is not defined
I have found similar issues and i understand where the problem is, but i need help finding the solution.
Here is my code:
protected override Expression<Func<List<object>, bool>> GetWhereClause()
{
var type = typeof(List<object>);
var parameterExpression = Expression.Parameter(type, "x");
Expression expressionBody = null;
if (Verified)
{
Expression<Func<List<object>, bool>> expression = x => x.Contains("Verified");
expressionBody = expression.Body;
}
if (GoodMatch)
{
Expression<Func<List<object>, bool>> expression = x => x.Contains("Good Match");
if (expressionBody != null)
expressionBody = Expression.Or(expressionBody, expression.Body);
else
expressionBody = expression.Body;
}
//More conditions here
if (expressionBody != null)
{
var whereClauseExp = Expression.Lambda<Func<List<object>, bool>>(expressionBody, parameterExpression);
return whereClauseExp;
}
return null;
}
Now, this method generates the desired where clause, but when i try to apply it, i get the mentioned Exception.
if (whereClause != null)
{
items = items.Where(whereClause.Compile());
//
}
I had a similar use case requiring dynamic where clauses and used Predicate Builder
Using it, you could do something like:*
private Expression<Func<List<T>, bool>> GetWhereClause<T>(T itemToFind){
var predicate = PredicateBuilder.False<List<T>>();
if(Verified) {
predicate = predicate.And(p => p.Contains(itemToFind));
}
if(GoodMatch) {
predicate = predicate.Or(p => p.Contains(itemToFind));
}
return predicate;
}
You cannot use a parameter from another expression 'as is' in a new expression. When you do:
Expression<Func<List<object>, bool>> expression = x => x.Contains("Verified");
expressionBody = expression.Body;
then you simply have the inline defined parameter x in the body. Now this parameter IS NOT the same parameter, that you defined before as
var parameterExpression = Expression.Parameter(type, "x");
even if they both have the name x, that is not enough. Expression trees have reference equality.
So to get it to work, just use a visitor, that will replace the parameter with yours. Create a visitor:
public class ParameterUpdateVisitor : ExpressionVisitor
{
private ParameterExpression _parameter;
public ParameterUpdateVisitor(ParameterExpression parameter)
{
_parameter = parameter;
}
protected override Expression VisitParameter(ParameterExpression node)
{
return _parameter;
}
}
and then in your code use it like:
Expression<Func<List<object>, bool>> expression = x => x.Contains("Verified");
var visitor = new ParameterUpdateVisitor(parameterExpression);
expressionBody = visitor.Visit(expression.Body);
Of course this for every part that comes from another expression tree.
NOTE!!! this visitor is extra simplified, just for your example. If you have expressions that might have methods that have their own parameters, then make sure to replace only the parameter that you want to!
E.g. it wont work for:
Expression<Func<List<object>, bool>> expression = x => x.Select(o => o.ToString()).Contains("Verified");
because this visitor will replace the 'o' too. If you have this case, then pass in the parameter you want to replace (e.g. x, that is expression.Parameters.First()) in the constructor too, and only replace in the overridden method if node == myOldParameter.
BTW: why do you need expression trees if you compile the thing at the end anyways?
I have looked at few examples here Calling a Method from an Expression and on MSDN but I have not been able to get the right method call/object type for Any() for the query below. I seem to be able to get the property call but not IEnumerable part of the child property.
billing_map_set_lu is the parent of billmaps_lu and is defined as an association in the Entity Framework.
The reason I am using expression trees is that
I need to be able to define the query at runtime with 1-n .SelectMany(p => p.billmaps_lu).Where(predicate) clauses. So I figured if I could build the expression trees I could handle all the different combinations I have for this system which are many.
var myResults = ctx.billing_map_set_lu
.Where(p => p.billmaps_lu.Any(b => b.billmap_columnname == "templatesittings_key" && b.billmap_columnvalue == 428264))
SelectMany(p => p.billmaps_lu)
.Where (b =>b.billmap_columnname =="locations_key" && b.billmap_columnvalue == 12445)
Select(z => z.billing_map_set_lu);
I have tried a quite a few attempts using the samples above...
ParameterExpression bms = Expression.Parameter(typeof(billmaps_lu));
Expression left1 = Expression.Property(bms, typeof(billmaps_lu).GetProperty("billmap_columnname"));
Expression right1 = Expression.Constant("templatesittings_key", typeof(string));
Expression InsideAny1 = Expression.Equal(left1, right1);
Expression left2 = Expression.Property(bms, typeof(billmaps_lu).GetProperty("billmap_columnvalue"));
Expression right2 = Expression.Constant(428264, typeof(int));
Expression InsideAny2 = Expression.Equal(left2, right2);
Expression myWhereClause1 = Expression.AndAlso(InsideAny1, InsideAny2);
The above part seems fine but when I try to do the .Any It is like I can't get the right property/method to get the right objects out. (I feel like I am on a physics problem where I am working with the wrong units.) I am hoping it is something simple that I am missing, I am pretty new to Expression Trees.. I have included non-working code to try to show you where my head is at and how someone can steer me in the right direction.
MethodInfo method = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(typeof(billing_map_set_lu).GetProperty("billmaps_lu").PropertyType);
ParameterExpression billMapSetParameter = Expression.Parameter(typeof(billing_map_set_lu), "p");
ParameterExpression billMaps = Expression.Parameter(typeof(billmaps_lu), "p1");
var myFunction = Expression.Lambda<Func<billmaps_lu, bool>>(Expression.Call(method, Expression.Property(billMapSetParameter, typeof(billing_map_set_lu).GetProperty("billmaps_lu")), myWhereClause1), billMaps)
Disclaimer, I haven't got any compiled working code.
2 problems.
First problem probably lies in:
ParameterExpression billMapSetParameter = Expression.Parameter(typeof(billing_map_set_lu), "p");
That's not a parameter you need in:
Expression.Lambda<Func<billmaps_lu, bool>>(Expression.Call(method, Expression.Property(**billMapSetParameter**, typeof(billing_map_set_lu).GetProperty("billmaps_lu")), myWhereClause1), billMaps)
Change the billMapSetParameter to the billMaps ParamterExpression, then you should be good to go. You are calling the PropertyExpression to obtain your billMapSet for you from the ParameterExpression.
2nd problem: (Not sure, but my gut feeling)
You may need to pass the Where clause as a ConstantExpression with type Expression<.Func<>>. .Any method takes two parameters, of which, the second is an Expression<.Func<>> (Or just a Func<>? can't remember).
var whereExpression = Expression.Lambda<.Func<.billmaps_lu, bool>>(myWhereClause1, bms);
var ce = Expression.Constant(whereExpression)
Then pass back ce into originally where you "myWhereClause1" is.
Cross finger it works
Edit- Scrap that, SHOW MI ZEH CODEZ
public class Foo
{
public List<string> Strings { get; set; }
}
class Program
{
static void Main(string[] args)
{
Func<Foo, bool> func =
a => a.Strings.Any(b => b == "asdf");
// b => b == "asdf";
var bParameter = Expression.Parameter(typeof (string));
var asdfConstant = Expression.Constant("asdf");
var compare = Expression.Equal(bParameter, asdfConstant);
var compareExpression = Expression.Lambda<Func<string, bool>>(compare, bParameter);
var ceCompareExpression = Expression.Constant(compareExpression.Compile());
// a => a.Strings.Any(compareExpression)
var parameter = Expression.Parameter(typeof (Foo));
var foosProperty = Expression.Property(parameter, typeof (Foo).GetProperty("Strings"));
MethodInfo method = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(typeof(string));
var anyMethod = Expression.Call(method, foosProperty, ceCompareExpression);
var lambdaExpression = Expression.Lambda<Func<Foo, bool>>(anyMethod, parameter);
// Test.
var foo = new Foo {Strings = new List<string> {"asdf", "fdsas"}};
Console.WriteLine(string.Format("original func result: {0}", func(foo)));
Console.Write(string.Format("constructed func result: {0}", lambdaExpression.Compile()(foo)));
Console.ReadKey();
}
}