We Keep Coding

In LinqToEntities, How to pass dynamic column name to DbFunctions.Like

I have an IQueryable<T> from my DbSet in Entity Framework. I am provided a "Fuzzy Search String", named searchText, like so:
public List<T> Search<T>(string searchText)
{
using (var context = ...)
{
var baseQuery = context.Set<T>().AsQueryable();
baseQuery = baseQuery.Where(x =>
DbFunctions.Like(x.PropertyName, searchText)
|| DbFunctions.Like(x.PropertyTwo, searchText)
|| DbFunctions.Like(x.PropertyThree, searchText)
|| DbFunctio..... etc
);
return baseQuery.ToList();
}
}
But given the generic nature, I don't know what properties there are on the type. I can provide an abstract method to somebody implementing this which allows them to give me a List of Properties (or even PropertyInfo or whatever else, I can figure that out). But I don't know how to dynamically create the expression. This is what I have so far:
var baseQuery = context.Set<T>().AsQueryable();
var expression = baseQuery.Expression;
var colName = "colName"; // Or names, I can iterate.
var parameter = Expression.Parameter(typeof(T), "x");
var selector = Expression.PropertyOrField(parameter, colName);
expression = Expression.Call(typeof(DbFunctions), nameof(DbFunctions.Like),
new Type[] { baseQuery.ElementType, selector.Type },
expression, Expression.Quote(Expression.Lambda(selector, parameter)));
The problem here is... well, it doesn't work to begin with. But mainly that I'm not using the searchText anywhere in it, and don't know how to plug it in. I THINK I'm close... but have spent an inordinate amount of time on it.

Hopefully I'm getting your query logic right: if you want to build a set of LIKE conditions based on known type and list of column names, you could try something like this:
static private MethodInfo dbLikeMethod = typeof(DbFunctions).GetMethod(nameof(DbFunctions.Like), BindingFlags.Public | BindingFlags.Static, null, new Type[] {typeof(string), typeof(string)}, null); // I am targeting DbFunctions.Like(string, string). You might want another overload (or even mix them up depending on your inputs)
public List<T> Search<T>(string searchText) where T: class
{
using (var context = new ...)
{
var baseQuery = context.Set<T>().AsQueryable().Where(CreateExpression<T>(searchText));// you could probably find a more elegant way of plugging it into your query
return baseQuery.ToList();
}
}
Expression<Func<T, bool>> CreateExpression<T>(string searchText) where T : class
{
var cols = new List<string> {
"PropertyName",
"PropertyTwo" // i understand you've got a way to figure out which strings you need here
};
var parameter = Expression.Parameter(typeof(T), "x");
var dbLikeCalls = cols.Select(colName => Expression.Call(dbLikeMethod, Expression.PropertyOrField(parameter, colName), Expression.Constant(searchText))); // for convenience, generate list of DbFunctions.Like(x.<Property>, searchText) expressions here
var aggregatedCalls = dbLikeCalls.Skip(1).Aggregate((Expression)dbLikeCalls.First(), (accumulate, call) => Expression.OrElse(accumulate, call)); // aggregate the list using || operators: use first item as a seed and keep adding onto it
return Expression.Lambda<Func<T, bool>>(aggregatedCalls, parameter);
}

How to consume a delegate within an expression

I'm struggling a bit to understand how to construct an expression that consumes a delegate. I'm new to expressions and embarrassingly wasn't able to create a unit test that replicated the problem I'm seeing, so hopefully the information below will be sufficient to explain the problem.
Consider the following classes:
public class Instance
{
internal Instance(string value)
{
Value = value;
}
public string Value { get; }
public Expression<Func<Container, bool>> IsContainerMatch => c => Selector(c.Item) == Value;
public Func<Item, string> Selector => i => i.Value;
}
public class Container
{
internal Container(Item item)
{
Item = item;
}
public Item Item { get; }
}
public class Item
{
internal Item(string value)
{
Value = value;
}
public string Value { get; }
}
The IsContainerMatch expression is used as an argument on a per-Instance basis for a third party method and is compiled/used at a later time. When the expression is actually called, however, I get an error that states that the variablecis referenced from scope '' but is not defined. If I'm not mistaken, this problem can be solved if I can incorporate the Selector delegate into the expression, so that the two have the same scope. (Please do correct me if I'm wrong!)
I came across this issue, but one fundamental difference I see is that the argument to my delegate is not a constant; it's determined at compile time. I haven't had much luck figuring out how to construct the expression for my scenario. Could someone provide me with a little guidance?
EDIT: This is the simplest test I can construct that fails - sorry it's not readily repeatable. The issue only occurs when I attempt to use the expression in conjunction with NHibernate; calling the func works fine when I use the method from #juharr. When I inject the expression into the .And statement, I get the error message variable c of type Container referenced from scope '', but it is not defined.
[Fact]
public void Test()
{
var instanceList = new Collection<Instance>{new Instance("test")};
var dbConfig = OracleDataClientConfiguration.Oracle10
.ConnectionString(c => c.Is("Data Source=(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = 10.11.12.13)(PORT = 1521))(CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = orcl.test.com)));Password=test;User ID=test;"))
.Driver<NHibernate.Driver.OracleManagedDataClientDriver>();
FluentConfiguration configuration = Fluently.Configure().Database(dbConfig)
.Mappings(m => m.FluentMappings.AddFromAssembly(Assembly.GetExecutingAssembly()));
var sessionFactory = configuration.BuildSessionFactory();
var session = sessionFactory.OpenSession();
var query = session.QueryOver<Container>();
foreach (var element in instanceList) query.And(c => element.Selector(c.Item) == element.Value);
var query2 = session.QueryOver<Container>();
foreach (var element in instanceList) query.And(element.IsContainerMatch);
}
EDIT #2: Note the query2 above; it is a second use case I have. The first query is the one that throws the exception, but my intent is to reuse the delegate for both the first query as well as in the expression for the second query.

Ok, lets unpack this;
public Expression<Func<Container, bool>> IsContainerMatch => c => Selector(c.Item) == Value;
IsContainerMatch is a property, returning a new instance of an expression on each get. The expression contains constant references to the Instance to access Value and Selector. This is roughly equivalent to;
public Expression<Func<Container, bool>> IsContainerMatch { get {
var inst = Expression.Constant(this);
var container = Expression.Parameter(typeof(Container), "c");
var selector = typeof(Instance).GetProperty("Selector");
var value = typeof(Instance).GetProperty("Value");
var item = typeof(Container).GetProperty("Item");
return Expression.Lambda<Func<Container, bool>>(
Expression.Equal(
Expression.Invoke(
Expression.MakeMemberAccess(inst, selector),
Expression.MakeMemberAccess(container, item)
),
Expression.MakeMemberAccess(inst, value)
),
container
);
} }
This is unlikely to be the true source of your exception. Somewhere else a new LambdaExpression has been constructed, perhaps from pieces of this Expression, with a reference to this ParameterExpression 'C'. But with a different parameter.
For example something like this might cause that Exception;
...
return Expression.Lambda<Func<Container, bool>>(
Expression.Equal(
Expression.Invoke(
Expression.MakeMemberAccess(inst, selector),
Expression.MakeMemberAccess(Expression.Parameter(typeof(Container), "X"), item)
),
Expression.MakeMemberAccess(inst, value)
),
Expression.Parameter(typeof(Container), "Y")
);
Clearly you are attempting to use a type of Expression that your 3rd party library doesn't support.
Now that you've updated the question to include NHibernate, it seems like what you're trying to achieve is;
foreach (var element in instanceList) query.And(c => c.Item.[Dynamic member] == element.Value);
So that the criteria can be evaluated efficiently by NHibernate. But since your Selector is a compiled Func, there's no way for NHibernate to look inside that method and translate this into an efficient query.

lambda using tuple via reflection

I have some code to sync Entity Framework POCOs between different database servers (source server is SQL Server, destination is MySQL). It is written to work generically using the [Key] attribute and the fact that the POCOs representing the data being synced know how to compare themselves.
The code is currently as follows:
var srcdbset = setprop.GetValue(src, null); var dstdbset = setprop.GetValue(dst, null);
var tabletype = srcdbset.GetType().GetGenericArguments().First();
var keys = tabletype.GetProperties().Where(p => p.GetCustomAttributesData().FirstOrDefault(a => a.AttributeType.Name == "KeyAttribute") != null).ToList();
var param = Expression.Parameter(tabletype); // TODO - support compound keys
var dt = typeof(Func<,>).MakeGenericType(tabletype, keys[0].PropertyType);
var df = Expression.Lambda(dt, Expression.Property(param, keys[0].Name), param).Compile();
var qtodict = typeof(Enumerable).GetMethods().Where(m => m.Name == "ToDictionary").First();
var gtodict = qtodict.MakeGenericMethod(new[] { tabletype, keys[0].PropertyType });
var srcdict = ((IDictionary)gtodict.Invoke(null, new object[] { srcdbset, df }));
var dstdict = ((IDictionary)gtodict.Invoke(null, new object[] { dstdbset, df }));
var qexcept = typeof(Enumerable).GetMethods().Where(m => m.Name == "Except").First();
var gexcept = qexcept.MakeGenericMethod(new[] { keys[0].PropertyType });
dynamic snotd = gexcept.Invoke(null, new object[] { srcdict.Keys, dstdict.Keys });
dynamic dnots = gexcept.Invoke(null, new object[] { dstdict.Keys, srcdict.Keys });
A little help - src is the source DbContext and dst is the destination DbContext and setprop is the PropertyInfo object of the DbSet being synced.
This question really isn't so much about Entity Framework, but linq and reflection. As you can see the TODO comment says - "support compound keys" - the code above works fine for POCOs with just one key, but to support composite keys, the lambda expression needs to change from something like:
dbcontext.Accounts.ToDictionary(a => a.AccountID);
to something like:
dbcontext.OntLocations.ToDictionary(l => Tuple.Create(l.OntID, l.LocationID, l.Index);
The two linq expressions immediately above are obviously written in a non generic way to make things simpler to explain - my question is - how do I write a generic lambda that does the tuple create? If anyone could point me in the right direction on that, I think the rest of the code would work as is.
Also, you are probably thinking - why aren't they using transactional replication - long story short - unable to find a product that works reliably - if anyone knows of one that works well from SQL Server to MySQL and doesn't require much/any downtime to the SQL Server to install I'm all ears.

The answer by #Dave M is fine, but the whole thing can be greatly simplified by emitting a call to one of the Tuple.Create method overloads using the following handy Expression.Call method overload
public static MethodCallExpression Call(
Type type,
string methodName,
Type[] typeArguments,
params Expression[] arguments
)
which will do the most of the work for you:
var source = Expression.Parameter(tabletype, "e");
var key = Expression.Call(typeof(Tuple), "Create",
keys.Select(pi => pi.PropertyType).ToArray(), // generic type arguments
keys.Select(pi => Expression.MakeMemberAccess(source, pi)).ToArray() // arguments
);
var keySelector = Expression.Lambda(key, source).Compile();
Later on, key.Type property can be used to get the tuple type.

First you will need to get the type of tuple you need to create, this has to be hard-coded per number of keys since the Tuple class is actually a different class depending on the number of items:
Type tupleType = null;
if(keys.Count == 1) tupleType = typeof(Tuple<>);
else if(keys.Count == 2) tupleType = typeof(Tuple<,>);
else if(keys.Count == 3) tupleType = typeof(Tuple<,,>);
else if(keys.Count == 4) tupleType = typeof(Tuple<,,,>);
//and so on
tupleType = tupleType.MakeGenericType(keys.Select(t=>t.PropertyType).ToArray());
Now you can make the Func as you had it above:
var lambdaFuncType = typeof(Func<,>).MakeGenericType(tabletype, tupleType);
Now build the expression to create the tuple. We'll need the tuple constructor and a property accessor expression for each key.
var parameterExpression = Expression.Parameter(tabletype);
var constructorExpression = Expression.New(tupleType.GetConstructors()[0],
keys.Select(t=>Expression.Property(parameterExpression, t.Name)).ToArray());
Now make the full lambda and compile it:
var compiledExpression = Expression.Lambda(lambdaFuncType, constructorExpression, parameterExpression).Compile();

Building nested conditional expression-trees

I'm trying to dynamically build some sql-queries depending on a given config to only query data needed:
When writing plain linq it would look like this:
var data = dbContext
.TableOne
.Select(t1 => new TableOneSelect
{
TableOneId = t1.TableOneId,
TableOneTableTwoReference = new[] { TableOne.FirstTableTwoReference.Invoke(t1) }
.Select(t2 => new TableTwoSelect
{
TableTowId = (Guid?)t2.TableTwoId,
// ... some more properties of t2
}).FirstOrDefault(),
// ... some more properties of t1
});
whereas TableOne.FirstTableTwoReference.Invoke(t1) is defined
public static Expression<Func<TableOne, TableTwo>> FirstTableTwoReference => (t1) => t1.TableTwoReferences.FirstOrDefault();
Currently I have the following for building the TableOne-part dynamically:
public Expression<Func<TableOne, TableOneSelect>> Init(TableOneConfig cfg)
{
var memberBindings = new List<MemberBinding>();
var selectType = typeof(TableOneSelect);
var newExpression = Expression.New(selectType);
var theEntity = Expression.Parameter(typeof(TableOne), "t1");
// decide if the property is needed and add to the object-initializer
if (cfg.Select("TableOneId"))
memberBindings.Add(Expression.Bind(selectType.GetProperty("TableOneId"), Expression.Property(theEntity, nameof("TableOneId"))));
// ... check other properties of TableOneSelect depending on given config
var memberInit = Expression.MemberInit(newExpression, memberBindings);
return Expression.Lambda<Func<tblTournament, EventResourceSelect>>(memberInit, theEntity);
}
same for TableTwo (different properties and different db-table).
This I can dynamically invoke like this
dbContext.TableOne.Select(t => TableOneHelper.Init(cfg).Invoke(t1));
whereas Invoke is the one from LinqKit.
But I get stuck with the inner part for the TableOneTableTwoReference where I need to make an enumeration to call the Init of TableTwoHelper but I don't get the point how this can be achieved.
I guess Expression.NewArrayInit(typeof(TableTwo), ...) would be step one. But I still get stuck in how to pass t1.TableTwoReferences.FirstOrDefault() to this array calling the Select on.

I guess Expression.NewArrayInit(typeof(TableTwo), ...) would be step one. But I still get stuck in how to pass t1.TableTwoReferences.FirstOrDefault() to this array calling the Select on.
As I understand, the question is what is the expression equivalent of
new[] { TableOne.FirstTableTwoReference.Invoke(t1) }
It's really simple. As you correctly stated, you'll need Expression.NewArrayInit expression. However, since it expects params Expression[] initializers, instead of LINQKit Invoke extension method you should use Expression.Invoke method to emit call to TableOne.FirstTableTwoReference lambda expression with the outer theEntity ("t1") parameter:
var t2Array = Expression.NewArrayInit(
typeof(TableTwo),
Expression.Invoke(TableOne.FirstTableTwoReference, theEntity));
The same way you can emit the Select expression:
var t2Selector = TableTwoHelper.Init(cfg2);
// t2Selector is Expression<Func<TableTwo, TableTwoSelect>>
var t2Select = Expression.Call(
typeof(Enumerable), "Select", new[] { t2Selector.Parameters[0].Type, t2Selector.Body.Type },
t2Array, t2Selector);
then FirstOrDefault call:
var t2FirstOrDefault = Expression.Call(
typeof(Enumerable), "FirstOrDefault", new[] { t2Selector.Body.Type },
t2Select);
and finally the outer member binding:
memberBindings.Add(Expression.Bind(
selectType.GetProperty("TableOneTableTwoReference"),
t2FirstOrDefault));
This will produce the equivalent of your "plain linq" approach.

Add the member binding...
memberBindings.Add(Expression.Bind(selectType.GetProperty("TableOneTableTwoReference"), BuildTableTwoExpression(theEntity)));
...and then build TableTwo's expression
private Expression BuildTableTwoExpression(ParameterExpression t1)
{
var arrayEx = Expression.NewArrayInit(typeof(TableTwo), Expression.Invoke(TableOne.FirstTableTwoReference, t1));
Expression<Func<TableTwo, TableTwoSelect>> selector = (t2 => new TableTwoSelect
{
TableTowId = (Guid?)t2.TableTwoId,
// ... some more properties of t2
});
Expression<Func<IEnumerable<TableTwo>, TableTwoSelect>> selectEx =
((t1s) => Enumerable.Select(t1s, selector.Compile()).FirstOrDefault());
return Expression.Invoke(selectEx, arrayEx);
}

Dynamically build select list from linq to entities query

I'm looking for a way to dynamically create a select list from a iQueryable object.
Concrete example, i want to do something like the following:
public void CreateSelectList(IQueryable(of EntityModel.Core.User entities), string[] columns)
{
foreach(var columnID in columns)
{
switch(columnID)
{
case "Type":
SelectList.add(e => e.UserType);
break;
case "Name":
SelectList.add(e => e.Name);
break;
etc....
}
}
var selectResult = (from u in entities select objSelectList);
}
So all properties are known, i however don't know beforehand what properties are to be selected. That will be passed via the columns parameter.
I know i'm going to run into issues with the type of the selectResult type, because when the select list is dynamic, the compiler doesn't know what the properties of the anonymous type needs to be.
If the above is not possible: The scenario I need it for is the following:
I'm trying to create a class that can be implemented to display a paged/filtered list of data. This data can be anything (depends on the implementations).The linq used is linq to entities. So they are directly linked to sql data. Now i want to only select the columns of the entities that i am actually showing in the list. Therefore i want the select to be dynamic. My entity might have a hundred properties, but if only 3 of them are shown in the list, i don't want to generate a query that selects the data of all 100 columns and then only uses 3 of them. If there is a different approach that I haven't thought of, I'm open to ideas
Edit:
Some clarifications on the contraints:
- The query needs to work with linq to entities (see question subject)
- an entity might contain 100 columns, so selecting ALL columns and then only reading the ones i need is not an option.
- The end user decides what columns to show, so the columns to select are determined at run time
- i need to create a SINGLE select, having multiple select statements means having multiple queries on the database, which i don't want

Dynamic select expression to a compile time known type can easily be build using Expression.MemberInit method with MemberBindings created using the Expression.Bind method.
Here is a custom extension method that does that:
public static class QueryableExtensions
{
public static IQueryable<TResult> Select<TResult>(this IQueryable source, string[] columns)
{
var sourceType = source.ElementType;
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = columns.Select(column => Expression.Bind(
resultType.GetProperty(column), Expression.PropertyOrField(parameter, column)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda(body, parameter);
return source.Provider.CreateQuery<TResult>(
Expression.Call(typeof(Queryable), "Select", new Type[] { sourceType, resultType },
source.Expression, Expression.Quote(selector)));
}
}
The only problem is what is the TResult type. In EF Core you can pass the entity type (like EntityModel.Core.User in your example) and it will work. In EF 6 and earlier, you need a separate non entity type because otherwise you'll get NotSupportedException - The entity or complex type cannot be constructed in a LINQ to Entities query.
UPDATE: If you want a to get rid of the string columns, I can suggest you replacing the extension method with the following class:
public class SelectList<TSource>
{
private List<MemberInfo> members = new List<MemberInfo>();
public SelectList<TSource> Add<TValue>(Expression<Func<TSource, TValue>> selector)
{
var member = ((MemberExpression)selector.Body).Member;
members.Add(member);
return this;
}
public IQueryable<TResult> Select<TResult>(IQueryable<TSource> source)
{
var sourceType = typeof(TSource);
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = members.Select(member => Expression.Bind(
resultType.GetProperty(member.Name), Expression.MakeMemberAccess(parameter, member)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda<Func<TSource, TResult>>(body, parameter);
return source.Select(selector);
}
}
with sample usage:
var selectList = new SelectList<EntityModel.Core.User>();
selectList.Add(e => e.UserType);
selectList.Add(e => e.Name);
var selectResult = selectList.Select<UserDto>(entities);

What you are going for is possible, but it's not simple. You can dynamically build EF queries using the methods and classes in the System.Linq.Expressions namespace.
See this question for a good example of how you can dynamically build your Select expression.

I believe this is what you need:
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA" });
entities.Add(new User { Name = "Second", Type = "TypeB" });
string[] columns = { "Name", "Type" };
var selectResult = new List<string>();
foreach (var columnID in columns)
{
selectResult.AddRange(entities.Select(e => e.GetType().GetProperty(columnID).GetValue(e, null).ToString()));
}
foreach (var result in selectResult)
{
Console.WriteLine(result);
}
This code outputs:
First
Second
TypeA
TypeB
UPDATE (according to comments)
// initialize alist of entities (User)
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA", SomeOtherField="abc" });
entities.Add(new User { Name = "Second", Type = "TypeB", SomeOtherField = "xyz" });
// set the wanted fields
string[] columns = { "Name", "Type" };
// create a set of properties of the User class by the set of wanted fields
var properties = typeof(User).GetProperties()
.Where(p => columns.Contains(p.Name))
.ToList();
// Get it with a single select (by use of the Dynamic object)
var selectResult = entities.Select(e =>
{
dynamic x = new ExpandoObject();
var temp = x as IDictionary<string, Object>;
foreach (var property in properties)
temp.Add(property.Name, property.GetValue(e));
return x;
});
// itterate the results
foreach (var result in selectResult)
{
Console.WriteLine(result.Name);
Console.WriteLine(result.Type);
}
This code outputs:
First
TypeA
Second
TypeB