I have a question for you regarding the creation of a Dynamic select query in Entity Framework.
I already have a dynamic query for the select based on rights etc. But for each table I get 30+ fields that I have to parse via the .GetType().GetProperties().
Its complex and its quite costly in terms of resource due to the amount of data we have.
I have a service that tells me which fields I should select for each table. I would like to find a way to transform that into the query but I can't find something that is really dynamic.
That is not dynamic but manual:
using (var context = new StackOverflowContext())
{
var posts = context.Posts
.Where(p => p.Tags == "<sql-server>")
.Select(p => new {p.Id, p.Title});
// Do something;
}
I need to say, select only those fields but only the fields with this names.
I have the field list in a list of string but that could be changed.
Could you please help me?
Here is a .Net Fiddle code (made by msbendtsen) that allows to dynamically select columns (properties).
https://dotnetfiddle.net/3IMR1r
The sample is written for linq to objects but it should work with entity frameworks.
The key section is:
internal static IQueryable SelectProperties<T>(this IQueryable<T> queryable, IEnumerable<string> propertyNames)
{
// get propertyinfo's from original type
var properties = typeof(T).GetProperties().Where(p => propertyNames.Contains(p.Name));
// Create the x => expression
var lambdaParameterExpression = Expression.Parameter(typeof(T));
// Create the x.<propertyName>'s
var propertyExpressions = properties.Select(p => Expression.Property(lambdaParameterExpression, p));
// Creating anonymous type using dictionary of property name and property type
var anonymousType = AnonymousTypeUtils.CreateType(properties.ToDictionary(p => p.Name, p => p.PropertyType));
var anonymousTypeConstructor = anonymousType.GetConstructors().Single();
var anonymousTypeMembers = anonymousType.GetProperties().Cast<MemberInfo>().ToArray();
// Create the new {} expression using
var anonymousTypeNewExpression = Expression.New(anonymousTypeConstructor, propertyExpressions, anonymousTypeMembers);
var selectLambdaMethod = GetExpressionLambdaMethod(lambdaParameterExpression.Type, anonymousType);
var selectBodyLambdaParameters = new object[] { anonymousTypeNewExpression, new[] { lambdaParameterExpression } };
var selectBodyLambdaExpression = (LambdaExpression)selectLambdaMethod.Invoke(null, selectBodyLambdaParameters);
var selectMethod = GetQueryableSelectMethod(typeof(T), anonymousType);
var selectedQueryable = selectMethod.Invoke(null, new object[] { queryable, selectBodyLambdaExpression }) as IQueryable;
return selectedQueryable;
}
Related
The ultimate objective is to have a piece of code that you can feed definitions of simple queries to and it generate data.
so given i can create this query
var query =
from cha in ds.Channels
select new object[]
{
cha.ChanLanguagestreamlocking,
cha.ChanMinimumtrailerduration,
from link in cha.Channelaudiolanguagelinks
select new object[]
{
link.ObjDatecreated
}
};
var data = query.ToArray();
how would i create it dynamically?
well i can get this far using a class stolen from another stack overflow question
public class SelectList<TSource>
{
private List<LambdaExpression> members = new List<LambdaExpression>();
public SelectList<TSource> Add<TValue>(Expression<Func<TSource, TValue>> selector)
{
members.Add(selector);
return this;
}
public Expression<Func<TSource, object[]>> ToDynamicColumns()
{
var parameter = Expression.Parameter(typeof(TSource), "e");
return Expression.Lambda<Func<TSource, object[]>>(
Expression.NewArrayInit(
typeof(object),
members.Select(m =>
Expression.Convert(Expression.Invoke(m, parameter), typeof(object))
)
),
parameter);
}
}
Imagine a case where the definition of which 'columns' to return is passed as a tree at runtime, this can then be mapped to predefined typessafe lambdas that define which columns to return and then the expression is created at runtime. So a hard coded version of this technique is this:
var channelColumns = new SelectList<Channel>();
channelColumns.Add(c => c.ChanLanguagestreamlocking);
channelColumns.Add(c => c.ChanMinimumtrailerduration);
var channelQuery =
ds.Channels.Select(channelColumns.ToDynamicColumns());
var bar = query.ToArray();
i.e. i can generate dynamic queries from the 'root' concept, but how do i generate the nested data.
If i do the obvious i.e. this
var audioColumns = new SelectList<Channelaudiolanguagelink>();
audioColumns.Add(a => a.ObjDatecreated);
var channelColumns = new SelectList<Channel>();
channelColumns.Add(c => c.ChanLanguagestreamlocking);
channelColumns.Add(c => c.ChanMinimumtrailerduration);
// next line causes an error
// Error CS1929 'ICollection<Channelaudiolanguagelink>' does not contain a definition for
// 'Select' and the best extension method overload
// 'Queryable.Select<Channel, object[]>(IQueryable<Channel>, Expression<Func<Channel, object[]>>)' requires a receiver of type 'IQueryable<Channel>' CSharpDb2Raw C:\Users\mark.nicholls\source\repos\scaffold2\CSharpDb2Raw\Program.cs 57 Active
channelColumns.Add(c => c.Channelaudiolanguagelinks.Select(channelColumns.ToDynamicColumns()));
var channelQuery =
ds.Channels.Select(channelColumns.ToDynamicColumns());
var bar = query.ToArray();
the error makes perfect sense.
c.Channelaudiolanguagelinks is an ICollection and so the select is looking for a Func<T,U> and I've given it an Expression<Func<T,U>>
(I don't really understand Expressions!)
Problem that you have defined method for IQueryable version of Select, so basic solution is simple - transform IEnumerable to IQueryable.
channelColumns.Add(c =>
c.Channelaudiolanguagelinks.AsQueryable().Select(audioColumns.ToDynamicColumns())
);
I'm looking for a way to dynamically create a select list from a iQueryable object.
Concrete example, i want to do something like the following:
public void CreateSelectList(IQueryable(of EntityModel.Core.User entities), string[] columns)
{
foreach(var columnID in columns)
{
switch(columnID)
{
case "Type":
SelectList.add(e => e.UserType);
break;
case "Name":
SelectList.add(e => e.Name);
break;
etc....
}
}
var selectResult = (from u in entities select objSelectList);
}
So all properties are known, i however don't know beforehand what properties are to be selected. That will be passed via the columns parameter.
I know i'm going to run into issues with the type of the selectResult type, because when the select list is dynamic, the compiler doesn't know what the properties of the anonymous type needs to be.
If the above is not possible: The scenario I need it for is the following:
I'm trying to create a class that can be implemented to display a paged/filtered list of data. This data can be anything (depends on the implementations).The linq used is linq to entities. So they are directly linked to sql data. Now i want to only select the columns of the entities that i am actually showing in the list. Therefore i want the select to be dynamic. My entity might have a hundred properties, but if only 3 of them are shown in the list, i don't want to generate a query that selects the data of all 100 columns and then only uses 3 of them. If there is a different approach that I haven't thought of, I'm open to ideas
Edit:
Some clarifications on the contraints:
- The query needs to work with linq to entities (see question subject)
- an entity might contain 100 columns, so selecting ALL columns and then only reading the ones i need is not an option.
- The end user decides what columns to show, so the columns to select are determined at run time
- i need to create a SINGLE select, having multiple select statements means having multiple queries on the database, which i don't want
Dynamic select expression to a compile time known type can easily be build using Expression.MemberInit method with MemberBindings created using the Expression.Bind method.
Here is a custom extension method that does that:
public static class QueryableExtensions
{
public static IQueryable<TResult> Select<TResult>(this IQueryable source, string[] columns)
{
var sourceType = source.ElementType;
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = columns.Select(column => Expression.Bind(
resultType.GetProperty(column), Expression.PropertyOrField(parameter, column)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda(body, parameter);
return source.Provider.CreateQuery<TResult>(
Expression.Call(typeof(Queryable), "Select", new Type[] { sourceType, resultType },
source.Expression, Expression.Quote(selector)));
}
}
The only problem is what is the TResult type. In EF Core you can pass the entity type (like EntityModel.Core.User in your example) and it will work. In EF 6 and earlier, you need a separate non entity type because otherwise you'll get NotSupportedException - The entity or complex type cannot be constructed in a LINQ to Entities query.
UPDATE: If you want a to get rid of the string columns, I can suggest you replacing the extension method with the following class:
public class SelectList<TSource>
{
private List<MemberInfo> members = new List<MemberInfo>();
public SelectList<TSource> Add<TValue>(Expression<Func<TSource, TValue>> selector)
{
var member = ((MemberExpression)selector.Body).Member;
members.Add(member);
return this;
}
public IQueryable<TResult> Select<TResult>(IQueryable<TSource> source)
{
var sourceType = typeof(TSource);
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = members.Select(member => Expression.Bind(
resultType.GetProperty(member.Name), Expression.MakeMemberAccess(parameter, member)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda<Func<TSource, TResult>>(body, parameter);
return source.Select(selector);
}
}
with sample usage:
var selectList = new SelectList<EntityModel.Core.User>();
selectList.Add(e => e.UserType);
selectList.Add(e => e.Name);
var selectResult = selectList.Select<UserDto>(entities);
What you are going for is possible, but it's not simple. You can dynamically build EF queries using the methods and classes in the System.Linq.Expressions namespace.
See this question for a good example of how you can dynamically build your Select expression.
I believe this is what you need:
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA" });
entities.Add(new User { Name = "Second", Type = "TypeB" });
string[] columns = { "Name", "Type" };
var selectResult = new List<string>();
foreach (var columnID in columns)
{
selectResult.AddRange(entities.Select(e => e.GetType().GetProperty(columnID).GetValue(e, null).ToString()));
}
foreach (var result in selectResult)
{
Console.WriteLine(result);
}
This code outputs:
First
Second
TypeA
TypeB
UPDATE (according to comments)
// initialize alist of entities (User)
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA", SomeOtherField="abc" });
entities.Add(new User { Name = "Second", Type = "TypeB", SomeOtherField = "xyz" });
// set the wanted fields
string[] columns = { "Name", "Type" };
// create a set of properties of the User class by the set of wanted fields
var properties = typeof(User).GetProperties()
.Where(p => columns.Contains(p.Name))
.ToList();
// Get it with a single select (by use of the Dynamic object)
var selectResult = entities.Select(e =>
{
dynamic x = new ExpandoObject();
var temp = x as IDictionary<string, Object>;
foreach (var property in properties)
temp.Add(property.Name, property.GetValue(e));
return x;
});
// itterate the results
foreach (var result in selectResult)
{
Console.WriteLine(result.Name);
Console.WriteLine(result.Type);
}
This code outputs:
First
TypeA
Second
TypeB
I have a query that looks like this:
var caseList = (from x in context.Cases
where allowedCaseIds.Contains(x => x.CaseId)
select new Case {
CaseId = x.CaseId,
NotifierId = x.NotifierId,
Notifier = x.NotifierId.HasValue ? new Notifier { Name = x.Notifier.Name } : null // This line throws exception
}).ToList();
A Case class can have 0..1 Notifier
The query above will result in the following System.NotSupportedException:
Unable to create a null constant value of type 'Models.Notifier'. Only entity types, enumeration types or primitive types are supported
in this context.
At the moment the only workaround I found is to loop the query result afterwards and manually populate Notifierlike this:
foreach (var c in caseList.Where(x => x.NotifierId.HasValue)
{
c.Notifier = (from x in context.Notifiers
where x.CaseId == c.CaseId
select new Notifier {
Name = x.Name
}).FirstOrDefault();
}
But I really don't want to do this because in my actual scenario it would generate hundreds of additional queries.
Is there any possible solution for a situation like this?.
I think you need to do that in two steps. First you can fetch only the data what you need with an anonymous type in a single query:
var caseList = (from x in context.Cases
where allowedCaseIds.Contains(x => x.CaseId)
select new {
CaseId = x.CaseId,
NotifierId = x.NotifierId,
NotifierName = x.Notifier.Name
}).ToList();
After that, you can work in memory:
List<Case> cases = new List<Case>();
foreach (var c in caseList)
{
var case = new Case();
case.CaseId = c.CaseId;
case.NotifierId = c.NotifierId;
case.NotifierName = c.NotifierId.HasValue ? c.NotifierName : null;
cases.Add(case);
}
You could try writing your query as a chain of function calls rather than a query expression, then put an .AsEnumerable() in between:
var caseList = context.Clases
.Where(x => allowedCaseIds.Contains(x.CaseId))
.AsEnumerable() // Switch context
.Select(x => new Case() {
CaseId = x.CaseId,
NotifierId = x.NotifierId,
Notifier = x.NotifierId.HasValue
? new Notifier() { Name = x.Notifier.Name }
: null
})
.ToList();
This will cause EF to generate an SQL query only up to the point where you put the .AsEnumerable(), further down the road, LINQ to Objects will do all the work. This has the advantage that you can use code that cannot be translated to SQL and should not require a lot of changes to your existing code base (unless you're using a lot of let expressions...)
I have data table "Car" which have 3 cols (owner, carType, colour). My question is how can i make the grouping portion more dynamic by using reflection. my idea is add the grouping col in to array, then use the reflection on the query grouping part. however i was struck at the reflection..
var gcols = new string[] { "owner", "carType" };
var reseult = dt.AsEnumerable()
.GroupBy(x => new
{
carType = x.Field<string>("carType"),
colour = x.Field<string>("colour")
})
.Select(x => new
{
CarType = x.Key.carType,
Colour = x.Key.colour,
count = x.Count()
})
.OrderBy(x => x.CarType).ToList();
If you added this extension method to object:
public static T Field<T>(this object source, string FieldName)
{
var type = source.GetType();
var field = type.GetField(FieldName);
return (T)field.GetValue(source);
}
You'd be able to use the syntax you've posted in your code.
I've not added any safety checking here so it'll need cleaning up, but it'll get you going. Ideally you'd want to check that the type of field is the same as T and a few other checks.
Using EntityFramework, I can get a list of entities of a certain type using the following syntax:
List<Customer> customers = ((IQueryable<Customer>)myEntities.Customers
.Where(c => c.Surname == strSurname)
.OrderBy(c => c.Surname)).ToList<Customer>();
I can then do something like this to end up with only the data I'm interested in:
var customerSummaries = from s in customers
select new
{
s.Surname, s.FirstName, s.Address.Postcode
};
I'm given a list of strings (based on user selection) of the fields (and tables where necessary) that comprise the requested summarized data. Eg for the above 'customerSummary' the list of strings provided would be: "Surname", "FirstName", "Address.Postcode".
My question is: How do I convert that list of strings into the syntax needed to extract only the specified fields?
If that can't be done, what would be a better type (than string) for the list of columns so I can extract the right info?
I guess I need to know the type that an EF [entity|table]'s [member|column|field] is, if that makes sense.
EDIT:
I tried the suggested answer - dynamic linq - using the following syntax
string parmList = "Surname, Firstname, Address.Postcode";
var customers = myEntities.Customers.Select(parmList)
.OrderBy("Address.Postcode");
but this results in: EntitySqlException was unhandled. 'Surname' could not be resolved in the current scope or context. Make sure that all referenced variables are in scope, that required schemas are loaded, and that namespaces are referenced correctly.
So, a follow-up question. Am I using Select properly? I have only seen examples using the Where and OrderBy clauses, but I think I'm doing it right based on those.
If my Select syntax is not the problem, can anyone see what is?
Edit 2: It was upside down. This works:
string parmList = "Surname, Firstname, Address.Postcode";
var customers = myEntities.Customers
.OrderBy("Address.Postcode")
.Select(parmList);
You can use dynamic linq, check it in here. It is also available on nuget
I suggest dynamic linq as #Cuong already posted.
But for simple Select projections and as a hand-made exercise in expressions...
Using LinqRuntimeTypeBuilder from How to create LINQ Expression Tree to select an anonymous type
(see the comments there for 'why' is that necessary)
And add this...
public static IQueryable SelectDynamic(this IQueryable source, IEnumerable<string> fieldNames)
{
Dictionary<string, PropertyInfo[]> sourceProperties = new Dictionary<string, PropertyInfo[]>();
foreach (var propertyPath in fieldNames)
{
var props = propertyPath.Split('.');
var name = props.Last();
PropertyInfo[] infos;
if (sourceProperties.TryGetValue(name, out infos))
name = string.Join("", props);
sourceProperties[name] = source.ElementType.GetDeepProperty(props);
}
Type dynamicType = LinqRuntimeTypeBuilder.GetDynamicType(sourceProperties.ToDictionary(x => x.Key, x => x.Value.Last().PropertyType));
ParameterExpression sourceItem = Expression.Parameter(source.ElementType, "t");
IEnumerable<MemberBinding> bindings = dynamicType.GetFields()
.Select(p => Expression.Bind(p, sourceItem.MakePropertyExpression(sourceProperties[p.Name]))).OfType<MemberBinding>();
Expression selector = Expression.Lambda(Expression.MemberInit(
Expression.New(dynamicType.GetConstructor(Type.EmptyTypes)), bindings), sourceItem);
MethodCallExpression selectExpression = Expression.Call(typeof(Queryable), "Select", new Type[] { source.ElementType, dynamicType }, Expression.Constant(source), selector);
return Expression.Lambda(selectExpression).Compile().DynamicInvoke() as IQueryable;
}
public static PropertyInfo[] GetDeepProperty(this Type type, params string[] props)
{
List<PropertyInfo> list = new List<PropertyInfo>();
foreach (var propertyName in props)
{
var info = type.GetProperty(propertyName);
type = info.PropertyType;
list.Add(info);
}
return list.ToArray();
}
public static Expression MakePropertyExpression(this ParameterExpression sourceItem, PropertyInfo[] properties)
{
Expression property = sourceItem;
foreach (var propertyInfo in properties)
property = Expression.Property(property, propertyInfo);
return property;
}
Use it like e.g.:
public static IEnumerable<object>
SelectAsEnumerable(this IQueryable entitySet, params string[] propertyPath)
{
return entitySet.SelectDynamic(propertyPath) as IEnumerable<object>;
}
var list = db.YourEntity.SelectAsEnumerable("Name", "ID", "TestProperty.ID").ToList();
Note:
You can use similar approach to expand for OrderBy etc. - but this wasn't meant to cover all angles (or any real queries)
Deep Property Expressions from this post of mine