We Keep Coding

Create a dynamic select function in Enity Framework

I have a question for you regarding the creation of a Dynamic select query in Entity Framework.
I already have a dynamic query for the select based on rights etc. But for each table I get 30+ fields that I have to parse via the .GetType().GetProperties().
Its complex and its quite costly in terms of resource due to the amount of data we have.
I have a service that tells me which fields I should select for each table. I would like to find a way to transform that into the query but I can't find something that is really dynamic.
That is not dynamic but manual:
using (var context = new StackOverflowContext())
{
var posts = context.Posts
.Where(p => p.Tags == "<sql-server>")
.Select(p => new {p.Id, p.Title});
// Do something;
}
I need to say, select only those fields but only the fields with this names.
I have the field list in a list of string but that could be changed.
Could you please help me?

Here is a .Net Fiddle code (made by msbendtsen) that allows to dynamically select columns (properties).
https://dotnetfiddle.net/3IMR1r
The sample is written for linq to objects but it should work with entity frameworks.
The key section is:
internal static IQueryable SelectProperties<T>(this IQueryable<T> queryable, IEnumerable<string> propertyNames)
{
// get propertyinfo's from original type
var properties = typeof(T).GetProperties().Where(p => propertyNames.Contains(p.Name));
// Create the x => expression
var lambdaParameterExpression = Expression.Parameter(typeof(T));
// Create the x.<propertyName>'s
var propertyExpressions = properties.Select(p => Expression.Property(lambdaParameterExpression, p));
// Creating anonymous type using dictionary of property name and property type
var anonymousType = AnonymousTypeUtils.CreateType(properties.ToDictionary(p => p.Name, p => p.PropertyType));
var anonymousTypeConstructor = anonymousType.GetConstructors().Single();
var anonymousTypeMembers = anonymousType.GetProperties().Cast<MemberInfo>().ToArray();
// Create the new {} expression using
var anonymousTypeNewExpression = Expression.New(anonymousTypeConstructor, propertyExpressions, anonymousTypeMembers);
var selectLambdaMethod = GetExpressionLambdaMethod(lambdaParameterExpression.Type, anonymousType);
var selectBodyLambdaParameters = new object[] { anonymousTypeNewExpression, new[] { lambdaParameterExpression } };
var selectBodyLambdaExpression = (LambdaExpression)selectLambdaMethod.Invoke(null, selectBodyLambdaParameters);
var selectMethod = GetQueryableSelectMethod(typeof(T), anonymousType);
var selectedQueryable = selectMethod.Invoke(null, new object[] { queryable, selectBodyLambdaExpression }) as IQueryable;
return selectedQueryable;
}

Dynamically selecting fields in a linq subquery

I am trying to dynamically filter the fields in a select clause similar to what was requested by a user as was described in the post dynamically selecting fields in a linq query. (I describe my problem and fully below so you don't need to go to the link)
What I am trying to do in short is to define a query and then at runtime determine what fields are part of the select statement and which are filterd out.
QueryFilter filter = new QueryFilter {
ShowClientInfo = false,
ShowPackaging = true,
ShowName = true,
ShowWeight = false
};
//Below is a sample linq query that works without desired filtering.
// Comments are to the right of the fields I want filterd with the object above
var testQuery = db.Client
.Where(c => c.ID == clientId)
.Select(c =>
new
{
ClientInfo = c, // <== Supress this
Packaging = c.ClientPackaging // <== Show this
.Where(cp => !cp.IsDeleted)
.Select(cp =>
new
{
Name = cp.Name, // <== Show this
Weight = cp.Weight // <== Suppress this
}
).ToList()
}
).FirstOrDefault();
The first answer of the related question was not sufficient, so I am not going to go into it further. It is insufficient because the linq statement would still query the database for specific fields, and would only just not assign the value on return.
The second awesome answer by Ivan Stoev is what I am trying to have expanded upon. This answer prevents the fields from even being part of the query to the database and is preferred. But this only works for the outermost select of the linq query. The part I am having trouble with is creating a helper method that will work on subqueries in the select statement.
The reason for the trouble, is the first use of the overload is of type IQueryable, the nested queries in the select are of type IEnumerable and thus will not map to the current extension of the answer.
Extension from other post mentioned above that works for the outermost select statement
public static class MyExtensions
{
public static IQueryable<TResult> Select<TSource, TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector, object options)
{
var memberInit = (MemberInitExpression)selector.Body;
var bindings = new List<MemberBinding>();
foreach (var binding in memberInit.Bindings)
{
var option = options.GetType().GetProperty("Show" + binding.Member.Name);
if (option == null || (bool)option.GetValue(options)) bindings.Add(binding);
}
var newSelector = Expression.Lambda<Func<TSource, TResult>>(
Expression.MemberInit(memberInit.NewExpression, bindings), selector.Parameters);
return source.Select(newSelector);
}
}
Here is my sample query that works with the first Select, but the nested select statements do NOT work as they need to be an IEnumerable extension.
public class QueryFilter
{
public bool ShowClientInfo {get;set;}
public bool ShowPackaging {get;set;}
public bool ShowName {get;set;}
public bool ShowWeight {get;set;}
}
QueryFilter filter = new QueryFilter {
ShowClientInfo = false,
ShowPackaging = true,
ShowName = true,
ShowWeight = false
};
var testQuery = db.Client
.Where(c => c.ID == clientId)
.Select(c =>
new
{
ClientInfo = c,
Packaging = c.ClientPackaging
.Where(cp => !cp.IsDeleted)
.Select(cp => // <==== This select does not work
new
{
Name = cp.Name,
Weight = cp.Weight,
packages = cp.ShipmentPackage
.Select(sp => // <==== Neither does this
new
{
Dimension_x = sp.Dimension_x,
Dimension_y = sp.Dimension_y,
Dimension_z = sp.Dimension_z
}, filter //<== additional filter parameter for the select statement
).ToList()
}, filter //<== additional filter parameter for the select statement
).ToList()
}, filter //<== additional filter parameter for the select statement
).FirstOrDefault();
The error I am getting on the two nested queries in the .Select is:
Method 'Select' has 1 parameter(s) but is invoked with 2 argument(s)
I need to create an IEnumerable version of the method that would work in this case.
How would I create the method as an IEnumerable?

Dynamically build select list from linq to entities query

I'm looking for a way to dynamically create a select list from a iQueryable object.
Concrete example, i want to do something like the following:
public void CreateSelectList(IQueryable(of EntityModel.Core.User entities), string[] columns)
{
foreach(var columnID in columns)
{
switch(columnID)
{
case "Type":
SelectList.add(e => e.UserType);
break;
case "Name":
SelectList.add(e => e.Name);
break;
etc....
}
}
var selectResult = (from u in entities select objSelectList);
}
So all properties are known, i however don't know beforehand what properties are to be selected. That will be passed via the columns parameter.
I know i'm going to run into issues with the type of the selectResult type, because when the select list is dynamic, the compiler doesn't know what the properties of the anonymous type needs to be.
If the above is not possible: The scenario I need it for is the following:
I'm trying to create a class that can be implemented to display a paged/filtered list of data. This data can be anything (depends on the implementations).The linq used is linq to entities. So they are directly linked to sql data. Now i want to only select the columns of the entities that i am actually showing in the list. Therefore i want the select to be dynamic. My entity might have a hundred properties, but if only 3 of them are shown in the list, i don't want to generate a query that selects the data of all 100 columns and then only uses 3 of them. If there is a different approach that I haven't thought of, I'm open to ideas
Edit:
Some clarifications on the contraints:
- The query needs to work with linq to entities (see question subject)
- an entity might contain 100 columns, so selecting ALL columns and then only reading the ones i need is not an option.
- The end user decides what columns to show, so the columns to select are determined at run time
- i need to create a SINGLE select, having multiple select statements means having multiple queries on the database, which i don't want

Dynamic select expression to a compile time known type can easily be build using Expression.MemberInit method with MemberBindings created using the Expression.Bind method.
Here is a custom extension method that does that:
public static class QueryableExtensions
{
public static IQueryable<TResult> Select<TResult>(this IQueryable source, string[] columns)
{
var sourceType = source.ElementType;
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = columns.Select(column => Expression.Bind(
resultType.GetProperty(column), Expression.PropertyOrField(parameter, column)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda(body, parameter);
return source.Provider.CreateQuery<TResult>(
Expression.Call(typeof(Queryable), "Select", new Type[] { sourceType, resultType },
source.Expression, Expression.Quote(selector)));
}
}
The only problem is what is the TResult type. In EF Core you can pass the entity type (like EntityModel.Core.User in your example) and it will work. In EF 6 and earlier, you need a separate non entity type because otherwise you'll get NotSupportedException - The entity or complex type cannot be constructed in a LINQ to Entities query.
UPDATE: If you want a to get rid of the string columns, I can suggest you replacing the extension method with the following class:
public class SelectList<TSource>
{
private List<MemberInfo> members = new List<MemberInfo>();
public SelectList<TSource> Add<TValue>(Expression<Func<TSource, TValue>> selector)
{
var member = ((MemberExpression)selector.Body).Member;
members.Add(member);
return this;
}
public IQueryable<TResult> Select<TResult>(IQueryable<TSource> source)
{
var sourceType = typeof(TSource);
var resultType = typeof(TResult);
var parameter = Expression.Parameter(sourceType, "e");
var bindings = members.Select(member => Expression.Bind(
resultType.GetProperty(member.Name), Expression.MakeMemberAccess(parameter, member)));
var body = Expression.MemberInit(Expression.New(resultType), bindings);
var selector = Expression.Lambda<Func<TSource, TResult>>(body, parameter);
return source.Select(selector);
}
}
with sample usage:
var selectList = new SelectList<EntityModel.Core.User>();
selectList.Add(e => e.UserType);
selectList.Add(e => e.Name);
var selectResult = selectList.Select<UserDto>(entities);

What you are going for is possible, but it's not simple. You can dynamically build EF queries using the methods and classes in the System.Linq.Expressions namespace.
See this question for a good example of how you can dynamically build your Select expression.

I believe this is what you need:
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA" });
entities.Add(new User { Name = "Second", Type = "TypeB" });
string[] columns = { "Name", "Type" };
var selectResult = new List<string>();
foreach (var columnID in columns)
{
selectResult.AddRange(entities.Select(e => e.GetType().GetProperty(columnID).GetValue(e, null).ToString()));
}
foreach (var result in selectResult)
{
Console.WriteLine(result);
}
This code outputs:
First
Second
TypeA
TypeB
UPDATE (according to comments)
// initialize alist of entities (User)
var entities = new List<User>();
entities.Add(new User { Name = "First", Type = "TypeA", SomeOtherField="abc" });
entities.Add(new User { Name = "Second", Type = "TypeB", SomeOtherField = "xyz" });
// set the wanted fields
string[] columns = { "Name", "Type" };
// create a set of properties of the User class by the set of wanted fields
var properties = typeof(User).GetProperties()
.Where(p => columns.Contains(p.Name))
.ToList();
// Get it with a single select (by use of the Dynamic object)
var selectResult = entities.Select(e =>
{
dynamic x = new ExpandoObject();
var temp = x as IDictionary<string, Object>;
foreach (var property in properties)
temp.Add(property.Name, property.GetValue(e));
return x;
});
// itterate the results
foreach (var result in selectResult)
{
Console.WriteLine(result.Name);
Console.WriteLine(result.Type);
}
This code outputs:
First
TypeA
Second
TypeB

How to store different GroupBy in one variable with minimal code duplication?

I have the following piece of code:
var newProducts = _summaryRepository.GetFilteredSummaries(manufacturerIds, countryIds, categoryIds,
null, widgetIds, startDate, null).Where(s => s.Product.ProductCountries.FirstOrDefault(pc => pc.CountryId == s.CountryId).CreatedAt >= startDate.Value)
.GroupBy(s => new { widgetId = s.widgetId, ProductId = s.ProductId });
If I have the condition Show all I want to take the following out of the GroupBY:
WidgetId = s.WidgetId
So the it would now be:
var newProducts = _summaryRepository.GetFilteredSummaries(manufacturerIds, countryIds, categoryIds,
null, widgetIds, startDate, null).Where(s => s.Product.ProductCountries.FirstOrDefault(pc => pc.CountryId == s.CountryId).CreatedAt >= startDate.Value)
.GroupBy(s => new {ProductId = s.ProductId });
There is a lot of code which is reliant on newProducts and when I have created an if statement and put the var newProducts in the outerscrope it stops everything working.
I know this is probably a silly question but how can I do this with the minimal repetitive code?
Is it that I'm declaring the variable wrong when I do this:
var newProducts;
if(model.allWidgets)
{newProducts = _summaryRepository.GetFilteredSummaries(manufacturerIds, countryIds, categoryIds,
null, WidgetIds, startDate, null).Where(s => s.Product.ProductCountries.FirstOrDefault(pc => pc.CountryId == s.CountryId).CreatedAt >= startDate.Value)
.GroupBy(s => new {ProductId = s.ProductId });}
else
{
newProducts = _summaryRepository.GetFilteredSummaries(manufacturerIds, countryIds, categoryIds,
null, WidgetIds, startDate, null).Where(s => s.Product.ProductCountries.FirstOrDefault(pc => pc.CountryId == s.CountryId).CreatedAt >= startDate.Value)
.GroupBy(s => new { WidgetId = s.WidgetId, ProductId = s.ProductId });
}

You can change your GroupBy to ignore s.WidgetId when model.allWidgets is true:
query.GroupBy(s => new { WidgetId = model.allWidgets ? 0 : s.WidgetId, ProductId = s.ProductId });

From LINQ Conditional Group, you can add a "null" to the grouping (WidgetId = 0), causing the GroupBy to return the same anonymous grouping type in both cases:
var newProducts = _summaryRepository.GetFilteredSummaries(...)
.Where(s => ...)
var groupedProducts = newProducts.GroupBy(s =>
{
if(model.allWidgets)
{
return new
{
ProductId = s.ProductId,
WidgetId = 0,
};
}
else
{
return new
{
ProductId = s.ProductId,
WidgetId = s.WidgetId,
};
}
});
And of course, as the just-added answer indicates, this can be greatly reduced using the conditional operator:
var groupedProducts = newProducts.GroupBy(s =>
new
{
ProductId = s.ProductId,
WidgetId = model.allWidgets ? 0 : s.WidgetId,
});
This type is still anonymous and thus not directly accessible from your code, so in order to return it from a method, introduce a class to hold the group:
public class ProductGrouping
{
public int ProductId { get; set; }
public int? WidgetId { get; set; }
}
public IGrouping<ProductGrouping, Summary> GetGroupedSummaries()
{
return _summaryRepository.GetFilteredSummaries(...)
.Where(s => ...)
.GroupBy(s => new ProductGrouping
{
ProductId = s.ProductId,
WidgetId = model.allWidgets ? (int?)null : s.WidgetId,
});
}

What you are trying to do is not really possible because your 2 queries return different anonymous types, unless you are willing to assign the results back into a variable of type object, which doesn't seem very useful (or dynamic, but then you lose the compile time benefits of using LINQ).
To be able to use the var keyword, the compiler must be able to determine the type on declaration, which means you have to provide an assignment right where you declare your variable.
As stated in the documentation for Implicitly Typed Local Variables:
The var keyword instructs the compiler to infer the type of the variable from the expression on the right side of the initialization statement.
And also...
It is important to understand that the var keyword does not mean "variant" and does not indicate that the variable is loosely typed, or late-bound. It just means that the compiler determines and assigns the most appropriate type.

Multiple linq query of order by to single one [duplicate]

This question already has answers here:
evaluate column name in linq where clause
(2 answers)
Closed 10 years ago.
I have multiple line condition for order by as below
if (enum1)
{
var = itemlist.orderby(r => r.column1)
}
else if (emum2)
{
var = itemlist.orderby(r => r.column2)
}
And so on.. Any way to do such thing dynamically.

What you can do is to better manage the column selection for sorting, e.g. using some collection that holds the Func<> for KeySelector.
E.g. if you have a class name 'SomeDTO' with four properties, Prop1,2,3 and 4. And four corresponding enum members.
var searchByMapping = new Dictionary<SearchByEnum,Func<SomeDTO, object>>();
searchByMapping.Add(SearchByEnum.Prop1, x => x.Prop1);
searchByMapping.Add(SearchByEnum.Prop2, x => x.Prop2);
searchByMapping.Add(SearchByEnum.Prop3, x => x.Prop3);
coll = coll.OrderBy(searchByMapping[searchByEnumParam]).ToList();
This is a not so dynamic approach but a typesafe one.

Supposing your enum can have only two values :
enum yourEnum( X=0, Y}
enum1 was somewhere set:
itemlist = enum1==Enum.X? itemlist.Orderby(r => r.column1): itemlist.Orderby(r => r.column2)

I did using below extension method.
public static class IQueryableExtensions
{
public static IQueryable<T> OrderBy<T>(this IQueryable<T> items, string propertyName)
{
var typeOfT = typeof(T);
var parameter = Expression.Parameter(typeOfT, "parameter");
var propertyType = typeOfT.GetProperty(propertyName).PropertyType;
var propertyAccess = Expression.PropertyOrField(parameter, propertyName);
var orderExpression = Expression.Lambda(propertyAccess, parameter);
var expression = Expression.Call(typeof(Queryable), "OrderBy", new Type[] { typeOfT, propertyType }, items.Expression, Expression.Quote(orderExpression));
return items.Provider.CreateQuery<T>(expression);
}
}