We Keep Coding

LINQ Query - How to parameterize a select distinct query

I'm having a hard time figuring out what I am missing when trying to parameterize a "Select" with "Distinct" query.
This is a sample of the code I have repeated multiple times with different items to select.
private void getCustomerCodeList(ObservableCollection<Model> FilteredData)
{
var distCustomerCode = FilteredData.Select(i => new { i.CustomerCode, i.FamilyName }).Distinct().OrderBy(x => x.FamilyName).ToList();
DistinctCustomerCodeList.Clear();
foreach (var item in distCustomerCode)
{
DistinctCustomerCodeList.Add(new Model() { CustomerCode = item.CustomerCode, FamilyName = item.FamilyName });
}
}
I am trying to convert this into one method where I can pass in the "Select" and the "Orderby" as parameters. The code below is as far as I was able to get, and it works fine if I pass in a lambda with one property, but errors as soon as I try to add a second.
public void getDistinct<TKey>(ObservableCollection<Model> FilteredData, Func<Model, TKey> myDistinctProperty, Func<TKey, TKey> mySortingProperty)
{
var distinct = FilteredData.Select(myDistinctProperty).Distinct().OrderBy(mySortingProperty).ToList();
DistinctCustomerCodeList.Clear();
foreach (var item in distinct )
{
DistinctCustomerCodeList.Add(new Model() { CustomerCode = item.ToString() });
}
}
If I call this method:
getDistinct<string>(FilteredData, x => x.CustomerCode, x => x);
it works fine
but if I try:
getDistinct<string>(FilteredData, i => new { i.CustomerCode, i.FamilyName }, x => x.FamilyName)
it errors.
Can anyone shed some light on this for me.
Thanks.
Update-
Here is the error message I am receiving.
'IEnumerable' does not contain a definition for 'OrderBy' and the best extension method overload 'ParallelEnumerable.OrderBy<Model, string>(ParallelQuery, Func<Model, string>)' requires a receiver of type 'ParallelQuery'
I made the changes based on the clarification that nalpnir provided and here is what my updated code looks like.
public void getDistinct<TKey>(ObservableCollection<Model> FilteredData, Func<Model, TKey> myDistinctProperty, Func<Model, string> mySortingProperty)
{
var distinct= FilteredData.Select(myDistinctProperty).Distinct().OrderBy(mySortingProperty).ToList();
DistinctCustomerCodeList.Clear();
foreach (var item in distinct)
{
DistinctCustomerCodeList.Add(new Model() { CustomerCode = item.ToString() });
}
}
This portion has the error provided above (FilteredData.Select(myDistinctProperty).Distinct())
and here is the updated method call:
getDistinct<Model>(FilteredData, i => new Model { FamilyName = i.FamilyName, CustomerCode = i.CustomerCode }, x => x.CustomerCode);
Update 2 -
Thank you all for your feedback.
I ended up using the answer from JeremyLakeman, as it fit best in the project I am working on.
The answer from nalpnir is awesome and I will try to implement it in the future.

Your first example turned each instance of Model into an anonymous type, then later back into a Model again. Instead, the generic distinct method should create a new Model with only the requested fields.
public void getDistinct<TOrder>(ObservableCollection<Model> FilteredData, Func<Model, Model> distinct, Func<Model, TOrder> sort)
{
var distinct = FilteredData.Select(distinct).Distinct().OrderBy(sort).ToList();
DistinctCustomerCodeList.Clear();
DistinctCustomerCodeList.AddRange(distinct);
}
Then this should work, with the compiler able to infer the generic types;
getDistinct(FilteredData, i => new Model{ i.CustomerCode, i.FamilyName }, x => x.FamilyName);

The reason is simple, the LINQ you are doing is wrong, for 2 reasons but the same reason itself:
Your TKey is set as a string, so when you say new { i.CustomerCode, i.FamilyName } it doesnt know how to go from that expression to a string.
The same occurs in the case of the sorting, x is of type String, and x doesnt know what x.CustomerName is.
What you should use is the Model as TKey, like this:
getDistinct<Model>(list, i => new Model { FamilyName = i.FamilyName, CustomerCode = i.CustomerCode }, x => x.CustomerCode);
UPDATE:
Since i noticed it didnt do what you wanted to do, which is to be able to get it without repetitions, I changed a bit the idea so you can get around it, and even made it more generic so you can implement it with any class you want, as long as you make a few changes which i ll explain
public static List<T> getDistinct<T, TKey>(ObservableCollection<T> FilteredData, Func<T, TKey> myDistinctProperty, Func<T, string> mySortingProperty)
where T:Model, new()
{
var DistinctCustomerCodeList = new List<T>();
var distinct = FilteredData.GroupBy(myDistinctProperty).Select(x => x.First()).OrderBy(mySortingProperty).ToList();
foreach (var item in distinct)
{
DistinctCustomerCodeList.Add(new T() { CustomerCode = item.CustomerCode, FamilyName = item.FamilyName });
}
return DistinctCustomerCodeList;
}
As you see now you have 2 Generics, One is T which i restricted to be of type Model, and the other one is another type whichever you want in this particular case it will accept new { x.CustomerCode, x.FamilyName } . I ll give you a link to a .Net fiddle so you know can see a demo
Demo: demo
You can toy with it. Basically whichever class you replace model for will work as long as you change the output of the type T, and change the Add inside the foreach. Inherited members of Model will also work so lets say for example you implement the following:
public class ModelExtended:Model
{
public int NewProp { get; set; }
}
And also changed the ObservableCollection To ObservableCollection will also work because they share some properties

The OrderBy() is expecting a generic type TKey but the parameter mySortingProperty is explicitly referencing Model. Update the parameter in your method definition as follows:
public void getDistinct<TKey>(ObservableCollection<Model> FilteredData, Func<Model, TKey> myDistinctProperty, Func<TKey, string> mySortingProperty)

Dynamic Linq Expression in .NET Core

First
I cannot use Entity Framework for this because this is a Desktop application in which all database calls are first routed through a Web API End Point. A requirement of this application is that all traffic is HTTP traffic.
Setup
Lets say I have an IEnumerable<BaseItem> where the BaseItem is a base model class that all models inherit from.
We also dynamically instantiate and populate our collection with a dynamic class and dynamic properties that match the column names and appropriate data types of the System.DataTable that is returned from the query.
These dynamic classes also inherit from BaseItem which exposes two functions:
void SetValue(string PropertyName, object Value)
object GetValue(string PropertyName).
Issue
I need to be able to write a linq query that dynamically selects properties from the collection based on the unique constraint columns from the table that the query referenced. I have that in a separate List, where the string values match the dynamic property names on the dynamic class.
I know the solution is probably to use an Expression Tree, but all the examples I find use actual property references, and in this case, the property can only be accessed and set through the GetValue, and SetValue functions. I'm having a hard time wrapping my head around how to achieve this. Below is an example of the linq query i want, if I KNEW the fields I wanted to return.
var Result = DefaultCollection.Select(x => new { BUSINESS_UNIT = x.GetValue("BUSINESS_UNIT"), FISCAL_YEAR = x.GetValue("FISCAL_YEAR") }).Distinct();
Again, I can't write this because I don't know which fields i need to include in the select list until runtime when I can reference the UniqueConstraint List.
How can I achieve this dynamically with Linq Expression?
Start To Finish Example
You've populated an IEnumerable<BaseItem> from a System.Data.DataTable and lets say the query that you ran was SELECT FIRST_NAME, LAST_NAME, EMAIL ADDRESS FROM TABLEA.
The IEnumerable is populated from automation in which a dynamic class is generated to represent a row in the table and dynamic properties are added to the class to represent each column in the table.
Accessing the properties is via methods in BaseItem string result = item.GetValue("FIRST_NAME") and item.SetValue("EMAIL_ADDRESS", "MyEmail#Email.com');
You also queried INFORMATION_SCHEMA to get unique constraints on TABLEA and you store it in a public class ConstraintModel object that has a public List<string> Columns representing the columns that make up the unique constraint. In this case, only one column "EMAIL_ADDRESS" is in the constraint.
If the user makes changes to the IEnumerable<BaseItem> and more than one item has the same EMAIL_ADDRESS, we know that the UPDATE statement back to SQL will fail. We want to check that the unique constraint hasn't been violated.
At the point the user saves, we want to run a "Validation Check" on the in memory IEnumerable object. Ideally we would write a linq query that essentailly did a SELECT SUM(1), EMAIL_ADDRESS GROUP BY EMAIL_ADDRESS, and if any of the results had more than 1 value, we know a duplicate exists. Obviously the above syntax is SQL not Linq, but you get the idea. I can't know at compile time what fields on the BaseItem i need to select and group by, so I need to leverage the Constraint.Columns object and use it to write a dynamic linq expression. Additionally, the properties on the Dynamic Class are only exposed through the methods that they inherit from BaseItem ... GetValue(string PropertyName) and SetValue(string PropertyName, object Value);

After reading your full example it sounds very much like you do not need dynamic linq at all to do what you're trying to do, but you do need to know what the Primary Key is, why not try something like:
Update I have replaced a simple pk lookup with a func PK lookup so that composite Primary Keys can be used.
List<string> primaryKeyColumns = new List<string> {"Id"}; // or whatever else
List<string> uniques = /// ConstraintModel.Columns;
List<BaseItem> baseItems = // your baseItems
// Now, wit
Func<BaseItem, BaseItem, bool> equalPrimaryKey = (left, right) => {
foreach(var pk in primaryKeyColumns) {
if(left.GetValue(pk) != right.GetValue(pk)) {
return false;
}
}
return true;
}; // or whatever else
var violators = baseItems.Select(x => {
return baseItems.Any(z => {
foreach(var unique in uniques) {
if (z.GetValue(unique) == x.GetValue(unique)
&& !equalPrimaryKey(x, z)) {
return true;
}
}
return false;
});
}).ToArray();
This will give you an array of base items that are duplicated by any unique column

Using a minified version of my EnumerableExpressionExt class for helping build Expression trees for Enumerable expressions, and my ExpressionExt helper class:
public static class EnumerableExpressionExt {
private static Type TEnumerable = typeof(Enumerable);
private static Type TypeGenArg(this Expression e, int n) => e.Type.GetGenericArguments()[n];
public static MethodCallExpression Distinct(this Expression src) => Expression.Call(TEnumerable, "Distinct", new[] { src.TypeGenArg(0) }, src);
public static MethodCallExpression Select(this Expression src, LambdaExpression resultFne) => Expression.Call(TEnumerable, "Select", new[] { src.TypeGenArg(0), resultFne.ReturnType }, src, resultFne);
}
public static class ExpressionExt {
public static ConstantExpression AsConst<T>(this T obj) => Expression.Constant(obj, typeof(T));
public static LambdaExpression AsLambda(this Expression body) => Expression.Lambda(body);
public static LambdaExpression Lambda(this ParameterExpression p1, Expression body) => Expression.Lambda(body, p1);
public static NewExpression New(this Type t, params Expression[] vals) => Expression.New(t.GetConstructors()[0], vals, t.GetProperties());
public static ParameterExpression Param(this Type t, string pName) => Expression.Parameter(t, pName);
}
You can create custom Expression extensions to translate your specific methods:
public static class MyExpressionExt {
public static MethodCallExpression GetValue(this Expression obj, string propName) =>
Expression.Call(obj, "GetValue", null, propName.AsConst());
}
You can build your Expression tree as follows:
// BaseItem x
var xParm = typeof(BaseItem).Param("x");
// typeof(new { object BUSINESS_UNIT, object FISCAL_YEAR })
var anonType = (new { BUSINESS_UNIT = default(object), FISCAL_YEAR = default(object) }).GetType();
// new { BUSINESS_UNIT = x.GetValue("BUSINESS_UNIT"), FISCAL_YEAR = x.GetValue("FISCAL_YEAR") }
var newExpr = anonType.New(xParm.GetValue("BUSINESS_UNIT"), xParm.GetValue("FISCAL_YEAR"));
// DefaultCollection.Select(x => newExpr)
var selExpr = DefaultCollection.AsConst().Select(xParm.Lambda(newExpr));
// DefaultCollection.Select(x => newExpr).Distinct()
var distExpr = selExpr.Distinct();
And you can test it (using LINQPad's Dump method) like this:
// () => DefaultCollection.Select(x => newExpr).Distinct()
var f = distExpr.AsLambda();
var fc = f.Compile();
fc.DynamicInvoke().Dump();
LINQPad is also very helpful for outputting compiler built Expression trees and seeing how they were built, or IQueryable<T> queries Expression member trees.
NOTE: I like not having to call AsConst() so I have additional helpers to make that easier as well.

Use IQueryable.OrderBy for a composite key without loading all the items?

I am retrieving some tuples from a database that are mapped to entity classes by means of Entity Framework.
For these entities, I have a key selector function (supplied at runtime by other developers) that I would like to pass to Queryable.OrderBy. The key selector function is provided upon "registration" of the entity type in my system - which happens by means of a method that looks roughly like this:
public void RegisterEntity<TEntity, TKey>(string entityName, TKey defaultKey, Func<TEntity, TKey> keySelectorFunc)
I would like to execute this OrderBy call before materializing the results to entity objects (i.e. in such a way that the OrderBy call still gets translated to SQL under the hood).
The problem is that the entities have composite keys, and thus, the key selector function will return a custom object instantiated in the function. You can imagine it like this:
var keySelectorFunc = e => new CustomKey(e.Value1, e.Value2);
As usual, Entity Framework does not like this (the usual "Only parameterless constructors and initializers are supported in LINQ to Entities" error).
Is there any way to use such a custom key selector function to return a custom key? Do I have to resort to anonymous classes? Or should I move the OrderBy call to a place after I have left the LINQ-to-Entities world?

In this particular case it would be easy to use Sort method of Generic List.
https://msdn.microsoft.com/en-us/library/3da4abas(v=vs.110).aspx
Sort method requires the type of the list to implement IComparable interface and it uses the implementation of CompareTo method from IComparable interface. Otherwise implementation of IComparer also can be passed to this method.
So if your entity class is already implemeting IComparable interface then this should surely work for you. You will have to to .ToList() on the IQueryable result of course before you can call the Sort method on it.
public class Category : IComparable<Category>
{
public int CategoryId { get; internal set; }
public string CategoryName { get; internal set; }
public int CompareTo(Category x)
{
return String.Compare(x.CategoryName, this.CategoryName, StringComparison.InvariantCulture);
}
}
List<Category> categories = new List<Category>();
categories.Add(new Category {CategoryName = "Cate1"});
categories.Add(new Category {CategoryName = "Cate2"});
categories.Sort();
foreach (var cat in categories)
{
Console.WriteLine(cat.CategoryName);
}
This displays me category names in reverse order based on the comparison logic I have written in the CompareTo method of Category Class.

In this case I think the best way is use a custom ExtensionMethod to avoid any overhead of coding or unnecessary complexity to do that.
See if it implementation can help you.
First we create your customkey class that is responsable to create the statement expressions:
class CustomKey
{
public CustomKey(params string[] value)
{
if(!value.Any())
throw new InvalidOperationException("Select at least one Property for this operation");
Values = new List<string>();
Values.AddRange(value);
}
private List<string> Values { get; set; }
// this method run throughout all property configured to create the expressions
public void ForEachProperty<TSource, TKey>(Action<Expression<Func<TSource, TKey>>, bool> method)
{
bool firstItem = true;
Values.ForEach(f =>
{
var expression = CreateExpression<TSource, TKey>(f);
method(expression, firstItem);
firstItem = false;
});
}
// this method is responsable to create each expression
Expression<Func<TSource, TKey>> CreateExpression<TSource, TKey>(string property)
{
var parameter = Expression.Parameter(typeof(TSource), "x");
var member = typeof(TSource).GetMember(property).FirstOrDefault();
Expression body = Expression.MakeMemberAccess(parameter, member);
return Expression.Lambda<Func<TSource, TKey>>(Expression.Convert(body, typeof(object)), parameter);
}
}
After that we create your custom ExtesionMethod, somethink like that:
public static class OrderByExtensionClass
{
// instead of try passing an expression, we pass our CustomKey object with the columns to sort.
// than this method create the apropriate OrderBy Expression statement
public static IOrderedQueryable<TSource> OrderBy<TSource>(this IQueryable<TSource> source, CustomKey customKey)
{
// the parameter isFirst is just to control where we are to build the expression
customKey.ForEachProperty<TSource, object>((expression, isFirst) =>
{
if (isFirst)
source = source.OrderBy(expression);
else
source = ((IOrderedQueryable<TSource>)source).ThenBy(expression);
});
return ((IOrderedQueryable<TSource>)source);
}
}
After that we just do:
CustomKey custom = new CustomKey("Name", "Age");
myEntityContext.People.OrderBy(custom).ToList()
I hope it can help you.

Part of the problem, I think, is that OrderBy wouldn't know what to do with a complex type. SQL Server knows how to order by primitive types, but that's about it. You would have to do something like ...OrderBy(x=>x.Field1).ThenBy(x=>x.Field2). You could write an extension method that takes the key, extracts the property names from the key, and builds the .OrderBy().ThenBy() expression, as long as you know what the key will be before executing the query. Otherwise yeah, you may have to materialize the results before ordering.

LINQ filter combining exact matches like SQL IN and StartsWith matches

I'm having product entity:
public class Product : DomainBase
{
public virtual string Name { get; set; }
}
And there should be option to select products by filter, which contains an array of names, like:
public static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var session = Database.OpenSession())
{
var products = session.Query<Product>();
var result = products.Where(product => names.Any(name => product.Name.Contains(name)));
return result.ToList();
}
}
but it throws
System.NotSupportedException: Specified method is not supported.
What is right approach, to accomplish such filtering?

Without knowing more about what database you're connecting to or what library (is it RavenDB.. having done a quick Google?) then it's hard to be completely sure what the problem is.
However, what I think is happening is that you are giving an expression to the IQueryable "Where" extension method and the library is trying to turn that into search criteria to run against the db.. and failing because "Any" is not supported in nested criteria like that (again, I'm guessing).
The LINQ expressions that may or may not be translated into the database language (eg. SQL) vary by the library that performs the translation and vary by the database being talked to.
For example, the following (which is basically what you want to do) works fine with Entity Framework:
private static void Test(IEnumerable<string> names)
{
using (var context = new NORTHWNDEntities())
{
foreach (var product in context.Products.Where(product => names.Any(name => product.ProductName.Contains(name))))
{
Console.WriteLine(product.ProductName);
}
}
Console.ReadLine();
}
One easy option for you is to change your code to
public static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var session = Database.OpenSession())
{
var products = session.Query<Product>();
return result = products.ToList().Where(product => names.Any(name => product.Name.Contains(name)));
}
}
This should work.. however, it will get all Products from the database and perform the filtering in-memory. This is less efficient than getting the database to perform the search.
An alternative would be to generate an "Expression<Func<Product, bool>>" filter yourself that is easier for the library that you're using to translate. If, instead, of a nested "Any" criteria, you could generate a simple set of "OR" name checks then there is a better change of it working. The following will achieve that - but it's quite a lot of code. If this is something that you need to do in several places then some of the code could be made more general and reused.
private static IEnumerable<Product> SearchArrayQueryLinq(IEnumerable<string> names)
{
using (var context = new NORTHWNDEntities())
{
return context.Products.Where(GetCombinedOrFilter(names)).ToList();
}
}
private static Expression<Func<Product, bool>> GetCombinedOrFilter(IEnumerable<string> names)
{
var filter = GetNameFilter(names.First());
foreach (var name in names.Skip(1))
filter = CombineFiltersAsOr(filter, GetNameFilter(name));
return filter;
}
private static Expression<Func<Product, bool>> GetNameFilter(string name)
{
return product => product.ProductName.Contains(name);
}
private static Expression<Func<Product, bool>> CombineFiltersAsOr(Expression<Func<Product, bool>> x, Expression<Func<Product, bool>> y)
{
// Combine two separate expressions into one by combining as "Or". In order for this to work, instead of there being a parameter
// for each expression, the parameter from the first expression must be shared between them both (otherwise things will go awry
// when this is translated into a database query) - this is why ParameterRebinder.ReplaceParameters is required.
var expressionParameter = x.Parameters.Single();
return Expression.Lambda<Func<Product, bool>>(
Expression.Or(x.Body, ParameterRebinder.ReplaceParameters(y.Body, toReplace: y.Parameters.Single(), replaceWith: expressionParameter)),
expressionParameter
);
}
// Borrowed and tweaked from https://blogs.msdn.microsoft.com/meek/2008/05/02/linq-to-entities-combining-predicates/
public sealed class ParameterRebinder : ExpressionVisitor
{
public static Expression ReplaceParameters(Expression expression, ParameterExpression toReplace, ParameterExpression replaceWith)
{
return new ParameterRebinder(toReplace, replaceWith).Visit(expression);
}
private readonly ParameterExpression _toReplace, _replaceWith;
private ParameterRebinder(ParameterExpression toReplace, ParameterExpression replaceWith)
{
_toReplace = toReplace;
_replaceWith = replaceWith;
}
protected override Expression VisitParameter(ParameterExpression p)
{
if (p == _toReplace)
p = _replaceWith;
return base.VisitParameter(p);
}
}
Update: I didn't notice your nhibernate tag - whoops! Using the criteria combining methods that nhibernate has is probably easier than all this.. :) I would have commented on your answer rather than updating my own but I haven't got the requisite 50 rep yet..

You are trying to mix both kinds of conditions and applying IEnumerable methods on string properties.
Your query should look like this:
var result = products.Where(product => names.Contains(product.Name));
to find exact matches.
For a combination of exact matches and StartsWith it should look like this:
var results = products.Where(product => (names.Contains(product.Name) || names.Any(name => name.StartsWith(product.Name))));

As I did a dive into NHibenrate documentation, it contains CriteriaAPI, so I came up to this
using (var session = Database.OpenSession())
{
var products = session.CreateCriteria<Product>();
if (names == null)
{
return products.List<Product>();
}
var orClause = Expression.Disjunction();
foreach (var name in names)
{
orClause.Add(Expression.Like(nameof(Product.Name), name, MatchMode.Start));
}
products.Add(orClause);
return products.List<Product>();
}

Best way to pass a list of property names as Lambda Expressions

I have a class MyDummyClass to which I'd like to pass some properties in form of a Lambda expression for a later evaluation. So what I can do something like
public class MyDummyClass<T>
{
public MyDummyClass(Expression<Func<T, object>> property)
{
...
}
...
}
..and then use that class like new MyDummyClass<Person>(x=>x.Name), right?
But then I'd like to pass not only a single property but a list of properties. So I'd write my class like
public class MyDummyClass<T>
{
public MyDummyClass(IEnumerable<Expression<Func<T, object>>> properties)
{
...
}
...
}
and I'd like to use it like new MyDummyClass<Person>(new[] { x=>x.Name, x=>x.Surname }) but unfortunately that doesn't work! Instead I have to write
new MyDummyClass<Person>
(new Expression<Func<Person, object>>[] { x=>x.Name, x=>x.Surname});
But this is a bit awkward to write, isn't it? Of course, using params would work, but this is just a sample out of a more complicated piece of code where using params is not an option.
Does anyone have a better option to come out of this??

Try using params instead:
public MyDummyClass(params Expression<Func<T, object>>[] properties)
Then you should be able to do:
var dummy = new DummyClass<Person>(x => x.Name, x => x.Surname);

You could try:
public class MyDummyClass<T>
{
public MyDummyClass(Expression<Func<T, object>> expression)
{
NewArrayExpression array = expression.Body as NewArrayExpression;
foreach( object obj in ( IEnumerable<object> )( array.Expressions ) )
{
Debug.Write( obj.ToString() );
}
}
}
And then you would call it like this:
MyDummyClass<Person> cls = new MyDummyClass<Person>( item => new[] { item.Name, item.Surname } );
The problem is this won't give you the value of the property because no actual Person instance it specified Doing a ToString on "obj" will give you the name of the property. I don't know if this is what you're after, but it maybe a starting point.