We Keep Coding

Dynamic LINQ Performance

I have a routine (written with the generous help of others here) that allows me to take a List objects, and using any number of properties in any order it dynamically builds a TreeView structure with a Count at each node. This dynamic ability is a firm user requirement.
So a source List of:
{Prop1 = "A", Prop2 = "I", Prop3 = "X"},
{Prop1 = "A", Prop2 = "J", Prop3 = "X"},
{Prop1 = "B", Prop2 = "I", Prop3 = "X"},
{Prop1 = "B", Prop2 = "I", Prop3 = "Y"},
{Prop1 = "C", Prop2 = "K", Prop3 = "Z"}
Gives the following when the Selection is by Prop1 by Prop3:
Total (5)
- A(2)
- - X(2)
- B(2)
- - X(1)
- - Y(1)
- C(1)
- - Z(1)
Functionally this works fine. However, the performance leaves a lot to be desired when the number of distinct values increases. For example - one particular run in a dataset with 5K objects and 1K distinct values in Prop1 will take several seconds.
Here is the routine:
public static class TreeBuilder
{
public static Dictionary<string, TreeItem> BuildTree<TSource>(List<TSource> source, List<string> columns)
{
return new Dictionary<string, TreeItem>()
{
{ "Total",
new TreeItem()
{
Key = "Total",
RawKey = "Total",
Count = source.Count,
Items = GroupBySelector<TSource>(source, columns, 0, "Total")
}
}
};
}
public static MethodInfo GetGenericMethod(this Type type, string name, Type[] genericTypeArgs, Type[] paramTypes)
{
foreach (MethodInfo method in type.GetMethods())
if (method.Name == name)
{
var pa = method.GetParameters();
if (pa.Length == paramTypes.Length)
{
var genericMethod = method.MakeGenericMethod(genericTypeArgs);
if (genericMethod.GetParameters().Select(p => p.ParameterType).SequenceEqual(paramTypes))
return genericMethod;
}
}
return null;
}
private static MethodInfo GetGroupByMethodStatically<TElement, TKey>()
=> typeof(Enumerable).GetGenericMethod("GroupBy", new[] { typeof(TElement), typeof(TKey) }, new[] { typeof(IEnumerable<TElement>), typeof(Func<TElement, TKey>) });
private static MethodInfo GetEnumerableMethod(string methodName, Type tElement, Type tTKey)
{
var tIElement = typeof(IEnumerable<>).MakeGenericType(tElement);
var tFunction = typeof(Func<,>).MakeGenericType(tElement, tTKey);
return typeof(Enumerable).GetGenericMethod(methodName, new[] { tElement, tTKey }, new[] { tIElement, tFunction });
}
private static MethodInfo GetEnumerableMethod(string methodName, Type tElement)
{
var tIELEMENT = typeof(IEnumerable<>).MakeGenericType(tElement);
return typeof(Enumerable).GetGenericMethod(methodName, new[] { tElement }, new[] { tIELEMENT });
}
public static Dictionary<string, TreeItem> GroupBySelector<TElement>(IEnumerable<TElement> source, IList<string> columnNames, int entry = 0, string key = "")
{
if (source == null) throw new ArgumentNullException(nameof(source));
List<string> columnParameters = columnNames[entry].Split('|').ToList();
string columnName = columnParameters[0];
if (columnName == null) throw new ArgumentNullException(nameof(columnName));
if (columnName.Length == 0) throw new ArgumentException(nameof(columnName));
int nextEntry = entry + 1;
var tElement = typeof(TElement);
var tIElement = typeof(IEnumerable<TElement>);
var keyParm = Expression.Parameter(tElement);
var prop = Expression.Property(keyParm, columnName);
var param = Expression.Parameter(tIElement, "p");
var groupByMethod = GetEnumerableMethod("GroupBy", tElement, prop.Type);
var groupByExpr = Expression.Lambda(prop, keyParm);
var bodyExprCall = Expression.Call(groupByMethod, param, groupByExpr);
var tSelectInput = typeof(IGrouping<,>).MakeGenericType(prop.Type, tElement);
var selectParam = Expression.Parameter(tSelectInput, "p");
var tKey = typeof(TreeItem).GetMember("Key").Single();
var tRawKey = typeof(TreeItem).GetMember("RawKey").Single();
var tCount = typeof(TreeItem).GetMember("Count").Single();
var tParentKey = typeof(TreeItem).GetMember("ParentKey").Single();
var tItems = typeof(TreeItem).GetMember("Items").Single();
Expression selectParamKey = Expression.Property(selectParam, "Key");
Expression selectParamRawKey = selectParamKey;
if (selectParamKey.Type != typeof(string))
{
var toStringMethod = selectParamKey.Type.GetMethod("ToString", Type.EmptyTypes);
selectParamKey = Expression.Call(selectParamKey, toStringMethod);
}
if (selectParamRawKey.Type != typeof(string))
{
var toStringMethod = selectParamRawKey.Type.GetMethod("ToString", Type.EmptyTypes);
selectParamRawKey = Expression.Call(selectParamRawKey, toStringMethod);
}
var countMethod = GetEnumerableMethod("Count", tElement);
var countMethodExpr = Expression.Call(countMethod, selectParam);
var concatFullKeyExpr = Expression.Call(typeof(string).GetMethod("Concat", new[] { typeof(string), typeof(string), typeof(string) }),
Expression.Constant(key),
Expression.Constant("|"),
selectParamRawKey);
var groupBySelectorMethod = GetGenericMethod(MethodBase.GetCurrentMethod().DeclaringType, "GroupBySelector", new[] { tElement }, new[] { tIElement, typeof(IList<string>), typeof(int), typeof(string) });
var groupBySelectorMethodExpr = Expression.Call(groupBySelectorMethod, selectParam, Expression.Constant(columnNames), Expression.Constant(nextEntry), concatFullKeyExpr);
var newMenuItemExpr = Expression.New(typeof(TreeItem));
var selectBodyExpr = Expression.MemberInit(newMenuItemExpr, new[] {
Expression.Bind(tKey, selectParamKey),
Expression.Bind(tRawKey, selectParamRawKey),
Expression.Bind(tParentKey, Expression.Constant(key) ),
Expression.Bind(tCount, countMethodExpr),
Expression.Bind(tItems, groupBySelectorMethodExpr)
});
var selectBodyExprLamba = Expression.Lambda(selectBodyExpr, selectParam);
var selectBodyLastExpr = Expression.MemberInit(newMenuItemExpr, new[] {
Expression.Bind(tKey, selectParamKey),
Expression.Bind(tRawKey, selectParamRawKey),
Expression.Bind(tParentKey, Expression.Constant(key) ),
Expression.Bind(tCount, countMethodExpr)
});
var selectBodyLastExprLambda = Expression.Lambda(selectBodyLastExpr, selectParam);
var selectMethod = GetEnumerableMethod("Select", tSelectInput, typeof(TreeItem));
bodyExprCall = Expression.Call(selectMethod, bodyExprCall, (nextEntry < columnNames.Count) ? selectBodyExprLamba : selectBodyLastExprLambda);
var selectParamout = Expression.Parameter(typeof(TreeItem), "o");
Expression selectParamKeyout = Expression.Property(selectParamout, "FullKey");
var selectParamKeyLambda = Expression.Lambda(selectParamKeyout, selectParamout);
var lmi = GetEnumerableMethod("ToDictionary", typeof(TreeItem), typeof(string));
bodyExprCall = Expression.Call(lmi, bodyExprCall, selectParamKeyLambda);
var returnFunc = Expression.Lambda<Func<IEnumerable<TElement>, Dictionary<string, TreeItem>>>(bodyExprCall, param).Compile();
return returnFunc(source);
}
}
The routine is used to take data from a DB table and convert it into a hierarchical structure for use in a WPF TreeView.
Dictionary<string, TreeItem> treeItems = new Dictionary<string, TreeItem>();
treeItems = TreeBuilder.BuildTree<IDBRecord>(DBService.GetDBRecordList(), PropertySortList);
Can anyone offer any advice on how to improve the performance of this routine? Or suggest any alternative way of achieving the same result in a more performant way?
Thanks

A couple of optimizations are possible. A lot of time is spent in the call to Compile and you are calling Compile for each key at each level in the tree, which is adding up to a lot of overhead, about 7 seconds on my tests of 5k items. I first changed the code to pull out all static Reflection that had fixed types, so it is only done once per program run. That only made a small difference, since building the Expression tree is not the main issue.
I then changed the method to separate building the Expression from compiling the Expression and calling the resultant lambda. This allowed me to modify the recursive call to the Expression builder to instead be an inline Invoke of a new lambda for the new level. Then I called compile once on the resulting Expression and executed it. The new version no longer takes the entry parameter, but it could be put back in.
This reduced the overall time from about 7.6 seconds to 0.14 seconds for around a 50x speedup. A test with all three properties resulted in a 280x speedup.
If it is still possible for repeated calls to the method, adding a cache would be of even more benefit, though a quick test only shows about 14% time savings, and in the hundredths of a second of real time.
static MemberInfo tKey = typeof(TreeItem).GetMember("Key").Single();
static MemberInfo tRawKey = typeof(TreeItem).GetMember("RawKey").Single();
static MemberInfo tCount = typeof(TreeItem).GetMember("Count").Single();
static MemberInfo tParentKey = typeof(TreeItem).GetMember("ParentKey").Single();
static MemberInfo tItems = typeof(TreeItem).GetMember("Items").Single();
// Concat(string, string, string)
static MethodInfo Concat3MI = ((Func<string, string, string, string>)String.Concat).Method;
// new TreeItem() { ... }
static NewExpression newMenuItemExpr = Expression.New(typeof(TreeItem));
// Enumerable.ToDictionary<TreeItem>(IEnumerable<TreeItem>, Func<TreeItem,string>)
static MethodInfo ToDictionaryMI = GetEnumerableMethod("ToDictionary", typeof(TreeItem), typeof(string));
static Expression<Func<IEnumerable<TElement>, Dictionary<string, TreeItem>>> BuildGroupBySelector<TElement>(IList<string> columnNames, int entry, Expression key) {
List<string> columnParameters = columnNames[entry].Split('|').ToList();
string columnName = columnParameters[0];
if (columnName == null) throw new ArgumentNullException(nameof(columnName));
if (columnName.Length == 0) throw new ArgumentException(nameof(columnName));
int nextEntry = entry + 1;
var tElement = typeof(TElement);
var tIElement = typeof(IEnumerable<TElement>);
// (TElement kp)
var keyParm = Expression.Parameter(tElement, "kp");
// kp.columnName
var prop = Expression.Property(keyParm, columnName);
// (IEnumerable<TElement> p)
var IEParam = Expression.Parameter(tIElement, "p");
// GroupBy<TElement>(IEnumerable<TElement>, Func<TElement, typeof(kp.columnName)>)
var groupByMethod = GetEnumerableMethod("GroupBy", tElement, prop.Type);
// kp => kp.columnName
var groupByExpr = Expression.Lambda(prop, keyParm);
// GroupBy(p, kp => kp.columnName)
var bodyExprCall = Expression.Call(groupByMethod, IEParam, groupByExpr);
// typeof(IGrouping<typeof(kp.columnName), TElement>)
var tSelectInput = typeof(IGrouping<,>).MakeGenericType(prop.Type, tElement);
// (IGrouping<typeof(kp.columnName), TElement> sp)
var selectParam = Expression.Parameter(tSelectInput, "sp");
// sp.Key
Expression selectParamKey = Expression.Property(selectParam, "Key");
Expression selectParamRawKey = selectParamKey;
if (selectParamKey.Type != typeof(string)) {
var toStringMethod = selectParamKey.Type.GetMethod("ToString", Type.EmptyTypes);
// sp.Key.ToString()
selectParamKey = Expression.Call(selectParamKey, toStringMethod);
selectParamRawKey = selectParamKey;
}
// Count<TElement>()
var countMethod = GetEnumerableMethod("Count", tElement);
// sp.Count()
var countMethodExpr = Expression.Call(countMethod, selectParam);
LambdaExpression selectBodyExprLamba;
if (nextEntry < columnNames.Count) {
// Concat(key, "|", sp.Key.ToString())
var concatFullKeyExpr = Expression.Call(Concat3MI, key, Expression.Constant("|"), selectParamRawKey);
// p# => p#.GroupBy().Select().ToDictionary()
var groupBySelectorLambdaExpr = BuildGroupBySelector<TElement>(columnNames, nextEntry, (Expression)concatFullKeyExpr);
// Invoke(p# => p#..., sp#)
var groupBySelectorInvokeExpr = Expression.Invoke(groupBySelectorLambdaExpr, selectParam);
var selectBodyExpr = Expression.MemberInit(newMenuItemExpr, new[] {
Expression.Bind(tKey, selectParamKey),
Expression.Bind(tRawKey, selectParamRawKey),
Expression.Bind(tParentKey, key ),
Expression.Bind(tCount, countMethodExpr),
Expression.Bind(tItems, groupBySelectorInvokeExpr)
});
// sp => new TreeItem { Key = sp.Key.ToString(), RawKey = sp.Key.ToString(), ParentKey = key, Count = sp.Count(), Items = Invoke(p# => p#..., sp)) }
selectBodyExprLamba = Expression.Lambda(selectBodyExpr, selectParam);
}
else { // Last Level
var selectBodyExpr = Expression.MemberInit(newMenuItemExpr, new[] {
Expression.Bind(tKey, selectParamKey),
Expression.Bind(tRawKey, selectParamRawKey),
Expression.Bind(tParentKey, key ),
Expression.Bind(tCount, countMethodExpr)
});
// sp => new TreeItem { Key = sp.Key.ToString(), RawKey = sp.Key.ToString(), ParentKey = key, Count = sp.Count() }
selectBodyExprLamba = Expression.Lambda(selectBodyExpr, selectParam);
}
// Enumerable.Select<IGrouping<typeof<kp.columnName>, TElement>>(IEnumerable<IGrouping<>>, Func<IGrouping<>, TreeItem>)
var selectMethod = GetEnumerableMethod("Select", tSelectInput, typeof(TreeItem));
// p.GroupBy(kp => kp => kp.columnName).Select(sp => ...)
bodyExprCall = Expression.Call(selectMethod, bodyExprCall, selectBodyExprLamba);
// (TreeItem o)
var selectParamout = Expression.Parameter(typeof(TreeItem), "o");
// o.FullKey
Expression selectParamKeyout = Expression.Property(selectParamout, "FullKey");
// o => o.FullKey
var selectParamKeyLambda = Expression.Lambda(selectParamKeyout, selectParamout);
// p.GroupBy(...).Select(...).ToDictionary(o => o.FullKey)
bodyExprCall = Expression.Call(ToDictionaryMI, bodyExprCall, selectParamKeyLambda);
// p => p.GroupBy(kp => kp => kp.columnName).Select(sp => ...).ToDictionary(o => o.FullKey)
return Expression.Lambda<Func<IEnumerable<TElement>, Dictionary<string, TreeItem>>>(bodyExprCall, IEParam);
}
public static Dictionary<string, TreeItem> GroupBySelector<TElement>(IEnumerable<TElement> source, IList<string> columnNames, string key = "") {
if (source == null) throw new ArgumentNullException(nameof(source));
// p => p.GroupBy(kp => kp => kp.columnName).Select(sp => ...).ToDictionary(o => o.FullKey)
var returnFunc = BuildGroupBySelector<TElement>(columnNames, 0, Expression.Constant(key)).Compile();
return returnFunc(source);
}

Include more rows in Pivot

I am using the extension method in below link to pivot my data:
https://techbrij.com/pivot-c-array-datatable-convert-column-to-row-linq
I am including the code from the link just in case somebody finds this question in the future and the link is dead:
public static DataTable ToPivotTable<T, TColumn, TRow, TData>(
this IEnumerable<T> source,
Func<T, TColumn> columnSelector,
Expression<Func<T, TRow>> rowSelector,
Func<IEnumerable<T>, TData> dataSelector)
{
DataTable table = new DataTable();
var rowName = ((MemberExpression)rowSelector.Body).Member.Name;
table.Columns.Add(new DataColumn(rowName));
var columns = source.Select(columnSelector).Distinct();
foreach (var column in columns)
table.Columns.Add(new DataColumn(column.ToString()));
var rows = source.GroupBy(rowSelector.Compile())
.Select(rowGroup => new
{
Key = rowGroup.Key,
Values = columns.GroupJoin(
rowGroup,
c => c,
r => columnSelector(r),
(c, columnGroup) => dataSelector(columnGroup))
});
foreach (var row in rows)
{
var dataRow = table.NewRow();
var items = row.Values.Cast<object>().ToList();
items.Insert(0, row.Key);
dataRow.ItemArray = items.ToArray();
table.Rows.Add(dataRow);
}
return table;
}
Referring to the example in the link, you get the pivoted data like;
var pivotTable = data.ToPivotTable(
item => item.Year,
item => item.Product,
items => items.Any() ? items.Sum(x=>x.Sales) : 0);
My question is, how can I include more rows into this query to return for example, ProductCode as well.. item => new {item.Product, item.ProductCode} does not work..
============== EDIT / 23 OCT 2018 ==============
Assuming my data is this;
With the help of the above mentioned code, I can manage to do this;
What I want to achieve is this (extra col: STOCKID or any other cols as well);

Anonymous types cannot be passed as generic parameters. Try defining your pivot key as a struct:
public struct PivotKey
{
public string Product;
public int ProductCode; // assuming your product codes are integers
}
This way you can take advantage of struct's default Equals and GetHashCode methods implementation in terms of the equality and hash code of all the fields.
Then, define the rowSelector as below:
item => new PivotKey { Product = item.Product, ProductCode = item.ProductCode}

Example: https://dotnetfiddle.net/mXr9sh
The issue seems to be fetching the row names from the expression, since it's only designed to handle one row. That can be fixed by this function:
public static IEnumerable<string> GetMemberNames<T1, T2>(Expression<Func<T1, T2>> expression)
{
var memberExpression = expression.Body as MemberExpression;
if (memberExpression != null)
{
return new[]{ memberExpression.Member.Name };
}
var memberInitExpression = expression.Body as MemberInitExpression;
if (memberInitExpression != null)
{
return memberInitExpression.Bindings.Select(x => x.Member.Name);
}
var newExpression = expression.Body as NewExpression;
if (newExpression != null)
{
return newExpression.Arguments.Select(x => (x as MemberExpression).Member.Name);
}
throw new ArgumentException("expression"); //use: `nameof(expression)` if C#6 or above
}
Once you have this function you can replace these lines:
var rowName = ((MemberExpression)rowSelector.Body).Member.Name;
table.Columns.Add(new DataColumn(rowName));
With this:
var rowNames = GetMemberNames(rowSelector);
rowNames.ToList().ForEach(x => table.Columns.Add(new DataColumn(x)));
One downside of this approach is the various values for these columns get returned concatenated in a single column; so you'll need to extract the data from the strings.
Resulting DataTable:
(displayed as JSON)
[
{
"StockId": "{ StockId = 65, Name = Milk }",
"Name": "3",
"Branch 1": "1",
"Branch 2": "0",
"Central Branch": null
},
{
"StockId": "{ StockId = 67, Name = Coffee }",
"Name": "0",
"Branch 1": "0",
"Branch 2": "22",
"Central Branch": null
}
]
Full Code Listing
using System;
using System.Data;
using System.Linq;
using System.Linq.Expressions;
using System.Collections.Generic;
using Newtonsoft.Json; //just for displaying output
public class Program
{
public static void Main()
{
var data = new[] {
new { StockId = 65, Name = "Milk", Branch = 23, BranchName = "Branch 1", Stock = 3 },
new { StockId = 65, Name = "Milk", Branch = 24, BranchName = "Branch 2", Stock = 1 },
new { StockId = 67, Name = "Coffee", Branch = 22, BranchName = "Central Branch", Stock = 22 }
};
var pivotTable = data.ToPivotTable(
item => item.BranchName,
item => new {item.StockId, item.Name},
items => items.Any() ? items.Sum(x=>x.Stock) : 0);
//easy way to view our pivotTable if using linqPad or similar
//Console.WriteLine(pivotTable);
//if not using linqPad, convert to JSON for easy display
Console.WriteLine(JsonConvert.SerializeObject(pivotTable, Formatting.Indented));
}
}
public static class PivotExtensions
{
public static DataTable ToPivotTable<T, TColumn, TRow, TData>(
this IEnumerable<T> source,
Func<T, TColumn> columnSelector,
Expression<Func<T, TRow>> rowSelector,
Func<IEnumerable<T>, TData> dataSelector)
{
DataTable table = new DataTable();
//foreach (var row in rowSelector()
var rowNames = GetMemberNames(rowSelector);
rowNames.ToList().ForEach(x => table.Columns.Add(new DataColumn(x)));
var columns = source.Select(columnSelector).Distinct();
foreach (var column in columns)
table.Columns.Add(new DataColumn(column.ToString()));
var rows = source.GroupBy(rowSelector.Compile())
.Select(rowGroup => new
{
Key = rowGroup.Key,
Values = columns.GroupJoin(
rowGroup,
c => c,
r => columnSelector(r),
(c, columnGroup) => dataSelector(columnGroup))
});
foreach (var row in rows)
{
var dataRow = table.NewRow();
var items = row.Values.Cast<object>().ToList();
items.Insert(0, row.Key);
dataRow.ItemArray = items.ToArray();
table.Rows.Add(dataRow);
}
return table;
}
public static IEnumerable<string> GetMemberNames<T1, T2>(Expression<Func<T1, T2>> expression)
{
var memberExpression = expression.Body as MemberExpression;
if (memberExpression != null)
{
return new[]{ memberExpression.Member.Name };
}
var memberInitExpression = expression.Body as MemberInitExpression;
if (memberInitExpression != null)
{
return memberInitExpression.Bindings.Select(x => x.Member.Name);
}
var newExpression = expression.Body as NewExpression;
if (newExpression != null)
{
return newExpression.Arguments.Select(x => (x as MemberExpression).Member.Name);
}
throw new ArgumentException("expression"); //use: `nameof(expression)` if C#6 or above
}
}

Pivot table with more columns

I was trying to pivot a list of records, then i came across this :
public static DataTable PivotTable<T, TColumn, TGroupRow, TRow, TData>(
this System.Collections.Generic.IEnumerable<T> list,
Func<T, TColumn> column,
Expression<Func<T, TRow>> row,
Func<IEnumerable<T>, TData> dataSelector)
{
DataTable table = new DataTable();
var rowName = ((MemberExpression)row.Body).Member.Name;
table.Columns.Add(new DataColumn(rowName));
var columns = list.Select(column).Distinct();
foreach (var col in columns)
table.Columns.Add(new DataColumn(col.ToString()));
var rows = list.GroupBy(row.Compile())
.Select(rowGroup => new
{
Key = rowGroup.Key,
Values = columns.GroupJoin(
rowGroup,
c => c,
r => column(r),
(c, columnGroup) => dataSelector(columnGroup))
});
foreach (var rowVal in rows)
{
var dataRow = table.NewRow();
var items = rowVal.Values.Cast<object>().ToList();
items.Insert(0, rowVal.Key);
dataRow.ItemArray = items.ToArray();
table.Rows.Add(dataRow);
}
return table;
}
This is some solution offered from this post http://techbrij.com/pivot-c-array-datatable-convert-column-to-row-linq
my client code would be like this
var _dataSourceMatrix = //Some logic that get the DataTable records
var _departmentList = //Some logic that get the DataTable records
var _joinDataSourceDepartmentList = from a in _dataSourceMatrix.AsEnumerable()
join b in _departmentList.AsEnumerable()
on a.Field<int>("EntityID") equals b.Field<int>("DepartmentID")
select new { EntityName = b.Field<string>("Name"), Period = a.Field<string>("Period"), Value = a.Field<double>("Value"), EntityID = a.Field<int>("EntityID") };
_dataSourceMatrix = _joinDataSourceDepartmentList .OrderBy(x => x.Period).PivotTable(x => x.Period, x => x.EntityName, x => x.Sum(y => y.Value));
However, this is just the 3 columns Fields : Period, EntityName and Value could be utilize. I wish to utilize the EntityID as well. It should be one of the key to group by the record with EntityName. Is there any way to modify the PivotTable function ?
The objective to be achieve should be like this :
EntityName Period1, Period2, Period3, EntityID
A Value1 Value2 Value3 1
B Value1 Value2 Value3 2

Best algorithm to determine added and removed items when comparing to collections

I am looking for the best algorithm to compare 2 collections and determine which element got added and which element got removed.
private string GetInvolvementLogging(ICollection<UserInvolvement> newInvolvement, ICollection<UserInvolvement> oldInvolvement)
{
//I defined the new and old dictionary's for you to know what useful data is inside UserInvolvement.
//Both are Dictionary<int, int>, because The Involvement is just a enum flag. Integer. UserId is also Integer.
var newDict = newInvolvement.ToDictionary(x => x.UserID, x => x.Involvement);
var oldDict = oldInvolvement.ToDictionary(x => x.UserID, x => x.Involvement);
//I Want to compare new to old -> and get 2 dictionaries: added and removed.
var usersAdded = new Dictionary<int, Involvement>();
var usersRemoved = new Dictionary<int, Involvement>();
//What is the best algoritm to accomplish this?
return GetInvolvementLogging(usersAdded, usersRemoved);
}
private string GetInvolvementLogging(Dictionary<int, Involvement> usersAdded, Dictionary<int, Involvement> usersRemoved)
{
//TODO: generate a string based on those dictionaries.
return "Change in userinvolvement: ";
}

Added elements are only in newDict removed only in oldDict
var intersection = newDict.Keys.Intersect(oldDict.Keys);
var added = newDict.Keys.Except(intersection);
var removed = oldDict.Keys.Except(intersection);
EDIT
I modify your base function, dictionaries is no neded.
Example UserInvolvement implementation
class UserInvolvement
{
public int UserId;
public string Name;
public string OtherInfo;
public override bool Equals(object obj)
{
if (obj == null)
{
return false;
}
UserInvolvement p = obj as UserInvolvement;
if ((System.Object)p == null)
{
return false;
}
return (UserId == p.UserId) && (Name == p.Name) && (OtherInfo == p.OtherInfo);
}
public override string ToString()
{
return $"{UserId} - {Name} - {OtherInfo}";
}
}
And example function:
private static string GetInvolvementLogging(ICollection<UserInvolvement> newInvolvement,
ICollection<UserInvolvement> oldInvolvement)
{
var intersection = newInvolvement.Select(x => x.UserId).Intersect(oldInvolvement.Select(x => x.UserId));
var addedIds = newInvolvement.Select(x => x.UserId).Except(intersection);
var removedIds = oldInvolvement.Select(x => x.UserId).Except(intersection);
List<UserInvolvement> modifiedUI = new List<UserInvolvement>();
foreach (var i in intersection)
{
var ni = newInvolvement.First(a => a.UserId == i);
var oi = oldInvolvement.First(a => a.UserId == i);
if (!ni.Equals(oi))
{
modifiedUI.Add(ni);
}
}
List<UserInvolvement> addedUI = newInvolvement.Where(x => addedIds.Contains(x.UserId)).Select(w => w).ToList();
List<UserInvolvement> removedUI = oldInvolvement.Where(x => removedIds.Contains(x.UserId)).Select(w => w).ToList();
StringBuilder sb = new StringBuilder();
sb.AppendLine("Added");
foreach (var added in addedUI)
{
sb.AppendLine(added.ToString());
}
sb.AppendLine("Removed");
foreach (var removed in removedUI)
{
sb.AppendLine(removed.ToString());
}
sb.AppendLine("Modified");
foreach (var modified in modifiedUI)
{
sb.AppendLine(modified.ToString());
}
return sb.ToString();
}
And my test function:
static void Main(string[] args)
{
List<UserInvolvement> newUI = new List<UserInvolvement>()
{
new UserInvolvement()
{
UserId = 1,
Name = "AAA",
OtherInfo = "QQQ"
},
new UserInvolvement()
{
UserId = 2,
Name = "BBB",
OtherInfo = "123"
},
new UserInvolvement()
{
UserId = 4,
Name = "DDD",
OtherInfo = "123ert"
}
};
List<UserInvolvement> oldUI = new List<UserInvolvement>()
{
new UserInvolvement()
{
UserId = 2,
Name = "BBBC",
OtherInfo = "123"
},
new UserInvolvement()
{
UserId = 3,
Name = "CCC",
OtherInfo = "QQ44"
},
new UserInvolvement()
{
UserId = 4,
Name = "DDD",
OtherInfo = "123ert"
}
};
string resp = GetInvolvementLogging(newUI, oldUI);
WriteLine(resp);
ReadKey();
WriteLine("CU");
}
Result is:
Added
1 - AAA - QQQ
Removed
3 - CCC - QQ44
Modified
2 - BBB - 123

You could try with Linq:
var usersAdded = newDict.Except(oldDict);
var usersRemoved = oldDict.Except(newDict);
If you need dictionaries as a result you can cast:
var usersAdded = newDict.Except(oldDict).ToDictionary(x => x.Key, x => x.Value);
var usersRemoved = oldDict.Except(newDict).ToDictionary(x => x.Key, x => x.Value);

Think best algorithm will be
foreach (var newItem in newDict)
if (!oldDict.ContainsKey(newItem.Key) || oldDict[newItem.Key]!=newItem.Value)
usersAdded.Add(newItem.Key, newItem.Value);
foreach (var oldItem in oldDict)
if (!newDict.ContainsKey(oldItem.Key) || newDict[oldItem.Key]!=oldItem.Value)
usersRemoved.Add(oldItem.Key, oldItem.Value);

Finally this is my implementation of GetInvolvementLogging:
(the implementation of the string builder method is irrelevant for my question here)
private string GetInvolvementLogging(ICollection<UserInvolvement> newInvolvement, ICollection<UserInvolvement> oldInvolvement)
{
//I defined the new and old dictionary's to focus on the relevant data inside UserInvolvement.
var newDict = newInvolvement.ToDictionary(x => x.UserID, x => (Involvement)x.Involvement);
var oldDict = oldInvolvement.ToDictionary(x => x.UserID, x => (Involvement)x.Involvement);
var intersection = newDict.Keys.Intersect(oldDict.Keys); //These are the id's of the users that were and remain involved.
var usersAdded = newDict.Keys.Except(intersection);
var usersRemoved = oldDict.Keys.Except(intersection);
var addedInvolvement = newDict.Where(x => usersAdded.Contains(x.Key)).ToDictionary(x => x.Key, x => x.Value);
var removedInvolvement = oldDict.Where(x => usersRemoved.Contains(x.Key)).ToDictionary(x => x.Key, x => x.Value);
//Check if the already involved users have a changed involvement.
foreach(var userId in intersection)
{
var newInvolvementFlags = newDict[userId];
var oldInvolvementFlags = oldDict[userId];
if ((int)newInvolvementFlags != (int)oldInvolvementFlags)
{
var xor = newInvolvementFlags ^ oldInvolvementFlags;
var added = newInvolvementFlags & xor;
var removed = oldInvolvementFlags & xor;
if (added != 0)
{
addedInvolvement.Add(userId, added);
}
if (removed != 0)
{
removedInvolvement.Add(userId, removed);
}
}
}
return GetInvolvementLogging(addedInvolvement, removedInvolvement);
}

Entity Framework query improved

I'm trying to make an android synchronization between client and ASP.NET MVC server. The logic is simple, my next method receives a data dictionary, where key = idGroup and value = LastMessageIdKnown, in the end I should get the next messages for each group what Id is higher than the LastMessageIdKnown (the value of my dictionary).
Right now I am iterating the map, for each key I do a query to my SQL database but this is inefficient, if I got N keys you can imagine what implying.
This is my current method
public Dictionary<int, List<Messages>> SynchronizedChatMessages(Dictionary<int, int> data)
{
Dictionary<int, List<Messages>> result = new Dictionary<int, List<Messages>>();
foreach(int item in data.Keys){
var idMessage= data[item];
var listMessages= _context.Messages.Where(x => x.Grupo_ID == item && x.ID > idMessage).ToList();
result.Add(item,listMessages);
}
return result;
}
How can I improve this query to get all what I need in an only and optimal way?
Thank you.

Here's an attempt that uses Predicates to make it so that there is only one Where against the whole collection of messages.
Note that I mocked this up without a database, so I am passing a List into the SynchronizedChatMessages function, whereas you have the context available.
What remains to be proven is that this way of doing things only generates one query to the database (since I did it in objects only). The whole program is further, below, but first, just the function showing use of predicates to achieve firing the Where only once.
public static Dictionary<int, List<Message>> SynchronizedChatMessages(List<Message> messages, Dictionary<int, int> data)
{
List<Predicate<Message>> predList = new List<Predicate<Message>>();
//Built of list of indivIdual predicates
foreach (var x in data)
{
var IdMessage = x.Key;
var lastMessageId = x.Value;
Predicate<Message> pred = m => m.IdGroup.Id == IdMessage && m.Id > lastMessageId;
predList.Add(pred);
}
//compose the predicates
Predicate<Message> compositePredicate = m =>
{
bool ret = false;
foreach (var pred in predList)
{
//If any of the predicates is true, the composite predicate is true (OR)
if (pred.Invoke(m) == true) { ret = true; break; }
}
return ret;
};
//do the query
var messagesFound = messages.Where(m => compositePredicate.Invoke(m)).ToList();
//get the individual distinct IdGroupIds
var IdGroupIds = messagesFound.Select(x => x.IdGroup.Id).ToList().Distinct().ToList();
//Create dictionary to return
Dictionary<int, List<Message>> result = new Dictionary<int, List<Message>>();
foreach (int i in IdGroupIds)
{
result.Add(i, messagesFound.Where(m => m.IdGroup.Id == i).ToList());
}
return result;
}
Here is the whole thing:
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApplication20
{
public class Program
{
public class Message
{
public int Id { get; set; }
public IdGroup IdGroup { get; set; }
}
public class IdGroup
{
public int Id { get; set; }
public List<Message> Messages { get; set; }
}
public static Dictionary<int, List<Message>> SynchronizedChatMessages(List<Message> messages, Dictionary<int, int> data)
{
List<Predicate<Message>> predList = new List<Predicate<Message>>();
//Built of list of indivIdual predicates
foreach (var x in data)
{
var IdMessage = x.Key;
var lastMessageId = x.Value;
Predicate<Message> pred = m => m.IdGroup.Id == IdMessage && m.Id > lastMessageId;
predList.Add(pred);
}
//compose the predicates
Predicate<Message> compositePredicate = m =>
{
bool ret = false;
foreach (var pred in predList)
{
//If any of the predicates is true, the composite predicate is true (OR)
if (pred.Invoke(m) == true) { ret = true; break; }
}
return ret;
};
//do the query
var messagesFound = messages.Where(m => compositePredicate.Invoke(m)).ToList();
//get the individual distinct IdGroupIds
var IdGroupIds = messagesFound.Select(x => x.IdGroup.Id).ToList().Distinct().ToList();
//Create dictionary to return
Dictionary<int, List<Message>> result = new Dictionary<int, List<Message>>();
foreach (int i in IdGroupIds)
{
result.Add(i, messagesFound.Where(m => m.IdGroup.Id == i).ToList());
}
return result;
}
public static void Main(string[] args)
{
var item1 = new IdGroup { Id = 2, Messages = new List<Message>() };
var item2 = new IdGroup { Id = 45, Messages = new List<Message>() };
var item3 = new IdGroup { Id = 36, Messages = new List<Message>() };
var item4 = new IdGroup { Id = 8, Messages = new List<Message>() };
var message1 = new Message { Id = 3, IdGroup = item1 };
var message2 = new Message { Id = 7, IdGroup = item1 };
var message3 = new Message { Id = 9, IdGroup = item1 };
item1.Messages.Add(message1);
item1.Messages.Add(message2);
item1.Messages.Add(message3);
var message4 = new Message { Id = 4, IdGroup = item2 };
var message5 = new Message { Id = 10, IdGroup = item2 };
var message6 = new Message { Id = 76, IdGroup = item2 };
item2.Messages.Add(message4);
item2.Messages.Add(message5);
item2.Messages.Add(message6);
var message7 = new Message { Id = 6, IdGroup = item3 };
var message8 = new Message { Id = 32, IdGroup = item3 };
item3.Messages.Add(message7);
item3.Messages.Add(message8);
var message9 = new Message { Id = 11, IdGroup = item4 };
var message10 = new Message { Id = 16, IdGroup = item4 };
var message11 = new Message { Id = 19, IdGroup = item4 };
var message12 = new Message { Id = 77, IdGroup = item4 };
item4.Messages.Add(message9);
item4.Messages.Add(message10);
item4.Messages.Add(message11);
item4.Messages.Add(message12);
List<IdGroup> items = new List<IdGroup> { item1, item2, item3, item4 };
List<Message> messages = new List<Message> { message1, message2, message3, message4, message5, message6,message7, message8, message9, message10, message11, message12};
Dictionary<int, int> lastMessagesPerItem = new Dictionary<int, int> { { 2, 3 }, { 45, 10 }, { 36, 6 }, { 8, 11 } };
var result = SynchronizedChatMessages(messages, lastMessagesPerItem);
var discard = Console.ReadKey();
}
}
}

Well it would be nice if this would work, but I doubt it that can be translated to a SQL statement in one go:
var toInsert =
from msg in _context.Messages
group msg by msg.Grupo_ID into g
where data.Keys.Contains(g.Key)
select new {
Item = g.Key,
Messages = g.Where(x => x.ID > data[g.Key])
};
I don't think the second Where clause x => x.ID > data[g.Key] can be translated.
So you may need to do this in two passes, like this:
// This is a single SQL query.
var groups =
from msg in _context.Messages
group msg by msg.Grupo_ID into g
where data.Keys.Contains(g.Key)
select new {
Item = g.Key,
// ordering helps us when we do the in-memory part.
Messages = g.OrderByDescending(x => x.ID).ToList()
};
// This iterates the result set in memory
foreach (var g in groups)
result.Add(
g.Item,
// input is ordered, we stop when an item is <= data[g.Item].
g.Messages.TakeWhile(m => m.ID > data[g.Item]).ToList())