We Keep Coding

C# comparing two large lists of items by a specific property

I have two large lists of items whos class look like this (both lists are of same type):
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
var oldList = new List<Items>(); // oldList
var newList = new List<Items>(); // new list
The old list contains items from database and the new list represents items fetched from API;
Both lists can be very large with 10000+ items in each (20000 total)
I need to compare items from newList against the items from "oldList" and see which items that have same itemID value, are of different "QuantitySold" value, and those that are of different "QuantitySold" value should be stored in third list called "differentQuantityItems".
I could just simply do double foreach list and compare values but since both of the lists are large the performance with double foreach loop is terrible and I can't do it...
Can someone help me out with this?
#YamamotoTetsua I'm already using a IEqualityComparer to get the desired result, however it doesn't gives the results that I'm expecting. Here is why...I have a first IEqualityComparer which looks like this:
public class MissingItemComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new MissingItemComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return x.ItemID == y.ItemID;
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
The usage of this IEqualityComparer basically gives me items from newList that are not present in my database like following:
var missingItems= newItems.Except(competitor.SearchedUserItems.ToList(), MissingItemComparer.Instance).ToList();
Now in this list I will have the list of items which are new from API and are not present in my DB...
Second IEqualityComparer is based on the different QuantitySold from old and new list:
public class ItemsComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new ItemsComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return (x.QuantitySold == y.QuantitySold);
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
Usage example:
var differentQuantityItems = newItems.Except(competitor.SearchedUserItems.ToList(), ItemsComparer.Instance).ToList();
The issue with these two equality comparers is that first one will for example return these itemID's that are missing:
123124124
123124421
512095902
And they indeed are missing from my oldList... However the second IEQualityComparer will also return these items as differentQuantity items, they indeed are, but the aren't present in the oldList.. So they shouldn't be included in the second list.

This is a perfect candidate for LINQ Join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID
where newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
This will return all new items which have corresponding old item with different QuantitySold. If you want to also include the new items without corresponding old item, then use left outer join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID into oldItems
from oldItem in oldItems.DefaultIfEmpty()
where oldItem == null || newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
In both cases, join operator is used to quickly correlate the items with the same ItemID. Then you can compare QuantitySold or any other properties.

This code will run in less than a second, even if there are no matches at all (also less than a second if everything is a match).
It will return all items that exists in both lists (i.e. same ItemID) but with a different QuantitySold.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApp5
{
class Program
{
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
static void Main(string[] args)
{
// Sample data
var oldList = new List<Items>();
oldList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 4 }));
var newList = new List<Items>();
newList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 5 }));
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold).Select(z => z.left);
Console.WriteLine(results.Count());
Console.ReadLine();
}
}
}
The use of z.left means only one of the items will be returned - if you want both the old and the new, instead use:
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold)
.Select(z => new[] { z.left, z.right })
.SelectMany(z => z);

From a big-O complexity point of view, just comparing the lists in a nested for loop would be in the class of O(n*m), being n the size of the list in the DB, and m the size of the list fetched from the API.
What you can do to improve your performance is to sort the two lists, that would cost O(n log(n) + m log(m)), and then you could find the new items in O(n + m). Therefore, the overall complexity of your algorithm would then be in the class of O(n log(n) + m log(m)).
Here's an idea of the time it would take, comparing the quadratic solution to the superlinear one.

You can think of using Except clause with custom written IEqualityComparer something like below
var oldList = new List<Item>(); // oldList
var newList = new List<Item>(); // new list
var distinctList = newList.Except(oldList,new ItemEqualityComparer()).ToList();
class ItemEqualityComparer : IEqualityComparer<Item>
{
public bool Equals(Item i1, Item i2)
{
if (i1.ItemID == i2.ItemID && i1.QuantitySold != i2.QuantitySold)
return false;
return true;
}
public int GetHashCode(Item item)
{
return item.ItemID.GetHashCode();
}
}
public class Item
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}

LINQ - Select statement on nested collection object

I have a class file as mentioned below:
public class TestA
{
public string Name {get; set;}
public List<TestA> TestCollection {get;set;}
}
above you can see type of TestCollection is the List of parent class.
Now I want to retrieve Names from each and every object.
I tried with SelectMany as shown below: but that doesn't work for me.
testAList.SelectMany(t => t.Name)
How can I achieve this ?
Here is the fiddle code: Fiddle Code

Assuming that testAList is a List<TestA> and you want the Name values of the TestA objects inside the TestCollection property of every TestA object inside testAList:
testAList.SelectMany(x => x.TestCollection.Select(y => y.Name))
Or, if you want the Name of the objects in testAList:
testAList.Select(x => x.Name)
Finally, if you want ALL the names, you will need to use more than just LINQ because of the recursive nature of your class:
IEnumerable<string> GetNames(List<TestA> testAList)
{
return testAList.Select(x => x.Name)
.Concat(testAList.SelectMany(x => GetNames(x.TestCollection)));
}
var names = GetNames(testAList);

Your collection is recursive, so you need t a recursive query to get its results:
IEnumerable<string> GetAllNames(TestA root) {
return GetNames(new[] {root}); // Forward to a method taking IEnumerable<T>
}
IEnumerable<string> GetAllNames(IEnumerable<TestA> tests) {
return tests
.Select(t => t.Name) // Names at this level
.Concat(tests.SelectMany(t => GetAllNames(t.TestCollection))); // Names of children
}
However, C# offers a better approach using yield return:
IEnumerable<string> GetAllNames(TestA node) {
yield return node.Name;
foreach (var childName in node.TestCollection.SelectMany(t => GetAllNames(t))) {
yield return childName;
}
}

Here is an alternate option that does not use linq but can provide the same result
IEnumerable<string> GetAllNames(TestA root) {
return GetAllNames(new[] { root });
}
IEnumerable<string> GetAllNames(IEnumerable<TestA> tests) {
var queue = new Queue<TestA>(tests);
while (queue.Count > 0) {
var current = queue.Dequeue();
yield return current.Name;
if (current.TestCollection != null) {
foreach (var child in current.TestCollection) {
queue.Enqueue(child);
}
}
}
}
The above based on your fiddle would produce
Parent_0
Parent_1
Parent_2
Parent_3
Parent_4
Child_0_0
Child_0_1
Child_1_0
Child_1_1
Child_2_0
Child_2_1
Child_3_0
Child_3_1
Child_4_0
Child_4_1
If you use a stack approach
IEnumerable<string> GetAllNamesStack(IEnumerable<TestA> tests) {
var stack = new Stack<TestA>(tests.Reverse());
while (stack.Count > 0) {
var current = stack.Pop();
yield return current.Name;
if (current.TestCollection != null) {
current.TestCollection.Reverse();
foreach (var child in current.TestCollection) {
stack.Push(child);
}
}
}
}
The output would be
Parent_0
Child_0_0
Child_0_1
Parent_1
Child_1_0
Child_1_1
Parent_2
Child_2_0
Child_2_1
Parent_3
Child_3_0
Child_3_1
Parent_4
Child_4_0
Child_4_1

traverse a "tree" dynamically given number of levels

I have an object called Team that has a property Children which represents a list of "sub teams" under that team
public class Team
{
public List<Team> Children {get;set;}
}
I currently have some code that does this to build up a single list of all teams but as you can see its "hard coded" in terms of "levels" it goes down:
Team topTeam = GetTopteam();
List<Team> allTeams = new List<Team>();
allTeams.Add(topTeam);
allTeams.AddRange(topTeam.Children);
var childrensChildren = topTeam.Children.SelectMany(r=>r.Children);
allTeams.AddRange(childrensChildren);
and it keeps going . .
I now want to make the "levels" configurable so something like this:
public IEnumberable<Team> GetTeams(int numberOfLevelsDown)
{
}
If I pass in 1, i only return a list of the topteam and its direct children.
If I pass in 2, I get the top team, the children and the children's children
and so on . . .
What is the most elegant way of traversing down levels in a tree dynamically?

You can use recursion to do this.
This kind of method signature is because of performance issues and it prevents generating a lot of intermediate lists.
public void GetTeams(List<Team> teams, Team team, int level)
{
if (level == 0)
return;
if (team.Children == null)
return;
foreach (var t in team.Children)
{
teams.Add(t);
GetTeams(teams, t, level - 1);
}
}
and use it like this
var list = new List<Team>();
Team topTeam = GetTopteam();
GetTeams(list, topTeam, 5);
//now you have teams in list

I have used this kind of thing many times.
public static IEnumerable<T> EnumerateDescendants<T>(this T root, Func<T, IEnumerable<T>> children)
{
yield return root;
foreach (var child in children(root).SelectMany(x => x.EnumerateDescendants(children)))
{
yield return child;
}
}
Use it like this:
var allTeams = rootTeam.EnumerateDescendants(x => x.Children);
You should be able to modify it something like this:
public static IEnumerable<T> EnumerateDescendants<T>(this T root, Func<T, IEnumerable<T>> children, int maxLevels, int currentLevel = 0)
{
if (currentLevel <= maxLevels)
{
yield return root;
foreach (var child in children(root).SelectMany(x => x.EnumerateDescendants(children, maxLevels, currentLevel + 1)))
{
yield return child;
}
}
}
You should be able to use it like this:
var allTeamsUpToLevel2 = rootTeam.EnumerateDescendants(x => x.Children, 2);
I may have the >= mixed up but something like this should work.

Get All Children to One List - Recursive C#

C# | .NET 4.5 | Entity Framework 5
I have a class in Entity Framework that looks like this:
public class Location
{
public long ID {get;set;}
public long ParentID {get;set;}
public List<Location> Children {get;set;}
}
ID is the identifier of the location, ParentID links it to a parent, and Children contains all of the children locations of the parent location. I'm looking for some easy way, likely recursively, to get all "Location" and their children to one single List containing the Location.ID's. I'm having trouble conceptualizing this recursively. Any help is appreciated.
This is what I have so far, its an extension to the entity class, but I believe it could be done better/simpler:
public List<Location> GetAllDescendants()
{
List<Location> returnList = new List<Location>();
List<Location> result = new List<Location>();
result.AddRange(GetAllDescendants(this, returnList));
return result;
}
public List<Location> GetAllDescendants(Location oID, ICollection<Location> list)
{
list.Add(oID);
foreach (Location o in oID.Children)
{
if (o.ID != oID.ID)
GetAllDescendants(o, list);
}
return list.ToList();
}
UPDATED
I ended up writing the recursion in SQL, throwing that in a SP, and then pulling that into Entity. Seemed cleaner and easier to me than using Linq, and judging by the comments Linq and Entity don't seem the best route to go. Thanks for all of the help!

You can do SelectMany
List<Location> result = myLocationList.SelectMany(x => x.Children).ToList();
You can use where condition for some selective results like
List<Location> result = myLocationList.Where(y => y.ParentID == someValue)
.SelectMany(x => x.Children).ToList();
If you only required Id's of Children you can do
List<long> idResult = myLocationList.SelectMany(x => x.Children)
.SelectMany(x => x.ID).ToList();

This will do the trick:
class Extensions
{
public static IEnumerable<T> SelectManyRecursive<T>(this IEnumerable<T> source, Func<T, IEnumerable<T>> selector)
{
var result = source.SelectMany(selector);
if (!result.Any())
{
return result;
}
return result.Concat(result.SelectManyRecursive(selector));
}
}
Use it like this:
List<Location> locations = new List<Location>();
//
// your code here to get locations
//
List<string> IDs = locations.SelectManyRecursive(l => l.Children).Select(l => l.ID).ToList();

Try this Extension method:
public static IEnumerable<T> Flatten<T, R>(this IEnumerable<T> source, Func<T, R> recursion) where R : IEnumerable<T>
{
return source.SelectMany(x => (recursion(x) != null && recursion(x).Any()) ? recursion(x).Flatten(recursion) : null)
.Where(x => x != null);
}
And you can use it like this:
locationList.Flatten(x => x.Children).Select(x => x.ID);

I had no Children prop in my model, so Nikhil Agrawal's answer doesn't work for me, so here is my solution.
With following model:
public class Foo
{
public int Id { get; set; }
public int? ParentId { get; set; }
// other props
}
You can get children of one item using:
List<Foo> GetChildren(List<Foo> foos, int id)
{
return foos
.Where(x => x.ParentId == id)
.Union(foos.Where(x => x.ParentId == id)
.SelectMany(y => GetChildren(foos, y.Id))
).ToList();
}
For ex.
List<Foo> foos = new List<Foo>();
foos.Add(new Foo { Id = 1 });
foos.Add(new Foo { Id = 2, ParentId = 1 });
foos.Add(new Foo { Id = 3, ParentId = 2 });
foos.Add(new Foo { Id = 4 });
GetChild(foos, 1).Dump(); // will give you 2 and 3 (ids)

I would like to contribute my own solution, which was modified from the references below:
public static IEnumerable<T> Flatten<T, R>(this IEnumerable<T> source, Func<T, R> recursion) where R : IEnumerable<T>
{
var flattened = source.ToList();
var children = source.Select(recursion);
if (children != null)
{
foreach (var child in children)
{
flattened.AddRange(child.Flatten(recursion));
}
}
return flattened;
}
Example:
var n = new List<FamilyMember>()
{
new FamilyMember { Name = "Dominic", Children = new List<FamilyMember>()
{
new FamilyMember { Name = "Brittany", Children = new List<FamilyMember>() }
}
}
}.Flatten(x => x.Children).Select(x => x.Name);
Output:
Dominic
Brittany
Class:
public class FamilyMember {
public string Name {get; set;}
public List<FamilyMember> Children { get; set;}
}
Ref. https://stackoverflow.com/a/21054096/1477388
Note: Can't find the other reference, but someone else on SO published an answer that I copied some code from.

Entity framework does not currently support recursion, and for that reason you can either
Rely on lazy loading child collections as you have done (beware the N+1 problem)
Query an arbitrary depth of objects (This will be an ugly query, though you could generate it using System.Linq.Expressions)
The only real option would be to avoid using LINQ to express the query, and instead resort to standard SQL.
Entity framework supports this scenario fairly well whether you're using code first or not.
For code-first, consider something along the lines of
var results = this.db.Database.SqlQuery<ResultType>(rawSqlQuery)
For model-first, consider using a defining query which I think is a good option as it allows further composition, or stored procedures.
To recursively get back data, you will need to understand recursive CTEs assuming you're using SQL Server, and that it is version 2005+
EDIT:
Here is the code for a recursive query to an arbitrary depth. I put this together just for fun, I doubt it would be very efficient!
var maxDepth = 5;
var query = context.Locations.Where(o => o.ID == 1);
var nextLevelQuery = query;
for (var i = 0; i < maxDepth; i++)
{
nextLevelQuery = nextLevelQuery.SelectMany(o => o.Children);
query = query.Concat(nextLevelQuery);
}
The flattened list is in the variable query

The accepted answer from #NikhilAgrawal will not recursively get all children and grandchildren as #electricalbah has pointed out.
I do miss the answer from #EricLippert that was given on Code Review.
https://codereview.stackexchange.com/a/5661/96658
static IEnumerable<T> DepthFirstTreeTraversal<T>(T root, Func<T, IEnumerable<T>> children)
{
var stack = new Stack<T>();
stack.Push(root);
while(stack.Count != 0)
{
var current = stack.Pop();
// If you don't care about maintaining child order then remove the Reverse.
foreach(var child in children(current).Reverse())
stack.Push(child);
yield return current;
}
}
Called like this:
static List<Location> AllChildren(Location start)
{
return DepthFirstTreeTraversal(start, c=>c.Children).ToList();
}
I made an example below with SelectMany. As you can see from Immediate Window you will not even get the Parent Id if you use that solution.

Create list to add all child using recursively
public static List list = new List();
recursive funtion
static void GetChild(int id) // Pass parent Id
{
using (var ctx = new CodingPracticeDataSourceEntities())
{
if (ctx.Trees.Any(x => x.ParentId == id))
{
var childList = ctx.Trees.Where(x => x.ParentId == id).ToList();
list.AddRange(childList);
foreach (var item in childList)
{
GetChild(item.Id);
}
}
}
}
Sample model
public partial class Tree
{
public int Id { get; set; }
public string Name { get; set; }
public Nullable<int> ParentId { get; set; }
}

Assuming Locations is a DbSet<Location> in your DB context, this will solve your problem "I'm looking for some easy way ... to get all 'Location' and their children to one single List containing the Location.ID's". Seems like I'm missing something, so please clarify if so.
dbContext.Locations.ToList()
// IDs only would be dbContext.Locations.Select( l => l.ID ).ToList()

This is my method for Flattening the children.
private Comment FlattenChildComments(Comment comment, ref Comment tempComment)
{
if (comment.ChildComments != null && comment.ChildComments.Any())
{
foreach (var childComment in comment.ChildComments)
{
tempComment.ChildComments.Add(childComment);
FlattenChildComments(childComment, ref tempComment);
}
}
comment.ChildComments = tempComment.ChildComments;
return comment;
}

For the people who needs something generic:
/// <summary>
/// Recursively enumerate all children, grandchildren etc... in a 1-dimentional IEnumerable
/// </summary>
/// <typeparam name="TModel">The type of the model</typeparam>
/// <param name="root">The root from which to enumerate children</param>
/// <param name="childSelector">The selector on how to select the children of the root.</param>
/// <returns>A 1-dimentional IEnumerable of all it's children, grandchildren etc.. recursively.</returns>
public static IEnumerable<TModel> EnumerateChildren<TModel>(TModel root, Func<TModel, IEnumerable<TModel>> childSelector)
{
var children = childSelector.Invoke(root);
if (children == null)
{
yield break;
}
foreach (var child in children)
{
yield return child;
foreach (var grandChild in EnumerateChildren(child, childSelector))
{
yield return grandChild;
}
}
}
Usage:
var location = GetLocation(); // Get your root.
var children = EnumerateChildren(location, l => l.Children);

Multiple columns in Linq

Problem: how to simplify the code below, as I'm aiming towards 30 different properties in the thing class.
The code looks for uniqueness in a 'thing' property.
public class thing
{
public string Name { get; set; }
public string Colour { get; set; }
public string Position { get; set; }
public string Height { get; set; }
}
public List<thing> SeeIfAnyInListHaveAUniqueSingleColumn(List<thing> listOfThings)
{
// try colour
IEnumerable<IGrouping<string, thing>> thingQuery2 = from t in listOfThings
group t by t.Colour;
List<thing> listOfThingsFound = new List<thing>();
foreach (var thingGroup in thingQuery2)
{
if (thingGroup.Count() == 1)
{
foreach (thing thing in thingGroup) // there is only going to be 1
listOfThingsFound.Add(thing);
}
}
// try position
IEnumerable<IGrouping<string, thing>> thingQuery3 = from t in listOfThings
group t by t.Position;
foreach (var thingGroup in thingQuery3)
{
if (thingGroup.Count() == 1)
{
foreach (thing thing in thingGroup) // there is only going to be 1
listOfThingsFound.Add(thing);
}
}
return listOfThingsFound;
}
Downloadable code on http://www.programgood.net/2010/11/06/FindingUniquenessInData.aspx

I think that if you abstract away the FindUnique operation, you can more easily write the tests:
static IEnumerable<T> FindDistinct<T, TKey>(this IEnumerable<T> source,
Func<T, TKey> keySelector)
{
return from item in source
group item by keySelector(item) into grp
where grp.Count() == 1
from single in grp
select single;
}
Then you can write:
var thingsWithUniqueName = listOfThings.FindDistinct(t => t.Name);
var thingsWithUniquePosition = listOfThings.FindDistinct(t => t.Position);
var thingsWithUniqueHeight = listOfThings.FindDistinct(t => t.Height);

You would like to write code like:
foreach var property in Thing.Properties
{
IEnumerable<IGrouping<string, thing>> thingQuery2 = from t in listOfThings
group t by t.property;
List<thing> listOfThingsFound = new List<thing>();
foreach (var thingGroup in thingQuery2)
{
if (thingGroup.Count() == 1)
{
foreach (thing thing in thingGroup) // there is only going to be 1
listOfThingsFound.Add(thing);
}
}
...
}
You can only do that through Reflection and that is something you should stay away from. The only thing I can think of is store the properties in some kind of collection, like a dictionary and iterate that.

I just noticed that Gabe has provided the same answer I was about to post. I thought I would post this anyway just to reinforce that this answer is a good strong use of LINQ. Please accept Gabe's answer rather than this one. Well done Gabe!
public static IEnumerable<T> WhereUniqueByKey<T, P>(
this IEnumerable<T> #this, Func<T, P> keySelector)
{
return #this
.GroupBy(keySelector)
.Where(gt => gt.Count() == 1)
.SelectMany(gt => gt, (_, t) => t);
}
As per Gabe's answer, my function is an extension method, and it needs to be defined in a static class. The only real difference between our answers is that Gabe has used the LINQ query syntax and I used direct LINQ method calls. The result is the same and so is the usage:
var thingsWithUniqueName = listOfThings.WhereUniqueByKey(t => t.Name);
var thingsWithUniquePosition = listOfThings.WhereUniqueByKey(t => t.Position);
var thingsWithUniqueHeight = listOfThings.WhereUniqueByKey(t => t.Height);