We Keep Coding

C# comparing two large lists of items by a specific property

I have two large lists of items whos class look like this (both lists are of same type):
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
var oldList = new List<Items>(); // oldList
var newList = new List<Items>(); // new list
The old list contains items from database and the new list represents items fetched from API;
Both lists can be very large with 10000+ items in each (20000 total)
I need to compare items from newList against the items from "oldList" and see which items that have same itemID value, are of different "QuantitySold" value, and those that are of different "QuantitySold" value should be stored in third list called "differentQuantityItems".
I could just simply do double foreach list and compare values but since both of the lists are large the performance with double foreach loop is terrible and I can't do it...
Can someone help me out with this?
#YamamotoTetsua I'm already using a IEqualityComparer to get the desired result, however it doesn't gives the results that I'm expecting. Here is why...I have a first IEqualityComparer which looks like this:
public class MissingItemComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new MissingItemComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return x.ItemID == y.ItemID;
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
The usage of this IEqualityComparer basically gives me items from newList that are not present in my database like following:
var missingItems= newItems.Except(competitor.SearchedUserItems.ToList(), MissingItemComparer.Instance).ToList();
Now in this list I will have the list of items which are new from API and are not present in my DB...
Second IEqualityComparer is based on the different QuantitySold from old and new list:
public class ItemsComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new ItemsComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return (x.QuantitySold == y.QuantitySold);
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
Usage example:
var differentQuantityItems = newItems.Except(competitor.SearchedUserItems.ToList(), ItemsComparer.Instance).ToList();
The issue with these two equality comparers is that first one will for example return these itemID's that are missing:
123124124
123124421
512095902
And they indeed are missing from my oldList... However the second IEQualityComparer will also return these items as differentQuantity items, they indeed are, but the aren't present in the oldList.. So they shouldn't be included in the second list.

This is a perfect candidate for LINQ Join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID
where newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
This will return all new items which have corresponding old item with different QuantitySold. If you want to also include the new items without corresponding old item, then use left outer join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID into oldItems
from oldItem in oldItems.DefaultIfEmpty()
where oldItem == null || newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
In both cases, join operator is used to quickly correlate the items with the same ItemID. Then you can compare QuantitySold or any other properties.

This code will run in less than a second, even if there are no matches at all (also less than a second if everything is a match).
It will return all items that exists in both lists (i.e. same ItemID) but with a different QuantitySold.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApp5
{
class Program
{
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
static void Main(string[] args)
{
// Sample data
var oldList = new List<Items>();
oldList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 4 }));
var newList = new List<Items>();
newList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 5 }));
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold).Select(z => z.left);
Console.WriteLine(results.Count());
Console.ReadLine();
}
}
}
The use of z.left means only one of the items will be returned - if you want both the old and the new, instead use:
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold)
.Select(z => new[] { z.left, z.right })
.SelectMany(z => z);

From a big-O complexity point of view, just comparing the lists in a nested for loop would be in the class of O(n*m), being n the size of the list in the DB, and m the size of the list fetched from the API.
What you can do to improve your performance is to sort the two lists, that would cost O(n log(n) + m log(m)), and then you could find the new items in O(n + m). Therefore, the overall complexity of your algorithm would then be in the class of O(n log(n) + m log(m)).
Here's an idea of the time it would take, comparing the quadratic solution to the superlinear one.

You can think of using Except clause with custom written IEqualityComparer something like below
var oldList = new List<Item>(); // oldList
var newList = new List<Item>(); // new list
var distinctList = newList.Except(oldList,new ItemEqualityComparer()).ToList();
class ItemEqualityComparer : IEqualityComparer<Item>
{
public bool Equals(Item i1, Item i2)
{
if (i1.ItemID == i2.ItemID && i1.QuantitySold != i2.QuantitySold)
return false;
return true;
}
public int GetHashCode(Item item)
{
return item.ItemID.GetHashCode();
}
}
public class Item
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}

Intersection of List of List

I have a list of lists which looks like the following
public class FilteredVM
{
public int ID { get; set; }
public string Name { get; set; }
public string Number { get; set; }
}
List<List<FilteredVM>> groupedExpressionResults = new List<List<FilteredVM>>();
I would like to Intersect the lists within this list based upon the ID's, whats the best way to tackle this?

Here's an optimized extension method:
public static HashSet<T> IntersectAll<T>(this IEnumerable<IEnumerable<T>> series, IEqualityComparer<T> equalityComparer = null)
{
if (series == null)
throw new ArgumentNullException("series");
HashSet<T> set = null;
foreach (var values in series)
{
if (set == null)
set = new HashSet<T>(values, equalityComparer ?? EqualityComparer<T>.Default);
else
set.IntersectWith(values);
}
return set ?? new HashSet<T>();
}
Use this with the following comparer:
public class FilteredVMComparer : IEqualityComparer<FilteredVM>
{
public static readonly FilteredVMComparer Instance = new FilteredVMComparer();
private FilteredVMComparer()
{
}
public bool Equals(FilteredVM x, FilteredVM y)
{
return x.ID == y.ID;
}
public int GetHashCode(FilteredVM obj)
{
return obj.ID;
}
}
Like that:
series.IntersectAll(FilteredVMComparer.Instance)
You could just write
series.Aggregate((a, b) => a.Intersect(b, FilteredVMComparer.Instance))
but it 'd be wasteful because it'd have to construct multiple sets.

Intersect will work when the type are dead equals, which in your case won't apply because you haven't implemented the GetHashCode and Equals methods, which is the best and complete way.
Thus, If you only intended to take elements that contains in both lists, than the following solution will suit you right.
Assuming list1 and list2 are type List<FilteredVM> than, The most simple way, will be doing this:
var intersectByIDs = list1.Where(elem => list2.Any(elem2 => elem2.ID == elem.ID));

If you are a fan of one-liner solutions you can use this:
List<FilteredVM> result = groupedExpressionResults.Aggregate((x, y) => x.Where(xi => y.Select(yi => yi.ID).Contains(xi.ID)).ToList());
And if you just want the IDs you can just add .Select(x => x.ID), like this:
var ids = groupedExpressionResults.Aggregate((x, y) => x.Where(xi => y.Select(yi => yi.ID).Contains(xi.ID)).ToList()).Select(x => x.ID);
Working Demo

Get distinct list values

i have a C# application in which i'd like to get from a List of Project objects , another List which contains distinct objects.
i tried this
List<Project> model = notre_admin.Get_List_Project_By_Expert(u.Id_user);
if (model != null) model = model.Distinct().ToList();
The list model still contains 4 identical objects Project.
What is the reason of this? How can i fix it?

You need to define "identical" here. I'm guessing you mean "have the same contents", but that is not the default definition for classes: the default definition is "are the same instance".
If you want "identical" to mean "have the same contents", you have two options:
write a custom comparer (IEqualityComparer<Project>) and supply that as a parameter to Distinct
override Equals and GetHashCode on Project
There are also custom methods like DistinctBy that are available lots of places, which is useful if identity can be determined by a single property (Id, typically) - not in the BCL, though. But for example:
if (model != null) model = model.DistinctBy(x => x.Id).ToList();
With, for example:
public static IEnumerable<TItem>
DistinctBy<TItem, TValue>(this IEnumerable<TItem> items,
Func<TItem, TValue> selector)
{
var uniques = new HashSet<TValue>();
foreach(var item in items)
{
if(uniques.Add(selector(item))) yield return item;
}
}

var newList =
(
from x in model
select new {Id_user= x.Id_user}
).Distinct();
or you can write like this
var list1 = model.DistinctBy(x=> x.Id_user);

How do you define identical? You should override Equals in Project with this definition (if you override Equals also override GetHashCode). For example:
public class Project
{
public int ProjectID { get; set; }
public override bool Equals(object obj)
{
var p2 = obj as Project;
if (p2 == null) return false;
return this.ProjectID == m2.ProjectID;
}
public override int GetHashCode()
{
return ProjectID;
}
}
Otherwise you are just checking reference equality.

The object's reference aren't equal. If you want to be able to do that on the entire object itself and not just a property, you have to implement the IEqualityComparer or IEquatable<T>.

Check this example: you need to use either Comparator or override Equals()
class Program
{
static void Main( string[] args )
{
List<Item> items = new List<Item>();
items.Add( new Item( "A" ) );
items.Add( new Item( "A" ) );
items.Add( new Item( "B" ) );
items.Add( new Item( "C" ) );
items = items.Distinct().ToList();
}
}
public class Item
{
string Name { get; set; }
public Item( string name )
{
Name = name;
}
public override bool Equals( object obj )
{
return Name.Equals((obj as Item).Name);
}
public override int GetHashCode()
{
return Name.GetHashCode();
}
}

Here's an answer from basically the same question that will help.
Explanation:
The Distinct() method checks reference equality for reference types. This means it is looking for literally the same object duplicated, not different objects which contain the same values.
Credits to #Rex M.

Isn't simpler to use one of the approaches shown below :) ?
You can just group your domain objects by some key and select FirstOrDefault like below.
More interesting option is to create some Comparer adapter that takes you domain object and creates other object the Comparer can use/work with out of the box. Base on the comparer you can create your custom linq extensions like in sample below. Hope it helps :)
[TestMethod]
public void CustomDistinctTest()
{
// Generate some sample of domain objects
var listOfDomainObjects = Enumerable
.Range(10, 10)
.SelectMany(x =>
Enumerable
.Range(15, 10)
.Select(y => new SomeClass { SomeText = x.ToString(), SomeInt = x + y }))
.ToList();
var uniqueStringsByUsingGroupBy = listOfDomainObjects
.GroupBy(x => x.SomeText)
.Select(x => x.FirstOrDefault())
.ToList();
var uniqueStringsByCustomExtension = listOfDomainObjects.DistinctBy(x => x.SomeText).ToList();
var uniqueIntsByCustomExtension = listOfDomainObjects.DistinctBy(x => x.SomeInt).ToList();
var uniqueStrings = listOfDomainObjects
.Distinct(new EqualityComparerAdapter<SomeClass, string>(x => x.SomeText))
.OrderBy(x=>x.SomeText)
.ToList();
var uniqueInts = listOfDomainObjects
.Distinct(new EqualityComparerAdapter<SomeClass, int>(x => x.SomeInt))
.OrderBy(x => x.SomeInt)
.ToList();
}
Custom comparer adapter:
public class EqualityComparerAdapter<T, V> : EqualityComparer<T>
where V : IEquatable<V>
{
private Func<T, V> _valueAdapter;
public EqualityComparerAdapter(Func<T, V> valueAdapter)
{
_valueAdapter = valueAdapter;
}
public override bool Equals(T x, T y)
{
return _valueAdapter(x).Equals(_valueAdapter(y));
}
public override int GetHashCode(T obj)
{
return _valueAdapter(obj).GetHashCode();
}
}
Custom linq extension (definition of DistinctBy extension method):
// Embedd this class in some specific custom namespace
public static class DistByExt
{
public static IEnumerable<T> DistinctBy<T,V>(this IEnumerable<T> enumerator,Func<T,V> valueAdapter)
where V : IEquatable<V>
{
return enumerator.Distinct(new EqualityComparerAdapter<T, V>(valueAdapter));
}
}
Definition of domain object used in test case:
public class SomeClass
{
public string SomeText { get; set; }
public int SomeInt { get; set; }
}

List<ViewClReceive> passData = (List<ViewClReceive>)TempData["passData_Select_BankName_List"];
passData = passData?.DistinctBy(b=>b.BankNm).ToList();
It will Works ......

Add items to a collection if the collection does NOT already contain it by comparing a property of the items?

Basically, how do I make it so I can do something similar to: CurrentCollection.Contains(...), except by comparing if the item's property is already in the collection?
public class Foo
{
public Int32 bar;
}
ICollection<Foo> CurrentCollection;
ICollection<Foo> DownloadedItems;
//LINQ: Add any downloaded items where the bar Foo.bar is not already in the collection?

You start by finding which elements are not already in the collection:
var newItems = DownloadedItems.Where(x => !CurrentCollection.Any(y => x.bar == y.bar));
And then just add them:
foreach(var item in newItems)
{
CurrentCollection.Add(item);
}
Note that the first operation may have quadratic complexity if the size of DownloadedItems is close to the size of CurrentCollection. If that ends up causing problems (measure first!), you can use a HashSet to bring the complexity down to linear:
// collect all existing values of the property bar
var existingValues = new HashSet<Foo>(from x in CurrentCollection select x.bar);
// pick items that have a property bar that doesn't exist yet
var newItems = DownloadedItems.Where(x => !existingValues.Contains(x.bar));
// Add them
foreach(var item in newItems)
{
CurrentCollection.Add(item);
}

You can use Enumerable.Except:
It will compare the two lists and return elements that appear only in the first list.
CurrentCollection.AddRange(DownloadedItems.Except(CurrentCollection));

Using R.Martinho Fernandes method and converting to 1 line:
CurrentCollection.AddRange(DownloadedItems.Where(x => !CurrentCollection.Any(y => y.bar== x.bar)));

You can call the Any method and pass a value to compare to whatever property of the type of object in the collection
if (!CurrentCollection.Any(f => f.bar == someValue))
{
// add item
}
a more complete solution could be:
DownloadedItems.Where(d => !CurrentCollection.Any(c => c.bar == d.bar)).ToList()
.ForEach(f => CurrentCollection.Add(f));

Or using All
CurrentCollection
.AddRange(DownloadedItems.Where(x => CurrentCollection.All(y => y.bar != x.bar)));

One thing that you can do also i think it is the easiest way is to une a HashSet instead of a List, by default the HashSet don't add redundant values.

List<int> current = new List<int> { 1, 2 };
List<int> add = new List<int> { 2, 3 };
current.AddRange(add.Except(current));
This will result in 1,2,3, using the default comparing.
This will also work for Foo if you change the compare behaviour:
public class Foo : IEquatable<Foo>
{
public Int32 bar;
public bool Equals(Foo other)
{
return bar == other.bar;
}
public override bool Equals(object obj) => Equals(obj as Foo);
public override int GetHashCode() => (bar).GetHashCode(); // (prop1,prop2,prop3).GetHashCode()
}
You could also implement an IEqualityComparer<Foo>, and pass it as second parameter to except
current.AddRange(add.Except(current, new FooComparer()));
public class FooComparer : IEqualityComparer<Foo>
{
public bool Equals(Foo x, Foo y)
{
return x.bar.Equals(y.bar);
}
public int GetHashCode(Foo obj)
{
return obj.bar.GetHashCode();
}
}

internal static class ExtensionMethod
{
internal static ICollection<T> AddIfExists<T>(this ICollection<T> list, ICollection<T> range)
{
foreach (T item in range)
{
if (!list.Contains(item))
list.Add(item);
}
return list;
}
}
ICollection<Foo> CurrentCollection;
ICollection<Foo> DownloadedItems;
CurrentCollection.AddIfExists(DownloadedItems)....

var newItems = DownloadedItems.Where(i => !CurrentCollection.Any(c => c.Attr == i.Attr));

You could do it like this:
CurrentCollection.Any(x => x.bar == yourGivenValue)

How to write generic IEnumerable<SelectListItem> extension method

I'm fairly new (ok, REALLy new) to generics but I love the idea of them. I am going to be having a few drop-down lists on a view and I'd like a generic way to take a list of objects and convert it to a list of SelectListItems
What I have now:
public static IEnumerable<SelectListItem> ToSelectListItems(
this IEnumerable<SpecificObject> items, long selectedId)
{
return
items.OrderBy(item => item.Name)
.Select(item =>
new SelectListItem
{
Selected = (item.Id == selectedId),
Text = item.Name,
Value = item.Id.ToString()
});
}
Trouble is, I'd need to repeat that code for each drop-down as the objects have different fields that represent the Text property of the SelectListItem
Here is what I'd like to accomplish:
public static IEnumerable<SelectListItem> ToSelectListItem<T>(this IEnumerable<T> items, string key, string value, int SelectedId) {
// I really have no idea how to proceed from here :(
}

Well, you could do something like this:
public static IEnumerable<SelectListItem> ToSelectListItems(
this IEnumerable<T> items,
Func<T,string> nameSelector,
Func<T,string> valueSelector,
Func<T,bool> selected)
{
return items.OrderBy(item => nameSelector(item))
.Select(item =>
new SelectListItem
{
Selected = selected(item),
Text = nameSelector(item),
Value = valueSelector(item)
});
}

You could pass in delegates to do the comparisons, and property retrieval. Something like this:
public static IEnumerable<SelectListItem> ToSelectListItem<T>(this IEnumerable<T> items,
int selectedId, Func<T,int> getId, Func<T, string> getName,
Func<T, string> getText, Func<T, string> getValue)
{
return
items.OrderBy(item => getName(item))
.Select(item =>
new SelectListItem
{
Selected = (getId(item) == selectedId),
Text = getText(item),
Value = getValue(item)
});
}
Then you would use it like so:
var selected = specificObjects.ToSelectListItem(10, s => s.Id, s=> s.Name, s => s.Name, s => s.Id.ToString());

In order for this to work as written, your type T will need to implement some interface which provides Name and Id properties:
public interface ISelectable
{
string Name { get; }
int Id { get; }
}
With this in place, you can do:
public static IEnumerable<SelectListItem> ToSelectListItems<T>(this IEnumerable<T> items, long selectedId)
where T : ISelectable
{
return
items.OrderBy(item => item.Name)
.Select(item =>
new SelectListItem
{
Selected = (item.Id == selectedId),
Text = item.Name,
Value = item.Id.ToString()
});
}
This is required in order to use the Name and Id properties within your extension method... You could, instead, provide a different means of receiving these (ie: passing delegates), but that may or may not increase the maintenance cost in your scenario.

//You need to use reflection to get the value of
public static IEnumerable<SelectListItem> GetListItems<T>
(this IEnumerable<T> items, int selectedVal)
{
return from item in items
select new SelectListItem
{
Text =item.GetPropValue("Name"),
Value = item.GetPropValue("Id"),
Selected =
item.GetPropValue("Id").Equals(selectedVal.ToString())
};
}
public static string GetPropValue<T> (this T item,string refName)
{
return
item.GetType().GetProperty(refName).GetValue(item).ToString();
}