We Keep Coding

C# comparing two large lists of items by a specific property

I have two large lists of items whos class look like this (both lists are of same type):
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
var oldList = new List<Items>(); // oldList
var newList = new List<Items>(); // new list
The old list contains items from database and the new list represents items fetched from API;
Both lists can be very large with 10000+ items in each (20000 total)
I need to compare items from newList against the items from "oldList" and see which items that have same itemID value, are of different "QuantitySold" value, and those that are of different "QuantitySold" value should be stored in third list called "differentQuantityItems".
I could just simply do double foreach list and compare values but since both of the lists are large the performance with double foreach loop is terrible and I can't do it...
Can someone help me out with this?
#YamamotoTetsua I'm already using a IEqualityComparer to get the desired result, however it doesn't gives the results that I'm expecting. Here is why...I have a first IEqualityComparer which looks like this:
public class MissingItemComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new MissingItemComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return x.ItemID == y.ItemID;
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
The usage of this IEqualityComparer basically gives me items from newList that are not present in my database like following:
var missingItems= newItems.Except(competitor.SearchedUserItems.ToList(), MissingItemComparer.Instance).ToList();
Now in this list I will have the list of items which are new from API and are not present in my DB...
Second IEqualityComparer is based on the different QuantitySold from old and new list:
public class ItemsComparer : IEqualityComparer<SearchedUserItems>
{
public static readonly IEqualityComparer<SearchedUserItems> Instance = new ItemsComparer();
public bool Equals(SearchedUserItems x, SearchedUserItems y)
{
return (x.QuantitySold == y.QuantitySold);
}
public int GetHashCode(SearchedUserItems x)
{
return x.ItemID.GetHashCode();
}
}
Usage example:
var differentQuantityItems = newItems.Except(competitor.SearchedUserItems.ToList(), ItemsComparer.Instance).ToList();
The issue with these two equality comparers is that first one will for example return these itemID's that are missing:
123124124
123124421
512095902
And they indeed are missing from my oldList... However the second IEQualityComparer will also return these items as differentQuantity items, they indeed are, but the aren't present in the oldList.. So they shouldn't be included in the second list.

This is a perfect candidate for LINQ Join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID
where newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
This will return all new items which have corresponding old item with different QuantitySold. If you want to also include the new items without corresponding old item, then use left outer join:
var differentQuantityItems =
(from newItem in newList
join oldItem in oldList on newItem.ItemID equals oldItem.ItemID into oldItems
from oldItem in oldItems.DefaultIfEmpty()
where oldItem == null || newItem.QuantitySold != oldItem.QuantitySold
select newItem).ToList();
In both cases, join operator is used to quickly correlate the items with the same ItemID. Then you can compare QuantitySold or any other properties.

This code will run in less than a second, even if there are no matches at all (also less than a second if everything is a match).
It will return all items that exists in both lists (i.e. same ItemID) but with a different QuantitySold.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApp5
{
class Program
{
public class Items
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}
static void Main(string[] args)
{
// Sample data
var oldList = new List<Items>();
oldList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 4 }));
var newList = new List<Items>();
newList.AddRange(Enumerable.Range(0, 20000).Select(z => new Items() { ItemID = z.ToString(), QuantitySold = 5 }));
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold).Select(z => z.left);
Console.WriteLine(results.Count());
Console.ReadLine();
}
}
}
The use of z.left means only one of the items will be returned - if you want both the old and the new, instead use:
var results = oldList.Join(newList,
left => left.ItemID,
right => right.ItemID,
(left, right) => new { left, right })
.Where(z => z.left.QuantitySold != z.right.QuantitySold)
.Select(z => new[] { z.left, z.right })
.SelectMany(z => z);

From a big-O complexity point of view, just comparing the lists in a nested for loop would be in the class of O(n*m), being n the size of the list in the DB, and m the size of the list fetched from the API.
What you can do to improve your performance is to sort the two lists, that would cost O(n log(n) + m log(m)), and then you could find the new items in O(n + m). Therefore, the overall complexity of your algorithm would then be in the class of O(n log(n) + m log(m)).
Here's an idea of the time it would take, comparing the quadratic solution to the superlinear one.

You can think of using Except clause with custom written IEqualityComparer something like below
var oldList = new List<Item>(); // oldList
var newList = new List<Item>(); // new list
var distinctList = newList.Except(oldList,new ItemEqualityComparer()).ToList();
class ItemEqualityComparer : IEqualityComparer<Item>
{
public bool Equals(Item i1, Item i2)
{
if (i1.ItemID == i2.ItemID && i1.QuantitySold != i2.QuantitySold)
return false;
return true;
}
public int GetHashCode(Item item)
{
return item.ItemID.GetHashCode();
}
}
public class Item
{
public string ItemID { get; set; }
public int QuantitySold { get; set; }
}

Write similar logic using generics

I have the following method which determines which cars I need to delete from the DB.
private List<CarDTO> BuildCarsToDelete(IList<CarDTO> newCars, IList<CarDTO> existingCars)
{
var missingCars = new List<CarDTO>();
var cars = newCars.Select(c => c.CarId);
var newCarIds = new HashSet<int>(cars);
foreach (var car in existingCars)
{
//If there are no new cars then it had some and they have been removed
if (newCars.Count() == 0)
{
missingCars.Add(car);
}
else
{
if (!newCarIds.Contains(car.CarId))
{
missingCars.Add(car);
}
}
}
return missingCars;
}
This works as I want - but if I want to achieve the same functionality for Customers or Apartments of other DTOs I will be copying a pasting the code but only changing the variable names and the Type of DTO around - is there a nicer way possible using generics which would keep the algorithm and logic as it is but allow me to use on any DTO?

If all the ids are of type int then you can do that by passing in a Func to determine the id.
private List<T> BuildToDelete<T>(
IList<T> newItems,
IList<T> existingItems,
Func<T, int> getId)
{
var missingItems = new List<T>();
var items = newItems.Select(getId);
var newItemIds = new HashSet<int>(items);
foreach (var item in existingItems)
{
if (newItems.Count() == 0)
{
missingItems.Add(item);
}
else
{
if (!newItemIds.Contains(getId(item)))
{
missingItems.Add(item);
}
}
}
return missingItems;
}
Then call as shown below:
var results = BuildToDelete(newCars, existingCars, c => c.CarId);

Assuming you use the interface approach mentioned in comments, a generic version could look something like this:
private List<TEntity> BuildEntitiesToDelete(IList<TEntity> newEntities, IList<TEntity> existingEntities) where TEntity : IEntityWithId
{
var missingEntities = new List<TEntity>();
var entities = newEntities.Select(e => e.Id);
var newEntityIds = new HashSet<int>(entities);
foreach (var entity in existingEntities)
{
if (entities.Count() == 0)
{
missingEntities.Add(entity);
}
else
{
if (!newEntityIds.Contains(entity.Id))
{
missingEntities.Add(entity);
}
}
}
return missingEntities;
}
IEntityWithId is probably a poor name for the interface, but I'll leave picking a better name up to you.

Try something cleaner:
1) create flexible equality comparer (need to add some null checking etc.)
public class FuncEqualityComparer<T> : IEqualityComparer<T>
{
Func<T, T, bool> comparer;
Func<T, int> hash;
public FuncEqualityComparer (Func<T, T, bool> comparer, Func<T, int> hash)
{
this.comparer = comparer;
this.hash = hash;
}
public bool Equals (T x, T y) => comparer (x, y);
public int GetHashCode (T obj) => hash (obj);
}
2) and now, just simply:
var carComparerByID = new FuncEqualityComparer<CarDTO> ((a, b) => a.CarId == b.CarId, x => x.CarId.GetHashCode ());
var result = existingCars.Except (newCars, carComparerByID).ToList ();

Adding new T to empty List<T> using reflection

I'm attempting to set add a new instance of an Officer class to a potentially empty list using reflection.
These are my classes
public class Report(){
public virtual ICollection<Officer> Officer { get; set; }
}
public class Officer(){
public string Name{ get; set; }
}
Simplified code snippet:
Report report = new Report()
PropertyInfo propertyInfo = report.GetType().GetProperty("Officer");
object entity = propertyInfo.GetValue(report, null);
if (entity == null)
{
//Gets the inner type of the list - the Officer class
Type type = propertyInfo.PropertyType.GetGenericArguments()[0];
var listType = typeof(List<>);
var constructedListType = listType.MakeGenericType(type);
entity = Activator.CreateInstance(constructedListType);
}
//The entity is now List<Officer> and is either just created or contains a list of
//Officers
//I want to check how many officers are in the list and if there are none, insert one
//Pseudo code:
if (entity.count = 0)
{
entity.add(new instance of type)
}
Much appreciated!

Use:
object o = Activator.CreateInstance(type); // "type" is the same variable you got a few lines above
((IList)entity).add(o);

You have two options:
1) Using dynamic:
dynamic list = entity;
if (list.Count = 0)
{
list.Add(new instance of type)
}
2) Using Reflection:
var countProp = entity.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance).First(p => p.Name == "Count");
var count = (int)countProp.GetValue(entity,null);
if(count == 0)
{
var method = entity.GetType().GetMethods(BindingFlags.Instance | BindingFlags.Public).First(m => m.Name == "Add");
method.Invoke(entity,new instance of type);
}

This isn't quite what you asked for but may accomplish the same task.
public static ICollection<T> EnsureListExistsAndHasAtLeastOneItem(ICollection<T> source)
where T : Officer, new()
{
var list = source ?? new List<T>();
if( list.Count == 0 ) list.Add(new T());
return list;
}
If Officer doesn't have a default constructor then you could add a factory callback
public static ICollection<T> EnsureListExistsAndHasAtLeastOneItem
(ICollection<T> source, Func<T> builder)
where T : Officer
{
var list = source ?? new List<T>();
if( list.Count == 0 ) list.Add(builder());
return list;
}

Just type your entity appropriately as a List<Officer> (or an appropriately more abstract type (such as IList)) and use as normal:
entity = Activator.CreateInstance(constructedListType) as IList;
But no need to check whether to insert or not, just insert:
entity.Insert(0, officer);
I'm assuming (based on the fact that you already know how to create instances using reflection) you're not having trouble creating the instance of type Officer.

Edit after re-reading over your question: This doesn't directly answer your question but is rather a suggestion of a different implementation.
You can easily get by without using reflection:
public class TestContainer<T>
{
private readonly List<T> _list;
public TestContainer()
{
_list = new List<T>();
}
public void Add()
{
_list.Add(default(T));
}
}
Then calling e.g.:
var t = new TestContainer<YourClass>();
t.Add();
t.Add();
t.Add();
you will have a list of 3 instances of YourClass by their default value

Add items to a collection if the collection does NOT already contain it by comparing a property of the items?

Basically, how do I make it so I can do something similar to: CurrentCollection.Contains(...), except by comparing if the item's property is already in the collection?
public class Foo
{
public Int32 bar;
}
ICollection<Foo> CurrentCollection;
ICollection<Foo> DownloadedItems;
//LINQ: Add any downloaded items where the bar Foo.bar is not already in the collection?

You start by finding which elements are not already in the collection:
var newItems = DownloadedItems.Where(x => !CurrentCollection.Any(y => x.bar == y.bar));
And then just add them:
foreach(var item in newItems)
{
CurrentCollection.Add(item);
}
Note that the first operation may have quadratic complexity if the size of DownloadedItems is close to the size of CurrentCollection. If that ends up causing problems (measure first!), you can use a HashSet to bring the complexity down to linear:
// collect all existing values of the property bar
var existingValues = new HashSet<Foo>(from x in CurrentCollection select x.bar);
// pick items that have a property bar that doesn't exist yet
var newItems = DownloadedItems.Where(x => !existingValues.Contains(x.bar));
// Add them
foreach(var item in newItems)
{
CurrentCollection.Add(item);
}

You can use Enumerable.Except:
It will compare the two lists and return elements that appear only in the first list.
CurrentCollection.AddRange(DownloadedItems.Except(CurrentCollection));

Using R.Martinho Fernandes method and converting to 1 line:
CurrentCollection.AddRange(DownloadedItems.Where(x => !CurrentCollection.Any(y => y.bar== x.bar)));

You can call the Any method and pass a value to compare to whatever property of the type of object in the collection
if (!CurrentCollection.Any(f => f.bar == someValue))
{
// add item
}
a more complete solution could be:
DownloadedItems.Where(d => !CurrentCollection.Any(c => c.bar == d.bar)).ToList()
.ForEach(f => CurrentCollection.Add(f));

Or using All
CurrentCollection
.AddRange(DownloadedItems.Where(x => CurrentCollection.All(y => y.bar != x.bar)));

One thing that you can do also i think it is the easiest way is to une a HashSet instead of a List, by default the HashSet don't add redundant values.

List<int> current = new List<int> { 1, 2 };
List<int> add = new List<int> { 2, 3 };
current.AddRange(add.Except(current));
This will result in 1,2,3, using the default comparing.
This will also work for Foo if you change the compare behaviour:
public class Foo : IEquatable<Foo>
{
public Int32 bar;
public bool Equals(Foo other)
{
return bar == other.bar;
}
public override bool Equals(object obj) => Equals(obj as Foo);
public override int GetHashCode() => (bar).GetHashCode(); // (prop1,prop2,prop3).GetHashCode()
}
You could also implement an IEqualityComparer<Foo>, and pass it as second parameter to except
current.AddRange(add.Except(current, new FooComparer()));
public class FooComparer : IEqualityComparer<Foo>
{
public bool Equals(Foo x, Foo y)
{
return x.bar.Equals(y.bar);
}
public int GetHashCode(Foo obj)
{
return obj.bar.GetHashCode();
}
}

internal static class ExtensionMethod
{
internal static ICollection<T> AddIfExists<T>(this ICollection<T> list, ICollection<T> range)
{
foreach (T item in range)
{
if (!list.Contains(item))
list.Add(item);
}
return list;
}
}
ICollection<Foo> CurrentCollection;
ICollection<Foo> DownloadedItems;
CurrentCollection.AddIfExists(DownloadedItems)....

var newItems = DownloadedItems.Where(i => !CurrentCollection.Any(c => c.Attr == i.Attr));

You could do it like this:
CurrentCollection.Any(x => x.bar == yourGivenValue)

How can I get LINQ to return the object which has the max value for a given property? [duplicate]

This question already has answers here:
How to perform .Max() on a property of all objects in a collection and return the object with maximum value [duplicate]
(9 answers)
How to use LINQ to select object with minimum or maximum property value
(20 answers)
Closed 7 years ago.
If I have a class that looks like:
public class Item
{
public int ClientID { get; set; }
public int ID { get; set; }
}
And a collection of those items...
List<Item> items = getItems();
How can I use LINQ to return the single "Item" object which has the highest ID?
If I do something like:
items.Select(i => i.ID).Max();
I'll only get the highest ID, when what I actually want returned is the Item object itself which has the highest ID? I want it to return a single "Item" object, not an int.

This will loop through only once.
Item biggest = items.Aggregate((i1,i2) => i1.ID > i2.ID ? i1 : i2);
Thanks Nick - Here's the proof
class Program
{
static void Main(string[] args)
{
IEnumerable<Item> items1 = new List<Item>()
{
new Item(){ ClientID = 1, ID = 1},
new Item(){ ClientID = 2, ID = 2},
new Item(){ ClientID = 3, ID = 3},
new Item(){ ClientID = 4, ID = 4},
};
Item biggest1 = items1.Aggregate((i1, i2) => i1.ID > i2.ID ? i1 : i2);
Console.WriteLine(biggest1.ID);
Console.ReadKey();
}
}
public class Item
{
public int ClientID { get; set; }
public int ID { get; set; }
}
Rearrange the list and get the same result

.OrderByDescending(i=>i.id).First()
Regarding the performance concern, it is very likely that this method is theoretically slower than a linear approach. However, in reality, most of the time we are not dealing with the data set that is big enough to make any difference.
If performance is a main concern, Seattle Leonard's answer should give you linear time complexity. Alternatively, you may also consider to start with a different data structure that returns the max value item at constant time.
First() will do the same as Take(1) but returns the item directly instead of an enumeration containing the item.

int max = items.Max(i => i.ID);
var item = items.First(x => x.ID == max);
This assumes there are elements in the items collection of course.

Use MaxBy from the morelinq project:
items.MaxBy(i => i.ID);

This is an extension method derived from #Seattle Leonard 's answer:
public static T GetMax<T,U>(this IEnumerable<T> data, Func<T,U> f) where U:IComparable
{
return data.Aggregate((i1, i2) => f(i1).CompareTo(f(i2))>0 ? i1 : i2);
}

In case you don't want to use MoreLINQ and want to get linear time, you can also use Aggregate:
var maxItem =
items.Aggregate(
new { Max = Int32.MinValue, Item = (Item)null },
(state, el) => (el.ID > state.Max)
? new { Max = el.ID, Item = el } : state).Item;
This remembers the current maximal element (Item) and the current maximal value (Item) in an anonymous type. Then you just pick the Item property. This is indeed a bit ugly and you could wrap it into MaxBy extension method to get the same thing as with MoreLINQ:
public static T MaxBy(this IEnumerable<T> items, Func<T, int> f) {
return items.Aggregate(
new { Max = Int32.MinValue, Item = default(T) },
(state, el) => {
var current = f(el.ID);
if (current > state.Max)
return new { Max = current, Item = el };
else
return state;
}).Item;
}

Or you can write your own extension method:
static partial class Extensions
{
public static T WhereMax<T, U>(this IEnumerable<T> items, Func<T, U> selector)
{
if (!items.Any())
{
throw new InvalidOperationException("Empty input sequence");
}
var comparer = Comparer<U>.Default;
T maxItem = items.First();
U maxValue = selector(maxItem);
foreach (T item in items.Skip(1))
{
// Get the value of the item and compare it to the current max.
U value = selector(item);
if (comparer.Compare(value, maxValue) > 0)
{
maxValue = value;
maxItem = item;
}
}
return maxItem;
}
}

try this:
var maxid = from i in items
group i by i.clientid int g
select new { id = g.Max(i=>i.ID }

In LINQ you can solve it the following way:
Item itemMax = (from i in items
let maxId = items.Max(m => m.ID)
where i.ID == maxId
select i).FirstOrDefault();

You could use a captured variable.
Item result = items.FirstOrDefault();
items.ForEach(x =>
{
if(result.ID < x.ID)
result = x;
});