We Keep Coding

Get unmatched items from first list when comparing two large lists of pocos

I have two lists of objects which have a common property called OrderNumber.
The first list has about 20000 items and the second list has about 1.5 million items.
I need an efficient way of finding items from list 1 which dont have a match in list 2. I am currently using Linq and it takes more than 20 mins to compute the solution. I am not able to find an efficient solution to this online.
My code so far
notmatched.AddRange(List1.Where(l1=> !list2.Select(l2=> l2.OrderNumber).Contains(l1.OrderNumber)).Select(l1 => new SomeObj
{
OrderNumber = l1.OrderNumber
}));

Using the built in Except extension provided by Linq is fast enough providing a custom IEqualityComparer. My implementation may not work for your use-case but given 1.5 million Poco classes in firstList, and 20k in secondList, it executes under 1 second.
Documentation for IEqualityComparer, Linq Except
DotnetFiddle - reduced numbers to work around memory limitations
// Classes used in test:
public interface IOrderNumber
{
string OrderNumber { get; set; }
}
public class Poco: IOrderNumber
{
public string OrderNumber { get; set; }
}
public class Podo: IOrderNumber
{
public string OrderNumber {get;set;}
}
public class DataEqualityComparer : IEqualityComparer<IOrderNumber>
{
public bool Equals(IOrderNumber p1, IOrderNumber p2)
{
var equal = GetHashCode(p1) == GetHashCode(p2);
return equal;
}
public int GetHashCode(IOrderNumber p1)
{
if (p1 == null)
return -1;
int hCode = p1.OrderNumber.GetHashCode();
return hCode.GetHashCode();
}
}
... then your code would look like this:
var firstList = Enumerable.Range(1, 1500000).Select(x => new Poco { OrderNumber = x.ToString() }).ToList();
var secondList = Enumerable.Range(50, 20000).Select(x => new Podo { OrderNumber = x.ToString() }).ToList();
Stopwatch sw = Stopwatch.StartNew();
var result = firstList.Except(secondList, new DataEqualityComparer()).ToList();
sw.Stop();
Console.WriteLine($"Duration: {sw.Elapsed:G}");

Here is a more simplified version of your solution, it uses less loops to do the job, but I'm not sure if this will make the process faster, Please let me know if it does
notmatched.AddRange(List1.Where(l1=> !list2.Any(l2=> l2.OrderNumber == l1.OrderNumber).Select(l1 => new SomeObj
{
OrderNumber = l1.OrderNumber
}));

USE Linq Except to compare objects. you must implement an IEquatable Interface for complex objects.
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
var first = new List<Poco> { new Poco {OrderNumber = "test", Orderid = 1 }, new Poco {OrderNumber = "test", Orderid = 1 } };
var second = new List<Poco> { new Poco {OrderNumber = "test", Orderid = 1 }, new Poco {OrderNumber = "test", Orderid = 1 } };
first.Except(second).Dump();
second.Except(first).Dump();
if ( !first.Except(second).Any() && !second.Except(first).Any())
{
Console.Write("first and second are equal");
} else {
Console.Write("equality failed");
}
}
}
public class Poco : IEquatable<Poco>
{
public string OrderNumber { get; set; }
public int Orderid {get; set;}
public bool Equals(Poco other)
{
if (other == null )
return false;
return this.Orderid == other.Orderid && this.OrderNumber == other.OrderNumber;
}
public override bool Equals(Object obj)
{
if (obj == null)
return false;
Poco poco = obj as Poco;
if (poco == null)
return false;
else
return Equals(obj);
}
public override int GetHashCode() => (Orderid, OrderNumber).GetHashCode();
}

Find two values in hierarchical object tree C# using Linq

I have the following structures:
using System.Collections.Generic;
public class Program
{
public static void Main()
{
var tourn = new Tournament();
var player = new Player() { Type = PlayerType.User };
var seat1 = new Seat() { Number = 1, Player = player } ;
tourn.Tables = new List<Table>() { new Table(){ Seats = new List<Seat>(){seat1} } };
//Console.WriteLine(tourn.Tables.Where((k=> k.Tables.Any(m=> m.Seats.Any(j=> j.Player == Player.User)))).Count());
// Get Table and Seat numbers for PlayerType.User
}
public class Seat{
public Player? Player {get;set;}
public int Number {get;set;}
}
public enum PlayerType {
User,
Bot
}
public class Tournament{
public List<Table> Tables {get;set;}
}
public class Table
{
public List<Seat> Seats {get;set;}
public int Number {get;set;}
}
public class Player
{
public PlayerType Type { get; set; }
}
}
I have want the table and seat number where the playertype = user
Is it possible to build a Linq query to do this in one statement?
(I have looked at many examples and can't seem to get it right)
Tried this:
var playerLocation = tables
.Select(seat => new
{
TableNumber = seat.TableNumber,
TableSeat = seat.Seats
.Where(s => s.Player is not null &&
s.Player.Type == PlayerType.User)
.Select(st => new
{
SeatNumber = st.Number
})
});
Fiddle (.NET Core): https://dotnetfiddle.net/CMjZs6

I cheated...
PlayerLocation location = new PlayerLocation();
foreach (Table table in tables)
foreach (Seat seat in table.Seats)
if (seat.Player is not null && seat.Player.Type == PlayerType.User)
{
location.TableNumber = table.TableNumber;
location.SeatNumber = seat.Number;
break;
}

Assuming you want the answer as an object with the table number and a list of seat numbers, it is straightforward:
var ans = tourn.Tables.Select(t => new {
t.Number,
Seats = t.Seats.Where(s => s.Player?.Type == PlayerType.User)
.Select(s => s.Number)
.ToList()
});

Based On your Updated Fiddle
var anonyType = tourn.Tables.Where(k=> k.Seats.Any(j=> j.Player?.Type == PlayerType.User)).Select(k=> new { TableNumber = k.Number, SeatNumbers = k.Seats.Select(j=> j.Number) }).FirstOrDefault();
Console.WriteLine(anonyType.TableNumber);
Console.WriteLine(String.Join(" , ", anonyType.SeatNumbers));

Linq Join Not Equal

I have 3 classes:
public class HoteAvail
{
public int HotelID { get; set; }
public string Name { get; set; }
public List<Room> Rooms { get; set; }
}
public class Room
{
public int RoomID { get; set; }
public string RoomName { get; set; }
}
public class DAL
{
public static List<HoteAvail> GetAll()
{
return new List<HoteAvail>()
{
new HoteAvail{HotelID=1,Name="Taj",Rooms=new List<Room>(){new Room{RoomID=1,RoomName="Deliux"}}},
new HoteAvail{HotelID=2,Name="x",Rooms=new List<Room>(){new Room{RoomID=2,RoomName="dd"},new Room{RoomID=1,RoomName="qq"}}},
new HoteAvail{HotelID=3,Name="y",Rooms=new List<Room>(){new Room{RoomID=3,RoomName="yy"},new Room{RoomID=1,RoomName="rr"}}},
};
}
public static List<HoteAvail> GetAllII()
{
return new List<HoteAvail>()
{
new HoteAvail{HotelID=1,Name="Taj",Rooms=new List<Room>(){new Room{RoomID=1,RoomName="Deliux"},new Room{RoomID=1,RoomName="pp"}}},
new HoteAvail{HotelID=4,Name="x",Rooms=new List<Room>(){new Room{RoomID=2,RoomName="dd"},new Room{RoomID=1,RoomName="qq"}}},
new HoteAvail{HotelID=5,Name="y",Rooms=new List<Room>(){new Room{RoomID=3,RoomName="yy"},new Room{RoomID=1,RoomName="rr"}}},
};
}
}
I want to join the values of DAL.GetAll() and DAL.GetAllII() and result should contain only those values whose HotelID doesnot matches.
The final result set should be like :
new HoteAvail{HotelID=2,Name="x",Rooms=new List<Room>(){new Room{RoomID=2,RoomName="dd"},new Room{RoomID=1,RoomName="qq"}}},
new HoteAvail{HotelID=3,Name="y",Rooms=new List<Room>(){new Room{RoomID=3,RoomName="yy"},new Room{RoomID=1,RoomName="rr"}}},
new HoteAvail{HotelID=4,Name="x",Rooms=new List<Room>(){new Room{RoomID=2,RoomName="dd"},new Room{RoomID=1,RoomName="qq"}}},
new HoteAvail{HotelID=5,Name="y",Rooms=new List<Room>(){new Room{RoomID=3,RoomName="yy"},new Room{RoomID=1,RoomName="rr"}}}
I tried:
var groupBNames = new HashSet<string>(DAL.GetAll().Select(x => x.HotelID.ToString()));
var filteredEmployees = DAL.GetAllII().Where(x => !groupBNames.Contains(x.HotelID.ToString()));
var resultList = from a in DAL.GetAll()
where !(DAL.GetAllII().Any(HotelID => HotelID == a))
select a;
But I am not getting any success. Thanks in advance.

I'd recommend doing 2 excepts using a custom IEqualityComparer. You can use this method to create the comparer:
// create a comparer to compare HotelAvail objects by hotelId
// see http://www.codeducky.org/10-utilities-c-developers-should-know-part-two/
// for the implementation of EqualityComparers.Create, which is a nice shortcut
var comparer = EqualityComparers.Create<HoteAvail>(ha => ha.HotelId); // compare by hotelId
var results =
// first take all entries in GetAll() NOT IN GetAllII()
DAL.GetAll().Except(DAL.GetAllII(), comparer)
// then add all entries in GetAllII() NOT IN GetAll()
.Concat(DAL.GetAllII()).Except(DAL.GetAll(), comparer);

You could implement a custom IEqualityComparer<HoteAvail>:
public class HoteAvailComparer: IEqualityComparer<HoteAvail>
{
public bool Equals(HoteAvail x, HoteAvail y)
{
return x != null && y != null && x.HotelID == y.HotelID;
}
public int GetHashCode(HoteAvail obj)
{
return obj.HotelID;
}
}
that you can use for Enumerable.Except which is efficient since it's using a set:
var resultList = DAL.GetAll().Except(DAL.GetAllII(), new HoteAvailComparer());
Console.WriteLine(String.Join(",", resultList.Select(h => h.HotelID))); // 2,3
Update
It is giving me HotelId 2, 3 where as I want to join the values of
DAL.GetAll() and DAL.GetAllII() and result should contain only those
values whose HotelID doesnot matchesi.e. The result should have
HotelId 2,3,4,5
Then you need to use Except from both perspectives:
var hotelComparer = new HoteAvailComparer();
var all1 = DAL.GetAll();
var all2 = DAL.GetAllII();
var resultList = all1.Except(all2, hotelComparer).Concat(all2.Except(all1, hotelComparer));
The desired result 2,3,4,5:
Console.WriteLine(String.Join(",", resultList.Select(h => h.HotelID)));
Of course you could also use Concat and GroupBy, but it's less efficient and maintainable:
resultList = all1.Concat(all2).GroupBy(h => h.HotelID)
.Where(g => g.Count() == 1)
.SelectMany(g => g);
You can use the IEqualityComparer<HoteAvail> for many other LINQ methods like GroupBy+Distinct,Join,Intersect etc.

How to use LINQ to group objects in two groups

Lets say that I have the class Invoice:
public class Invoice
{
public int PartNumber { get; set; }
public string PartDescription { get; set; }
public int Quantity { get; set; }
public decimal Price { get; set; }
}
and then I have an array of its objects in the variable arrayOfInvoices .
If I had to Group invoices in two groups- invoices with Unit price below 12 and invoices with unit price above or equal to 12 and display details about invoices in each group sorted in ascending order of the price, how could I do that ?

You can simply do something like this:
var results =
from inv in arrayOfInvoices
orderby inv.Price
group inv by inv.Price < 12;
Or if you prefer fluent syntax:
var results = arrayOfInvoices.OrderBy(inv => inv.Price)
.GroupBy(inv => inv.Price < 12);
To group the invoices into three or more 'buckets', you can use BinarySearch:
var priceBoundaries = new[] { 12m, 20m };
var results =
from inv in arrayOfInvoices
orderby inv.Price
let index = Array.BinarySearch(priceBoundaries, inv.Price)
group inv by (index < 0 ? ~index : index + 1);
Or use side effects, like this:
var priceBoundaries = new[] { 12m, 20m, Decimal.MaxValue }; // Note the addition of MaxValue
var i = 0;
var results =
from inv in arrayOfInvoices
orderby inv.Price
group inv by (inv.Price < priceBoundaries[i] ? i : ++i);
This is generally bad practice, but should perform better than the BinarySearch method above.

If using group function is a pain (sometimes it gets annoying), then you can also use "Where"
var invoices = new List<Invoice> ();
var group1= invoices.Where(i=> i.Price<12).Orderby(i=> i.Price).ToList();
var group2= invoices.Where(i=> i.Price>=12).Orderby(i=> i.Price).ToList();

You can encapsulate the concept of the range in a class:
private class PriceRange<T> : IEquatable<PriceRange<T>>
{
public T Min { get; set; }
public T Max { get; set; }
public override int GetHashCode()
{
unchecked
{
int hash = 17;
hash = hash * 31 + Min.GetHashCode();
hash = hash * 31 + Max.GetHashCode();
return hash;
}
}
public override bool Equals(object obj)
{
return Equals(obj as PriceRange<T>);
}
public bool Equals(PriceRange<T> other)
{
if (other == null) return false;
if (!Min.Equals(other.Min)) return false;
if (!Max.Equals(other.Max)) return false;
return true;
}
}
Then use a map function to map the prices of each of your invoices to the appropriate range:
private class PriceRangeFactory<T>
{
public PriceRangeFactory(T[] rangeCutoffs)
{
_RangeCutoffs = rangeCutoffs;
}
private T[] _RangeCutoffs;
public PriceRange<T> Map(T price)
{
var index = Array.BinarySearch(_RangeCutoffs, price);
// Assume that the _RangeCutoffs that we have fully cover all possible values for T.
if (index < 0) index = ~index;
return new PriceRange<T>() { Min = _RangeCutoffs[index - 1], Max = _RangeCutoffs[index] };
}
}
Then use GroupBy with that map function:
var rangeFactory = new PriceRangeFactory<decimal>(new decimal[] { decimal.MinValue, 12, 20, decimal.MaxValue });
var grouped = foos.GroupBy(a => rangeFactory.Map(a.Price));
You get a list of IGroupings each keyed by a range that you specified, with the appropriate objects that fit into that range attached.
Now, obviously the code above is not quite production level, but it should be enough to get you started. As for why its not production level:
There's no check to assert that the ranges supplied to the PriceRangeFactory actually do fully cover all possible values.
It always assumes that the range is described as > X and <= Y.
There's no checking around Range.Min < Range.Max.
There's no assert that the list of range cutoffs supplied to the PriceRangeFactory are ordered correctly (required for BinarySearch).
There are bound to be some edge cases I haven't covered.

How do you use linq to xml to find matching nodes in two different xml files

I just asked another question here and the answer was spot on.
But that addressed what was essentially a syntax problem. Now I need help with an actual resolution.
This is the same code from the previou question (fixed up and with stuff added).
XElement FILE1 = XElement.Load (#"..\FILE1.XML");
XElement FILE2 = XElement.Load (#"..\FILE2.XML");
var orders = from file1 in FILE1.Descendants("Players").Elements("Player")
select new {
name=new {
clientID=ulm.Element("ClientID").Value,
firstName=file1.Element("FirstName").Value,
lastName=file1.Element("LastName").Value
}
};
var orders2 =
from file2 in FILE2.Descendants("Players").Elements("Player")
select new {
name=new {
clientID=ulm.Element("ClientID").Value,
firstName=file2.Element("FirstName").Value,
lastName=file2.Element("LastName").Value
}
};
var matchingResults = from i in orders from j in orders2 where (i.name.firstName==j.name.firstName && i.name.lastName==j.name.lastName)
select i;
matchingResults.Dump()
To make it interesting I have added a ClientID to each sequence result before trying to match them up.
What I need is to know does a Player node from order EXISTS in a player node from orders2. Or does it NOT EXIST? Ideally I would also be able to CHOOSE the selection criteria for the NOT EXISTS/EXISTS check. (LastName, or FirstName && LastName, or ClientID only, etc.)
I have NO IDEA how to go about this. Thanks for your help.

Use Enumerable.Intersect or Enumerable.Except
Use concrete types instead of anonymous types.
To be able to choose the selection criteria you can create a parameterized IEqualityComparer<Client>.
class Client
{
public string ClientID { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
}
[Flags]
enum Criteria {
ClientID, FirstName, LastName
}
class ClientEqualityComparer : IEqualityComparer<Client> {
private Criteria criteria;
public ClientEqualityComparer(Criteria criteria) {
this.criteria = criteria;
}
#region IEqualityComparer<Client> Membres
public bool Equals(Client x, Client y)
{
if (criteria.HasFlag(Criteria.ClientID) && x.ClientID != y.ClientID)
return false;
if (criteria.HasFlag(Criteria.FirstName) && x.FirstName != y.FirstName)
return false;
if (criteria.HasFlag(Criteria.LastName) && x.LastName != y.LastName)
return false;
return true;
}
public int GetHashCode(Client obj)
{
int hash = 17;
if (criteria.HasFlag(Criteria.ClientID))
hash = hash * 31 + obj.ClientID;
if (criteria.HasFlag(Criteria.FirstName))
hash = hash * 31 + obj.FirstName;
if (criteria.HasFlag(Criteria.LastName))
hash = hash * 31 + obj.LastName;
}
#endregion
}
static void Main(string[] args)
{
IEnumerable<Client> orders;
IEnumerable<Client> orders2;
//...
var matchingIdFn = orders.Intersect(orders2,
new ClientEqualityComparer(Criteria.ClientID | Criteria.FirstName));
var matchingIdFnLn = orders.Intersect(orders2,
new ClientEqualityComparer(Criteria.ClientID | Criteria.FirstName | Criteria.LastName));
var different = orders.Except(orders2, new ClientEqualityComparer(Criteria.ClientID | Criteria.FirstName));
}
var orders = from file1 in FILE1.Descendants("Players").Elements("Player")
select new Client{
ClientID=ulm.Element("ClientID").Value,
FirstName=file1.Element("FirstName").Value,
LastName=file1.Element("LastName").Value
};

I second Ahmed KRAIEM's suggestion to use Intersect or Except
Here's another solution that lets you compare using any arbitrary lambda:
void Main()
{
XElement FILE1 = XElement.Parse(
#"<Root>
<Players>
<Player><ClientId>1</ClientId><FirstName>Bob</FirstName><LastName>Smith</LastName></Player>
<Player><ClientId>2</ClientId><FirstName>John</FirstName><LastName>Smith</LastName></Player>
</Players>
</Root>");
XElement FILE2 = XElement.Parse(
#"<Root>
<Players>
<Player><ClientId>2</ClientId><FirstName>Bob</FirstName><LastName>Smith</LastName></Player>
<Player><ClientId>3</ClientId><FirstName>Mike</FirstName><LastName>Smith</LastName></Player>
</Players>
</Root>");
var orders = from file1 in FILE1.Descendants("Players").Elements("Player")
select new Player(Int32.Parse(file1.Element("ClientId").Value), file1.Element("FirstName").Value, file1.Element("LastName").Value);
var orders2 = from file2 in FILE2.Descendants("Players").Elements("Player")
select new Player(Int32.Parse(file2.Element("ClientId").Value), file2.Element("FirstName").Value, file2.Element("LastName").Value);
//orders.Dump();
//orders2.Dump();
var exists = orders2.Intersect(orders, new LambdaEqualityComparer<Player>((i, j) => i.FirstName == j.FirstName && i.LastName == j.LastName));
// or
//var exists = orders2.Intersect(orders, new LambdaEqualityComparer<Player>((i, j) => i.ClientId == j.ClientId));
exists.Dump();
var notExists = orders2.Except(orders, new LambdaEqualityComparer<Player>((i, j) => i.FirstName == j.FirstName && i.LastName == j.LastName));
// or
//var notExists = orders2.Except(orders, new LambdaEqualityComparer<Player>((i, j) => i.ClientId == j.ClientId));
notExists.Dump();
}
public class Player
{
public int ClientId { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public Player(int clientId, string firstName, string lastName)
{
ClientId = clientId;
FirstName = firstName;
LastName = lastName;
}
}
public class LambdaEqualityComparer<T> : IEqualityComparer<T>
{
private Func<T, T, bool> EqualityComparer { get; set; }
public LambdaEqualityComparer(Func<T, T, bool> equalityComparer)
{
EqualityComparer = equalityComparer;
}
public bool Equals(T x, T y)
{
return EqualityComparer(x, y);
}
public int GetHashCode(T obj)
{
// If the hash codes are different, then Equals never gets called. Make sure Equals is always called by making sure the hash codes are always the same.
// (Underneath, the .NET code is using a set and the not (!) of a Find method to determine if the set doesn't already contain the item and should be added.
// Find is not bothering to call Equals unless it finds a hash code that matches.)
//return obj.GetHashCode();
return 0;
}
}
Note that if you don't want to create a Player object, you can remove it completely, populate your anonymous objects in orders and orders2 just like you were doing before, and create a new LambdaEqualityComparer<dynamic> instead of new LambdaEqualityComparer<Player>, though it will be slower due to reflection calls on dynamic.