We Keep Coding

Get unmatched items from first list when comparing two large lists of pocos

I have two lists of objects which have a common property called OrderNumber.
The first list has about 20000 items and the second list has about 1.5 million items.
I need an efficient way of finding items from list 1 which dont have a match in list 2. I am currently using Linq and it takes more than 20 mins to compute the solution. I am not able to find an efficient solution to this online.
My code so far
notmatched.AddRange(List1.Where(l1=> !list2.Select(l2=> l2.OrderNumber).Contains(l1.OrderNumber)).Select(l1 => new SomeObj
{
OrderNumber = l1.OrderNumber
}));

Using the built in Except extension provided by Linq is fast enough providing a custom IEqualityComparer. My implementation may not work for your use-case but given 1.5 million Poco classes in firstList, and 20k in secondList, it executes under 1 second.
Documentation for IEqualityComparer, Linq Except
DotnetFiddle - reduced numbers to work around memory limitations
// Classes used in test:
public interface IOrderNumber
{
string OrderNumber { get; set; }
}
public class Poco: IOrderNumber
{
public string OrderNumber { get; set; }
}
public class Podo: IOrderNumber
{
public string OrderNumber {get;set;}
}
public class DataEqualityComparer : IEqualityComparer<IOrderNumber>
{
public bool Equals(IOrderNumber p1, IOrderNumber p2)
{
var equal = GetHashCode(p1) == GetHashCode(p2);
return equal;
}
public int GetHashCode(IOrderNumber p1)
{
if (p1 == null)
return -1;
int hCode = p1.OrderNumber.GetHashCode();
return hCode.GetHashCode();
}
}
... then your code would look like this:
var firstList = Enumerable.Range(1, 1500000).Select(x => new Poco { OrderNumber = x.ToString() }).ToList();
var secondList = Enumerable.Range(50, 20000).Select(x => new Podo { OrderNumber = x.ToString() }).ToList();
Stopwatch sw = Stopwatch.StartNew();
var result = firstList.Except(secondList, new DataEqualityComparer()).ToList();
sw.Stop();
Console.WriteLine($"Duration: {sw.Elapsed:G}");

Here is a more simplified version of your solution, it uses less loops to do the job, but I'm not sure if this will make the process faster, Please let me know if it does
notmatched.AddRange(List1.Where(l1=> !list2.Any(l2=> l2.OrderNumber == l1.OrderNumber).Select(l1 => new SomeObj
{
OrderNumber = l1.OrderNumber
}));

USE Linq Except to compare objects. you must implement an IEquatable Interface for complex objects.
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
var first = new List<Poco> { new Poco {OrderNumber = "test", Orderid = 1 }, new Poco {OrderNumber = "test", Orderid = 1 } };
var second = new List<Poco> { new Poco {OrderNumber = "test", Orderid = 1 }, new Poco {OrderNumber = "test", Orderid = 1 } };
first.Except(second).Dump();
second.Except(first).Dump();
if ( !first.Except(second).Any() && !second.Except(first).Any())
{
Console.Write("first and second are equal");
} else {
Console.Write("equality failed");
}
}
}
public class Poco : IEquatable<Poco>
{
public string OrderNumber { get; set; }
public int Orderid {get; set;}
public bool Equals(Poco other)
{
if (other == null )
return false;
return this.Orderid == other.Orderid && this.OrderNumber == other.OrderNumber;
}
public override bool Equals(Object obj)
{
if (obj == null)
return false;
Poco poco = obj as Poco;
if (poco == null)
return false;
else
return Equals(obj);
}
public override int GetHashCode() => (Orderid, OrderNumber).GetHashCode();
}

Best approach to compare if one list is subset of another in C#

I have the below two classes:
public class FirstInner
{
public int Id { get; set; }
public string Type { get; set; }
public string RoleId { get; set; }
}
public class SecondInner
{
public int Id { get; set; }
public string Type { get; set; }
}
Again, there are lists of those types inside the below two classes:
public class FirstOuter
{
public int Id { get; set; }
public string Name { get; set; }
public string Title { get; set; }
public List<FirstInner> Inners { get; set; }
}
public class SecondOuter
{
public int Id { get; set; }
public string Name { get; set; }
public List<SecondInner> Inners { get; set; }
}
Now, I have list of FirstOuter and SecondOuter. I need to check if FirstOuter list is a subset of SecondOuter list.
Please note:
The names of the classes cannot be changed as they are from different systems.
Some additional properties are present in FirstOuter but not in SecondOuter. When comparing subset, we can ignore their presence in SecondOuter.
No.2 is true for FirstInner and SecondInner as well.
List items can be in any order---FirstOuterList[1] could be found in SecondOuterList[3], based on Id, but inside that again need to compare that FirstOuterList[1].FirstInner[3], could be found in SecondOuterList[3].SecondInner[2], based on Id.
I tried Intersect, but that is failing as the property names are mismatching. Another solution I have is doing the crude for each iteration, which I want to avoid.
Should I convert the SecondOuter list to FirstOuter list, ignoring the additional properties?
Basically, here is a test data:
var firstInnerList = new List<FirstInner>();
firstInnerList.Add(new FirstInner
{
Id = 1,
Type = "xx",
RoleId = "5"
});
var secondInnerList = new List<SecondInner>();
secondInner.Add(new SecondInner
{
Id = 1,
Type = "xx"
});
var firstOuter = new FirstOuter
{
Id = 1,
Name = "John",
Title = "Cena",
Inners = firstInnerList
}
var secondOuter = new SecondOuter
{
Id = 1,
Name = "John",
Inners = secondInnerList,
}
var firstOuterList = new List<FirstOuter> { firstOuter };
var secondOuterList = new List<SecondOuter> { secondOuter };
Need to check if firstOuterList is part of secondOuterList (ignoring the additional properties).

So the foreach way that I have is:
foreach (var item in firstOuterList)
{
var secondItem = secondOuterList.Find(so => so.Id == item.Id);
//if secondItem is null->throw exception
if (item.Name == secondItem.Name)
{
foreach (var firstInnerItem in item.Inners)
{
var secondInnerItem = secondItem.Inners.Find(sI => sI.Id == firstInnerItem.Id);
//if secondInnerItem is null,throw exception
if (firstInnerItem.Type != secondInnerItem.Type)
{
//throw exception
}
}
}
else
{
//throw exception
}
}
//move with normal flow
Please let me know if there is any better approach.

First, do the join of firstOuterList and secondOuterList
bool isSubset = false;
var firstOuterList = new List<FirstOuter> { firstOuter };
var secondOuterList = new List<SecondOuter> { secondOuter };
var jointOuterList = firstOuterList.Join(
secondOuterList,
p => new { p.Id, p.Name },
m => new { m.Id, m.Name },
(p, m) => new { FOuterList = p, SOuterList = m }
);
if(jointOuterList.Count != firstOuterList.Count)
{
isSubset = false;
return;
}
foreach(var item in jointOuterList)
{
var jointInnerList = item.firstInnerList.Join(
item.firstInnerList,
p => new { p.Id, p.Type },
m => new { m.Id, m.type },
(p, m) => p.Id
);
if(jointInnerList.Count != item.firstInnerList.Count)
{
isSubset = false;
return;
}
}
Note: I am assuming Id is unique in its outer lists. It means there will not be multiple entries with same id in a list. If no, then we need to use group by in above query

I think to break the question down..
We have two sets of Ids, the Inners and the Outers.
We have two instances of those sets, the Firsts and the Seconds.
We want Second's inner Ids to be a subset of First's inner Ids.
We want Second's outer Ids to be a subset of First's outer Ids.
If that's the case, these are a couple of working test cases:
[TestMethod]
public void ICanSeeWhenInnerAndOuterCollectionsAreSubsets()
{
HashSet<int> firstInnerIds = new HashSet<int>(GetFirstOuterList().SelectMany(outer => outer.Inners.Select(inner => inner.Id)).Distinct());
HashSet<int> firstOuterIds = new HashSet<int>(GetFirstOuterList().Select(outer => outer.Id).Distinct());
HashSet<int> secondInnerIds = new HashSet<int>(GetSecondOuterList().SelectMany(outer => outer.Inners.Select(inner => inner.Id)).Distinct());
HashSet<int> secondOuterIds = new HashSet<int>(GetSecondOuterList().Select(outer => outer.Id).Distinct());
bool isInnerSubset = secondInnerIds.IsSubsetOf(firstInnerIds);
bool isOuterSubset = secondOuterIds.IsSubsetOf(firstOuterIds);
Assert.IsTrue(isInnerSubset);
Assert.IsTrue(isOuterSubset);
}
[TestMethod]
public void ICanSeeWhenInnerAndOuterCollectionsAreNotSubsets()
{
HashSet<int> firstInnerIds = new HashSet<int>(GetFirstOuterList().SelectMany(outer => outer.Inners.Select(inner => inner.Id)).Distinct());
HashSet<int> firstOuterIds = new HashSet<int>(GetFirstOuterList().Select(outer => outer.Id).Distinct());
HashSet<int> secondInnerIds = new HashSet<int>(GetSecondOuterList().SelectMany(outer => outer.Inners.Select(inner => inner.Id)).Distinct());
HashSet<int> secondOuterIds = new HashSet<int>(GetSecondOuterList().Select(outer => outer.Id).Distinct());
firstInnerIds.Clear();
firstInnerIds.Add(5);
firstOuterIds.Clear();
firstOuterIds.Add(5);
bool isInnerSubset = secondInnerIds.IsSubsetOf(firstInnerIds);
bool isOuterSubset = secondOuterIds.IsSubsetOf(firstOuterIds);
Assert.IsFalse(isInnerSubset);
Assert.IsFalse(isOuterSubset);
}
private List<FirstOuter> GetFirstOuterList() { ... }
private List<SecondOuter> GetSecondOuterList() { ... }

linq query to create new objects

assumed i have a class hierarchy where basically every class consists of a property item (like string) and a property list item of the nested class:
public class Master
{
public String myName;
public List<Detail> myDetails;
}
public class Detail
{
public String myDetailDescription;
public List<DetailParts> parts;
}
public class DetailParts
{
public int PartNumber;
public String PartDescription;
}
Now i would like to have a linq query that gives me
an list of objects that consists of these elements
public class Result
{
public String myName; // from Master
public String myDetailDescription; // from Detail
public int PartNumber; // from DetailParts
}
It is a silly question, I know, but i cant find useful links...
[Edit #1]: Better code sample:
public class Master
{
public String myName;
public List<Detail> myDetails = new List<Detail>();
}
public class Detail
{
public String myDetailDescription;
public List<DetailParts> parts = new List<DetailParts>();
}
public class DetailParts
{
public int PartNumber;
public String PartDescription;
}
// List of Masters
List<Master> master = new List<Master>();
// Two detail parts for the Details
DetailParts dp1 = new DetailParts { PartDescription = "dp1" };
DetailParts dp2 = new DetailParts { PartDescription = "dp2" };
// once again two details for the Master
Detail d1 = new Detail { myDetailDescription = "d1" };
Detail d2 = new Detail{ myDetailDescription = "d2"};
// Assign the Parts
d1.parts.Add(dp1);
d1.parts.Add(dp2);
d2.parts.Add(dp1);
d2.parts.Add(dp2);
Master m1 = new Master { myName = "m1" };
Master m2 = new Master { myName = "m2" };
m1.myDetails.Add(d1);
m1.myDetails.Add(d2);
m2.myDetails.Add(d1);
m2.myDetails.Add(d2);
master.Add(m1);
master.Add(m2);
// given a value for `myName` and `myDetailDescription` i
// would be able to get a list with the `DetailParts` objects:
var t = master.Where(a => a.myName == "m2")
.Select(a => a.myDetails).FirstOrDefault()
.Where(b => b.myDetailDescription == "d1")
.Select(c => c.parts).FirstOrDefault();

You are looking for SelectMany:
List<Result> results = masterList
.SelectMany(m => m.myDetails
.SelectMany(d => d.parts
.Select(dp => new Result
{
PartNumber = dp.PartNumber,
myDetailDescription = d.myDetailDescription,
myName = m.myName
})))
.ToList();
A more readable version with LINQ's query syntax:
var resultQuery = from m in masterList
from d in m.myDetails
from dp in d.parts
select new Result
{
PartNumber = dp.PartNumber,
myDetailDescription = d.myDetailDescription,
myName = m.myName
};
List<Result> results = resultQuery.ToList();

Build hierarchy from strings C#

I have a collection of strings:
"Alberton;Johannesburg"
"Allendale;Phoenix"
"Brackenhurst;Alberton"
"Cape Town;"
"Durban;"
"Johannesburg;"
"Mayville;Durban"
"Phoenix;Durban"
"Sandton;Johannesburg"
that I want to structure into a hierarchical structure in the fastest possible manner, like:
Johannesburg
Alberton
Brackenhurst
Sandton
Cape Town
Durban
Phoenix
Allandale
Mayville
Currently I have nested for loops and checks, but was hoping I could achieve this with a single LAMBDA query?
The above mentioned strings are in a List.

I prepared lambda-like solution, but you should really think if it's more readable/efficient then your current one:
Helper Extension Method:
public static class ChildrenGroupExtensions
{
public static List<CityInfo> GetChildren(this IEnumerable<IGrouping<string, City>> source, string parentName)
{
var cities = source.SingleOrDefault(g => g.Key == parentName);
if (cities == null)
return new List<CityInfo>();
return cities.Select(c => new CityInfo { Name = c.Name, Children = source.GetChildren(c.Name) }).ToList();
}
}
Helper Classes:
public class City
{
public string Name { get; set; }
public string Parent { get; set; }
}
public class CityInfo
{
public string Name { get; set; }
public List<CityInfo> Children { get; set; }
}
Usage:
var groups = (from i in items
let s = i.Split(new[] { ';' })
select new City { Name = s[0], Parent = s[1] }).GroupBy(e => e.Parent);
var root = groups.GetChildren(string.Empty);
Where items is your List<string>
You can look the results with simple helper method like that one:
private static void PrintTree(List<CityInfo> source, int level)
{
if (source != null)
{
source.ForEach(c =>
{
Enumerable.Range(1, level).ToList().ForEach(i => Console.Write("\t"));
Console.WriteLine(c.Name);
PrintTree(c.Children, level + 1);
});
}
}
And the results are:
Cape Town
Durban
Mayville
Phoenix
Allendale
Johannesburg
Alberton
Brackenhurst
Sandton

You haven't specified any specific data structure so I just used a class called Area with a list of children of itself. Also, it's in 2 lines of linq. There is also no check to see if an area is a child of 2 separate parents as the code is. Here's the code for the test I used(Relevant lines in-between the equals comments):
[TestFixture]
public class CitiesTest
{
[Test]
public void Test()
{
var strings = new List<string>
{
"Alberton;Johannesburg",
"Allendale;Phoenix",
"Brackenhurst;Alberton",
"Cape Town;",
"Durban;",
"Johannesburg;",
"Mayville;Durban",
"Phoenix;Durban",
"Sandton;Johannesburg"
};
//===================================================
var allAreas = strings.SelectMany(x=>x.Split(';')).Where(x=>!string.IsNullOrWhiteSpace(x)).Distinct().ToDictionary(x=>x, x=>new Area{Name = x});
strings.ForEach(area =>
{
var areas = area.Split(';');
if (string.IsNullOrWhiteSpace(areas[1]))
return;
var childArea = allAreas[areas[0]];
if (!allAreas[areas[1]].Children.Contains(childArea))
allAreas[areas[1]].Children.Add(childArea);
childArea.IsParent = false;
});
var result = allAreas.Select(x=>x.Value).Where(x => x.IsParent);
//===================================================
}
public class Area
{
public string Name;
public bool IsParent;
public List<Area> Children { get; set; }
public Area()
{
Children = new List<Area>();
IsParent = true;
}
}
}

Select distinct values from a list using LINQ in C#

I have a collection of Employee
Class Employee
{
empName
empID
empLoc
empPL
empShift
}
My list contains
empName,empID,empLoc,empPL,empShift
E1,1,L1,EPL1,S1
E2,2,L2,EPL2,S2
E3,3,L3,EPL3,S3
E4,4,L1,EPL1,S1
E5,5,L5,EPL5,S5
E6,6,L2,EPL2,S2
I need to take the employees having distinct values
empLoc,empPL,empShift.
Is there is any way to achieve this using LINQ ?

You can use GroupBy with anonymous type, and then get First:
list.GroupBy(e => new {
empLoc = e.empLoc,
empPL = e.empPL,
empShift = e.empShift
})
.Select(g => g.First());

You could implement a custom IEqualityComparer<Employee>:
public class Employee
{
public string empName { get; set; }
public string empID { get; set; }
public string empLoc { get; set; }
public string empPL { get; set; }
public string empShift { get; set; }
public class Comparer : IEqualityComparer<Employee>
{
public bool Equals(Employee x, Employee y)
{
return x.empLoc == y.empLoc
&& x.empPL == y.empPL
&& x.empShift == y.empShift;
}
public int GetHashCode(Employee obj)
{
unchecked // overflow is fine
{
int hash = 17;
hash = hash * 23 + (obj.empLoc ?? "").GetHashCode();
hash = hash * 23 + (obj.empPL ?? "").GetHashCode();
hash = hash * 23 + (obj.empShift ?? "").GetHashCode();
return hash;
}
}
}
}
Now you can use this overload of Enumerable.Distinct:
var distinct = employees.Distinct(new Employee.Comparer());
The less reusable, robust and efficient approach, using an anonymous type:
var distinctKeys = employees.Select(e => new { e.empLoc, e.empPL, e.empShift })
.Distinct();
var joined = from e in employees
join d in distinctKeys
on new { e.empLoc, e.empPL, e.empShift } equals d
select e;
// if you want to replace the original collection
employees = joined.ToList();

You can try with this code
var result = (from item in List
select new
{
EmpLoc = item.empLoc,
EmpPL= item.empPL,
EmpShift= item.empShift
})
.ToList()
.Distinct();

I was curious about which method would be faster:
Using Distinct with a custom IEqualityComparer or
Using the GroupBy method described by Cuong Le.
I found that depending on the size of the input data and the number of groups, the Distinct method can be a lot more performant. (as the number of groups tends towards the number of elements in the list, distinct runs faster).
Code runs in LinqPad!
void Main()
{
List<C> cs = new List<C>();
foreach(var i in Enumerable.Range(0,Int16.MaxValue*1000))
{
int modValue = Int16.MaxValue; //vary this value to see how the size of groups changes performance characteristics. Try 1, 5, 10, and very large numbers
int j = i%modValue;
cs.Add(new C{I = i, J = j});
}
cs.Count ().Dump("Size of input array");
TestGrouping(cs);
TestDistinct(cs);
}
public void TestGrouping(List<C> cs)
{
Stopwatch sw = Stopwatch.StartNew();
sw.Restart();
var groupedCount = cs.GroupBy (o => o.J).Select(s => s.First()).Count();
groupedCount.Dump("num groups");
sw.ElapsedMilliseconds.Dump("elapsed time for using grouping");
}
public void TestDistinct(List<C> cs)
{
Stopwatch sw = Stopwatch.StartNew();
var distinctCount = cs.Distinct(new CComparerOnJ()).Count ();
distinctCount.Dump("num distinct");
sw.ElapsedMilliseconds.Dump("elapsed time for using distinct");
}
public class C
{
public int I {get; set;}
public int J {get; set;}
}
public class CComparerOnJ : IEqualityComparer<C>
{
public bool Equals(C x, C y)
{
return x.J.Equals(y.J);
}
public int GetHashCode(C obj)
{
return obj.J.GetHashCode();
}
}

Try,
var newList =
(
from x in empCollection
select new {Loc = x.empLoc, PL = x.empPL, Shift = x.empShift}
).Distinct();