We Keep Coding

LINQ distinct with IEquatable is not returning distinct result [duplicate]

This question already has answers here:
LINQ's Distinct() on a particular property
(23 answers)
Closed 19 days ago.
class Program
{
static void Main(string[] args)
{
List<Book> books = new List<Book>
{
new Book
{
Name="C# in Depth",
Authors = new List<Author>
{
new Author
{
FirstName = "Jon", LastName="Skeet"
},
new Author
{
FirstName = "Jon", LastName="Skeet"
},
}
},
new Book
{
Name="LINQ in Action",
Authors = new List<Author>
{
new Author
{
FirstName = "Fabrice", LastName="Marguerie"
},
new Author
{
FirstName = "Steve", LastName="Eichert"
},
new Author
{
FirstName = "Jim", LastName="Wooley"
},
}
},
};
var temp = books.SelectMany(book => book.Authors).Distinct();
foreach (var author in temp)
{
Console.WriteLine(author.FirstName + " " + author.LastName);
}
Console.Read();
}
}
public class Book
{
public string Name { get; set; }
public List<Author> Authors { get; set; }
}
public class Author
{
public string FirstName { get; set; }
public string LastName { get; set; }
public override bool Equals(object obj)
{
return true;
//if (obj.GetType() != typeof(Author)) return false;
//else return ((Author)obj).FirstName == this.FirstName && ((Author)obj).FirstName == this.LastName;
}
}
This is based on an example in "LINQ in Action". Listing 4.16.
This prints Jon Skeet twice. Why? I have even tried overriding Equals method in Author class. Still Distinct does not seem to work. What am I missing?
Edit:
I have added == and != operator overload too. Still no help.
public static bool operator ==(Author a, Author b)
{
return true;
}
public static bool operator !=(Author a, Author b)
{
return false;
}

LINQ Distinct is not that smart when it comes to custom objects.
All it does is look at your list and see that it has two different objects (it doesn't care that they have the same values for the member fields).
One workaround is to implement the IEquatable interface as shown here.
If you modify your Author class like so it should work.
public class Author : IEquatable<Author>
{
public string FirstName { get; set; }
public string LastName { get; set; }
public bool Equals(Author other)
{
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}
public override int GetHashCode()
{
int hashFirstName = FirstName == null ? 0 : FirstName.GetHashCode();
int hashLastName = LastName == null ? 0 : LastName.GetHashCode();
return hashFirstName ^ hashLastName;
}
}
Try it as DotNetFiddle

The Distinct() method checks reference equality for reference types. This means it is looking for literally the same object duplicated, not different objects which contain the same values.
There is an overload which takes an IEqualityComparer, so you can specify different logic for determining whether a given object equals another.
If you want Author to normally behave like a normal object (i.e. only reference equality), but for the purposes of Distinct check equality by name values, use an IEqualityComparer. If you always want Author objects to be compared based on the name values, then override GetHashCode and Equals, or implement IEquatable.
The two members on the IEqualityComparer interface are Equals and GetHashCode. Your logic for determining whether two Author objects are equal appears to be if the First and Last name strings are the same.
public class AuthorEquals : IEqualityComparer<Author>
{
public bool Equals(Author left, Author right)
{
if((object)left == null && (object)right == null)
{
return true;
}
if((object)left == null || (object)right == null)
{
return false;
}
return left.FirstName == right.FirstName && left.LastName == right.LastName;
}
public int GetHashCode(Author author)
{
return (author.FirstName + author.LastName).GetHashCode();
}
}

Another solution without implementing IEquatable, Equals and GetHashCode is to use the LINQs GroupBy method and to select the first item from the IGrouping.
var temp = books.SelectMany(book => book.Authors)
.GroupBy (y => y.FirstName + y.LastName )
.Select (y => y.First ());
foreach (var author in temp){
Console.WriteLine(author.FirstName + " " + author.LastName);
}

There is one more way to get distinct values from list of user defined data type:
YourList.GroupBy(i => i.Id).Select(i => i.FirstOrDefault()).ToList();
Surely, it will give distinct set of data

Distinct() performs the default equality comparison on objects in the enumerable. If you have not overridden Equals() and GetHashCode(), then it uses the default implementation on object, which compares references.
The simple solution is to add a correct implementation of Equals() and GetHashCode() to all classes which participate in the object graph you are comparing (ie Book and Author).
The IEqualityComparer interface is a convenience that allows you to implement Equals() and GetHashCode() in a separate class when you don't have access to the internals of the classes you need to compare, or if you are using a different method of comparison.

You've overriden Equals(), but make sure you also override GetHashCode()

The Above answers are wrong!!!
Distinct as stated on MSDN returns the default Equator which as stated The Default property checks whether type T implements the System.IEquatable interface and, if so, returns an EqualityComparer that uses that implementation. Otherwise, it returns an EqualityComparer that uses the overrides of Object.Equals and Object.GetHashCode provided by T
Which means as long as you overide Equals you are fine.
The reason you're code is not working is because you check firstname==lastname.
see https://msdn.microsoft.com/library/bb348436(v=vs.100).aspx and https://msdn.microsoft.com/en-us/library/ms224763(v=vs.100).aspx

You can achieve this several ways:
1. You may to implement the IEquatable interface as shown Enumerable.Distinct Method or you can see #skalb's answer at this post
2. If your object has not unique key, You can use GroupBy method for achive distinct object list, that you must group object's all properties and after select first object.
For example like as below and working for me:
var distinctList= list.GroupBy(x => new {
Name= x.Name,
Phone= x.Phone,
Email= x.Email,
Country= x.Country
}, y=> y)
.Select(x => x.First())
.ToList()
MyObject class is like as below:
public class MyClass{
public string Name{get;set;}
public string Phone{get;set;}
public string Email{get;set;}
public string Country{get;set;}
}
3. If your object's has unique key, you can only use the it in group by.
For example my object's unique key is Id.
var distinctList= list.GroupBy(x =>x.Id)
.Select(x => x.First())
.ToList()

You can use extension method on list which checks uniqueness based on computed Hash.
You can also change extension method to support IEnumerable.
Example:
public class Employee{
public string Name{get;set;}
public int Age{get;set;}
}
List<Employee> employees = new List<Employee>();
employees.Add(new Employee{Name="XYZ", Age=30});
employees.Add(new Employee{Name="XYZ", Age=30});
employees = employees.Unique(); //Gives list which contains unique objects.
Extension Method:
public static class LinqExtension
{
public static List<T> Unique<T>(this List<T> input)
{
HashSet<string> uniqueHashes = new HashSet<string>();
List<T> uniqueItems = new List<T>();
input.ForEach(x =>
{
string hashCode = ComputeHash(x);
if (uniqueHashes.Contains(hashCode))
{
return;
}
uniqueHashes.Add(hashCode);
uniqueItems.Add(x);
});
return uniqueItems;
}
private static string ComputeHash<T>(T entity)
{
System.Security.Cryptography.SHA1CryptoServiceProvider sh = new System.Security.Cryptography.SHA1CryptoServiceProvider();
string input = JsonConvert.SerializeObject(entity);
byte[] originalBytes = ASCIIEncoding.Default.GetBytes(input);
byte[] encodedBytes = sh.ComputeHash(originalBytes);
return BitConverter.ToString(encodedBytes).Replace("-", "");
}

The Equal operator in below code is incorrect.
Old
public bool Equals(Author other)
{
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}
NEW
public override bool Equals(Object obj)
{
var other = obj as Author;
if (other is null)
{
return false;
}
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}

Instead of
var temp = books.SelectMany(book => book.Authors).Distinct();
Do
var temp = books.SelectMany(book => book.Authors).DistinctBy(f => f.Property);

Linq - Select distinct objects [duplicate]

This question already has answers here:
Distinct not working with LINQ to Objects [duplicate]
(11 answers)
Closed 4 years ago.
I am trying to extract distinct objects by their values to have the unique CurrencyISO I have in the .csv.
public List<CurrencyDetail> InitGSCheckComboCurrency()
{
var lines = File.ReadAllLines("Data/AccountsGSCheck.csv");
var data = (from l in lines.Skip(1)
let split = l.Split(',')
select new CurrencyDetail
{
ISOName = split[3],
ISOCode = split[3]
}).Distinct();
List<CurrencyDetail> lstSrv = new List<CurrencyDetail>();
lstSrv = data.ToList();
return lstSrv;
}
However, the distinct function does not work for this and I end up with duplications.

You would need to define the Equals and GetHashCode of CurrencyDetail to do what you want. Quick and dirty solution:
var data = (from l in lines.Skip(1)
let split = l.Split(',')
select new
{
ISOName = split[3],
ISOCode = split[3]
}).Distinct()
.Select(x => new CurrencyDetail
{
ISOName = x.ISOName,
ISOCode = x.ISOCode
};
Anonymous types (the first new { ... }) automatically define sensible Equals() and GetHashCode(). Normally I wouldn't do this, because you are creating objects to then discard them. For this reason it is a quick and dirty solution.
Note that you are using twice split[3]... an error?
Now, a fully equatable version of CurrencyDetail could be:
public class CurrencyDetail : IEquatable<CurrencyDetail>
{
public string ISOName { get; set; }
public string ISOCode { get; set; }
public override bool Equals(object obj)
{
// obj is object, so we can use its == operator
if (obj == null)
{
return false;
}
CurrencyDetail other = obj as CurrencyDetail;
if (object.ReferenceEquals(other, null))
{
return false;
}
return this.InnerEquals(other);
}
public bool Equals(CurrencyDetail other)
{
if (object.ReferenceEquals(other, null))
{
return false;
}
return this.InnerEquals(other);
}
private bool InnerEquals(CurrencyDetail other)
{
// Here we know that other != null;
if (object.ReferenceEquals(this, other))
{
return true;
}
return this.ISOName == other.ISOName && this.ISOCode == other.ISOCode;
}
public override int GetHashCode()
{
unchecked
{
// From http://stackoverflow.com/a/263416/613130
int hash = 17;
hash = hash * 23 + (this.ISOName != null ? this.ISOName.GetHashCode() : 0);
hash = hash * 23 + (this.ISOCode != null ? this.ISOCode.GetHashCode() : 0);
return hash;
}
}
}
With this you can use the Distinct() as used by your code.

Linq distinct doesn't call Equals method

I have the following class
public class ModInfo : IEquatable<ModInfo>
{
public int ID { get; set; }
public string MD5 { get; set; }
public bool Equals(ModInfo other)
{
return other.MD5.Equals(MD5);
}
public override int GetHashCode()
{
return MD5.GetHashCode();
}
}
I load some data into a list of that class using a method like this:
public void ReloadEverything() {
var beforeSort = new List<ModInfo>();
// Bunch of loading from local sqlite database.
// not included since it's reload boring to look at
var modinfo = beforeSort.OrderBy(m => m.ID).AsEnumerable().Distinct().ToList();
}
Problem is the Distinct() call doesn't seem to do it's job. There are still objects which are equals each other.
Acording to this article: https://msdn.microsoft.com/en-us/library/vstudio/bb348436%28v=vs.100%29.aspx
that is how you are supposed to make distinct work, however it doesn't seem to be calling to Equals method on the ModInfo object.
What could be causing this to happen?
Example values:
modinfo[0]: id=2069, MD5 =0AAEBF5D2937BDF78CB65807C0DC047C
modinfo[1]: id=2208, MD5 = 0AAEBF5D2937BDF78CB65807C0DC047C
I don't care which value gets chosen, they are likely to be the same anyway since the md5 value is the same.

You also need to override Object.Equals, not just implement IEquatable.
If you add this to your class:
public override bool Equals(object other)
{
ModInfo mod = other as ModInfo;
if (mod != null)
return Equals(mod);
return false;
}
It should work.
See this article for more info: Implementing IEquatable Properly
EDIT: Okay, here's a slightly different implementation based on best practices with GetHashCode.
public class ModInfo : IEquatable<ModInfo>
{
public int ID { get; set; }
public string MD5 { get; set; }
public bool Equals(ModInfo other)
{
if (other == null) return false;
return (this.MD5.Equals(other.MD5));
}
public override int GetHashCode()
{
unchecked
{
int hash = 13;
hash = (hash * 7) + MD5.GetHashCode();
return hash;
}
}
public override bool Equals(object obj)
{
ModInfo other = obj as ModInfo;
if (other != null)
{
return Equals(other);
}
else
{
return false;
}
}
}
You can verify it:
ModInfo mod1 = new ModInfo {ID = 1, MD5 = "0AAEBF5D2937BDF78CB65807C0DC047C"};
ModInfo mod2 = new ModInfo {ID = 2, MD5 = "0AAEBF5D2937BDF78CB65807C0DC047C"};
// You should get true here
bool areEqual = mod1.Equals(mod2);
List<ModInfo> mods = new List<ModInfo> {mod1, mod2};
// You should get 1 result here
mods = mods.Distinct().ToList();
What's with those specific numbers in GetHashCode?

Add
public bool Equals(object other)
{
return this.Equals(other as ModInfo)
}
Also see here the recommendations how to implement the equality members: https://msdn.microsoft.com/en-us/library/ms173147(v=vs.80).aspx

How to compare two lists that contain objects that contain Dictionaries?

I have two Lists (List “A” and List “B”) that hold objects of type “KeyStore”, which is shown below:
public class KeyStore
{
public Dictionary<string, string> PrimaryKeys { get; set; }
public KeyStore(string pkName, string pkValue)
{
PrimaryKeys = new Dictionary<string, string> {{pkName, pkValue}};
}
public KeyStore()
{
PrimaryKeys = new Dictionary<string, string>();
}
}
I need to look at each record in List “A” and see if there is a matching record in List “B”. If there is, then this record needs to be stored in a new list that contains just matching records. A match is considered true if a record’s PrimaryKeys dictionary contains the same number of entries and the same key value combination as a record in List “B”. The order of the entries in the dictionary is not important in testing for equality. If there is a record in List “A” that does not have a match in List “B”, then this needs to be stored in a new list that will only contain records found in List “A”.
Previously I did something similar when I had Lists of strings where I used “Intersect” and “Except” to create lists of matched and non-matched records. I’m assuming that now that I need to compare these KeyStore objects I need to go up a level of complexity. Can anyone offer a solution or advise on how I should approach this problem?
EDIT 1 ----------------
Based on comments, I have created a class that implements IEqualityComparer, as shown below:
class KeyStoreComparer : IEqualityComparer<KeyStore>
{
public bool Equals(KeyStore x, KeyStore y)
{
if (x != null && x.PrimaryKeys.Count == y.PrimaryKeys.Count)
{
return x.PrimaryKeys.Keys.All(k => y.PrimaryKeys.ContainsKey(k)) &&
x.PrimaryKeys.Keys.All(k => x.PrimaryKeys[k].Equals(y.PrimaryKeys[k]));
}
return false;
}
public int GetHashCode(KeyStore obj)
{
return ReferenceEquals(obj, null) ? 0 : obj.GetHashCode();
}
}
I have created some dummy data but when the "Intersect" command is run the above code is never called. Any ideas where I am going wrong?
var ListA = new List<KeyStore>();
ListA.Add(new KeyStore("a", "b"));
ListA.Add(new KeyStore("c", "d"));
var ListB = new List<KeyStore>();
ListB.Add(new KeyStore("a", "b"));
ListB.Add(new KeyStore("x", "y"));
var g = ListA.Intersect(ListB, new KeyStoreComparer());
The code in the "Equals" and "GetHashCode" may not be correct but I'm just trying to get it to get as far as running it before I can improve it.
EDIT 2 ---------------------------------------
I have made various changes to the KeyStore class as shown in the example by “fox” on this page. I still don’t get the overridden functions to be called. As an experiment I tried this:
var result = ListA.Equals(ListB);
When I do this the overridden functions in the KeyStor class don’t run. But if I do this:
var result = ListA[0].Equals(ListB[0]);
The overridden functions do run and give the expected result. Anyone know how I can get this to work for all items in the lists rather than just for individual records?
EDIT 3 ---------------------------------------
The problem I am seeing is that the override works fine for single items, eg:
var a = new KeyStore("a", "b");
var b = new KeyStore("a", "b");
var c = a.Equals(b);
When I run the above my break point on the KeyStore "Equals" function is hit. As soon as I try to do something similar but with a List of KeyStore, the breakpoint is no longer hit. Do I need to do something extra when working with Lists?

public class KeyStore
{
public Dictionary<string, string> PrimaryKeys { get; set; }
public KeyStore(string pkName, string pkValue)
{
PrimaryKeys = new Dictionary<string, string> { { pkName, pkValue } };
}
public KeyStore()
{
PrimaryKeys = new Dictionary<string, string>();
}
public override bool Equals(object obj)
{
// If parameter is null return false.
if (obj == null)
return false;
// If parameter cannot be cast to KeyStore return false.
KeyStore targetKeyStore = obj as KeyStore;
if (targetKeyStore == null)
return false;
return PrimaryKeys.OrderBy(pk => pk.Key).SequenceEqual(targetKeyStore.PrimaryKeys.OrderBy(pk => pk.Key));
}
public override int GetHashCode()
{
StringBuilder content = new StringBuilder();
foreach (var item in PrimaryKeys.OrderBy(pk => pk.Key))
content.AppendFormat("{0}-{1}", item.Key, item.Value);
return content.ToString().GetHashCode();
}
}
Eric Lippert's guide on implementing GetHashCode wrote that "equal items have equal hashes". GetHashCode() implementation above just to show concept, might not suitable for production code.

Overriding the ToString method will help simplify your code quite a bit. See if this helps:
public class KeyStore
{
public SortedDictionary<string, string> PrimaryKeys
{
get;
set;
}
public KeyStore(string pkName, string pkValue)
{
PrimaryKeys = new SortedDictionary<string, string> { { pkName, pkValue } };
}
public KeyStore()
{
PrimaryKeys = new SortedDictionary<string, string>();
}
public override bool Equals(object obj)
{
if(obj == null || (KeyStore)obj == null)
return false;
KeyStore temp = (KeyStore)obj;
return ToString() == temp.ToString();
}
public override int GetHashCode()
{
return ToString().GetHashCode();
}
public override string ToString()
{
return PrimaryKeys.Count.ToString() + " : \n" + string.Join("\n",(from kvp in PrimaryKeys
let s = kvp.Key + " - " + kvp.Value
select s));
}
}
List<KeyStore> Lista = new List<KeyStore>
{
new KeyStore("testa","testa1"),
new KeyStore("testb","testb1"),
new KeyStore("testc", "testc1")
};
List<KeyStore> Listb = new List<KeyStore>
{
new KeyStore("testa","testa1"),
new KeyStore("testd","testb1"),
new KeyStore("testc", "testa1"),
new KeyStore("teste", "teste1")
};
var Listc = Lista.Intersect(Listb).ToList();
var Listd = Lista.Except(Listb).ToList();
?Listc
Count = 1
[0]: {1 :
testa - testa1}
?Listd
Count = 2
[0]: {1 :
testb - testb1}
[1]: {1 :
testc - testc1}

Attempting to merge two Lists<> via Union. Still have duplicates

I have two Lists:
List<L1>, List<L2>
L1 = { detailId = 5, fileName = "string 1" }{ detailId = 5, fileName = "string 2" }
L2 = { detailId = 5, fileName = "string 2" }{ detailId = 5, fileName = "string 3" }
That I want to combine them with no duplicates:
List<L3>
L1 = { detailId = 5, fileName = "string 1" }{ detailId = 5, fileName = "string 2" }{ detailId = 5, fileName = "string 3" }
I've tried:
L1.Union(L2).ToList();
L1.Concat(L2).Distinct().ToList();
But both return with duplicates (1, 2, 2, 3).
Not sure what I'm missing.
edit Here's the method. It takes one list and creates another from a delimited string, and tries to combine them.
private List<Files> CombineList(int detailId, string fileNames)
{
List<Files> f1 = new List<Files>();
List<Files> f2 = new List<Files>();
f1 = GetFiles(detailId, false);
if (f1[0].fileNames != "")
{
string[] names = fileNames.Split('|');
for (int i = 0; i < names.Length; i++)
{
Files x = new Files();
x.detailId = detailId;
x.fileNames = names[i];
f2.Add(x);
}
List<Files> f3 = f1.Union(f2).ToList();
}
return f3;
}

From MSDN, for Union :
The default equality comparer, Default, is used to compare values of
the types that implement the IEqualityComparer(Of T) generic
interface. To compare a custom data type, you need to implement this
interface and provide your own GetHashCode and Equals methods for the
type.
link
Since you use a custom type, you need to either override the GetHashCode and Equals or provide an object that implements the IEqualityComparer interface (preferably IEquatable) and provide it as a second parameter to Union.
Here's a simple example of implementing such a class.

I don't like overriding the Files class equals object and getHashCode since you are interfering with the object. Let another object do that and just pass it in. This way if you have an issue with it later on, just swap it out and pass another IEqualityComparer
Here's an example you can just test out
public void MainMethod()
{
List<Files> f1 = new List<Files>() { new Files() { detailId = 5, fileName = "string 1" }, new Files() { detailId = 5, fileName = "string 2" } };
List<Files> f2 = new List<Files>() { new Files() { detailId = 5, fileName = "string 2" }, new Files() { detailId = 5, fileName = "string 3" } };
var f3 = f1.Union(f2, new FilesComparer()).ToList();
foreach (var f in f3)
{
Console.WriteLine(f.detailId + " " + f.fileName);
}
}
public class Files
{
public int detailId;
public string fileName;
}
public class FilesComparer : IEqualityComparer<Files>
{
public bool Equals(Files x, Files y)
{
return x.fileName == y.fileName;
}
public int GetHashCode(Files obj)
{
return obj.fileName.GetHashCode();
}
}

If your elements do not implement some kind of comparison interface (Object.Equals, IEquatable, IComparable, etc.) then any equality test between them will involve ReferenceEquals, in which two different objects are two different objects, even if all their members contain identical values.

If you're merging a list of objects, you will need to define some criteria for the equality comparisons. The example below demonstrates this:
class MyModelTheUniqueIDComparer : IEqualityComparer<MyModel>
{
public bool Equals(MyModel x, MyModel y)
{
return x.SomeValue == y.SomeValue && x.OtherValue == y.OtherValue;
}
// If Equals() returns true for a pair of objects
// then GetHashCode() must return the same value for these objects.
public int GetHashCode(MyModel myModel)
{
unchecked
{
int hash = 17;
hash = hash * 31 + myModel.SomeValue.GetHashCode();
hash = hash * 31 + myModel.OtherValue.GetHashCode();
return hash;
}
}
}
Then you can call to get the result:
var result = q1.Union(q2, new MyModelTheUniqueIDComparer());

From MSDN Enumerable.Union Method:
If you want to compare sequences of objects of a custom data type, you
have to implement the IEqualityComparer < T > generic interface in the
class.
A sample implementation specific to your Files class so that the Union works correctly when merging two collections of custom type:
public class Files : IEquatable<Files>
{
public string fileName { get; set; }
public int detailId { get; set; }
public bool Equals(Files other)
{
//Check whether the compared object is null.
if (Object.ReferenceEquals(other, null)) return false;
//Check whether the compared object references the same data.
if (Object.ReferenceEquals(this, other)) return true;
//Check whether the products' properties are equal.
return detailId.Equals(other.detailId) && fileName.Equals(other.fileName);
}
// If Equals() returns true for a pair of objects
// then GetHashCode() must return the same value for these objects.
public override int GetHashCode()
{
//Get hash code for the fileName field if it is not null.
int hashFileName = fileName == null ? 0 : fileName.GetHashCode();
//Get hash code for the detailId field.
int hashDetailId = detailId.GetHashCode();
//Calculate the hash code for the Files object.
return hashFileName ^ hashDetailId;
}
}