We Keep Coding

LINQ distinct with IEquatable is not returning distinct result [duplicate]

This question already has answers here:
LINQ's Distinct() on a particular property
(23 answers)
Closed 19 days ago.
class Program
{
static void Main(string[] args)
{
List<Book> books = new List<Book>
{
new Book
{
Name="C# in Depth",
Authors = new List<Author>
{
new Author
{
FirstName = "Jon", LastName="Skeet"
},
new Author
{
FirstName = "Jon", LastName="Skeet"
},
}
},
new Book
{
Name="LINQ in Action",
Authors = new List<Author>
{
new Author
{
FirstName = "Fabrice", LastName="Marguerie"
},
new Author
{
FirstName = "Steve", LastName="Eichert"
},
new Author
{
FirstName = "Jim", LastName="Wooley"
},
}
},
};
var temp = books.SelectMany(book => book.Authors).Distinct();
foreach (var author in temp)
{
Console.WriteLine(author.FirstName + " " + author.LastName);
}
Console.Read();
}
}
public class Book
{
public string Name { get; set; }
public List<Author> Authors { get; set; }
}
public class Author
{
public string FirstName { get; set; }
public string LastName { get; set; }
public override bool Equals(object obj)
{
return true;
//if (obj.GetType() != typeof(Author)) return false;
//else return ((Author)obj).FirstName == this.FirstName && ((Author)obj).FirstName == this.LastName;
}
}
This is based on an example in "LINQ in Action". Listing 4.16.
This prints Jon Skeet twice. Why? I have even tried overriding Equals method in Author class. Still Distinct does not seem to work. What am I missing?
Edit:
I have added == and != operator overload too. Still no help.
public static bool operator ==(Author a, Author b)
{
return true;
}
public static bool operator !=(Author a, Author b)
{
return false;
}

LINQ Distinct is not that smart when it comes to custom objects.
All it does is look at your list and see that it has two different objects (it doesn't care that they have the same values for the member fields).
One workaround is to implement the IEquatable interface as shown here.
If you modify your Author class like so it should work.
public class Author : IEquatable<Author>
{
public string FirstName { get; set; }
public string LastName { get; set; }
public bool Equals(Author other)
{
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}
public override int GetHashCode()
{
int hashFirstName = FirstName == null ? 0 : FirstName.GetHashCode();
int hashLastName = LastName == null ? 0 : LastName.GetHashCode();
return hashFirstName ^ hashLastName;
}
}
Try it as DotNetFiddle

The Distinct() method checks reference equality for reference types. This means it is looking for literally the same object duplicated, not different objects which contain the same values.
There is an overload which takes an IEqualityComparer, so you can specify different logic for determining whether a given object equals another.
If you want Author to normally behave like a normal object (i.e. only reference equality), but for the purposes of Distinct check equality by name values, use an IEqualityComparer. If you always want Author objects to be compared based on the name values, then override GetHashCode and Equals, or implement IEquatable.
The two members on the IEqualityComparer interface are Equals and GetHashCode. Your logic for determining whether two Author objects are equal appears to be if the First and Last name strings are the same.
public class AuthorEquals : IEqualityComparer<Author>
{
public bool Equals(Author left, Author right)
{
if((object)left == null && (object)right == null)
{
return true;
}
if((object)left == null || (object)right == null)
{
return false;
}
return left.FirstName == right.FirstName && left.LastName == right.LastName;
}
public int GetHashCode(Author author)
{
return (author.FirstName + author.LastName).GetHashCode();
}
}

Another solution without implementing IEquatable, Equals and GetHashCode is to use the LINQs GroupBy method and to select the first item from the IGrouping.
var temp = books.SelectMany(book => book.Authors)
.GroupBy (y => y.FirstName + y.LastName )
.Select (y => y.First ());
foreach (var author in temp){
Console.WriteLine(author.FirstName + " " + author.LastName);
}

There is one more way to get distinct values from list of user defined data type:
YourList.GroupBy(i => i.Id).Select(i => i.FirstOrDefault()).ToList();
Surely, it will give distinct set of data

Distinct() performs the default equality comparison on objects in the enumerable. If you have not overridden Equals() and GetHashCode(), then it uses the default implementation on object, which compares references.
The simple solution is to add a correct implementation of Equals() and GetHashCode() to all classes which participate in the object graph you are comparing (ie Book and Author).
The IEqualityComparer interface is a convenience that allows you to implement Equals() and GetHashCode() in a separate class when you don't have access to the internals of the classes you need to compare, or if you are using a different method of comparison.

You've overriden Equals(), but make sure you also override GetHashCode()

The Above answers are wrong!!!
Distinct as stated on MSDN returns the default Equator which as stated The Default property checks whether type T implements the System.IEquatable interface and, if so, returns an EqualityComparer that uses that implementation. Otherwise, it returns an EqualityComparer that uses the overrides of Object.Equals and Object.GetHashCode provided by T
Which means as long as you overide Equals you are fine.
The reason you're code is not working is because you check firstname==lastname.
see https://msdn.microsoft.com/library/bb348436(v=vs.100).aspx and https://msdn.microsoft.com/en-us/library/ms224763(v=vs.100).aspx

You can achieve this several ways:
1. You may to implement the IEquatable interface as shown Enumerable.Distinct Method or you can see #skalb's answer at this post
2. If your object has not unique key, You can use GroupBy method for achive distinct object list, that you must group object's all properties and after select first object.
For example like as below and working for me:
var distinctList= list.GroupBy(x => new {
Name= x.Name,
Phone= x.Phone,
Email= x.Email,
Country= x.Country
}, y=> y)
.Select(x => x.First())
.ToList()
MyObject class is like as below:
public class MyClass{
public string Name{get;set;}
public string Phone{get;set;}
public string Email{get;set;}
public string Country{get;set;}
}
3. If your object's has unique key, you can only use the it in group by.
For example my object's unique key is Id.
var distinctList= list.GroupBy(x =>x.Id)
.Select(x => x.First())
.ToList()

You can use extension method on list which checks uniqueness based on computed Hash.
You can also change extension method to support IEnumerable.
Example:
public class Employee{
public string Name{get;set;}
public int Age{get;set;}
}
List<Employee> employees = new List<Employee>();
employees.Add(new Employee{Name="XYZ", Age=30});
employees.Add(new Employee{Name="XYZ", Age=30});
employees = employees.Unique(); //Gives list which contains unique objects.
Extension Method:
public static class LinqExtension
{
public static List<T> Unique<T>(this List<T> input)
{
HashSet<string> uniqueHashes = new HashSet<string>();
List<T> uniqueItems = new List<T>();
input.ForEach(x =>
{
string hashCode = ComputeHash(x);
if (uniqueHashes.Contains(hashCode))
{
return;
}
uniqueHashes.Add(hashCode);
uniqueItems.Add(x);
});
return uniqueItems;
}
private static string ComputeHash<T>(T entity)
{
System.Security.Cryptography.SHA1CryptoServiceProvider sh = new System.Security.Cryptography.SHA1CryptoServiceProvider();
string input = JsonConvert.SerializeObject(entity);
byte[] originalBytes = ASCIIEncoding.Default.GetBytes(input);
byte[] encodedBytes = sh.ComputeHash(originalBytes);
return BitConverter.ToString(encodedBytes).Replace("-", "");
}

The Equal operator in below code is incorrect.
Old
public bool Equals(Author other)
{
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}
NEW
public override bool Equals(Object obj)
{
var other = obj as Author;
if (other is null)
{
return false;
}
if (FirstName == other.FirstName && LastName == other.LastName)
return true;
return false;
}

Instead of
var temp = books.SelectMany(book => book.Authors).Distinct();
Do
var temp = books.SelectMany(book => book.Authors).DistinctBy(f => f.Property);

Proper way to write GetHashCode() when Equality Comparer is based on OR operation?

I'm trying to write an Equality Comparer for a simple class with 3 fields, like so:
public class NumberClass
{
public int A { get; set; }
public int B { get; set; }
public int C { get; set; }
}
My condition for two objects of NumberClass to be equal is if Obj1.A == Obj2.A || Obj1.B == Obj2.B (in other words, OR), Obj1 and Obj2 being instances of NumberClass.
I can easily write the Equals() of my comparer as follows, but I don't know what to do with my GetHashCode() method.
public bool Equals(NumberClass x, NumberClass y)
{
if (x.A == y.A || x.B == y.B)
return true;
else
return false;
}
public int GetHashCode(NumberClass obj)
{
return ???
}
If my condition for equality was AND instead of OR, I could write my GetHashCode() as follows, taken from this SO answer.
public int GetHashCode(NumberClass obj)
{
unchecked
{
int hash = 17;
if (obj != null)
{
hash = hash * 23 + obj.A.GetHashCode();
hash = hash * 23 + obj.B.GetHashCode();
}
return hash;
}
}
But that obviously wouldn't work for OR since only one of A or B being equal is sufficient for my equality condition to be true.
One workaround I could think of is always returning the same value in GetHashCode() which would be sufficient for LINQ operations such as Distinct(), but I feel like there should be another way as that has its own shortcomings.
What's the proper way to handle this situation?
P.S.
For testing, imagine my Main() is the following:
static void Main(string[] args)
{
List<NumberClass> list = new List<NumberClass>();
list.Add(new NumberClass { A = 1, B = 2, C = 3 });
list.Add(new NumberClass { A = 1, B = 22, C = 33 });
var distinct = list.Distinct(new NumberComparer());
Console.ReadKey();
}
I expect distinct to contain only the first element in the list.

There is no solution for your situation. Your objects violate assumptions that are necessary for an equality comparer to work, for example, it assumes that equality is going to be transitive, but that's not true of your implementation of equality.
You simply won't be able to use any hash-based algorithms so long as you have "fuzzy" equality like that.

How to count distinct trees in a list of trees

I have a tree structure as follows:
public class TAGNode
{
public string Val;
public string Type = "";
private List<TAGNode> childs;
public IList<TAGNode> Childs
{
get { return childs.AsReadOnly(); }
}
public TAGNode AddChild(string val)
{
TAGNode tree = new TAGNode(val);
tree.Parent = this;
childs.Add(tree);
return tree;
}
public override bool Equals(object obj)
{
var t = obj as TAGNode;
bool eq = Val == t.Val && childs.Count == t.Childs.Count;
if (eq)
{
for (int i = 0; i < childs.Count; i++)
{
eq &= childs[i].Equals(t.childs[i]);
}
}
return eq;
}
}
I have a list of such trees which can contain repeated trees, by repeated I mean they have the same structure with the same labels. Now I want to select distinct trees from this list. I tried
etrees = new List<TAGNode>();
TAGNode test1 = new TAGNode("S");
test1.AddChild("A").AddChild("B");
test1.AddChild("C");
TAGNode test2 = new TAGNode("S");
test2.AddChild("A").AddChild("B");
test2.AddChild("C");
TAGNode test3 = new TAGNode("S");
test3.AddChild("A");
test3.AddChild("B");
etrees.Add(test1);
etrees.Add(test2);
etrees.Add(test3);
var results = etrees.Distinct();
label1.Text = results.Count() + " unique trees";
This returns the count of all the trees (3) while I expect 2 distinct trees! I think maybe I should implement a suitable Equals function for it, but as I tested it doesn't care what Equals returns!

I think maybe I should implement a suitable Equals function for it
Correct.
but as I tested it doesn't care what Equals returns!
Because you have to implement a matching GetHashCode! It doesn't need to include all the items used inside the Equals, in your case Val could be sufficient. Remember, all you need is to return one and the same hash code for the potentially equal items. The items with different hash codes are considered non equal and never checked with Equals.
So something like this should work:
public bool Equals(TAGNode other)
{
if ((object)this == (object)other) return true;
if ((object)other == null) return false;
return Val == other.Val && childs.SequenceEqual(other.childs);
}
public override bool Equals(object obj) => Equals(obj as TAGNode);
public override int GetHashCode() => Val?.GetHashCode() ?? 0;
Once you do that, you can also "mark" your TAGNode as IEquatable<TAGNode>, to let the default equality comparer directly call the Equals(TAGNode other) overload.

see https://msdn.microsoft.com/en-us/library/bb348436(v=vs.100).aspx
If you want to return distinct elements from sequences of objects of some custom data type, you have to implement the IEquatable generic interface in the class. The following code example shows how to implement this interface in a custom data type and provide GetHashCode and Equals methods.
you need to impliment IEquatable for TagNode

try following for GetHashCode. I updated the method below to make more robust. Was afraid original answer may not create unique has values.
private int GetHashCode(TAGNode node)
{
string hash = node.Val;
foreach(TAGNode child in node.childs)
{
hash += GetHashStr(child);
}
return hash.GetHashCode();
}
private string GetHashStr(TAGNode node)
{
string hash = node.Val;
foreach (TAGNode child in node.childs)
{
hash += ":" + GetHashStr(child);
}
return hash;
}

Avoiding duplicates in a HashSet of custom types in C#

I have the following custom class deriving from Tuple:
public class CustomTuple : Tuple<List<string>, DateTime?>
{
public CustomTuple(IEnumerable<string> strings, DateTime? time)
: base(strings.OrderBy(x => x).ToList(), time)
{
}
}
and a HashSet<CustomTuple>. The problem is that when I add items to the set, they are not recognised as duplicates. i.e. this outputs 2, but it should output 1:
void Main()
{
HashSet<CustomTuple> set = new HashSet<CustomTuple>();
var a = new CustomTuple(new List<string>(), new DateTime?());
var b = new CustomTuple(new List<string>(), new DateTime?());
set.Add(a);
set.Add(b);
Console.Write(set.Count); // Outputs 2
}
How can I override the Equals and GetHashCode methods to cause this code to output a set count of 1?

You should override GetHashCode and Equals virtual methods defined in System.Object class.
Please remember that:
If two objects are logically "equal" then they MUST have the same hash code!
If two objects have the same hashcode, then it is not mandatory to have your objects equal.
Also, i've noticed an architectural problem in your code:
List is a mutable type but overriding Equals and GetHashCode usually makes your class logically to behave like a value type. So having "Item1" a mutable type and behaving like a value type is very dangerous. I suggest replacing your List with a ReadOnlyCollection . Then you would have to make a method that checks whether two ReadOnlyCollections are Equal.
For the GetHashCode () method, just compose a string from all string items found in Item1 then append a string that represents the Hash code for datetime then finally call on the concatenated result the "GetHashCode ()" overrided on string method. So normally you would have:
override int GetHashCode () {
return (GetHashCodeForList (Item1) + (Item2 ?? DateTime.MinValue).GetHashCode ()).GetHashCode ();
}
And the GetHashCodeForList method would be something like this:
private string GetHashCodeForList (IEnumerable <string> lst) {
if (lst == null) return string.Empty;
StringBuilder sb = new StringBuilder ();
foreach (var item in lst) {
sb.Append (item);
}
return sb.ToString ();
}
Final note: You could cache the GetHashCode result since it is relative expensive to get and your entire class would became immutable (if you replace List with a readonly collection).

A HashSet<T> will first call GetHashCode, so you need to work on that first. For an implementation, see this answer: https://stackoverflow.com/a/263416/1250301
So a simple, naive, implementation might look like this:
public override int GetHashCode()
{
unchecked
{
int hash = 17;
hash = hash * 23 + this.Item2.GetHashCode();
foreach (var s in this.Item1)
{
hash = hash * 23 + s.GetHashCode();
}
return hash;
}
}
However, if your lists are long, then this might not be efficient enough. So you'll have to decide where to compromise depending on how tolerant you are of collisions.
If the result of GetHashCode for two items are the same, then, and only then, will it call Equals. An implementation of Equals is going to need to compare the items in the list. Something like this:
public override bool Equals(object o1)
{
var o = o1 as CustomTuple;
if (o == null)
{
return false;
}
if (Item2 != o.Item2)
{
return false;
}
if (Item1.Count() != o.Item1.Count())
{
return false;
}
for (int i=0; i < Item1.Count(); i++)
{
if (Item1[i] != o.Item1[i])
{
return false;
}
}
return true;
}
Note that we check the date (Item2) first, because that's cheap. If the date isn't the same, we don't bother with anything else. Next we check the Count on both collections (Item1). If they don't match, there's no point iterating the collections. Then we loop through both collections and compare each item. Once we find one that doesn't match, we return false because there is no point continuing to look.
As pointed out in George's answer, you also have the problem that your list is mutable, which will cause problems with your HashSet, for example:
var a = new CustomTuple(new List<string>() {"foo"} , new DateTime?());
var b = new CustomTuple(new List<string>(), new DateTime?());
set.Add(a);
set.Add(b);
// Hashset now has two entries
((List<string>)a.Item1).Add("foo");
// Hashset still has two entries, but they are now identical.
To solve that, you need to force your IEnumerable<string> to be readonly. You could do something like:
public class CustomTuple : Tuple<IReadOnlyList<string>, DateTime?>
{
public CustomTuple(IEnumerable<string> strings, DateTime? time)
: base(strings.OrderBy(x => x).ToList().AsReadOnly(), time)
{
}
public override bool Equals(object o1)
{
// as above
}
public override int GetHashCode()
{
// as above
}
}

This is is what I went for, which outputs 1 as desired:
private class CustomTuple : Tuple<List<string>, DateTime?>
{
public CustomTuple(IEnumerable<string> strings, DateTime? time)
: base(strings.OrderBy(x => x).ToList(), time)
{
}
public override bool Equals(object obj)
{
if (obj == null || GetType() != obj.GetType())
{
return false;
}
var that = (CustomTuple) obj;
if (Item1 == null && that.Item1 != null || Item1 != null && that.Item1 == null) return false;
if (Item2 == null && that.Item2 != null || Item2 != null && that.Item2 == null) return false;
if (!Item2.Equals(that.Item2)) return false;
if (that.Item1.Count != Item1.Count) return false;
for (int i = 0; i < Item1.Count; i++)
{
if (!Item1[i].Equals(that.Item1[i])) return false;
}
return true;
}
public override int GetHashCode()
{
int hash = 17;
hash = hash*23 + Item2.GetHashCode();
return Item1.Aggregate(hash, (current, s) => current*23 + s.GetHashCode());
}
}

Should I Overload == Operator?

How does the == operator really function in C#? If it used to compare objects of class A, will it try to match all of A's properties, or will it look for pointers to the same memory location (or maybe something else)?
Let's create a hypothetical example. I'm writing an application that utilizes the Twitter API, and it has a Tweet class, which has all the properties of a single tweet: text, sender, date&time, source, etc. If I want to compare objects of class Tweet for equivalence, can I just use:
Tweet a, b;
if (a == b)
{
//do something...
}
Will that check for equivalence of all the properties of the Tweet class between a and b?
If not, would the correct approach be to overload the == operator to explicitly check for equivalence of all the fields?
UPDATE: From the first two answers, am I right in assuming:
If the == operator or Equals method is not overloaded for a class, the == operator for the object class is used.
The == operator for the object class checks for equality in memory location.
I have to overload the == operator or the Equals method to accomplish this task.
In the overload, I have to check for equivalence in properties manually, so there is no way to do it semi-automatically, say, in a loop, right?
UPDATE #2: Yuriy made a comment that it is possible to do check for equivalence in properties in the == operator with reflection. How can this be done? Could you give me some sample code? Thanks!

For reference types, the default implementations of both the == operator and the Equals() method will simply check that both objects have the same reference, and are therefore the same instance.
If you want to check the contents of two different objects are equal then you must write the code to do it yourself, one way or another. It would be possible to do with reflection (the MbUnit test framework does something along these lines) but with a heavy performance penalty and a good chance that it wouldn't do quite what you expected anyway, and you should implement == or Equals and GetHashCode by hand.

MSDN has a good example of how to do it:
public override bool Equals(object o)
{
try
{
return (bool) (this == (DBBool) o);
}
catch
{
return false;
}
}
Then you overload the == and !=:
// Equality operator. Returns dbNull if either operand is dbNull,
// otherwise returns dbTrue or dbFalse:
public static DBBool operator ==(DBBool x, DBBool y)
{
if (x.value == 0 || y.value == 0) return dbNull;
return x.value == y.value? dbTrue: dbFalse;
}
// Inequality operator. Returns dbNull if either operand is
// dbNull, otherwise returns dbTrue or dbFalse:
public static DBBool operator !=(DBBool x, DBBool y)
{
if (x.value == 0 || y.value == 0) return dbNull;
return x.value != y.value? dbTrue: dbFalse;
}
And don't forget to overload the GetHash method.
Edit:
I wrote the following quick sample for using reflection in a compare. This would have to be much more comprehensive, I might try doing a blog on it if people want me to:
public class TestEquals
{
private int _x;
public TestEquals(int x)
{
this._x = x;
}
public override bool Equals(object obj)
{
TestEquals te = (TestEquals)obj;
if (te == null) return false;
foreach (var field in typeof(TestEquals)
.GetFields(BindingFlags.NonPublic | BindingFlags.Instance))
{
if (!field.GetValue(this).Equals(field.GetValue(te)))
return false;
}
return true;
}
}

The proper approach is the overload the equals method of the Tweet class in addition to the == operator, as described here.

Will that check for equivalence of all the properties of the Tweet class between a and b?
No
If not, would the correct approach be to overload the == operator to explicitly check for equivalence of all the fields?
You can either overload the == operator, or overload the Equals function.
Edit
#Yuriy gave a good example for compating all the non public variables. Since i also wrote an example, here it is (mine compares properties)
class TwitterItem
{
private string myValue = "default value";
public string Value1
{
get { return myValue; }
set { myValue = value; }
}
public string Value2
{
get { return myValue; }
set { myValue = value; }
}
public string Value3
{
get { return myValue; }
set { myValue = value; }
}
public override bool Equals(object obj)
{
if (base.Equals(obj)) return true;
Type type = typeof(TwitterItem);
PropertyInfo[] properties = type.GetProperties();
foreach (PropertyInfo property in properties)
{
if (false == property.GetValue(this, null).Equals(property.GetValue(obj, null)))
return false;
}
return true;
}
}

You can compare the properties using reflection:
var a = new Entity() { Name = "test", ID = "1" };
var b = new Entity() { Name = "test", ID = "1" };
var c = new Entity() { Name = "test", ID = "2" };
System.Diagnostics.Debug.WriteLine(a.Equals(b));//Returns true
System.Diagnostics.Debug.WriteLine(a.Equals(c));//Returns false
public class Entity
{
public string Name { get; set; }
public string ID { get; set; }
public override bool Equals(object obj)
{
var t = obj.GetType();
foreach (var p in t.GetProperties())
{
if (t.GetProperty(p.Name).GetValue(obj, null) != t.GetProperty(p.Name).GetValue(this, null))
return false;
}
return true;
}
}