We Keep Coding

How to compare list efficiently?

I'm currently working on a web application in asp.net. In certain api-calls it is necessary to compare ListA with a ListB of Lists to determine if ListA has the same elements of any List in ListB. In other words: If ListA is included in ListB.
Both collections are queried with Linq of an EF-Code-First db. ListB has either one matching List or none, never more than one. In the worst case ListB has millions of elements, so the comparison needs to be scalable.
Instead of doing nested foreach loops, i'm looking for a pure linq query, which will let the db do the work. (before i consider multi column index)
To illustrate the structure:
//In reality Lists are queried of EF
var ListA = new List<Element>();
var ListB = new List<List<Element>>();
List<Element> solution;
bool flag = false;
foreach (List e1 in ListB) {
foreach(Element e2 in ListA) {
if (e1.Any(e => e.id == e2.id)) flag = true;
else {
flag = false;
break;
}
}
if(flag) {
solution = e1;
break;
}
}
Update Structure
Since its a EF Database i'll provide the relevant Object Structure. I'm not sure if i'm allowed to post real code, so this example is still generic.
//List B
class Result {
...
public int Id;
public virtual ICollection<Curve> curves;
...
}
class Curve {
...
public int Id;
public virtual Result result;
public int resultId;
public virtual ICollection<Point> points;
...
}
public class Point{
...
public int Id;
...
}
The controller (for the api-call) wants to serve the right Curve-Object. To identify the right Object, a filter (ListA) is provided (which is in fact a Curve Object)
Now the filter (ListA) needs to be compared to the List of Curves in Result (ListB)
The only way to compare the Curves is by comparing the Points both have.
(So infact comparing Lists)
Curves have around 1 - 50 Points.
Result can have around 500.000.000 Curves
It's possible to compare by Object-Identity here, because all Objects (even the filter) is re-queried of the db.
I'm looking for a way to implement this mechanism, not how to get around this situation. (e.g. by using multi column index (altering the table))
(for illustration purposes):
class controller {
...
public Response serveRequest(Curve filter) {
foreach(Curve c in db.Result.curves) {
if(compare(filter.points , c.points)) return c;
}
}
}

Use Except:
public static bool ContainsAllItems(IList<T> listA, IList<T> listB)
{
return !listB.Except(listA).Any();
}
the above method will tell if listA contains all the elements of listB or not..and the complexity is much faster than O(n*m) approach.

Try this:
bool isIn = ListB.Any(x=>x.Count==ListA.Count && ListA.All(y=>x.Contains(y)));
or, if you want the element
var solution = ListB.FirstOrDefault(x=>x.Count==ListA.Count && ListA.All(y=>x.Contains(y)));

I have something for you:
var db = new MyContext();
var a = db.LoadList(); // or whatever
var b = new List<IQueryable<Entities>>(db.LoadListOfLists()/*or whatever*/);
b.Any(x => x.Count.Equals(a.Count) & x.All(y => a.Any(z => z.Id == y.Id)));

Because performance is concern, I would suggest convert your listA to lookup/dictionary before comparing Ex-
var listALookup = listA.ToLookup(item => item.Id);
var result = listB.FirstOrDefault(childList => childList.Count == listA.Count && childList.All(childListItem => listALookup.Contains(childListItem.Id)));
Lookup.Contain is O(1) while List.Contains is O(n)
Better option is to perform this comparison at db level, to reduce loading unnecessary data.

What to do to get only one List?

Hello i have a method that compares the objects of 2 Lists for differences. Right now this works but only for one property at a time.
Here is the Method:
public SPpowerPlantList compareTwoLists(string sqlServer, string database, DateTime timestampCurrent, string noteCurrent, DateTime timestampOld, string noteOld)
{
int count = 0;
SPpowerPlantList powerPlantListCurrent = loadProjectsAndComponentsFromSqlServer(sqlServer, database, timestampCurrent, noteCurrent);
SPpowerPlantList powerPlantListOld = loadProjectsAndComponentsFromSqlServer(sqlServer, database, timestampOld, noteOld);
SPpowerPlantList powerPlantListDifferences = new SPpowerPlantList();
count = powerPlantListOld.Count - powerPlantListCurrent.Count;
var differentObjects = powerPlantListCurrent.Where(p => !powerPlantListOld.Any(l => p.mwWeb == l.mwWeb)).ToList();
foreach (var differentObject in differentObjects)
{
powerPlantListDifferences.Add(differentObject);
}
return powerPlantListDifferences;
}
This works and i get 4 Objects in the new List. The Problem is that i have a few other properties that i need to compare. Instead of mwWeb for example name. When i try to change it i need for every new property a new List and a new Foreach-Loop.
e.g.
int count = 0;
SPpowerPlantList powerPlantListCurrent = loadProjectsAndComponentsFromSqlServer(sqlServer, database, timestampCurrent, noteCurrent);
SPpowerPlantList powerPlantListOld = loadProjectsAndComponentsFromSqlServer(sqlServer, database, timestampOld, noteOld);
SPpowerPlantList powerPlantListDifferences = new SPpowerPlantList();
SPpowerPlantList powerPlantListDifferences2 = new SPpowerPlantList();
count = powerPlantListOld.Count - powerPlantListCurrent.Count;
var differentObjects = powerPlantListCurrent.Where(p => !powerPlantListOld.Any(l => p.mwWeb == l.mwWeb)).ToList();
var differentObjects2 = powerPlantListCurrent.Where(p => !powerPlantListOld.Any(l => p.shortName == l.shortName)).ToList();
foreach (var differentObject in differentObjects)
{
powerPlantListDifferences.Add(differentObject);
}
foreach (var differentObject in differentObjects2)
{
powerPlantListDifferences2.Add(differentObject);
}
return powerPlantListDifferences;
Is there a way to prevent this? or to make more querys and get only 1 List with all different Objects back?
I tried it with except and intersect but that didnt worked.
So any help or advise would be great and thx for your time.
PS: If there is something wrong with my question-style please say it to me becouse i try to learn to ask better questions.

You may be able to simply chain the properties that you wanted to compare within your Where() clause using OR statements :
// This should get you any elements that have different A properties, B properties, etc.
var different = current.Where(p => !old.Any(l => p.A == l.A || p.B == l.B))
.ToList();
If that doesn't work and you really want to use the Except() or Intersect() methods to properly compare the objects, you could write your own custom IEqualityComparer<YourPowerPlant> to use to properly compare them :
class PowerPlantComparer : IEqualityComparer<YourPowerPlant>
{
// Powerplants are are equal if specific properties are equal.
public bool Equals(YourPowerPlant x, YourPowerPlant y)
{
// Check whether the compared objects reference the same data.
if (Object.ReferenceEquals(x, y)) return true;
//Check whether any of the compared objects is null.
if (Object.ReferenceEquals(x, null) || Object.ReferenceEquals(y, null))
return false;
// Checks the other properties to compare (examples using mwWeb and shortName)
return x.mwWeb == y.mwWeb && x.shortName == y.shortName;
}
// If Equals() returns true for a pair of objects
// then GetHashCode() must return the same value for these objects.
public int GetHashCode(YourPowerPlant powerPlant)
{
// Check whether the object is null
if (Object.ReferenceEquals(powerPlant, null)) return 0;
// Get hash code for the mwWeb field if it is not null.
int hashA = powerPlant.mwWeb == null ? 0 : powerPlant.mwWeb.GetHashCode();
// Get hash code for the shortName field if it is not null.
int hashB = powerPlant.shortName == null ? 0 : powerPlant.shortName.GetHashCode();
// Calculate the hash code for the product.
return hashA ^ hashB;
}
}
and then you could likely use something like one of the following depending on your needs :
var different = current.Except(old,new PowerPlantComparer());
or :
var different = current.Intersect(old,new PowerPlantComparer());

One way is to use IEqualityComparer as Rion Williams suggested, if you'd like a more flexible solution you can split logic in to two parts. First create helper method that accepts two lists, and function where you can define what properties you wish to compare. For example :
public static class Helper
{
public static SPpowerPlantList GetDifference(this SPpowerPlantList current, SPpowerPlantList old, Func<PowerPlant, PowerPlant, bool> func)
{
var diff = current.Where(p => old.All(l => func(p, l))).ToList();
var result = new SPpowerPlantList();
foreach (var item in diff) result.Add(item);
return result;
}
}
And use it :
public SPpowerPlantList compareTwoLists(string sqlServer, string database,
DateTime timestampCurrent, string noteCurrent,
DateTime timestampOld, string noteOld)
{
var powerPlantListCurrent = ...;
var powerPlantListOld = ...;
var diff = powerPlantListCurrent.GetDifference(
powerPlantListOld,
(x, y) => x.mwWeb != y.mwWeb ||
x.shortName != y.shortName);
return diff;
}
P.S. if it better suits your needs, you could move method inside of existing class :
public class MyClass
{
public SPpowerPlantList GetDifference(SPpowerPlantList current, SPpowerPlantList old, Func<PowerPlant, PowerPlant, bool> func)
{
...
}
}
And call it (inside of class) :
var result = GetDifference(currentValues, oldValues, (x, y) => x.mwWeb != y.mwWeb);

The easiest way to do this would be to compare some unique identifier (ID)
var differentObjects = powerPlantListCurrent
.Where(p => !powerPlantListOld.Any(l => p.Id == l.Id)
.ToList();
If the other properties might have been updated and you want to check that too, you'll have to compare all of them to detect changes made to existing elements:
Implement a camparison-method (IComparable, IEquatable, IEqualityComparer, or override Equals) or, if that's not possible because you didn't write the class yourself (code generated or external assembly), write a method to compare two of those SPpowerPlantList elements and use that instead of comparing every single property in Linq. For example:
public bool AreThoseTheSame(SPpowerPlantList a,SPpowerPlantList b)
{
if(a.mwWeb != b.mwWeb) return false;
if(a.shortName != b.shortName) return false;
//etc.
return true;
}
Then replace your difference call with this:
var differentObjects = powerPlantListCurrent
.Where(p => !powerPlantListOld.Any(l => AreThoseTheSame(p,l))
.ToList();

Return objects which contain Lists, based on the those lists matching

I have Cell objects which contain a List<int> called PossibleValues. I'm trying to find a way to get a list of cells in which all members have matching PossibleValues. I currently have:
foreach (var cell in group)
{
var cellsWithMatchingPossibleValues = group.Where(c => c.PossibleValues == cell.PossibleValues);
}
Unfortunately this isn't working, I suspect my linq statement isn't comparing the contents of PossibleValues, but instead comparing a reference of some kind, so that even in the case where both lists are composed of 3 and nothing else, cellsWithMatchingPossibleValues ends up only containing one cell, although I'm not certain, or sure how to get around that.
To formalise the question:
How can I return objects which contain Lists based on the those lists matching?

You can implement your own IEqualityComparer<Cell> for your Cell class that states equality when the PossibleValues are equal like this:
public class CellComparer : IEqualityComparer<Cell>
{
public bool Equals(Cell x, Cell y)
{
if (ReferenceEquals(x, null)) return ReferenceEquals(y, null);
if (ReferenceEquals(y, null)) return false;
return x.PossibleValues.SequenceEqual(y.PossibleValues);
}
public int GetHashCode(Cell obj)
{
if (obj == null) return 0;
unchecked
{
int hash = 1;
foreach (int h in obj.PossibleValues.Select(v => v?.GetHashCode() ?? 0))
hash = (hash * 397) ^ h;
return hash;
}
}
}
Then you can use this for a simple LINQ grouping like this:
var cellsGroupedByEqualValues = group.GroupBy(c => c, new CellComparer());
This returns an IEnumerable<IGrouping<Cell,Cell>> and you can iterate through it and receive the number of matching cells:
foreach(var groupedCells in cellsGroupedByEqualValues)
Console.WriteLine(groupedCells.Count());
But these contain duplicates since GroupBy generates a IGrouping for every Cell and adds all matching cells to that. (still trying to find a good way around that)
But for now you can tell for every Cell how many other Cells with the same list of values there are.

You can implement IEqualityComparer interface and use GroupBy method.
Here you can find good GetHashCode for List, and here how to compare a lists.
public class PossibleValuesCellComparer : IEqualityComparer<Cell>
{
public bool Equals(Cell x, Cell y)
{
return Enumerable.SequenceEqual(x.PossibleValues.OrderBy(t => t), y.PossibleValues.OrderBy(t => t));
}
public int GetHashCode(Cell cell)
{
var list = cell.PossibleValues.OrderBy(t => t);
unchecked
{
int hash = 19;
foreach (var obj in list)
{
hash = hash * 31 + obj.GetHashCode();
}
return hash;
}
}
}
....
var g2 = group.GroupBy(x => x, new PossibleValuesCellComparer());

How to Check All Values in Dictionary is same in C#?

I have a Dictionary, I want to write a method to check whether all values are same in this Dictionary.
Dictionary Type:
Dictionary<string, List<string>>
List {1,2,3}`and {2,1,3} are same in my case.
I have done this previously for simple datatype values, but I can not find logic for new requirement, please help me.
For simple values:
MyDict.GroupBy(x => x.Value).Where(x => x.Count() > 1)
I have also written a Generic Method to compare two datatypes in this way.
// 1
// Require that the counts are equal
if (a.Count != b.Count)
{
return false;
}
// 2
// Initialize new Dictionary of the type
Dictionary<T, int> d = new Dictionary<T, int>();
// 3
// Add each key's frequency from collection A to the Dictionary
foreach (T item in a)
{
int c;
if (d.TryGetValue(item, out c))
{
d[item] = c + 1;
}
else
{
d.Add(item, 1);
}
}
// 4
// Add each key's frequency from collection B to the Dictionary
// Return early if we detect a mismatch
foreach (T item in b)
{
int c;
if (d.TryGetValue(item, out c))
{
if (c == 0)
{
return false;
}
else
{
d[item] = c - 1;
}
}
else
{
// Not in dictionary
return false;
}
}
// 5
// Verify that all frequencies are zero
foreach (int v in d.Values)
{
if (v != 0)
{
return false;
}
}
// 6
// We know the collections are equal
return true;

Implement an IEqualityComparer for List<string> that compares two list based on their content. Then just use Distinct on Values and check the count:
dictionary.Values.Distinct(new ListEqualityComparer()).Count() == 1

This should do the trick
var lists = dic.Select(kv => kv.Value.OrderBy(x => x)).ToList();
var first = lists.First();
var areEqual = lists.Skip(1).All(hs => hs.SequenceEqual(first));
You'll need to add some checks to make this work for the empty case.
...or if you want to take #Selman's approach here's an implementation of the IEqualityComparer:
class SequenceComparer<T>:IEqualityComparer<IEnumerable<T>>
{
public bool Equals(IEnumerable<T> left, IEnumerable<T> right)
{
return left.OrderBy(x => x).SequenceEqual(right.OrderBy(x => x));
}
public int GetHashCode(IEnumerable<T> item)
{
//no need to sort because XOR is commutative
return item.Aggregate(0, (acc, val) => val.GetHashCode() ^ acc);
}
}
You could make a variant of this combining the best of both approaches using a HashSet<T> that might be considerably more efficient in the case that you have many candidates to test:
HashSet<IEnumerable<int>> hs = new HashSet<IEnumerable<int>>(new SequenceComparer<int>());
hs.Add(dic.First().Value);
var allEqual = dic.All(kvp => !hs.Add(kvp.Value));
This uses the feature of HashSets that disallows adding more than one item that is considered equal with an item already in the set. We make the HashSet use the custom IEqualityComparer above...
So we insert an arbitrary item from the dictionary before we start, then the moment another item is allowed into the set (i.e. hs.Add(kvp.Value) is true), we can say that there's more than one item in the set and bail out early. .All does this automatically.

Selman22's answer works perfectly - you can also do this for your Dictionary<string, List<string>> without having to implement an IEqualityComparer yourself:
var firstValue = dictionary.Values.First().OrderBy(x => x);
return dictionary.Values.All (x => x.OrderBy(y => y).SequenceEqual(firstValue));
We compare the first value to every other value, and check equality in each case. Note that List<string>.OrderBy(x => x) simply sorts the list of strings alphabetically.

Its not the fastest sdolution, but its works for me:
bool AreEqual = l1.Intersect(l2).ToList().Count() == l1.Count() && l1.Count() == l2.Count();

search List<string> for string .StartsWith()

I have a
List<string>
with 1500 strings. I am now using the following code to pull out only string that start with the string prefixText.
foreach(string a in <MYLIST>)
{
if(a.StartsWith(prefixText, true, null))
{
newlist.Add(a);
}
}
This is pretty fast, but I'm looking for google fast. Now my question is if I arrange the List in alphabetical order, then compare char by char can I make this faster? Or any other suggestions on making this faster?

Thus 1500 is not really a huge number binary search on sorted list would be enough probably.
Nevertheless most efficient algorithms for prefix search are based on the data structure named Trie or Prefix Tree. See: http://en.wikipedia.org/wiki/Trie
Following picture demonstrates the idea very briefly:
For c# implementation see for instance .NET DATA STRUCTURES FOR PREFIX STRING SEARCH AND SUBSTRING (INFIX) SEARCH TO IMPLEMENT AUTO-COMPLETION AND INTELLI-SENSE

You can use PLINQ (Parallel LINQ) to make the execution faster:
var newList = list.AsParallel().Where(x => x.StartsWith(prefixText)).ToList()

If you have the list in alpabetical order, you can use a variation of binary search to make it a lot faster.
As a starting point, this will return the index of one of the strings that match the prefix, so then you can look forward and backward in the list to find the rest:
public static int BinarySearchStartsWith(List<string> words, string prefix, int min, int max) {
while (max >= min) {
int mid = (min + max) / 2;
int comp = String.Compare(words[mid].Substring(0, prefix.Length), prefix);
if (comp < 0) {
min = mid + 1;
} else if (comp > 0) {
max = mid - 1;
} else {
return mid;
}
}
return -1;
}
int index = BinarySearchStartsWith(theList, "pre", 0, theList.Count - 1);
if (index == -1) {
// not found
} else{
// found
}
Note: If you use a prefix that is longer than any of the strings that are compared, it will break, so you might need to figure out how you want to handle that.

So many approches were analyzed to achive minimum data capacity and high performance. The first place is: all prefixes are stored in dictionary: key - prefix, values - items appropriate for prefix.
Here simple implementation of this algorithm:
public class Trie<TItem>
{
#region Constructors
public Trie(
IEnumerable<TItem> items,
Func<TItem, string> keySelector,
IComparer<TItem> comparer)
{
this.KeySelector = keySelector;
this.Comparer = comparer;
this.Items = (from item in items
from i in Enumerable.Range(1, this.KeySelector(item).Length)
let key = this.KeySelector(item).Substring(0, i)
group item by key)
.ToDictionary( group => group.Key, group => group.ToList());
}
#endregion
#region Properties
protected Dictionary<string, List<TItem>> Items { get; set; }
protected Func<TItem, string> KeySelector { get; set; }
protected IComparer<TItem> Comparer { get; set; }
#endregion
#region Methods
public List<TItem> Retrieve(string prefix)
{
return this.Items.ContainsKey(prefix)
? this.Items[prefix]
: new List<TItem>();
}
public void Add(TItem item)
{
var keys = (from i in Enumerable.Range(1, this.KeySelector(item).Length)
let key = this.KeySelector(item).Substring(0, i)
select key).ToList();
keys.ForEach(key =>
{
if (!this.Items.ContainsKey(key))
{
this.Items.Add(key, new List<TItem> { item });
}
else if (this.Items[key].All(x => this.Comparer.Compare(x, item) != 0))
{
this.Items[key].Add(item);
}
});
}
public void Remove(TItem item)
{
this.Items.Keys.ToList().ForEach(key =>
{
if (this.Items[key].Any(x => this.Comparer.Compare(x, item) == 0))
{
this.Items[key].RemoveAll(x => this.Comparer.Compare(x, item) == 0);
if (this.Items[key].Count == 0)
{
this.Items.Remove(key);
}
}
});
}
#endregion
}

1500 is usually too few:
you could search it in parallel with a simple divide and conquer of the problem. Search each half of the list in two (or divide into three, four, ..., parts) different jobs/threads.
Or store the strings in a (not binary) tree instead. Will be O(log n).
sorted in alphabetical order you can do a binary search (sort of the same as the previous one)

You can accelerate a bit by comparing the first character before invoking StartsWith:
char first = prefixText[0];
foreach(string a in <MYLIST>)
{
if (a[0]==first)
{
if(a.StartsWith(prefixText, true, null))
{
newlist.Add(a);
}
}
}

I assume that the really fastest way would be to generate a dictionary with all possible prefixes from your 1500 strings, effectively precomputing the results for all possible searches that will return non-empty. Your search would then be simply a dictionary lookup completing in O(1) time. This is a case of trading memory (and initialization time) for speed.
private IDictionary<string, string[]> prefixedStrings;
public void Construct(IEnumerable<string> strings)
{
this.prefixedStrings =
(
from s in strings
from i in Enumerable.Range(1, s.Length)
let p = s.Substring(0, i)
group s by p
).ToDictionary(
g => g.Key,
g => g.ToArray());
}
public string[] Search(string prefix)
{
string[] result;
if (this.prefixedStrings.TryGetValue(prefix, out result))
return result;
return new string[0];
}

Have you tried implementing a Dictionary and comparing the results? Or, if you do put the entries in alphabetical order, try a binary search.

The question to me is whether or not you'll need to do this one time or multiple times.
If you only find the StartsWithPrefix list one time, you can't get faster then leaving the original list as is and doing myList.Where(s => s.StartsWith(prefix)). This looks at every string one time so it's O(n)
If you need to find the StartsWithPrefix list several times, or maybe you're going to want to add or remove strings to the original list and update the StartsWithPrefix list then you should sort the original list and use binary search. But this will be sort time + search time = O(n log n) + 2 * O(log n)
If you did the binary search method, you would find the indexes of the first occurrence of your prefix and the last occurrence via search. Then do mySortedList.Skip(n).Take(m-n) where n is first index and m is last index.
Edit:
Wait a minute, we're using the wrong tool for the job. Use a Trie! If you put all your strings into a Trie instead of the list, all you have to do is walk down the trie with your prefix and grab all the words underneath that node.

I would go with using Linq:
var query = list.Where(w => w.StartsWith("prefixText")).Select(s => s).ToList();