We Keep Coding

'Dictionary Types' In .Net

I've used generic dictionaries in C# a fair bit. Things like:
var example = new Dictionary<int, string> {
{ 0, "Test0" },
{ 1, "Test1" } };
I vaguely remember being told that, before generics came along, you could use a Hashtable(). Basically the same thing, but without a specific type (so value types are going to be boxed, I think).
var example2 = new Hashtable {
{0, "Test0"},
{1, "Test1"} };
And there are questions like this one discussing why we prefer Dictionary over Hashtables (Why is Dictionary preferred over hashtable?).
But what about all the other 'dictionary' types?
SortedDictionary<K,V> - Seems to work like Dictionary but it's .Keys collection is sorted. I'm not sure why you'd care though.
OrderedDictionary is non-generic like a Hashtable, but I can't wrap my head around what's different than a Hashtable. http://msdn.microsoft.com/en-us/library/system.collections.specialized.ordereddictionary.aspx mentions that it's keys are not sorted like a SortedDictionary, so I just plain don't see why or when to use this.
ListDictionary - Smaller/Faster than Hashtable (but is it faster than a generic Dictionary?) when the number of elements is less than 10. Again, I'm at a loss for when you'd use this.
I'm also confused about SortedList<K,V>. When I hear List I don't think key/value pairs (maybe I should?). It implements IDictionary<TKey,TValue>. From this question, I can see that it differs from SortedDictionary in it's performance characteristics (What's the difference between SortedList and SortedDictionary?)
Can Someone Briefly Explain When To Use Which Dictionary Type?
For the sake of simplicity, assume I have access to .Net 4.5 or higher...so maybe there is no situation where Hashtable is useful any more?

Both Dictionary and Hashtable indicate the use of some kind of indexing of the data. Consulting the index takes some time, making it slower at small numbers of elements.
A List does not use an index, and items are typically added at the end. When inserting items, the other items "physically" move to create room for the new element, and when removing items, the other items move to close the gap.
A Dictionary typically does not preserve order, and may contain gaps in memory. When adding items, these gaps may be filled by the new item. Iterating over the Dictionary would then return the items in a different order.
Sorting is a different kind of ordering - it does not preserve the order in which items were added, but follows rules to determine the place of added items.
It's funny that ArrayList became List<T> when the genericalisation happened, and Hashtable became Dictionary<T, U> - both removing the technical aspect from the name, leaving only the name of the abstraction.

Use Dictionary<TKey,TValue>. There's no reason to use the older non-generic hash table.
Ordered Dictionary
If the insertion order of the items in the dictionary matter, then use the OrderedDictionary.
Say I have a mapping of children to their favorite ice cream.
OrderedDictioanry childToIcecream = new OrderedDictionary();
childToIcecream["Jake"] = "Vanilla";
childToIcecream["Kevin"] = "Chocolate";
childToIcecream["Megan"] = "Strawberry";
Each day one child gets an extra scoop in rotation. We could take the day number (Sunday = 0, Monday = 1..) mod it by the number of children, and pull their index from the dictionary to select whose lucky day it is. This of course only works if the dictionary maintains the order. Otherwise I would need a separate List<string> just for maintaining the order. You get key/value pairs and order in one container.
It's unfortunate there's no generic ordered dictionary, but someone posted an implementation here,
Sorted Dictionary
Same for sorted dictionary. If you had a requirement that the key/value pairs needed to be sorted this would save you time to keep it always sorted rather than have to do an expensive sort operating when you needed it to be.
SortedDictionary<char, string> letterToWord = new SortedDictionary<char, string>();
letterToWord['b'] = "bat";
letterToWord['c'] = "cat";
letterToWord['a'] = "apple";
Say you have a dictionary like the above, except the user can build the letter associations at runtime. You always want to display it in alphabetical order, so it makes sense to always keep it sorted as each new item is added.
TLDR; Always use Dictionary<TKey, TValue> unless you have a circumstance that requires it to be ordered or sorted.

What is the quickest way to compare a C# Dictionary to a 'gold standard' Dictionary for equality?

I have a known-good Dictionary, and at run time I need to create a new Dictionary and run a check to see if it has the same key-value pairs as the known-good Dictionary (potentially inserted in different orders), and take one path if it does and another if it doesn't. I don't necessarily need to serialize the entire known-good Dictionary (I could use a hash, for example), but I need some on-disk data that has enough information about the known-good Dictionary to allow for comparison, if not for recreation. What is the quickest way to do this? I can use a SortedDictionary, but the amount of time required to initialize and add values counts in the speed of this task.
Concrete example:
Consider a Dictionary<String,List<String>> that looks something like this (in no particular order, obviously):
{ {"key1", {"value1", "value2"} }, {"key2", {"value3", "value4"} } }
I create that Dictionary once and save some form of information about it on disk (a full serialization, a hash, whatever). Then, at runtime, I do the following:
Dictionary<String,List<String>> d1 = new Dictionary<String,List<String>> ();
Dictionary<String,List<String>> d2 = new Dictionary<String,List<String>> ();
Dictionary<String,List<String>> d3 = new Dictionary<String,List<String>> ();
String key11 = "key1";
String key12 = "key1";
String key13 = "key1";
String key21 = "key2";
String key22 = "key2";
String key23 = "key2";
List<String> value11 = new List<String> {"value1", "value2"};
List<String> value12 = new List<String> {"value1", "value2"};
List<String> value13 = new List<String> {"value1", "value2"};
List<String> value21 = new List<String> {"value3", "value4"};
List<String> value22 = new List<String> {"value3", "value4"};
List<String> value23 = new List<String> {"value3", "value5"};
dict1.add(key11, value11);
dict1.add(key21, value21);
dict2.add(key22, value22);
dict2.add(key12, value12);
dict3.add(key13, value13);
dict3.add(key23, value23);
dict1.compare(fileName); //Should return true
dict2.compare(fileName); //Should return true
dict3.compare(fileName); //Should return false
Again, if the overall time from startup to the return from compare() is quicker, I can change this code to use a SortedDictionary (or anything else) instead, but I can't guarantee ordering and I need some consistent comparison. compare() could load a serialization and iterate through the dictionaries, it could serialize the in-memory dictionary and compare the serialization to the file name, or it could do any number of other things.

Solution one: use set equality.
If the dictionaries are of different sizes, you know they are unequal.
If they are of the same size then build a mutable hash set of keys from one dictionary. Remove from it all the keys from the other dictionary. If you attempted to remove a key that wasn't there, then the key sets are unequal and you know which key was the problem.
Alternatively, build two hash sets and take their intersection; the resulting intersection should be the size of the original sets.
This takes O(n) time and O(n) space.
Once you know that the key sets are equal then go through all the keys one at a time, fetch the values, and do comparison of the values. Since the values are sequences, use SequenceEquals. This takes O(n) time and O(1) space.
Solution two: sort the keys
Again, if the dictionaries are of different size, you know they are unequal.
If they are of the same size, sort both sets of keys and do a SequenceEquals on them; if the sequences of keys are unequal then the dictionaries are unequal.
This takes O(n lg n) time and O(n) space.
If that succeeds, then again, go through the keys one at a time and compare the values.
Solution three:
Again, check the dictionaries to see if they are the same size.
If they are, then iterate over the keys of one dictionary and check to see if the key exists in the other dictionary. If it does not, then they are not equal. If it does, then check the corresponding values for equality.
This is O(n) in time and O(1) in space.
How to choose amongst these possible solutions? It depends on what the likely failure mode is, and whether you need to know what the missing or extra key is. If the likely failure mode is a bad key then it might be more performant to choose a solution that concentrates on finding the bad key first, and only checking for bad values if all the keys turn out to be OK. If the likely failure mode is a bad value, then the third solution is probably best, since it prioritizes checking values early.

Due to my comments on the accepted answer, here's a stricter check.
goodDictionary.Keys.All(k=>
{
List<string> otherVal;
if(!testDictionary.TryGetValue(k,out otherVal))
{
return false;
}
return goodDictionary[k].SequenceEquals(otherVal);
})

If you already have serialisation, then take the hash (I recommend SHA-1) of each serialised dictionary and then compare them.

I don't think there is a magic bullet here; you just need to do a lookup for each key pair:
public bool IsDictionaryAMatch(Dictionary<string, List<string>> dictionaryToCheck)
{
foreach(var kvp in dictionaryToCheck)
{
// Do the Keys Match
if(!goodDictionary.Exists(x => x.Key == kvp.Key))
return false;
foreach(var valueElement in kvp.Value)
{
// Do the Values in each list match
if(!goodDictionary[kvp.Key].Exists(x => x == valueElement))
return false;
}
}
return true;
}

Well, at some point you need to compare that each key has the same value, but before that you can do quick things, like checking to see how many keys each dictionary has, then checking that the list of keys match. Those should be fairly quick, and if either of those tests fail you can abort the more expensive testing.
After that, you might be able to build separate lists of keys and then fire off a Paraells query to compare the actual values.

Datastructures, C#: ~O(1) lookup with range keys?

I have a dataset. This dataset will serve a lookup table. Given a number, I should be able to lookup a corresponding value for that number.
The dataset (let's say its CSV) has a few caveats though. Instead of:
1,ABC
2,XYZ
3,LMN
The numbers are ranges (- being "through", not minus):
1-3,ABC // 1, 2, and 3 = ABC
4-8,XYZ // 4, 5, 6, 7, 8 = XYZ
11-11,LMN // 11 = LMN
All the numbers are signed ints. No ranges overlap with another ranges. There are some gaps; there are ranges that aren't defined in the dataset (like 9 and 10 in the last snippet above).
`
How might I model this dataset in C# so that I have the most-performant lookup while keeping my in-memory footprint low?
The only option I've come up with suffers from overconsumption of memory. Let's say my dataset is:
1-2,ABC
4-6,XYZ
Then I create a Dictionary<int,string>() whose key/values are:
1/ABC
2/ABC
4/XYZ
5/XYZ
6/XYZ
Now I have hash performance-lookup, but tons of wasted space in the hash table.
Any ideas? Maybe just use PLINQ instead and hope for good performance? ;)

If your dictionary is going to truly store a wide range of key values, an approach that expands all possible ranges into explicit keys will rapidly consume more memory than you likely have available.
You're best option is to use a data structure that supports some variation of binary search (or other O(log N) lookup technique). Here's a link to a generic RangeDictionary for .NET that uses an OrderedList internally, and has O(log N) performance.
Achieving constant-time O(1) lookup requires that you expand all ranges into explicit keys. This requires both a lot of memory, and can actually degrade performance when you need to split or insert a new range. This probably isn't what you want.

You can create a doubly-indirected lookup:
Dictionary<int, int> keys;
Dictionary<int, string> values;
Then store the data like this:
keys.Add(1, 1);
keys.Add(2, 1);
keys.Add(3, 1);
//...
keys.Add(11, 3);
values.Add(1, "ABC");
//...
values.Add(3, "LMN");
And then look the data up:
return values[keys[3]]; //returns "ABC"
I'm not sure how much memory footprint this will save with trivial strings, but once you get beyond "ABC" it should help.
EDIT
After Dan Tao's comment below, I went back and checked on what he was asking about. The following code:
var abc = "ABC";
var def = "ABC";
Console.WriteLine(ReferenceEquals(abc, def));
will write "True" to the console. Which means that the either the compiler or the runtime (clarification?) is maintaining the reference to "ABC", and assigns it as the value of both variables.
After reading up some more on Interned strings, if you're using string literals to populate the dictionary, or Interning computed strings, it will in fact take more space to implement my suggestion than the original dictionary would have taken. If you're not using Interned strings, then my solution should take less space.
FINAL EDIT
If you're treating your strings correctly, there should be no excess memory usage from the original Dictionary<int, string> because you can assign them to a variable and then assign that reference as the value (or, if you need to, because you can Intern them)
Just make sure your assignment code includes an intermediate variable assignment:
while (thereAreStringsLeftToAssign)
{
var theString = theStringToAssign;
foreach (var i in range)
{
strings.Add(i, theString);
}
}

As arootbeer has mentioned in his answer, the following code does not create multiple instances of the string "ABC"; rather, it interns a single instance and assigns a reference to that instance to each KeyValuePair<int, string> in dictionary:
var dictionary = new Dictionary<int, string>();
dictionary[0] = "ABC";
dictionary[1] = "ABC";
dictionary[2] = "ABC";
// etc.
OK, so in the case of string literals, you're only using one string instance per range of keys. Is there a scenario where this wouldn't be the case--that is, where you would be using a separate string instance for each key within the range (this is what I assume you're concerned about when you speak of "overconsumption of memory")?
Honestly, I don't think so. There are scenarios where multiple equivalent string instances may be created without the benefit of interning, yes. But I can't imagine these scenarios would affect what you're trying to do here.
My reasoning is this: you want to assign certain values to different ranges of keys, right? So any time you are defining a key-range-value pairing of this sort, you have a single value and several keys. The single part is what leads me to doubt that you'll ever have multiple instances of the same string, unless it is defined as the value for more than one range.
To illustrate: yes, the following code will instantiate two identical strings:
string x = "ABC";
Console.Write("Type 'ABC' and press Enter: ");
string y = Console.ReadLine();
Console.WriteLine(Equals(x, y));
Console.WriteLine(ReferenceEquals(x, y));
The above program, assuming the user follows instructions and types "ABC," outputs True, then False. So you might think, "Ah, so when a string is only provided at run-time, it isn't interned! So this could be where my values could be duplicated!"
But... again: I don't think so. It all comes back to the fact that you are going to be assigning a single value to a range of keys. So let's say your values come from user input; then your code would look something like this:
var dictionary = new Dictionary<int, string>();
int start, count;
GetRange(out start, out count);
string value = GetValue();
foreach (int key in Enumerable.Range(start, count))
{
// Look, you're using the same string instance to assign
// to each key... how could it be otherwise?
dictionary[key] = value;
}
Now, if you were actually thinking more along the lines of what LBushkin mentions in his answer--that you may potentially have huge ranges, making it impractical to define a KeyValuePair<int, string> for each key within that range (e.g., if you have a range of 1-1000000)--then I would agree that you're best off with some sort of data structure that bases its lookup on a binary search. If that's more your scenario, say so and I will be happy to offer more ideas on that front. (Or you could just take a look at the link LBushkin already posted.)

Use a balanced ordered tree (or something similar) mapping start-of-range to end-of-range and data. This will be easy to implement for non-overlapping ranges.

arootbeer has a good solution, but one you may find confusing to work with.
Another choice is to use a reference type instead of a string, so that you point to the same reference
class StringContainer {
public string Value { get; set; }
}
Dictionary<int, StringContainer> values;
var value1 = new StringContainer { Value = "ABC" };
values.Add(1, value1);
values.Add(2, value1);
They will both point to the same instance of StringContainer
EDIT: Thanks for the comments everyone. This method handles value types other than string, so it might be useful for more than the given example. Also, it is my understanding that strings don't always behave in the manner you would expect from reference values, but I could be wrong.

Random Access by Position of Key/Value Pairs in .NET (C#)

I am currently developing a program that uses C#'s Dictionary container (specifically, SortedDictionary). This container works very well for my purposes except for one specific case because I want random access. Specifically, I am generating a random position using a pseudorandom number generator and I need to be able to access that value in the SortedDictionary. At the point that this happens, I do not have a key value.
I could potentially switch to a List which would solve this problem, but would create problems in the rest of the algorithm where SortedDictionary works quite well. Any suggestions/solutions would be much appreciated.
I am currently developing Visual Studio 2005.
Thank you.

You can use a SortedList and it has a Values collection which you may access through an integer index.

public TValue GetRandomElement<TKey, TValue>(SortedDictionary<TKey, TValue> dict)
{
Random randGen = new Random();
int randIndex = randGen.Next(dict.Values.Count);
int i = 0;
foreach (TValue value in dict.Values)
{
if (i++ == randIndex)
return value;
}
// this shouldn't happen unless I have a bug above or you are accessing the dictionary from multiple threads
return default(TValue);
}
Blindly enumerating the ValueCollection is not the most efficient thing in the world. But it gets the job done. If this is a frequent operation in your scenario, you should consider a hybrid data structure that has the performance characteristics needed for both dictionary lookup and random access.

Linq could do this for you:
int n = GetRandomIndex();
object item = dictionary.ElementAt(n).Value;

You don't provide enough information to come up with a solution. How many elements, how often are you going to do this, do you have memory/speed constraints? BTree, SortedList, inserting special nodes in the SortedDictionary could all be useful

Will pulling a random key work?
var randValue = myDictionary.Values.ToList()[myRandomInt];
Edit:
Seems the keys collection and values collection are both IEnumerables so you can't use [] operators. This is the best it gets it seems.
Edit:
Without Linq... Perhaps expensive, but you could copyto array and then pull a value at an index
System.Collections.Generic.KeyValuePair<string, int>[] dictCopy = new System.Collections.Generic.KeyValuePair<string, int>[myDictionary.Count];
myDictionary.CopyTo(dictCopy, 0);
var randValue = dictCopy[myRandomInt].Value;

Performance when checking for duplicates

I've been working on a project where I need to iterate through a collection of data and remove entries where the "primary key" is duplicated. I have tried using a
List<int>
and
Dictionary<int, bool>
With the dictionary I found slightly better performance, even though I never need the Boolean tagged with each entry. My expectation is that this is because a List allows for indexed access and a Dictionary does not. What I was wondering is, is there a better solution to this problem. I do not need to access the entries again, I only need to track what "primary keys" I have seen and make sure I only perform addition work on entries that have a new primary key. I'm using C# and .NET 2.0. And I have no control over fixing the input data to remove the duplicates from the source (unfortunately!). And so you can have a feel for scaling, overall I'm checking for duplicates about 1,000,000 times in the application, but in subsets of no more than about 64,000 that need to be unique.

They have added the HashSet class in .NET 3.5. But I guess it will be on par with the Dictionary. If you have less than say a 100 elements a List will probably perform better.

Edit: Nevermind my comment. I thought you're talking about C++. I have no idea if my post is relevant in the C# world..
A hash-table could be a tad faster. Binary trees (that's what used in the dictionary) tend to be relative slow because of the way the memory gets accessed. This is especially true if your tree becomes very large.
However, before you change your data-structure, have you tried to use a custom pool allocator for your dictionary? I bet the time is not spent traversing the tree itself but in the millions of allocations and deallocations the dictionary will do for you.
You may see a factor 10 speed-boost just plugging a simple pool allocator into the dictionary template. Afaik boost has a component that can be directly used.
Another option: If you know only 64.000 entries in your integers exist you can write those to a file and create a perfect hash function for it. That way you can just use the hash function to map your integers into the 0 to 64.000 range and index a bit-array.
Probably the fastest way, but less flexible. You have to redo your perfect hash function (can be done automatically) each time your set of integers changes.

I don't really get what you are asking.
Firstly is just the opposite of what you say. The dictionary has indexed access (is a hash table) while de List hasn't.
If you already have the data in a dictionary then all keys are unique, there can be no duplicates.
I susspect you have the data stored in another data type and you're storing it into the dictionary. If that's the case the inserting the data will work with two dictionarys.
foreach (int key in keys)
{
if (!MyDataDict.ContainsKey(key))
{
if (!MyDuplicatesDict.ContainsKey(key))
MyDuplicatesDict.Add(key);
}
else
MyDataDict.Add(key);
}

If you are checking for uniqueness of integers, and the range of integers is constrained enough then you could just use an array.
For better packing you could implement a bitmap data structure (basically an array, but each int in the array represents 32 ints in the key space by using 1 bit per key). That way if you maximum number is 1,000,000 you only need ~30.5KB of memory for the data structure.
Performs of a bitmap would be O(1) (per check) which is hard to beat.

There was a question awhile back on removing duplicates from an array. For the purpose of the question performance wasn't much of a consideration, but you might want to take a look at the answers as they might give you some ideas. Also, I might be off base here, but if you are trying to remove duplicates from the array then a LINQ command like Enumerable.Distinct might give you better performance than something that you write yourself. As it turns out there is a way to get LINQ working on .NET 2.0 so this might be a route worth investigating.

If you're going to use a List, use the BinarySearch:
// initailize to a size if you know your set size
List<int> FoundKeys = new List<int>( 64000 );
Dictionary<int,int> FoundDuplicates = new Dictionary<int,int>();
foreach ( int Key in MyKeys )
{
// this is an O(log N) operation
int index = FoundKeys.BinarySearch( Key );
if ( index < 0 )
{
// if the Key is not in our list,
// index is the two's compliment of the next value that is in the list
// i.e. the position it should occupy, and we maintain sorted-ness!
FoundKeys.Insert( ~index, Key );
}
else
{
if ( DuplicateKeys.ContainsKey( Key ) )
{
DuplicateKeys[Key]++;
}
else
{
DuplicateKeys.Add( Key, 1 );
}
}
}
You can also use this for any type for which you can define an IComparer by using an overload: BinarySearch( T item, IComparer< T > );