We Keep Coding

Create Hashset with a large number of elements (1M)

I have to create a HashSet with the elements from 1 to N+1, where N is a large number (1M).
For example, if N = 5, the HashSet will have then integers {1, 2, 3, 4, 5, 6 }.
The only way I have found is:
HashSet<int> numbers = new HashSet<int>(N);
for (int i = 1; i <= (N + 1) ; i++)
{
numbers.Add(i);
}
Are there another faster (more efficient) ways to do it?

6 is a tiny number of items so I suspect the real problem is adding a few thousand items. The delays in this case are caused by buffer reallocations, not the speed of Add itself.
The solution to this is to specify even an approximate capacity when constructing the HashSet :
var set=new HashSet<int>(1000);
If, and only if, the input implements ICollection<T>, the HashSet<T>(IEnumerable<T>) constructor will check the size of input collection and use it as its capacity:
if (collection is ICollection<T> coll)
{
int count = coll.Count;
if (count > 0)
{
Initialize(count);
}
}
Explanation
Most containers in .NET use buffers internally to store data. This is far faster than implementing containers using pointers, nodes etc due to CPU cache and RAM access delays. Accessing the next item in the CPU's cache is far faster than chasing a pointer in RAM in all CPUs.
The downside is that each time the buffer is full a new one will have to be allocated. Typically, this buffer will have twice the size of the original buffer. Adding items one by one can result in log2(N) reallocations. This works fine for a moderate number of items but can result in a lot of orphaned buffers when adding eg 1000 items one by one. All those temporary buffers will have to be garbage collected at some point, causing additional delays.

Here's the code to test the three options:
var N = 1000000;
var trials = new List<(int method, TimeSpan duration)>();
for (var i = 0; i < 100; i++)
{
var sw = Stopwatch.StartNew();
HashSet<int> numbers1 = new HashSet<int>(Enumerable.Range(1, N + 1));
sw.Stop();
trials.Add((1, sw.Elapsed));
sw = Stopwatch.StartNew();
HashSet<int> numbers2 = new HashSet<int>(N);
for (int n = 1; n < N + 1; n++)
numbers2.Add(n);
sw.Stop();
trials.Add((2, sw.Elapsed));
HashSet<int> numbers3 = new HashSet<int>(N);
foreach (int n in Enumerable.Range(1, N + 1))
numbers3.Add(n);
sw.Stop();
trials.Add((3, sw.Elapsed));
}
for (int j = 1; j <= 3; j++)
Console.WriteLine(trials.Where(x => x.method == j).Average(x => x.duration.TotalMilliseconds));
Typical output is this:
31.314788
16.493208
16.493208
It is nearly twice as fast to preallocate the capacity of the HashSet<int>.
There is no difference between the traditional loop and a LINQ foreach option.

To build on #Enigmativity's answer, here's a proper benchmark using BenchmarkDotNet:
public class Benchmark
{
private const int N = 1000000;
[Benchmark]
public HashSet<int> EnumerableRange() => new HashSet<int>(Enumerable.Range(1, N + 1));
[Benchmark]
public HashSet<int> NoPreallocation()
{
var result = new HashSet<int>();
for (int n = 1; n < N + 1; n++)
{
result.Add(n);
}
return result;
}
[Benchmark]
public HashSet<int> Preallocation()
{
var result = new HashSet<int>(N);
for (int n = 1; n < N + 1; n++)
{
result.Add(n);
}
return result;
}
}
public class Program
{
public static void Main(string[] args)
{
BenchmarkRunner.Run(typeof(Program).Assembly);
}
}
With the results:
Method
Mean
Error
StdDev
EnumerableRange
29.17 ms
0.743 ms
2.179 ms
NoPreallocation
23.96 ms
0.471 ms
0.775 ms
Preallocation
11.68 ms
0.233 ms
0.665 ms
As we can see, using linq is a bit slower than not using linq (as expected), and pre-allocating saves a significant amount of time.

Why do I get an index of -1 for C# array?

I get the following result for the code I added right after the output:
Initializing
2.4
Searching
Index: 55504605
Time: 0.0374
Index: 21891944
Time: 0.0178
Index: 56663763
Time: 0.0425
Index: 37441319
Time: 0.0261
Index: -1
Time: 0.0676
Index: 9344095
Time: 0.0062
namespace ConsoleApplication2
{
class Program
{
static void Main(string[] args)
{
var sw = new Stopwatch();
int[] big = new int[100000000];
Console.WriteLine("Initializing");
sw.Start();
var r = new Random(0);
for (int i=0; i < big.Length; ++i)
{
big[i] = r.Next(big.Length);
}
sw.Stop();
Console.WriteLine(sw.Elapsed.ToString("s\\.f"));
Console.WriteLine();
Console.WriteLine("Searching");
for (int i=0; i<6; ++i)
{
int searchFor = r.Next(big.Length);
sw.Reset();
sw.Start();
int index = Array.IndexOf(big, searchFor);
sw.Stop();
Console.WriteLine("Index: {0}", index);
Console.WriteLine("Time: {0:s\\.ffff}", sw.Elapsed);
}
Console.Read();
}
}
}
I do not understand why I get an index of -1 for the 5th iteration. Isn't the code returning the location of the match within the array? Isn't array numbered from 0 to 100,000,000?
I am not sure if I can ask a follow up question here. The question is about array binary search implemented in the CLR of C# namespace. According to this link if the number is not found, the search returns the complement of the index of an array element. It says if the number is less than one or two elements, the result will the bit wise complement of the index of the first element which is greater. If the value is larger than all the elements, the result will be bit wise complement of the index of the last element of the array. I want to know what good is that result and what is the logic behind returning the bit wise complement of these values?
Below is my results and my modified code:
namespace ConsoleApplication2
{
class Program
{
static void Main(string[] args)
{
var sw = new Stopwatch();
int[] big = new int[100000000];
Console.WriteLine("Initializing");
sw.Start();
var r = new Random(0);
for (int i=0; i < big.Length; ++i)
{
big[i] = r.Next(big.Length);
}
sw.Stop();
Console.WriteLine(sw.Elapsed.ToString("s\\.f"));
Console.WriteLine(sw.ElapsedTicks.ToString("s\\.f"));
Console.WriteLine();
Console.WriteLine("Searching");
for (int i=0; i<6; ++i)
{
int searchFor = r.Next(big.Length);
sw.Reset();
sw.Start();
int index = Array.IndexOf(big, searchFor);
sw.Stop();
Console.WriteLine("Index: {0}", index);
Console.WriteLine("Time: {0:s\\.ffff}", sw.Elapsed);
}
Console.WriteLine();
Console.WriteLine("Sorting");
sw.Reset();
sw.Start();
Array.Sort(big);
sw.Stop();
Console.WriteLine(sw.Elapsed.ToString("s\\.f"));
Console.WriteLine();
Console.WriteLine("Searching (binary)");
for (int i=0; i<6; ++i)
{
int searchFor = r.Next() % big.Length;
sw.Reset();
sw.Start();
int index = Array.BinarySearch(big, searchFor);
sw.Stop();
Console.WriteLine("Index: {0}", index);
Console.WriteLine("Time: {0:s\\.fffffff}", sw.Elapsed);
}
Console.ReadLine();
}
}
}
Index: 55504605
Time: 0.0460
Index: 21891944
Time: 0.0147
Index: 56663763
Time: 0.0377
Index: 37441319
Time: 0.0248
Index: -1
Time: 0.0755
Index: 9344095
Time: 0.0068
Sorting
16.5
Searching (binary)
Index: 8990721
Time: 0.0000844
Index: 4404823
Time: 0.0000046
Index: 52683151
Time: 0.0000059
Index: -37241611
Time: 0.0000238
Index: -49384544
Time: 0.0000021
Index: 88243160
Time: 0.0000064
Just a couple of qualifying statements: 1- The above code is not mine. I am just trying to understand it. 2- If I used any terms incorrectly, please let me know as I am in learning process.

Isn't array numbered from 0 to 100,000,000?
No. You initialize an array of 100000000 items with 100000000 random numbers.
int[] big = new int[100000000];
for (int i=0; i < big.Length; ++i)
{
big[i] = r.Next(big.Length);
}
You then try to find the index of six random numbers:
for (int i=0; i<6; ++i)
{
int searchFor = r.Next(big.Length);
int index = Array.IndexOf(big, searchFor);
However, there's no guarantee that the big array contains all numbers between 0 and 100000000. Random.Next() can return duplicate values, meaning some other numbers from that range are missing.
So there is a chance that r.Next(big.Length) in the second loop returns a number that isn't in the array, hence the return value of -1.
If you actually want to shuffle the numbers from 0-99999999, then instead generate a list containing those numbers, and shuffle it.

Why is my ElapsedMilliseconds always zero here?

So I'm trying to measure the performance of the hash set I created versus the performance of the same elements in a List and in the following block of code
Stopwatch Watch = new Stopwatch();
long tList = 0, tHset = 0; // ms
foreach ( string Str in Copy )
{
// measure time to look up string in ordinary list
Watch.Start();
if ( ListVersion.Contains(Str) ) { }
Watch.Stop();
tList += Watch.ElapsedMilliseconds;
// now measure time to look up same string in my hash set
Watch.Reset();
Watch.Start();
if ( this.Contains(Str) ) { }
Watch.Stop();
tHset += Watch.ElapsedMilliseconds;
Watch.Reset();
}
int n = Copy.Count;
Console.WriteLine("Average milliseconds to look up in List: {0}", tList / n);
Console.WriteLine("Average milliseconds to look up in hashset: {0}", tHset / n);
it is outputing 0 for both. Any idea why this is? Relevant documentation: https://msdn.microsoft.com/en-us/library/system.diagnostics.stopwatch(v=vs.110).aspx

That's because the operation is faster than precision of the Stapwatch.
Instead of measuring each of the Contains call separately measure a group of them:
Stopwatch Watch = new Stopwatch();
long tList = 0, tHset = 0; // ms
// measure time to look up string in ordinary list
Watch.Start();
foreach ( string Str in Copy )
{
if ( ListVersion.Contains(Str) ) { }
}
Watch.Stop();
tList = Watch.ElapsedMilliseconds;
// now measure time to look up same string in my hash set
Watch.Reset();
Watch.Start();
foreach ( string Str in Copy )
{
if ( this.Contains(Str) ) { }
}
Watch.Stop();
tHset = Watch.ElapsedMilliseconds;
Console.WriteLine("Total milliseconds to look up in List: {0}", tList);
Console.WriteLine("Total milliseconds to look up in hashset: {0}", tHset);
As you can see, I also changed the code to print total time spent instead of average. With operations so fast performance is usually presented in Xs per Y operations instead of average. E.g. 40ms per 10 million lookups.
Also, it's possible that in Release mode parts of your code will be optimized away, because it doesn't actually do anything. Consider counting number of elements for which Contains returns true and printing that number out at the end.

You can keep your code as it is and instead of doing :
Watch.ElapsedMilliseconds
You do this :
Watch.Elapsed.TotalMilliseconds
This way you will have the fractional part of the millisecond

C#: Any() vs Count() for an empty list

A question posted earlier got me thinking. Would Any() and Count() perform similarly when used on an empty list?
As explained here, both should go through the same steps of GetEnumerator()/MoveNext()/Dispose().
I tested this out using quick program on LINQPad:
static void Main()
{
var list = new List<int>();
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
for (int i = 0; i < 10000; i++)
list.Any();
stopwatch.Stop();
Console.WriteLine("Time elapsed for Any() : {0}", stopwatch.Elapsed);
stopwatch = new Stopwatch();
stopwatch.Start();
for (int i = 0; i < 10000; i++)
list.Count();
stopwatch.Stop();
Console.WriteLine("Time elapsed for Count(): {0}", stopwatch.Elapsed);
}
And the general result seems to indicate that Count() is faster in this situation. Why is that?
I'm not sure if I got the benchmark right, I would appreciate any correction if not.
Edit: I understand that it would make more sense semantically. The first link I've posted in the question shows a situation where it does make sense to do use Count() directly since the value would be used, hence the question.

The Count() method is optimized for ICollection<T> type, so the pattern GetEnumerator()/MoveNext()/Dispose() is not used.
list.Count();
Is translated to
((ICollection)list).Count;
Whereas the Any() has to build an enumerator.
So the Count() method is faster.
Here a benchmarks for 4 differents IEnumerable instance. The MyEmpty looks like IEnumerable<T> MyEmpty<T>() { yield break; }
iterations : 100000000
Function Any() Count()
new List<int>() 4.310 2.252
Enumerable.Empty<int>() 3.623 6.975
new int[0] 3.960 7.036
MyEmpty<int>() 5.631 7.194
As casperOne said in the comment, Enumerable.Empty<int>() is ICollection<int>, because it is an array, and arrays are not good with the Count() extension because the cast to ICollection<int> is not trivial.
Anyway, for a homemade empty IEnumerable, we can see what we expected, that Count() is slower than Any(), due to the overhead of testing if the IEnumerable is a ICollection.
Complete benchmark:
class Program
{
public const long Iterations = (long)1e8;
static void Main()
{
var results = new Dictionary<string, Tuple<TimeSpan, TimeSpan>>();
results.Add("new List<int>()", Benchmark(new List<int>(), Iterations));
results.Add("Enumerable.Empty<int>()", Benchmark(Enumerable.Empty<int>(), Iterations));
results.Add("new int[0]", Benchmark(new int[0], Iterations));
results.Add("MyEmpty<int>()", Benchmark(MyEmpty<int>(), Iterations));
Console.WriteLine("Function".PadRight(30) + "Any()".PadRight(10) + "Count()");
foreach (var result in results)
{
Console.WriteLine("{0}{1}{2}", result.Key.PadRight(30), Math.Round(result.Value.Item1.TotalSeconds, 3).ToString().PadRight(10), Math.Round(result.Value.Item2.TotalSeconds, 3));
}
Console.ReadLine();
}
public static Tuple<TimeSpan, TimeSpan> Benchmark(IEnumerable<int> source, long iterations)
{
var anyWatch = new Stopwatch();
anyWatch.Start();
for (long i = 0; i < iterations; i++) source.Any();
anyWatch.Stop();
var countWatch = new Stopwatch();
countWatch.Start();
for (long i = 0; i < iterations; i++) source.Count();
countWatch.Stop();
return new Tuple<TimeSpan, TimeSpan>(anyWatch.Elapsed, countWatch.Elapsed);
}
public static IEnumerable<T> MyEmpty<T>() { yield break; }
}

C# FindAll VS Where Speed

Anyone know any speed differences between Where and FindAll on List. I know Where is part of IEnumerable and FindAll is part of List, I'm just curious what's faster.

The FindAll method of the List<T> class actually constructs a new list object, and adds results to it. The Where extension method for IEnumerable<T> will simply iterate over an existing list and yield an enumeration of the matching results without creating or adding anything (other than the enumerator itself.)
Given a small set, the two would likely perform comparably. However, given a larger set, Where should outperform FindAll, as the new List created to contain the results will have to dynamically grow to contain additional results. Memory usage of FindAll will also start to grow exponentially as the number of matching results increases, where as Where should have constant minimal memory usage (in and of itself...excluding whatever you do with the results.)

FindAll is obviously slower than Where, because it needs to create a new list.
Anyway, I think you really should consider Jon Hanna comment - you'll probably need to do some operations on your results and list would be more useful than IEnumerable in many cases.
I wrote small test, just paste it in Console App project. It measures time/ticks of: function execution, operations on results collection(to get perf. of 'real' usage, and to be sure that compiler won't optimize unused data etc. - I'm new to C# and don't know how it works yet,sorry).
Notice: every measured function except WhereIENumerable() creates new List of elements. I might be doing something wrong, but clearly iterating IEnumerable takes much more time than iterating list.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
namespace Tests
{
public class Dummy
{
public int Val;
public Dummy(int val)
{
Val = val;
}
}
public class WhereOrFindAll
{
const int ElCount = 20000000;
const int FilterVal =1000;
const int MaxVal = 2000;
const bool CheckSum = true; // Checks sum of elements in list of resutls
static List<Dummy> list = new List<Dummy>();
public delegate void FuncToTest();
public static long TestTicks(FuncToTest function, string msg)
{
Stopwatch watch = new Stopwatch();
watch.Start();
function();
watch.Stop();
Console.Write("\r\n"+msg + "\t ticks: " + (watch.ElapsedTicks));
return watch.ElapsedTicks;
}
static void Check(List<Dummy> list)
{
if (!CheckSum) return;
Stopwatch watch = new Stopwatch();
watch.Start();
long res=0;
int count = list.Count;
for (int i = 0; i < count; i++) res += list[i].Val;
for (int i = 0; i < count; i++) res -= (long)(list[i].Val * 0.3);
watch.Stop();
Console.Write("\r\n\nCheck sum: " + res.ToString() + "\t iteration ticks: " + watch.ElapsedTicks);
}
static void Check(IEnumerable<Dummy> ieNumerable)
{
if (!CheckSum) return;
Stopwatch watch = new Stopwatch();
watch.Start();
IEnumerator<Dummy> ieNumerator = ieNumerable.GetEnumerator();
long res = 0;
while (ieNumerator.MoveNext()) res += ieNumerator.Current.Val;
ieNumerator=ieNumerable.GetEnumerator();
while (ieNumerator.MoveNext()) res -= (long)(ieNumerator.Current.Val * 0.3);
watch.Stop();
Console.Write("\r\n\nCheck sum: " + res.ToString() + "\t iteration ticks :" + watch.ElapsedTicks);
}
static void Generate()
{
if (list.Count > 0)
return;
var rand = new Random();
for (int i = 0; i < ElCount; i++)
list.Add(new Dummy(rand.Next(MaxVal)));
}
static void For()
{
List<Dummy> resList = new List<Dummy>();
int count = list.Count;
for (int i = 0; i < count; i++)
{
if (list[i].Val < FilterVal)
resList.Add(list[i]);
}
Check(resList);
}
static void Foreach()
{
List<Dummy> resList = new List<Dummy>();
int count = list.Count;
foreach (Dummy dummy in list)
{
if (dummy.Val < FilterVal)
resList.Add(dummy);
}
Check(resList);
}
static void WhereToList()
{
List<Dummy> resList = list.Where(x => x.Val < FilterVal).ToList<Dummy>();
Check(resList);
}
static void WhereIEnumerable()
{
Stopwatch watch = new Stopwatch();
IEnumerable<Dummy> iEnumerable = list.Where(x => x.Val < FilterVal);
Check(iEnumerable);
}
static void FindAll()
{
List<Dummy> resList = list.FindAll(x => x.Val < FilterVal);
Check(resList);
}
public static void Run()
{
Generate();
long[] ticks = { 0, 0, 0, 0, 0 };
for (int i = 0; i < 10; i++)
{
ticks[0] += TestTicks(For, "For \t\t");
ticks[1] += TestTicks(Foreach, "Foreach \t");
ticks[2] += TestTicks(WhereToList, "Where to list \t");
ticks[3] += TestTicks(WhereIEnumerable, "Where Ienum \t");
ticks[4] += TestTicks(FindAll, "FindAll \t");
Console.Write("\r\n---------------");
}
for (int i = 0; i < 5; i++)
Console.Write("\r\n"+ticks[i].ToString());
}
}
class Program
{
static void Main(string[] args)
{
WhereOrFindAll.Run();
Console.Read();
}
}
}
Results(ticks) - CheckSum enabled(some operations on results), mode: release without debugging(CTRL+F5):
- 16,222,276 (for ->list)
- 17,151,121 (foreach -> list)
- 4,741,494 (where ->list)
- 27,122,285 (where ->ienum)
- 18,821,571 (findall ->list)
CheckSum disabled (not using returned list at all):
- 10,885,004 (for ->list)
- 11,221,888 (foreach ->list)
- 18,688,433 (where ->list)
- 1,075 (where ->ienum)
- 13,720,243 (findall ->list)
Your results can be slightly different, to get real results you need more iterations.

UPDATE(from comment): Looking through that code I agree, .Where should have, at worst, equal performance but almost always better.
Original answer:
.FindAll() should be faster, it takes advantage of already knowing the List's size and looping through the internal array with a simple for loop. .Where() has to fire up an enumerator (a sealed framework class called WhereIterator in this case) and do the same job in a less specific way.
Keep in mind though, that .Where() is enumerable, not actively creating a List in memory and filling it. It's more like a stream, so the memory use on something very large can have a significant difference. Also, you could start using the results in a parallel fashion much faster using there .Where() approach in 4.0.

Where is much, much faster than FindAll. No matter how big the list is, Where takes exactly the same amount of time.
Of course Where just creates a query. It doesn't actually do anything, unlike FindAll which does create a list.

The answer from jrista makes senses. However, the new list adds the same objects, thus just growing with reference to existing objects, which should not be that slow.
As long as 3.5 / Linq extension are possible, Where stays better anyway.
FindAll makes much more sense when limited with 2.0