We Keep Coding

Why is in C# a nested foreach loop more performant than a SelectMany combined with a single foreach loop or a GroupBy? [duplicate]

I am writing a Mesh Rendering manager and thought it would be a good idea to group all of the meshes which use the same shader and then render these while I'm in that shader pass.
I am currently using a foreach loop, but wondered if utilising LINQ might give me a performance increase?

Why should LINQ be faster? It also uses loops internally.
Most of the times, LINQ will be a bit slower because it introduces overhead. Do not use LINQ if you care much about performance. Use LINQ because you want shorter better readable and maintainable code.

LINQ-to-Objects generally is going to add some marginal overheads (multiple iterators, etc). It still has to do the loops, and has delegate invokes, and will generally have to do some extra dereferencing to get at captured variables etc. In most code this will be virtually undetectable, and more than afforded by the simpler to understand code.
With other LINQ providers like LINQ-to-SQL, then since the query can filter at the server it should be much better than a flat foreach, but most likely you wouldn't have done a blanket "select * from foo" anyway, so that isn't necessarily a fair comparison.
Re PLINQ; parallelism may reduce the elapsed time, but the total CPU time will usually increase a little due to the overheads of thread management etc.

LINQ is slower now, but it might get faster at some point. The good thing about LINQ is that you don't have to care about how it works. If a new method is thought up that's incredibly fast, the people at Microsoft can implement it without even telling you and your code would be a lot faster.
More importantly though, LINQ is just much easier to read. That should be enough reason.

It should probably be noted that the for loop is faster than the foreach. So for the original post, if you are worried about performance on a critical component like a renderer, use a for loop.
Reference:
In .NET, which loop runs faster, 'for' or 'foreach'?

You might get a performance boost if you use parallel LINQ for multi cores. See Parallel LINQ (PLINQ) (MSDN).

I was interested in this question, so I did a test just now. Using .NET Framework 4.5.2 on an Intel(R) Core(TM) i3-2328M CPU # 2.20GHz, 2200 Mhz, 2 Core(s) with 8GB ram running Microsoft Windows 7 Ultimate.
It looks like LINQ might be faster than for each loop. Here are the results I got:
Exists = True
Time = 174
Exists = True
Time = 149
It would be interesting if some of you could copy & paste this code in a console app and test as well.
Before testing with an object (Employee) I tried the same test with integers. LINQ was faster there as well.
public class Program
{
public class Employee
{
public int id;
public string name;
public string lastname;
public DateTime dateOfBirth;
public Employee(int id,string name,string lastname,DateTime dateOfBirth)
{
this.id = id;
this.name = name;
this.lastname = lastname;
this.dateOfBirth = dateOfBirth;
}
}
public static void Main() => StartObjTest();
#region object test
public static void StartObjTest()
{
List<Employee> items = new List<Employee>();
for (int i = 0; i < 10000000; i++)
{
items.Add(new Employee(i,"name" + i,"lastname" + i,DateTime.Today));
}
Test3(items, items.Count-100);
Test4(items, items.Count - 100);
Console.Read();
}
public static void Test3(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item.id == idToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test4(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Exists(e => e.id == idToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
#region int test
public static void StartIntTest()
{
List<int> items = new List<int>();
for (int i = 0; i < 10000000; i++)
{
items.Add(i);
}
Test1(items, -100);
Test2(items, -100);
Console.Read();
}
public static void Test1(List<int> items,int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item == itemToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test2(List<int> items, int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Contains(itemToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
}

This is actually quite a complex question. Linq makes certain things very easy to do, that if you implement them yourself, you might stumble over (e.g. linq .Except()). This particularly applies to PLinq, and especially to parallel aggregation as implemented by PLinq.
In general, for identical code, linq will be slower, because of the overhead of delegate invocation.
If, however, you are processing a large array of data, and applying relatively simple calculations to the elements, you will get a huge performance increase if:
You use an array to store the data.
You use a for loop to access each element (as opposed to foreach or linq).
Note: When benchmarking, please everyone remember - if you use the same array/list for two consecutive tests, the CPU cache will make the second one faster. *

Coming in .NET core 7 are some significant updates to LINQ performance of .Min .Max, .Average and .Sum
Reference: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/#linq
Here is a benchmark from the post.
If you compare to a ForEach loop, than it becomes apparent that in .NET 6 the ForEach loop was faster and in .NET 7 the LINQ methods:
this was the code of the benchmark using BenchmarkDotNet
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
public class Program
{
public static void Main()
{
BenchmarkRunner.Run<ForEachVsLinq>();
}
}
[SimpleJob(RuntimeMoniker.Net60)]
[SimpleJob(RuntimeMoniker.Net70)]
[MemoryDiagnoser(false)]
public class ForEachVsLinq
{
private int[] _intArray;
[GlobalSetup]
public void Setup()
{
var random = new Random();
var randomItems = Enumerable.Range(0, 500).Select(_ => random.Next(999));
this._intArray = randomItems.ToArray();
}
[Benchmark]
public void ForEachMin()
{
var min = int.MaxValue;
foreach (var i in this._intArray)
{
if ( i < min)
min = i;
}
Console.WriteLine(min);
}
[Benchmark]
public void Min()
{
var min = this._intArray.Min();
Console.WriteLine(min);
}
[Benchmark]
public void ForEachMax()
{
var max = 0;
foreach (var i in this._intArray)
{
if (i > max)
max = i;
}
Console.WriteLine(max);
}
[Benchmark]
public void Max()
{
var max = this._intArray.Max();
Console.WriteLine(max);
}
[Benchmark]
public void ForEachSum()
{
var sum = 0;
foreach (var i in this._intArray)
{
sum += i;
}
Console.WriteLine(sum);
}
[Benchmark]
public void Sum()
{
var sum = this._intArray.Sum();
Console.WriteLine(sum);
}
}
In .NET Core 6 and earlier versions the mentioned methods are slower than doing your own foreach loop and finding the min, max value, average or summarizing the objects in the array.
But in .NET Core 7, the performance increase makes these buildin LINQ methods actually a lot faster.
Nick Chapsas shows this in a benchmark video on YouTupe
So if you want to calculate the sum, min, max or average value, you should use the LINQ methods instead of a foreach loop from .NET Core 7 onwards (at least, from a performance point of view)

Why not use `dynamic` instead of reflection when the property is known?

This question is similar to this one, but assuming that we know the member name at compile time.
Assuming that we have a class
public class MyClass
{
public string TheProperty { get; set; }
}
and in another method, we want to set the TheProperty member of an instance of that class, but we don't know the type of the instance at compile time, we only know the property name at compile time.
So, as I see it, there are two ways to do that now:
object o = new MyClass(); // For simplicity.
o.GetType().GetProperty("TheProperty").SetValue(o, "bar"); // (1)
((dynamic) o).TheProperty = "bar"; // (2)
I measured this test case using the System.Diagnostics.Stopwatch class to find out that reflection took 475 ticks and the way using dynamic took 0 ticks, therefore being about as fast as a direct call to new MyClass().TheProperty = "bar".
Since I have almost never seen the second way, I am a little confused and my questions now are:
Is there a lapse of thought or anything?
Should the second way be preferred over the first or the other way around? I don't see any disadvantages of using the second way; both (1) and (2) would throw exceptions if the property would not have been found, wouldn't they?
Why does the second way seem to be used so rarely even though seemingly being the faster?

(...)reflection took 475 ticks and the way using dynamic took 0 ticks(...)
That is simply false. The problem is that you are not understanding how dynamic works. I will assume you are correctly setting up the benchmark:
Running in Release mode with optimizations turned on and without the debugger.
You are jitting the methods before actually measuring times.
And here comes the key part you are probably not doing:
Jit the dynamic test without actually performing the dynamic runtime binding.
And why is 3 important? Because the runtime will cache the dynamic call and reuse it! So in a naive benchmark implementation, if you are doing things right, you will incurr the cost of the initial dynamic call jitting the method and therefore you won't measure it.
Run the following benchmark:
public static void Main(string[] args)
{
var repetitions = 1;
var isWarmup = true;
var foo = new Foo();
//warmup
SetPropertyWithDynamic(foo, isWarmup); //JIT method without caching the dynamic call
SetPropertyWithReflection(foo); //JIT method
var s = ((dynamic)"Hello").Substring(0, 2); //Start up the runtime compiler
for (var test = 0; test < 10; test++)
{
Console.WriteLine($"Test #{test}");
var watch = Stopwatch.StartNew();
for (var i = 0; i < repetitions; i++)
{
SetPropertyWithDynamic(foo);
}
watch.Stop();
Console.WriteLine($"Dynamic benchmark: {watch.ElapsedTicks}");
watch = Stopwatch.StartNew();
for (var i = 0; i < repetitions; i++)
{
SetPropertyWithReflection(foo);
}
watch.Stop();
Console.WriteLine($"Reflection benchmark: {watch.ElapsedTicks}");
}
Console.WriteLine(foo);
Console.ReadLine();
}
static void SetPropertyWithDynamic(object o, bool isWarmup = false)
{
if (isWarmup)
return;
((dynamic)o).TheProperty = 1;
}
static void SetPropertyWithReflection(object o)
{
o.GetType().GetProperty("TheProperty").SetValue(o, 1);
}
public class Foo
{
public int TheProperty { get; set; }
public override string ToString() => $"Foo: {TheProperty}";
}
Spot the difference between the first run and the subsequent ones?

How to populate an IEnumerable with consecutive uints

Is this a clean and correct way to generate a list of consecutive uints?
The cast looks kind of ugly, but I'm a beginner...might be there is a method without casting around?
public class Test
{
static readonly IEnumerable<uint> AvailableChannels
= (IEnumerable<uint>)Enumerable.Range(1,1000);
}

static readonly IEnumerable<uint> AvailableChannels
= Enumerable.Range(1,1000)
.Select(i => (uint)i)
.ToList();
It's still a cast though ...
EDIT
The .ToList() is so the full list doesn't need to be recreated every time you loop over it. (OK, a 1000 uints isn't much, but it's the principle of it - if they were classes you would create new ones every time and get unexpected results, like lost changes)
EDIT2
The Cast<uint>() doesn't work at runtime ("Specified cast is not valid"). Changed to a .Select to perform the cast.

You can write your own enumerable method :
public static IEnumerable<uint> Foo(
uint startValue =0,
uint maxValue = uint.MaxValue
)
{
uint index = startValue;
while(index < maxValue) {
yield return index++;
}
}
public static void Main()
{
var myUints = Foo().Take(100);
var myUints2 = Foo(startValue:0, maxValue:1000);
var myUints3 = Foo(0, 1000);
foreach(uint x in myUints) {
Console.WriteLine(x);
}
}
A side note: If performance is a critical point in your application, you may read this question : Why is Enumerable.Range faster than a direct yield loop? (especially the answer marked as the answer)

Is IEnumerable.Any faster than a for loop with a break?

We experienced some slowness in our code opening a form and it was possibly due to a for loop with a break that was taking a long time to execute. I switched this to an IEnumerable.Any() and saw the form open very quickly. I am now trying to figure out if making this change alone increased performance or if it was accessing the ProductIDs property more efficiently. Should this implementation be faster, and if so, why?
Original Implementation:
public bool ContainsProduct(int productID) {
bool containsProduct = false;
for (int i = 0; i < this.ProductIDs.Length; i++) {
if (productID == this.ProductIDs[i]) {
containsProduct = true;
break;
}
}
return containsProduct;
}
New Implementation:
public bool ContainsProduct(int productID) {
return this.ProductIDs.Any(t => productID == t);
}

Call this an educated guess:
this.ProductIDs.Length
This probably is where the slowness lies. If the list of ProductIDs gets retrieved from database (for example) on every iteration in order to get the Length it would indeed be very slow. You can confirm this by profiling your application.
If this is not the case (say ProductIDs is in memory and Length is cached), then both should have an almost identical running time.

First implementation is slightly faster (enumeration is slightly slower than for loop). Second one is a lot more readable.
UPDATE
Oded's answer is possibly correct and well done for spotting it. The first one is slower here since it involves database roundtrip. Otherwise, it is slightly faster as I said.
UPDATE 2 - Proof
Here is a simple code showing why first one is faster:
public static void Main()
{
int[] values = Enumerable.Range(0, 1000000).ToArray();
int dummy = 0;
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
for (int i = 0; i < values.Length; i++)
{
dummy *= i;
}
stopwatch.Stop();
Console.WriteLine("Loop took {0}", stopwatch.ElapsedTicks);
dummy = 0;
stopwatch.Reset();
stopwatch.Start();
foreach (var value in values)
{
dummy *= value;
}
stopwatch.Stop();
Console.WriteLine("Iteration took {0}", stopwatch.ElapsedTicks);
Console.Read();
}
Here is output:
Loop took 12198
Iteration took 20922
So loop is twice is fast as iteration/enumeration.

I think they would be more or less identical. I usually refer to Jon Skeet's Reimplementing LINQ to Objects blog series to get an idea of how the extension methods work. Here's the post for Any() and All()
Here's the core part of Any() implementation from that post
public static bool Any<TSource>(
this IEnumerable<TSource> source,
Func<TSource, bool> predicate)
{
...
foreach (TSource item in source)
{
if (predicate(item))
{
return true;
}
}
return false;
}

This post assumes that ProductIDs is a List<T> or an array. So I'm talking about Linq-to-objects.
Linq is usually slower but shorter/more readable than conventional loop based code. A factor of 2-3 depending on what you're doing is typical.
Can you refactor your code to make this.ProductIDs a HashSet<T>? Or at least sort the array so you can use a binary search. Your problem is that you're performing a linear search, which is slow if there are many products.

I think the below implementation would be a little faster than the corresponding linq implementation, but very minor though
public bool ContainsProduct(int productID) {
var length = this.ProductIDs.Length;
for (int i = 0; i < length; i++) {
if (productID == this.ProductIDs[i]) {
return true;
}
}
return false;
}

The difference will be generally in memory usage then speed.
But generally you should use for loop when you know that you will be using all elements of array in other cases you should try to use while or do while.
I think that this solution use minimum resources
int i = this.ProductIDs.Length - 1;
while(i >= 0) {
if(this.ProductIDs[i--] == productId) {
return true;
}
}
return false;

Is a LINQ statement faster than a 'foreach' loop?

I am writing a Mesh Rendering manager and thought it would be a good idea to group all of the meshes which use the same shader and then render these while I'm in that shader pass.
I am currently using a foreach loop, but wondered if utilising LINQ might give me a performance increase?

Why should LINQ be faster? It also uses loops internally.
Most of the times, LINQ will be a bit slower because it introduces overhead. Do not use LINQ if you care much about performance. Use LINQ because you want shorter better readable and maintainable code.

LINQ-to-Objects generally is going to add some marginal overheads (multiple iterators, etc). It still has to do the loops, and has delegate invokes, and will generally have to do some extra dereferencing to get at captured variables etc. In most code this will be virtually undetectable, and more than afforded by the simpler to understand code.
With other LINQ providers like LINQ-to-SQL, then since the query can filter at the server it should be much better than a flat foreach, but most likely you wouldn't have done a blanket "select * from foo" anyway, so that isn't necessarily a fair comparison.
Re PLINQ; parallelism may reduce the elapsed time, but the total CPU time will usually increase a little due to the overheads of thread management etc.

LINQ is slower now, but it might get faster at some point. The good thing about LINQ is that you don't have to care about how it works. If a new method is thought up that's incredibly fast, the people at Microsoft can implement it without even telling you and your code would be a lot faster.
More importantly though, LINQ is just much easier to read. That should be enough reason.

It should probably be noted that the for loop is faster than the foreach. So for the original post, if you are worried about performance on a critical component like a renderer, use a for loop.
Reference:
In .NET, which loop runs faster, 'for' or 'foreach'?

You might get a performance boost if you use parallel LINQ for multi cores. See Parallel LINQ (PLINQ) (MSDN).

I was interested in this question, so I did a test just now. Using .NET Framework 4.5.2 on an Intel(R) Core(TM) i3-2328M CPU # 2.20GHz, 2200 Mhz, 2 Core(s) with 8GB ram running Microsoft Windows 7 Ultimate.
It looks like LINQ might be faster than for each loop. Here are the results I got:
Exists = True
Time = 174
Exists = True
Time = 149
It would be interesting if some of you could copy & paste this code in a console app and test as well.
Before testing with an object (Employee) I tried the same test with integers. LINQ was faster there as well.
public class Program
{
public class Employee
{
public int id;
public string name;
public string lastname;
public DateTime dateOfBirth;
public Employee(int id,string name,string lastname,DateTime dateOfBirth)
{
this.id = id;
this.name = name;
this.lastname = lastname;
this.dateOfBirth = dateOfBirth;
}
}
public static void Main() => StartObjTest();
#region object test
public static void StartObjTest()
{
List<Employee> items = new List<Employee>();
for (int i = 0; i < 10000000; i++)
{
items.Add(new Employee(i,"name" + i,"lastname" + i,DateTime.Today));
}
Test3(items, items.Count-100);
Test4(items, items.Count - 100);
Console.Read();
}
public static void Test3(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item.id == idToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test4(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Exists(e => e.id == idToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
#region int test
public static void StartIntTest()
{
List<int> items = new List<int>();
for (int i = 0; i < 10000000; i++)
{
items.Add(i);
}
Test1(items, -100);
Test2(items, -100);
Console.Read();
}
public static void Test1(List<int> items,int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item == itemToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test2(List<int> items, int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Contains(itemToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
}

This is actually quite a complex question. Linq makes certain things very easy to do, that if you implement them yourself, you might stumble over (e.g. linq .Except()). This particularly applies to PLinq, and especially to parallel aggregation as implemented by PLinq.
In general, for identical code, linq will be slower, because of the overhead of delegate invocation.
If, however, you are processing a large array of data, and applying relatively simple calculations to the elements, you will get a huge performance increase if:
You use an array to store the data.
You use a for loop to access each element (as opposed to foreach or linq).
Note: When benchmarking, please everyone remember - if you use the same array/list for two consecutive tests, the CPU cache will make the second one faster. *

Coming in .NET core 7 are some significant updates to LINQ performance of .Min .Max, .Average and .Sum
Reference: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/#linq
Here is a benchmark from the post.
If you compare to a ForEach loop, than it becomes apparent that in .NET 6 the ForEach loop was faster and in .NET 7 the LINQ methods:
this was the code of the benchmark using BenchmarkDotNet
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
public class Program
{
public static void Main()
{
BenchmarkRunner.Run<ForEachVsLinq>();
}
}
[SimpleJob(RuntimeMoniker.Net60)]
[SimpleJob(RuntimeMoniker.Net70)]
[MemoryDiagnoser(false)]
public class ForEachVsLinq
{
private int[] _intArray;
[GlobalSetup]
public void Setup()
{
var random = new Random();
var randomItems = Enumerable.Range(0, 500).Select(_ => random.Next(999));
this._intArray = randomItems.ToArray();
}
[Benchmark]
public void ForEachMin()
{
var min = int.MaxValue;
foreach (var i in this._intArray)
{
if ( i < min)
min = i;
}
Console.WriteLine(min);
}
[Benchmark]
public void Min()
{
var min = this._intArray.Min();
Console.WriteLine(min);
}
[Benchmark]
public void ForEachMax()
{
var max = 0;
foreach (var i in this._intArray)
{
if (i > max)
max = i;
}
Console.WriteLine(max);
}
[Benchmark]
public void Max()
{
var max = this._intArray.Max();
Console.WriteLine(max);
}
[Benchmark]
public void ForEachSum()
{
var sum = 0;
foreach (var i in this._intArray)
{
sum += i;
}
Console.WriteLine(sum);
}
[Benchmark]
public void Sum()
{
var sum = this._intArray.Sum();
Console.WriteLine(sum);
}
}
In .NET Core 6 and earlier versions the mentioned methods are slower than doing your own foreach loop and finding the min, max value, average or summarizing the objects in the array.
But in .NET Core 7, the performance increase makes these buildin LINQ methods actually a lot faster.
Nick Chapsas shows this in a benchmark video on YouTupe
So if you want to calculate the sum, min, max or average value, you should use the LINQ methods instead of a foreach loop from .NET Core 7 onwards (at least, from a performance point of view)