We Keep Coding

C# LINQ performance when extension method called inside where clause

I have a LINQ query like this
public static bool CheckIdExists(int searchId)
{
return itemCollection.Any(item => item.Id.Equals(searchId.ConvertToString()));
}
item.Id is a string while searchId is an int. .ConvertToString() is an extension which which converts int to string
Code for ConvertToString:
public static string ConvertToString(this object input)
{
return Convert.ToString(input, CultureInfo.InvariantCulture);
}
Now my query is, does searchId.ConvertToString() gets executed for each item in itemCollection?
Is computing searchId.ConvertToString() beforehand and calling the method like below improves performance?
public static bool CheckIdExists(int searchId)
{
string sId=searchId.ConvertToString();
return itemCollection.Any(item => item.Id.Equals(sId));
}
How to debug these two scenarios and observe their performances?

I re-generated the scenarios you talked about in your question. I tried following code and got this output.
But this is how you can debug this.
static List<string> itemCollection = new List<string>();
static void Main(string[] args)
{
for (int i = 0; i < 10000000; i++)
{
itemCollection.Add(i.ToString());
}
var watch = new Stopwatch();
watch.Start();
Console.WriteLine(CheckIdExists(580748));
watch.Stop();
Console.WriteLine($"Took {watch.ElapsedMilliseconds}");
var watch1 = new Stopwatch();
watch1.Start();
Console.WriteLine(CheckIdExists1(580748));
watch1.Stop();
Console.WriteLine($"Took {watch1.ElapsedMilliseconds}");
Console.ReadLine();
}
public static bool CheckIdExists(int searchId)
{
return itemCollection.Any(item => item.Equals(ConvertToString(searchId)));
}
public static bool CheckIdExists1(int searchId)
{
string sId =ConvertToString(searchId);
return itemCollection.Any(item => item.Equals(sId));
}
public static string ConvertToString(int input)
{
return Convert.ToString(input, CultureInfo.InvariantCulture);
}
OUTPUT:
True
Took 170
True
Took 11

How long it takes is the ultimate guide. You can create a stopwatch to log the performance of any code. Just use the ElapsedMilliseconds to see how long has been taken. For very short operations I suggest using very long loops to get a more accurate length of time.
var watch = new Stopwatch();
watch.Start();
/// CODE HERE (IDEALLY IN A LONG LOOP)
Debub.WriteLine($"Took {watch.ElapsedMilliseconds}");

Yes, it should be faster to get the string once. But I guess that compiler does optimize that thing for you (I just suspect this, don't ave anything to back it up. I just remeber that compilers are very good at detecting things that are not changing).
And no, it's not computed for every item, since LINQ method Any does not necessarily check all items. It return true for the first matching item. The only scenario when it checks all items, is where for none the lambda returns true.
If you want to test the speed difference,make sure to have more data - otherwise the difference may be too small.
Just do:
itemCollection = Enumerable.Range(0, 1000).SelectMany(x => itemCollection).ToList() // or array or whatever the type of collection you have
Than measure the times with StopWatch, just like #RobSedgwick said

I think you have two solution:
1- make log and meke inside this log datetime.now
2- you can use the diagnostic Tools tab
hopefully this help you

Why not use `dynamic` instead of reflection when the property is known?

This question is similar to this one, but assuming that we know the member name at compile time.
Assuming that we have a class
public class MyClass
{
public string TheProperty { get; set; }
}
and in another method, we want to set the TheProperty member of an instance of that class, but we don't know the type of the instance at compile time, we only know the property name at compile time.
So, as I see it, there are two ways to do that now:
object o = new MyClass(); // For simplicity.
o.GetType().GetProperty("TheProperty").SetValue(o, "bar"); // (1)
((dynamic) o).TheProperty = "bar"; // (2)
I measured this test case using the System.Diagnostics.Stopwatch class to find out that reflection took 475 ticks and the way using dynamic took 0 ticks, therefore being about as fast as a direct call to new MyClass().TheProperty = "bar".
Since I have almost never seen the second way, I am a little confused and my questions now are:
Is there a lapse of thought or anything?
Should the second way be preferred over the first or the other way around? I don't see any disadvantages of using the second way; both (1) and (2) would throw exceptions if the property would not have been found, wouldn't they?
Why does the second way seem to be used so rarely even though seemingly being the faster?

(...)reflection took 475 ticks and the way using dynamic took 0 ticks(...)
That is simply false. The problem is that you are not understanding how dynamic works. I will assume you are correctly setting up the benchmark:
Running in Release mode with optimizations turned on and without the debugger.
You are jitting the methods before actually measuring times.
And here comes the key part you are probably not doing:
Jit the dynamic test without actually performing the dynamic runtime binding.
And why is 3 important? Because the runtime will cache the dynamic call and reuse it! So in a naive benchmark implementation, if you are doing things right, you will incurr the cost of the initial dynamic call jitting the method and therefore you won't measure it.
Run the following benchmark:
public static void Main(string[] args)
{
var repetitions = 1;
var isWarmup = true;
var foo = new Foo();
//warmup
SetPropertyWithDynamic(foo, isWarmup); //JIT method without caching the dynamic call
SetPropertyWithReflection(foo); //JIT method
var s = ((dynamic)"Hello").Substring(0, 2); //Start up the runtime compiler
for (var test = 0; test < 10; test++)
{
Console.WriteLine($"Test #{test}");
var watch = Stopwatch.StartNew();
for (var i = 0; i < repetitions; i++)
{
SetPropertyWithDynamic(foo);
}
watch.Stop();
Console.WriteLine($"Dynamic benchmark: {watch.ElapsedTicks}");
watch = Stopwatch.StartNew();
for (var i = 0; i < repetitions; i++)
{
SetPropertyWithReflection(foo);
}
watch.Stop();
Console.WriteLine($"Reflection benchmark: {watch.ElapsedTicks}");
}
Console.WriteLine(foo);
Console.ReadLine();
}
static void SetPropertyWithDynamic(object o, bool isWarmup = false)
{
if (isWarmup)
return;
((dynamic)o).TheProperty = 1;
}
static void SetPropertyWithReflection(object o)
{
o.GetType().GetProperty("TheProperty").SetValue(o, 1);
}
public class Foo
{
public int TheProperty { get; set; }
public override string ToString() => $"Foo: {TheProperty}";
}
Spot the difference between the first run and the subsequent ones?

ThreadLocal performance vs using parameters

I am currently implementing a runtime (i.e. a collection of functions) for a formulas language. Some formulas need a context to be passed to them and I created a class called EvaluationContext which contains all properties I need access to at runtime.
Using ThreadLocal<EvaluationContext> seems like a good option to make this context available to the runtime functions. The other option is to pass the context as a parameter to the functions that need it.
I prefer using ThreadLocal but I was wondering if there is any performance penalty as opposed to passing the evaluation context via method parameters.

I created the program below and it is faster to use parameters rather than the ThreadLocal field.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace TestThreadLocal
{
internal class Program
{
public class EvaluationContext
{
public int A { get; set; }
public int B { get; set; }
}
public static class FormulasRunTime
{
public static ThreadLocal<EvaluationContext> Context = new ThreadLocal<EvaluationContext>();
public static int SomeFunction()
{
EvaluationContext ctx = Context.Value;
return ctx.A + ctx.B;
}
public static int SomeFunction(EvaluationContext context)
{
return context.A + context.B;
}
}
private static void Main(string[] args)
{
Stopwatch stopwatch = Stopwatch.StartNew();
int N = 10000;
Task<int>[] tasks = new Task<int>[N];
int sum = 0;
for (int i = 0; i < N; i++)
{
int x = i;
tasks[i] = Task.Factory.StartNew(() =>
{
//Console.WriteLine("Starting {0}, thread {1}", x, Thread.CurrentThread.ManagedThreadId);
FormulasRunTime.Context.Value = new EvaluationContext {A = 0, B = x};
return FormulasRunTime.SomeFunction();
});
sum += i;
}
Task.WaitAll(tasks);
Console.WriteLine("Using ThreadLocal: It took {0} millisecs and the sum is {1}", stopwatch.ElapsedMilliseconds, tasks.Sum(t => t.Result));
Console.WriteLine(sum);
stopwatch = Stopwatch.StartNew();
for (int i = 0; i < N; i++)
{
int x = i;
tasks[i] = Task.Factory.StartNew(() =>
{
return FormulasRunTime.SomeFunction(new EvaluationContext { A = 0, B = x });
});
}
Task.WaitAll(tasks);
Console.WriteLine("Using parameter: It took {0} millisecs and the sum is {1}", stopwatch.ElapsedMilliseconds, tasks.Sum(t => t.Result));
Console.ReadKey();
}
}
}

Going on costa's answer;
If you try N as 10000000,
int N = 10000000;
you will see there is not much of a difference (around 107.4 to 103.4 seconds).
If the value gets bigger the difference becomes smaller.
So, if you do not mind a three seconds slowness, i think it is the difference between the usability and the taste.
PS: In the code, int return types must be converted to long.

I consider the ThreadLocal design to be dirty, yet creative. It is definitely going to be faster to use parameters but performance should not be your only concern. Parameters will be much clearer to understand. I recommend you go with parameters.

There will not be any performance impact, but you will not be able to do any parallel computations in this case (which can be quite useful especially in formulas domain).
If you definitely don't want to do it you can go for ThreadLocal.
Otherwise I would suggest you look at the "state monad" "pattern" that will allow you to seamlessly pass your state (context) through your computations (formulas) without having any explicit parameters.

I think you'll find that in a head-to-head comparison, accessing a ThreadLocal<> takes substantially longer than accessing a parameter, but in the end it might not be a significant difference - it all depends what else you're doing.

Compiled C# lambda expression performance with imbrication

Considering this class:
/// <summary>
/// Dummy implementation of a parser for the purpose of the test
/// </summary>
class Parser
{
public List<T> ReadList<T>(Func<T> readFunctor)
{
return Enumerable.Range(0, 10).Select(i => readFunctor()).ToList();
}
public int ReadInt32()
{
return 12;
}
public string ReadString()
{
return "string";
}
}
I try to generate the following call with a compiled lambda expression tree:
Parser parser = new Parser();
List<int> list = parser.ReadList(parser.ReadInt32);
However, the peformance is not quite the same...
class Program
{
private const int MAX = 1000000;
static void Main(string[] args)
{
DirectCall();
LambdaCall();
CompiledLambdaCall();
}
static void DirectCall()
{
Parser parser = new Parser();
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < MAX; i++)
{
List<int> list = parser.ReadList(parser.ReadInt32);
}
sw.Stop();
Console.WriteLine("Direct call: {0} ms", sw.ElapsedMilliseconds);
}
static void LambdaCall()
{
Parser parser = new Parser();
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < MAX; i++)
{
List<int> list = parser.ReadList(() => parser.ReadInt32());
}
sw.Stop();
Console.WriteLine("Lambda call: {0} ms", sw.ElapsedMilliseconds);
}
static void CompiledLambdaCall()
{
var parserParameter = Expression.Parameter(typeof(Parser), "parser");
var lambda = Expression.Lambda<Func<Parser, List<int>>>(
Expression.Call(
parserParameter,
typeof(Parser).GetMethod("ReadList").MakeGenericMethod(typeof(int)),
Expression.Lambda(
typeof(Func<int>),
Expression.Call(
parserParameter,
typeof(Parser).GetMethod("ReadInt32")))),
parserParameter);
Func<Parser, List<int>> func = lambda.Compile();
Parser parser = new Parser();
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < MAX; i++)
{
List<int> list = func(parser);
}
sw.Stop();
Console.WriteLine("Compiled lambda call: {0} ms", sw.ElapsedMilliseconds);
}
}
These are the results in milliseconds on my computer :
Direct call: 647 ms
Lambda call: 641 ms
Compiled lambda call: 5861 ms
I don't understand why the compiled lambda call is so slow.
And I forgot to say that my test is run in release mode with the "Optimize Code" option enabled.
Update: Changed benchmarking based on DateTime.Now to Stopwatch.
Does anyone know how to tweak the lambda expression to obtain a better performance in the compiled lambda call ?

The test is invalid for two reasons:
DateTime.Now isn't accurate enough for micro-benchmarking short tests.
Use the Stopwatch class instead. When I do so, I get the following results (using MAX = 100000), in milliseconds:
Lambda call: 86.3196
Direct call: 74.057
Compiled lambda call: 814.2178
Indeed, the "direct call" is faster than the "lambda call", which makes sense - the "direct call" involves calls to a delegate that refers directly to a method on a Parser object. The "lambda call" requires a call to a delegate that refers to a method on a compiler-generated closure object, which in turn calls the method on the Parser object. This extra indirection introduces a minor speed-bump.
The "Compiled lambda call" isn't the same as the "Lambda call"
The "Lambda" looks like this:
() => parser.ReadInt32()
whereas the "Compiled lambda" looks like this:
parser => parser.ReadList(() => parser.ReadInt32())
There's an extra step in there: To create the embedded delegate for the inner lambda. In a tight loop, this is expensive.
EDIT:
I went ahead and inspected the IL of the "lambda" vs the "compiled lambda" and decompiled them back to "simpler" C# (see: Viewing the IL code generated from a compiled expression).
For the "non compiled" lambda, it looks like this:
for (int i = 0; i < 100000; i++)
{
if (CS$<>9__CachedAnonymousMethodDelegate1 == null)
{
CS$<>9__CachedAnonymousMethodDelegate1 = new Func<int>(CS$<>8__locals3.<LambdaCall>b__0);
}
CS$<>8__locals3.parser.ReadList<int>(CS$<>9__CachedAnonymousMethodDelegate1);
}
Note that a single delegate is created once and cached.
Whereas for the "compiled lambda", it looks like this:
Func<Parser, List<int>> func = lambda.Compile();
Parser parser = new Parser();
for (int i = 0; i < 100000; i++)
{
func(parser);
}
Where the target of the delegate is:
public static List<int> Foo(Parser parser)
{
object[] objArray = new object[] { new StrongBox<Parser>(parser) };
return ((StrongBox<Parser>) objArray[0]).Value.ReadList<int>
(new Func<int>(dyn_type.<ExpressionCompilerImplementationDetails>{1}lambda_method));
}
Note that although the "outer" delegate is created only once and cached, a new "inner" delegate is created on every iteration of the loop. Not to mention other allocations for the object array and the StrongBox<T> instance.

The primary reason the compiled lambda is slower is because the delegate is created over and over again. Anonymous delegates are a special breed: they are only used in one location. So the compiler can do some special optimizations, like caching the value the first time the delegate is called. This is what is happening here.
I was not able to reproduce the large difference between the direct call and the lambda call. In fact, in my measurements the direct call is slightly faster.
When doing benchmarks like this, you may want to use a more accurate timer. The Stopwatch class in System.Diagnostics is ideal. You may also want to increase your number of iterations. The code as is only runs for a few milliseconds.
Also, the first of the three cases will incur a slight overhead from JIT'ing the Parser class. Try running the first case twice and see what happens. Or better still: use the number of iterations as a parameter in each method, and call each method first for 1 iteration, so they all start on a level playing field.

Is a LINQ statement faster than a 'foreach' loop?

I am writing a Mesh Rendering manager and thought it would be a good idea to group all of the meshes which use the same shader and then render these while I'm in that shader pass.
I am currently using a foreach loop, but wondered if utilising LINQ might give me a performance increase?

Why should LINQ be faster? It also uses loops internally.
Most of the times, LINQ will be a bit slower because it introduces overhead. Do not use LINQ if you care much about performance. Use LINQ because you want shorter better readable and maintainable code.

LINQ-to-Objects generally is going to add some marginal overheads (multiple iterators, etc). It still has to do the loops, and has delegate invokes, and will generally have to do some extra dereferencing to get at captured variables etc. In most code this will be virtually undetectable, and more than afforded by the simpler to understand code.
With other LINQ providers like LINQ-to-SQL, then since the query can filter at the server it should be much better than a flat foreach, but most likely you wouldn't have done a blanket "select * from foo" anyway, so that isn't necessarily a fair comparison.
Re PLINQ; parallelism may reduce the elapsed time, but the total CPU time will usually increase a little due to the overheads of thread management etc.

LINQ is slower now, but it might get faster at some point. The good thing about LINQ is that you don't have to care about how it works. If a new method is thought up that's incredibly fast, the people at Microsoft can implement it without even telling you and your code would be a lot faster.
More importantly though, LINQ is just much easier to read. That should be enough reason.

It should probably be noted that the for loop is faster than the foreach. So for the original post, if you are worried about performance on a critical component like a renderer, use a for loop.
Reference:
In .NET, which loop runs faster, 'for' or 'foreach'?

You might get a performance boost if you use parallel LINQ for multi cores. See Parallel LINQ (PLINQ) (MSDN).

I was interested in this question, so I did a test just now. Using .NET Framework 4.5.2 on an Intel(R) Core(TM) i3-2328M CPU # 2.20GHz, 2200 Mhz, 2 Core(s) with 8GB ram running Microsoft Windows 7 Ultimate.
It looks like LINQ might be faster than for each loop. Here are the results I got:
Exists = True
Time = 174
Exists = True
Time = 149
It would be interesting if some of you could copy & paste this code in a console app and test as well.
Before testing with an object (Employee) I tried the same test with integers. LINQ was faster there as well.
public class Program
{
public class Employee
{
public int id;
public string name;
public string lastname;
public DateTime dateOfBirth;
public Employee(int id,string name,string lastname,DateTime dateOfBirth)
{
this.id = id;
this.name = name;
this.lastname = lastname;
this.dateOfBirth = dateOfBirth;
}
}
public static void Main() => StartObjTest();
#region object test
public static void StartObjTest()
{
List<Employee> items = new List<Employee>();
for (int i = 0; i < 10000000; i++)
{
items.Add(new Employee(i,"name" + i,"lastname" + i,DateTime.Today));
}
Test3(items, items.Count-100);
Test4(items, items.Count - 100);
Console.Read();
}
public static void Test3(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item.id == idToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test4(List<Employee> items, int idToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Exists(e => e.id == idToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
#region int test
public static void StartIntTest()
{
List<int> items = new List<int>();
for (int i = 0; i < 10000000; i++)
{
items.Add(i);
}
Test1(items, -100);
Test2(items, -100);
Console.Read();
}
public static void Test1(List<int> items,int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = false;
foreach (var item in items)
{
if (item == itemToCheck)
{
exists = true;
break;
}
}
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
public static void Test2(List<int> items, int itemToCheck)
{
Stopwatch s = new Stopwatch();
s.Start();
bool exists = items.Contains(itemToCheck);
Console.WriteLine("Exists=" + exists);
Console.WriteLine("Time=" + s.ElapsedMilliseconds);
}
#endregion
}

This is actually quite a complex question. Linq makes certain things very easy to do, that if you implement them yourself, you might stumble over (e.g. linq .Except()). This particularly applies to PLinq, and especially to parallel aggregation as implemented by PLinq.
In general, for identical code, linq will be slower, because of the overhead of delegate invocation.
If, however, you are processing a large array of data, and applying relatively simple calculations to the elements, you will get a huge performance increase if:
You use an array to store the data.
You use a for loop to access each element (as opposed to foreach or linq).
Note: When benchmarking, please everyone remember - if you use the same array/list for two consecutive tests, the CPU cache will make the second one faster. *

Coming in .NET core 7 are some significant updates to LINQ performance of .Min .Max, .Average and .Sum
Reference: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/#linq
Here is a benchmark from the post.
If you compare to a ForEach loop, than it becomes apparent that in .NET 6 the ForEach loop was faster and in .NET 7 the LINQ methods:
this was the code of the benchmark using BenchmarkDotNet
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
public class Program
{
public static void Main()
{
BenchmarkRunner.Run<ForEachVsLinq>();
}
}
[SimpleJob(RuntimeMoniker.Net60)]
[SimpleJob(RuntimeMoniker.Net70)]
[MemoryDiagnoser(false)]
public class ForEachVsLinq
{
private int[] _intArray;
[GlobalSetup]
public void Setup()
{
var random = new Random();
var randomItems = Enumerable.Range(0, 500).Select(_ => random.Next(999));
this._intArray = randomItems.ToArray();
}
[Benchmark]
public void ForEachMin()
{
var min = int.MaxValue;
foreach (var i in this._intArray)
{
if ( i < min)
min = i;
}
Console.WriteLine(min);
}
[Benchmark]
public void Min()
{
var min = this._intArray.Min();
Console.WriteLine(min);
}
[Benchmark]
public void ForEachMax()
{
var max = 0;
foreach (var i in this._intArray)
{
if (i > max)
max = i;
}
Console.WriteLine(max);
}
[Benchmark]
public void Max()
{
var max = this._intArray.Max();
Console.WriteLine(max);
}
[Benchmark]
public void ForEachSum()
{
var sum = 0;
foreach (var i in this._intArray)
{
sum += i;
}
Console.WriteLine(sum);
}
[Benchmark]
public void Sum()
{
var sum = this._intArray.Sum();
Console.WriteLine(sum);
}
}
In .NET Core 6 and earlier versions the mentioned methods are slower than doing your own foreach loop and finding the min, max value, average or summarizing the objects in the array.
But in .NET Core 7, the performance increase makes these buildin LINQ methods actually a lot faster.
Nick Chapsas shows this in a benchmark video on YouTupe
So if you want to calculate the sum, min, max or average value, you should use the LINQ methods instead of a foreach loop from .NET Core 7 onwards (at least, from a performance point of view)