I'm working on a project at work where there's a performance issue with the code.
I've got some changes I think will improve performance, but no real way of gauging how my changes affect it.
I wrote a unit test that does things the way they're currently implemented, with a Stopwatch to monitor how fast the function runs. I've also wrote a similar unit test that does things slightly differently.
If the tests are ran together, one takes 1s to complete, the other takes 73 ms.
If the tests are ran separately, they both take around 1s to complete (yea.. that change i made didn't seem to change much).
If the tests are identical, I have the same issue, one runs faster than the other.
Is visual studio doing something behind the scenes to improve performance? Can I turn it off if it is?
I've tried moving tests into different files, which didn't fix the issue I'm having.
I'd like to be able to run all the tests, but have them run as if there's only one test running at a time.
My guess: it's likely down to dll loading and JIT compiling
1. Assembly loading.
.NET lazily loads assemblies (dll's). If you add reference to FooLibrary, it doesn't mean it gets loaded when your code loads.
Instead, what happens is that the first time you call a function or instantiate a class from FooLibrary, then the CLR will go and load the dll it lives in. This involves searching for it in the filesystem, possible security checks, etc.
If your code is even moderately complex, then the "first test" can often end up causing dozens of assemblies to get loaded, which obviously takes some time.
Subsequent tests appear fast because everything's already loaded.
2. JIT Compiling
Remember, your .NET assemblies don't contain code that the CPU can directly execute. Whenever you call any .NET function, the CLR takes the MSIL bytecode and compiles it into executable machine code, and then it goes and runs this machine code. It does this on a per-function basis.
So, if you consider that the first time you call any function, there will be a small delay while it JIT compiles, these things can add up. This can be particularly bad if you're calling a lot of functions or initializing a big third party library (think entity framework, etc).
As above, subsequent tests appear fast, because many of the functions will have already been JIT compiled, and cached in memory.
So, how can you get around this?
You can improve the assembly loading time by having fewer assemblies. This means fewer file searches and so on. The microsoft .NET performance guidelines go into more detail.
Also, I believe installing them in the global assembly cache may (??) help, but I haven't tested that at all so please take it with a large grain of salt.
Installing into the GAC requires administrative permissions and is quite a heavyweight operation. You don't want to be doing it during development, as it will cause you problems (assemblies get loaded from the GAC in preference to the filesystem, so you can end up loading old copies of your code without realizing it).
You can improve the JIT time by using ngen to pre-compile your assemblies. However, like with the GAC, this requires administrative permissions and takes some time, so you do not want to do it during development either.
My advice?
Firstly, measuring performance in unit tests is not a particularly good or reliable thing to be doing. Who knows what else visual studio is doing in the background that may or may not affect your tests.
Once you've got your code you're trying to benchmark out into a standalone app, have it loop and run all the tests twice, and discard the first result :-)
"Premature optimization is the root of all evil."
If you didn't measure before, how do you know you are fixing anything now? How do you even know you had a problem that needed to be solved?
Unit tests are for operational correctness. They could be used for performance, but I would not depend on that because many other factors come into play at run-time.
Your best bet is to get a profiler (or use one that comes with VS) and start measuring.
Related
I noticed that sometimes a .net 4.0 c# application takes a long time to start, without any apparent reason. Can can I determine what's actually happening, what modules are loaded? I'm using a number of external assemblies. Can putting them into the GAC improve performances?
Is .NET 4 slower than .NET 2?
.NET programs have two distinct start-up behaviors. They are called cold-start and warm-start. The cold-start is the slow one, you'll get it when no .NET program was started before. Or when the program you start is large and was never run before. The operating system has to find the assembly files on disk, they won't be available in the file system cache (RAM). That takes a while, hard disks are slow and there are a lot of files to find. A small do-nothing Winforms app has to load 51 DLLs to get started. A do-nothing WPF app weighs in at 77 DLLs.
You get a warm start when the assembly files were loaded before, not too long ago. The assembly file data now comes from RAM instead of the slow disk, that's zippedy-doodah. The only startup overhead is now the jitter.
There's little you can do about cold starts, the assemblies have to come of the disk one way or another. A fast disk makes a Big difference, SSDs are especially effective. Using ngen.exe to pre-jit an assembly actually makes the problem worse, it creates another file that needs to be found and loaded. Which is the reason that Microsoft recommends not prejitting small assemblies. Seeing this problem with .NET 4 programs is also highly indicated, you don't have a lot of programs that bind to the version 4 CLR and framework assemblies. Not yet anyway, this solves itself over time.
There's another way this problem automatically disappears. The Windows SuperFetch feature will start to notice that you often load the CLR and the jitted Framework assemblies and will start to pre-load them into RAM automatically. The same kind of trick that the Microsoft Office and Adobe Reader 'optimizers' use. They are also programs that have a lot of DLL dependencies. Unmanaged ones, the problem isn't specific to .NET. These optimizers are crude, they preload the DLLs when you login. Which is the 'I'm really important, screw everything else' approach to working around the problem, make sure you disable them so they don't crowd out the RAM space that SuperFetch could use.
The startup time is most likely due to the runtime JIT compiling assembly IL into machine code for execution. It can also be affected by the debugger - as another answerer has suggested.
Excluding that - I'll talk about an application ran 'in the wild' on a user's machine, with no debugger etc.
The JIT compiler in .Net 4 is, I think it's fair to say, better than in .Net 2 - so no; it's not slower.
You can improve this startup time significantly by running ngen on your application's assemblies - this pre-compiles the EXEs and DLLs into native images. However you lose some flexibility by doing this and, in general, there is not much point.
You should see the startup time of some MFC apps written in C++ - all native code, and yet depending on how they are linked they can take just as long.
It does, of course, also depend on what an application is actually doing at startup!
I dont think putting your assemblies in GAC will boot the performance.
If possible do logging for each instruction you have written on Loading or Intialize events which may help you to identify which statement is actually taking time and with this you can identify the library which is taking time in loading.
I've encountered the following paragraph:
“Debug vs. Release setting in the IDE when you compile your code in Visual Studio makes almost no difference to performance… the generated code is almost the same. The C# compiler doesn’t really do any optimization. The C# compiler just spits out IL… and at the runtime it’s the JITer that does all the optimization. The JITer does have a Debug/Release mode and that makes a huge difference to performance. But that doesn’t key off whether you run the Debug or Release configuration of your project, that keys off whether a debugger is attached.”
The source is here and the podcast is here.
Can someone direct me to a Microsoft article that can actually prove this?
Googling "C# debug vs release performance" mostly returns results saying "Debug has a lot of performance hit", "release is optimized", and "don't deploy debug to production".
Partially true. In debug mode, the compiler emits debug symbols for all variables and compiles the code as is. In release mode, some optimizations are included:
unused variables do not get compiled at all
some loop variables are taken out of the loop by the compiler if they are proven to be invariants
code written under #debug directive is not included, etc.
The rest is up to the JIT.
Full list of optimizations here courtesy of Eric Lippert.
There is no article which "proves" anything about a performance question. The way to prove an assertion about the performance impact of a change is to try it both ways and test it under realistic-but-controlled conditions.
You're asking a question about performance, so clearly you care about performance. If you care about performance then the right thing to do is to set some performance goals and then write yourself a test suite which tracks your progress against those goals. Once you have a such a test suite you can then easily use it to test for yourself the truth or falsity of statements like "the debug build is slower".
And furthermore, you'll be able to get meaningful results. "Slower" is meaningless because it is not clear whether it's one microsecond slower or twenty minutes slower. "10% slower under realistic conditions" is more meaningful.
Spend the time you would have spent researching this question online on building a device which answers the question. You'll get far more accurate results that way. Anything you read online is just a guess about what might happen. Reason from facts you gathered yourself, not from other people's guesses about how your program might behave.
I can’t comment on the performance but the advice “don’t deploy debug to production” still holds simply because debug code usually does quite a few things differently in large products. For one thing, you might have debug switches active and for another there will probably be additional redundant sanity checks and debug outputs that don’t belong in production code.
From msdn social
It is not well documented, here's what
I know. The compiler emits an
instance of the
System.Diagnostics.DebuggableAttribute.
In the debug version, the
IsJitOptimizerEnabled property is
True, in the release version it is
False. You can see this attribute in
the assembly manifest with ildasm.exe
The JIT compiler uses this attribute
to disable optimizations that would
make debugging difficult. The ones
that move code around like
loop-invariant hoisting. In selected
cases, this can make a big difference
in performance. Not usually though.
Mapping breakpoints to execution
addresses is the job of the debugger.
It uses the .pdb file and info
generated by the JIT compiler that
provides the IL instruction to code
address mapping. If you would write
your own debugger, you'd use
ICorDebugCode::GetILToNativeMapping().
Basically debug deployment will be slower since the JIT compiler optimizations are disabled.
What you read is quite valid. Release is usually more lean due to JIT optimization, not including debug code (#IF DEBUG or [Conditional("DEBUG")]), minimal debug symbol loading and often not being considered is smaller assembly which will reduce loading time. Performance different is more obvious when running the code in VS because of more extensive PDB and symbols that are loaded, but if you run it independently, the performance differences may be less apparent. Certain code will optimize better than other and it is using the same optimizing heuristics just like in other languages.
Scott has a good explanation on inline method optimization here
See this article that give a brief explanation why it is different in ASP.NET environment for debug and release setting.
One thing you should note, regarding performance and whether the debugger is attached or not, something that took us by surprise.
We had a piece of code, involving many tight loops, that seemed to take forever to debug, yet ran quite well on its own. In other words, no customers or clients where experiencing problems, but when we were debugging it seemed to run like molasses.
The culprit was a Debug.WriteLine in one of the tight loops, which spit out thousands of log messages, left from a debug session a while back. It seems that when the debugger is attached and listens to such output, there's overhead involved that slows down the program. For this particular code, it was on the order of 0.2-0.3 seconds runtime on its own, and 30+ seconds when the debugger was attached.
Simple solution though, just remove the debug messages that was no longer needed.
In msdn site...
Release vs. Debug configurations
While you are still working on your
project, you will typically build your
application by using the debug
configuration, because this
configuration enables you to view the
value of variables and control
execution in the debugger. You can
also create and test builds in the
release configuration to ensure that
you have not introduced any bugs that
only manifest on one type of build or
the other. In .NET Framework
programming, such bugs are very rare,
but they can occur.
When you are ready to distribute your
application to end users, create a
release build, which will be much
smaller and will usually have much
better performance than the
corresponding debug configuration. You
can set the build configuration in the
Build pane of the Project Designer, or
in the Build toolbar. For more
information, see Build Configurations.
I recently run into a performance issue. The products full list was taking too much time, about 80 seconds. I tuned the DB, improved the queries and there wasn't any difference. I decided to create a TestProject and I found out that the same process was executed in 4 seconds. Then I realized the project was in Debug mode and the test project was in Release mode. I switched the main project to Release mode and the products full list only took 4 seconds to display all the results.
Summary: Debug mode is far more slower than run mode as it keeps debugging information. You should always deploy in Relase mode. You can still have debugging information if you include .PDB files. That way you can log errors with line numbers, for example.
To a large extent, that depends on whether your app is compute-bound, and it is not always easy to tell, as in Lasse's example. If I've got the slightest question about what it's doing, I pause it a few times and examine the stack. If there's something extra going on that I didn't really need, that spots it immediately.
Debug and Release modes have differences. There is a tool Fuzzlyn: it is a fuzzer which utilizes Roslyn to generate random C# programs. It runs these programs on .NET core and ensures that they give the same results when compiled in debug and release mode.
With this tool it was found and reported a lot of bugs.
My project is developed in Visual Studio 08, in C#. It's a standalone desktop application, about 60k lines of code.
Once upon a time I loved working on this software - now that the compliation time has grown to approx 2 minutes, it becomes a far less enjoyable experience...
I think that my lack of experience in C# may be a factor; I have developed everything under one namespace for example - would having a well structured codebase enable the compiler to recompile only the necessary parts of the code when changes are made? Or do I need to separate sections into separate projects/DLLs to force this to happen?
How much of a difference would upgrading to the latest quad-core processor make?
The other thought is, perhaps this is a typical thing for programmers to deal with - is a long compile time like this simply something that must be managed?
Thanks in advance.
Things that increase compile time:
The number of projects in a solution makes more difference than the number of files in a particular project.
Custom build tasks can make a huge difference, especially if they are generating code or running post-build analysis (FxCop, StyleCop, Code Contracts).
Native code projects take longer to build.
A single project containing 60K lines of C# code with no special build features enabled should compile in seconds on any machine made in the past 5+ years.
I'm surprised that 60k lines of code take 2 minutes to compile. I have an application that is 500,000 lines of code, and it only takes about a minute and a half. Make sure you are not doing full rebuilds each time, and make sure you are not cleaning the solution between builds. A normal build should perform an incremental build, only recompiling code that has changed since the last build (along with anything affected by that change.)
Perhaps some other factors might include heavy use of large resources (images?), broad-sweeping changes in the lowest level libraries (i.e. those used by everything else), etc. Generally speaking, on a relatively modern machine, compiling 60,000 lines of C# code should take less than a minute on average, unless you are rebuilding the entire solution.
There is this thread about hardware to improve compile time. Also this really excellent blog post from Scott Guthrie on looking at hard drive speed for performance.
Splitting your project up into multiple projects will help. Only those projects that have changes (and projects that depend on it) will need recompilation.
A single namespace however, shouldn't affect compile time. However, if you do split up your project into multiple projects/assemblies, then a single namespace is definitely not a good idea.
Upgrading to a faster CPU will probably help, but you might find that faster I/O (better disks, RAID, etc will be more useful).
And yes, avoiding long compile times are one of the things developers need to take care of. When it comes to productivity, do whatever you can (better tools, bigger screens, faster machines, etc...)
Has anybody here ever used ngen? Where? why? Was there any performance improvement? when and where does it make sense to use it?
I don't use it day-to-day, but it is used by tools that want to boost performance; for example, Paint.NET uses NGEN during the installer (or maybe first use). It is possible (although I don't know for sure) that some of the MS tools do, too.
Basically, NGEN performs much of the JIT for an assembly up front, so that there is very little delay on a cold start. Of course, in most typical usage, not 100% of the code is ever reached, so in some ways this does a lot of unnecessary work - but it can't tell that ahead of time.
The downside, IMO, is that you need to use the GAC to use NGEN; I try to avoid the GAC as much as possible, so that I can use robocopy-deployment (to servers) and ClickOnce (to clients).
Yes, I've seen performance improvements. My measurements indicated that it did improve startup performance if I also put my assemblies into the GAC since my assemblies are all strong named. If your assemblies are strong named, NGen won't make any difference without using the GAC. The reason for this is that if you have strong named assemblies that are not in the GAC, then the .NET runtime validates that your strong named assembly hasn't been tampered with by loading the whole managed assembly from disk so it can validate it circumventing one of the major benefits of NGen.
This wasn't a very good option for my application since we rely on common assemblies from our company (that are also strong named). The common assemblies are used by many products that use many different versions, putting them in the GAC meant that if one of our applications didn't say "use specific version" of one of the common assemblies it would load the GAC version regardless of what version was in its executing directory. We decided that the benefits of NGen weren't worth the risks.
Ngen mainly reduces the start-up time of .NET app and application's working set. But it's have some disadvantages (from CLR Via C# of Jeffrey Richter):
No Intellectual Property Protection
NGen'd files can get out of sync
Inferior Load-Time Performance (Rebasing/Binding)
Inferior Execution-Time Performance
Due to all of the issues just listed, you should be very cautious when considering the use of
NGen.exe. For server-side applications, NGen.exe makes little or no sense because only the
first client request experiences a performance hit; future client requests run at high speed. In
addition, for most server applications, only one instance of the code is required, so there is no
working set benefit.
For client applications, NGen.exe might make sense to improve startup time or to reduce
working set if an assembly is used by multiple applications simultaneously. Even in a case in
which an assembly is not used by multiple applications, NGen'ing an assembly could improve
working set. Moreover, if NGen.exe is used for all of a client application's assemblies, the CLR
will not need to load the JIT compiler at all, reducing working set even further. Of course, if
just one assembly isn't NGen'd or if an assembly's NGen'd file can't be used, the JIT compiler
will load, and the application's working set increases.
ngen is mostly known for improving startup time (by eliminating JIT compilation). It might improve (by reducing JIT time) or decrease overall performance of the application (since some JIT optimizations won't be available).
.NET Framework itself uses ngen for many assemblies upon installation.
i have used it but just for research purpose. use it ONLY if you are sure about the cpu architecture of your deployment environment (it wont change)
but let me tell you JIT compilation is not too bad and if you have deployments across multiple cpu environments (for example a windows client application which is updated often) THEN DO NOT USE NGEN. thats coz a valid ngen cache depends upon many attributes. if one of these fail, your assembly falls back to jit again
JIT is a clear winner in such cases, as it optimizes code on the fly based on the cpu architecture its running on. (for eg it can detect if there are more then 1 cpu)
and clr is getting better with every release, so in short stick with JIT unless you are dead sure of your deployment environment - even then your performance gains would hardly justify using ngen.exe (probably gains would be in few hundred ms) - imho - its not worth the efforts
also check this real nice link on this topic - JIT Compilation and Performance - To NGen or Not to NGen?
Yes. Used on a WPF application to speed up startup time. Startup time went from 9 seconds to 5 seconds. Read about it in my blog :
I recently discovered how great NGEN can be for performance. The
application I currently work on has a data access layer (DAL) that is
generated. The database schema is quite large, and we also generate
some of the data (list of values) directly into the DAL. Result: many
classes with many fields, and many methods. JIT overhead often showed
up when profiling the application, but after a search on JIT compiling
and NGEN I though it wasn’t worth it. Install-time overhead, with
management my major concern, made me ignore the signs and focus on
adding more functionality to the application instead. When we changed
architecture to “Any CPU” running on 64 bit machines things got worse:
We experienced hang in our application for up to 10 seconds on a
single statement, with the profiler showing only JIT overhead on the
problem-area. NGEN solved the problem: the statement went from 10
seconds to 1 millisecond. This statement was not part of the
startup-procedure, so I was eager to find out what NGEN’ing the whole
application could do to the startup time. It went from 8 seconds to
3.5 seconds.
Conclusion: I really recommend giving NGEN a try on your application!
As an addition to Mehrdad Afshari's comment about JIT compilation. If serializing a class with many properties via the XmlSerializer and on a 64-bit system a SGEN, NGEN combo has a potentially huge (in our case gigabytes and minutes) effect.
More info here:
XmlSerializer startup HUGE performance loss on 64bit systems see Nick Martyshchenko's answer especially.
Yes, I tried it with a small single CPU-intensive exe and with ngen it was slightly slower!
I installed and uninstalled the ngen image multiple times and ran a benchmark.
I always got the following times reproducable +/- 0.1s:
33.9s without,
35.3s with
I'm writing a DSP application in C# (basically a multitrack editor). I've been profiling it for quite some time on different machines and I've noticed some 'curious' things.
On my home machine, the first run of the playback loop takes up about 50%-60% of the available time, (I assume it's due to the JIT doing its job), then for the subsequent loops it goes down to a steady 5% consumption. The problem is, if I run the application on a slower computer, the first run takes up more than the available time, causing the playback to get interrupted and messing the output audio, which is unacceptable. After that, it goes down to a 8%-10% consumption.
Even after the first run, the application keeps calling some time-consuming routines from time to time (every 2 seconds more or less), which causes the steady 5% consumption to experience very short peaks of 20%-25%. I've noticed that if I let the application run for a while these peaks will also go down to a 7%-10%. (I'm not sure if it's due to the JIT recompiling these portions of code).
So, I have a serious problem with the JIT. While the application will behave nicely even in very slow machines, these 'compiling storms' are going to be a big problem. I'm trying to figure out how to resolve this issue and I've come up with an idea, which is to mark all the 'sensible' routines with an attribute that will tell the application to 'squeeze' them beforehand during start-up, so they'll be fully optimized when they're really needed. But this is only an idea (and I don't like it too much either) and I wonder if there's a better solution to the whole problem.
I'd like to hear what you guys think.
(NGEN the application is not an option, I like and want all the JIT optimizations I can get.)
EDIT:
Memory consumption and garbage collection kicks are not an issue, I'm using object pools and the maximum peak of memory during playback is 304 Kb.
You can trigger the JIT compiler to compile your entire set of assemblies during your application's initialization routine using the PrepareMethod ... method (without having to use NGen).
This solution is described in more detail here: Forcing JIT Compilation During Runtime.
The initial speed indeed sounds like Fusion+JIT, which would be helped by ILMerge (for Fusion) and NGEN (for JIT); you could always play a silent track through the system at startup so that this does all the hard work without the user noticing any distortion?
NGEN is a good option; is there a reason you can't use it?
The issues you mention after the initial load do not sound like they are related to JIT. Perhaps garbage collection.
Have you tried profiling? Both CPU and memory (collections)?
As Marc mentioned, the ongoing spikes do not sound like JIT issues. Other things to look for:
Garbage collection - are you allocating memory during your audio processing? If you're creating a lot of garbage, or even objects which survive a Gen 0 collection, this might cause noticible spikes. It sounds like you are doing some kind of pre-allocation, but watch out for hidden allocations in library code (even a foreach loop can allocate!)
Denormals. There is an issue with certain types of processors when dealing with very small floating point numbers which can cause CPU spikes. See http://www.musicdsp.org/files/denormal.pdf for details.
Edit:
Even if you don't want to use NGen, at least compare an NGen'd version so you can see what difference JITing makes
If you believe you are being impacted by JIT, then precompile your app with NGEN and run the tests again. There is no JIT overhead in code that has been compiled by NGEN. If you still see spikes in the NGEN'd app, then you know they are not caused by JIT.