I'm trying to tune the performance of my application. And I'm curious what methods are taking the longest to process, and thus should be looked over for any optimization opportunities.
Are there any existing free tools that will help me visualize the call stack and how long each method is taking to complete? I'm thinking something that displays the call stack as a stacked bar graph or a treemap so it's easy to see that MethodA() took 10 second to complete because it called MethodB() and Method(C) that took 3 and 7 seconds to complete.
Does something like this exist?
Yes, they are called performance profilers. Check out RedGate ANTS and JetBrains' dotTrace, not free but pretty cheap and way better than any free alternative I have seen.
Several profilers are available. I'm only familiar with one (the one included in Visual Studio Team Suite), but it's not free. I must say, it's been powerful/reliable enough that I've had no desire to try switching to another. The other one I've heard about is Red Gate's .NET profiler (also not free). Both of these have feature sets including, but definitely not limited to, what you're after. If this is for commercial application development, definitely check them out. :)
The way this is measured is by either instrumentation or sampling within a profiler.
Instrumentation
Instrumentation does the rough equivalent of rewriting all your code to do the following:
public void Foo()
{
//Code
}
into
public void Foo()
{
InformTheProfilingCodeThatFooStarted();
//Code
InformTheProfilingCodeThatFooEnded();
}
Since the profiler knows when everything starts and stops it can managed a stack of when thisngs start and stop and supply this information later. Many allow you to do this at a line level (by doing much the same thing but instrumenting even more before each line.
This gets your 100% accurate information on the 'call graph' in your application but does so at the cost of: preventing inlining of methods and adding considerable overhead to each method call.
Sampling
An alternate approach is Sampling.
Instead of trying to get 100% accurate call graphs but with less than accurate actual times this approach instead works on the basis that, if it check on a regular basis what is going on in your app it can give you a good idea of how much time it sepnds in various functions without actually having to spend much effort working this out. Most sampling profilers know how to 'parse' the call stack when they interrupt the program so they can still give a reasonable idea of what is calling which function and how much time this seems to take but will not be able to tell you whether this was (say) 10 calls to Foo() which did ten calls to Bar() or one call to Foo() which was in the one call to Bar() which just happened to last so long it was sampled 10 times.
Both approaches have their pros and cons and solve different problems. In general the sampling method is the best one to start with first since it is less invasive and thus should give more accurate information on what is taking time which is often the most important first question before working out why.
I know of only one free sampling profiler for .net code which is the free redistributable profiler which is linked with the VS 2008 Team System release (but which can be downloaded separately). The resulting output cannot be easily viewed with anything but the (very expensive) Team System edition of Visual Studio.
Red Gate ANTS does not support sampling (at this time), Jet Brains (dotTrace) and MS Visual Studio Team System have profilers that support both styles. Which you prefer on a cost benefit basis is a matter of opinion.
This is the method I use. If you have an IDE with a pause button it costs nothing and works very well.
What it tells you is roughly what % of wall-clock time is spent in each routine, and more precisely, in each statement. That is more important than the average duration of executing the routine or statement, because it automatically factors in the invocation count. By sampling wall-clock time it automatically includes CPU, IO, and other kinds of system time.
Even more importantly, if you look at the samples in which your routine is on the call stack, you can see not only what it is doing, but why. The reason that is important is that what you are really looking for is time being spent that could be replaced with something faster. Without the "why" information, you have to guess what that is.
BTW: This technique is little-known mainly because professors do not teach it (even if they know it) because they seldom have to work with monstrous software like we have in the real world, so they treat gprof as the foundation paradigm of profiling.
Here's an example of using it.
P.S. Expect the percents to add up to a lot more than 100%. The way to think about the percents is, if a statement or routine is on the stack X% of the time (as estimated from a small number of samples), that is roughly how much wall-clock time will shrink if the statement or routine can be made to take a lot less time.
Related
I find timing my code for performance to be a bit of a headache and I have to assume this is a common problem for folks.
Adding in a bunch of code to time my loops is great but often there is more overhead in adding the timing checks than adding a parallel task.
making a loop parallel can take seconds whereas adding in timing chunks requires several lines of code and be dangerous if forgotten.
So... My question is why dont IDE's include this ability to time chunks of code like a break point?
Something like i mark a section of code and tell it to dump the timing output to a console with some options to make it more readable. averages and such.
I know break points can seriously hamper performance but there must be an easier way to time specific chunks of code then to write all my own timing routines for each chunk of code i want to check.
First, VS 2015 does offer timing indicators while debugging.
(source: s-msft.com)
But I say don't use it for serious profiling purposes. Use it to get a very rough idea of what's going on.
Timing anything in a debug build yields unreliable figures, because of many factors (disabled compiler optimizations, disabled JIT optimizations, nop instructions everywhere, breakpoints, etc). Garbage in, garbage out.
Just do your profiling properly, it's a hard thing to get right. And use a real profiler for that if you need detailed performance data.
I'm curious how does a typical C# profiler work?
Are there special hooks in the virtual machine?
Is it easy to scan the byte code for function calls and inject calls to start/stop timer?
Or is it really hard and that's why people pay for tools to do this?
(as a side note i find a bit interesting bec it's so rare - google misses the boat completely on the search "how does a c# profiler work?" doesn't work at all - the results are about air conditioners...)
There is a free CLR Profiler by Microsoft, version 4.0.
https://www.microsoft.com/downloads/en/details.aspx?FamilyID=be2d842b-fdce-4600-8d32-a3cf74fda5e1
BTW, there's a nice section in the CLR Profiler doc that describes how it works, in detail, page 103. There's source as part of distro.
Is it easy to scan the byte code for
function calls and inject calls to
start/stop timer?
Or is it really hard and that's why
people pay for tools to do this?
Injecting calls is hard enough that tools are needed to do it.
Not only is it hard, it's a very indirect way to find bottlenecks.
The reason is what a bottleneck is is one or a small number of statements in your code that are responsible for a good percentage of time being spent, time that could be reduced significantly - i.e. it's not truly necessary, i.e. it's wasteful.
IF you can tell the average inclusive time of one of your routines (including IO time), and IF you can multiply it by how many times it has been called, and divide by the total time, you can tell what percent of time the routine takes.
If the percent is small (like 10%) you probably have bigger problems elsewhere.
If the percent is larger (like 20% to 99%) you could have a bottleneck inside the routine.
So now you have to hunt inside the routine for it, looking at things it calls and how much time they take. Also you want to avoid being confused by recursion (the bugaboo of call graphs).
There are profilers (such as Zoom for Linux, Shark, & others) that work on a different principle.
The principle is that there is a function call stack, and during all the time a routine is responsible for (either doing work or waiting for other routines to do work that it requested) it is on the stack.
So if it is responsible for 50% of the time (say), then that's the amount of time it is on the stack,
regardless of how many times it was called, or how much time it took per call.
Not only is the routine on the stack, but the specific lines of code costing the time are also on the stack.
You don't need to hunt for them.
Another thing you don't need is precision of measurement.
If you took 10,000 stack samples, the guilty lines would be measured at 50 +/- 0.5 percent.
If you took 100 samples, they would be measured as 50 +/- 5 percent.
If you took 10 samples, they would be measured as 50 +/- 16 percent.
In every case you find them, and that is your goal.
(And recursion doesn't matter. All it means is that a given line can appear more than once in a given stack sample.)
On this subject, there is lots of confusion. At any rate, the profilers that are most effective for finding bottlenecks are the ones that sample the stack, on wall-clock time, and report percent by line. (This is easy to see if certain myths about profiling are put in perspective.)
1) There's no such thing as "typical". People collect profile information by a variety of means: time sampling the PC, inspecting stack traces, capturing execution counts of methods/statements/compiled instructions, inserting probes in code to collect counts and optionally calling contexts to get profile data on a call-context basis. Each of these techniques might be implemented in different ways.
2) There's profiling "C#" and profiling "CLR". In the MS world, you could profile CLR and back-translate CLR instruction locations to C# code. I don't know if Mono uses the same CLR instruction set; if they did not, then you could not use the MS CLR profiler; you'd have to use a Mono IL profiler. Or, you could instrument C# source code to collect the profiling data, and then compile/run/collect that data on either MS, Mono, or somebody's C# compatible custom compiler, or C# running in embedded systems such as WinCE where space is precious and features like CLR-built-ins tend to get left out.
One way to instrument source code is to use source-to-source transformations, to map the code from its initial state to code that contains data-collecting code as well as the original program. This paper on instrumenting code to collect test coverage data shows how a program transformation system can be used to insert test coverage probes by inserting statements that set block-specific boolean flags when a block of code is executed. A counting-profiler substitutes counter-incrementing instructions for those probes. A timing profiler inserts clock-snapshot/delta computations for those probes. Our C# Profiler implements both counting and timing profiling for C# source code both ways; it also collect the call graph data by using more sophisticated probes that collect the execution path. Thus it can produce timing data on call graphs this way. This scheme works anywhere you can get your hands on a halfway decent resolution time value.
This is a link to a lengthy article that discusses both instrumentation and sampling methods:
http://smartbear.com/support/articles/aqtime/profiling/
Anyone can recommend a LOP on Windows? Similar to Linux's OProfile or to OS X's Shark.
must be able to sample non-instrumented binaries
capable of resolving CLR stacks
preferable delayed PDB resolution of symbols
impact low enough to be able to get a decent reading on live, production systems
The Visual Studio Team Suite profiler is amazing. It's so good at its job that it makes me seem better at mine.
Redgate has a performance profiler and memory profiler which I haven't used.
Automated QA's AQTime has saved my butt. I used it to figure out a problem with a .NET web service calling some nasty old C code, and it did it well.
This is what I use. Although it is not suitable for live production use, it answers your other needs.
For live production use, you need something that samples the stack. In my opinion, it's OK if it has some small overhead. My goal is to discover the activities that need optimization, and for that I'm willing to pay a temporary price in speed.
There is always one or more intervals of interest, like the interval between when a request is received, and the response goes out. It's surprising how few samples you need in such an interval to find out what's taking the time.
High precision of timing is not needed. If there is something X going on that, through optimization, would save you, say, 50% of the interval, that is roughly the fraction of samples that will show you X.
I wish (I dont know if its possible) to build profiling support into my code instead of using some external profiler. I have heard that there is some profiler api that is used by most of the profiler writers. Can that api be used to profile from within the code that is being executed? Are there any other considerations?
If you don't want to use a regular profiler, you could have your application output performance counters.
You may find this blog entry useful to get started: Link
The EQATEC Profiler builds an instrumented version of your app that will run and collect profiling statistics entirely by itself - you don't need to attach the profiler. By default your app will simply dump the statistics into plaintext xml-files.
This means that you can build a profiled version of your app, deploy it at your customer's site, and have them run it and send back the statistics-reports to you. No need for them to install anything special or run a profiler or anything.
Also, if you can reach your deployed app's machine via a network-connection and it allows incoming connections then you can even take snapshots of the running profiled app yourself, sitting at home with the profiler. All you need is a socket-connection - you decide the port-number yourself and the control-protocol itself is plain http, so it's pretty likely to make it past even content-filtering gateways.
The .NET framework profiler API is a COM object that intercepts calls before .NET handles them. My understanding is that it cannot be hosted in managed (C#) code.
Depending on what you want to do, you can insert Stopwatch timers to measure length of calls, or add Performance Counters to your application so that you can monitor the performance of the application from the Performance Monitor.
There's a GameDev article that discusses how to build profiling infrastructure in a C++ program. You may be able to adapt this approach to work with C# provided objects created on the stack are freed on exit instead of left for the garbage collector
http://www.gamedev.net/reference/programming/features/enginuity3/
Even if you can't take the whole technique, there may be some useful ideas.
What I've done when I can't use my favorite technique is this. It's clumsy and gives low-resolution information, but it works. First, have a global stack of strings. This is in C, but you can adapt it to C#:
int nStack = 0;
char* stack[10000];
Then, on entry and exit to each routine you have source code for, push/pop the name of the routine:
void EveryFunction(){
int iStack = nStack++; stack[iStack] = "EveryFunction";
... code inside function
nStack = iStack; stack[iStack] = NULL;
}
So now stack[0..nStack] keeps a running call stack (minus the line numbers of where functions are called from), so it's not as good as a real call stack, but better than nothing.
Now you need a way to take snapshots of it at random or pseudo-random times. Have another global variable and a routine to look at it:
time_t timeToSnap;
void CheckForSnap(){
time_t now = time(NULL);
if (now >= timeToSnap){
if (now - timeToSnap > 10000) timeToSnap = now; // don't take snaps since 1970
timeToSnap += 1; // setup time for next snapshot
// print stack to snapshot file
}
}
Now, sprinkle calls to CheckForSnap throughout your code, especially in the low-level routines. When the run is finished, you have a file of stack samples. You can look at those for unexpected behavior. For example, any function showing up on a significant fraction of samples has inclusive time roughly equal to that fraction.
Like I said, this is better than nothing. It does have shortcomings:
It does not capture line-numbers where calls come from, so if you find a function with suspiciously large time, you need to rummage within it for the time-consuming code.
It adds significant overhead of it's own, namely all the calls to time(NULL), so when you have removed all your big problems, it will be harder to find the small ones.
If your program spends significant time waiting for I/O or for user input, you will see a bunch of samples piled up after that I/O. If it's file I/O, that's useful information, but if it's user input, you will have to discard those samples, because all they say is that you take time.
It is important to understand a few things:
Contrary to popular accepted wisdom, accuracy of time measurement (and thus a large number of samples) is not important. What is important is that samples occur during the time when you are waiting for the program to do its work.
Also contrary to accepted wisdom, you are not looking for a call graph, you don't need to care about recursion, you don't need to care about how many milliseconds any routine takes or how many times it is called, and you don't need to care about the distinction between inclusive and exclusive time, or the distinction between CPU and wall-clock time. What you do need to care about is, for any routine, what percent of time it is on the stack, because that is how much time it is responsible for, in the sense that if you could somehow make that routine take no time, that is how much your total time would decrease.
I am trying to make the loading part of a C# program faster. Currently it takes like 15 seconds to load up.
On first glimpse, things that are done during the loading part includes constructing many 3rd Party UI components, loading layout files, xmls, DLLs, resources files, reflections, waiting for WndProc... etc.
I used something really simple to see the time some part takes,
i.e. breakpointing at a double which holds the total milliseconds of a TimeSpan which is the difference of a DateTime.Now at the start and a DateTime.Now at the end.
Trying that a few times will give me sth like,
11s 13s 12s 12s 7s 11s 12s 11s 7s 13s 7s.. (Usually 12s, but 7s sometimes)
If I add SuspendLayout, BeginUpdate like hell; call things in reflections once instead of many times; reduce some redundant redundant computation redundancy. The time are like 3s 4s 3s 4s 3s 10s 4s 4s 3s 4s 10s 3s 10s.... (Usually 4s, but 10s sometimes)
In both cases, the times are not consistent but more like, a bimodal distribution? It really made me unsure whether my correction of the code is really making it faster.
So I would like to know what will cause such result.
Debug mode?
The "C# hve to compile/interpret the code on the 1st time it runs, but the following times will be faster" thing?
The waiting of WndProc message?
The reflections? PropertyInfo? Reflection.Assembly?
Loading files? XML? DLL? resource file?
UI Layouts?
(There are surely no internet/network/database access in that part)
Thanks.
Profiling by stopping in the debugger is not a reliable way to get timings, as you've discovered.
Profiling by writing times to a log works fine, although why do all this by hand when you can just launch the program in dotTrace? (Free trial, fully functional).
Another thing that works when you don't have access to a profiler is what I call the binary approach - look at what happens in the code and try to disable about half of it by using comments. Note the effect on the running time. If it appears significant, repeat this process with half of that half, and so on recursively until you narrow in on the most significant piece of work. The difficulty is in simulating the side effects of the missing code so that that the remaining code can still work, so this is still harder than using a debugger, but can be quicker than adding a lot of manually time logging, because the binary approach lets you zero in on the slowest place in logarithmic time.
Raymond Chen's advise is good here. When people ask him "How can I make my application start up faster?" he says "Do less stuff."
(And ALWAYS profile the release build - profiling the debug build is generally a wasted effort).
Profile it. you can use eqatec its free
Well, the best thing is to run your application through a profiler and see what the bottlenecks are. I've personally used dotTrace, there are plenty of others you can find on the web.
Debug mode turns off a lot of JIT optimizations, so apps will run a lot slower than release builds. Whatever the mode, JITting has to happen, so I'd discount that as a significant factor. Time to read files from disk can vary based on the OS's caching mechanism, and whether you're doing a cold start or a warm start.
If you have to use timers to profile, I'd suggest repeating the experiment a large number of times and taking the average.
Profiling you code is definitely the best way to identify which areas are taking the longest to run.
As for the other part of your question about the inconsistent timings: timings in an multitasking O/S are inherently inconsistent, and working with managed code throws the garbage collector into the equation too. It could be that the GC is kicking in during your timing which will obviously slow things down.
If you want to try and get a "purer" timing try putting a GC collect before you start your timers, this way it is less likely to start in your timing section. Do remember to remove the timers after, as second guessing when the GC should run normally results in poorer performance.
Apart from the obvious (profiling), which will tell you precisely where time is being spent, there are some other points that spring to mind:
To get reasonable timing results with the approach you are using, run a release build of your program, and have it dump the timing results to a file (e.g. with Trace.WriteLine). Timing a debug version will give you spurious results. When running the timing tests, quit all other applications (including your debugger) to minimise the load on your computer and get more consistent results. Run the program many times and look at the average timings. Finally, bear in mind that Windows caches a lot of stuff, so the first run will be slow and subsequent runs will be much faster. This will at least give you a more consistent basis to tell whether your improvements are making a significant difference.
Don't try and optimise code that shouldn't be run in the first place - Can you defer any of the init tasks? You may find that some of the work can simply be removed from the init sequence. e.g. if you are loading a data file, check whether it is needed immediately - if not, then you could load it the first time it is needed instead of during program startup.