We Keep Coding

How to trace memory allocation in C#

Are there any methods to trace how much memory is allocated, de-allocated and retrieved by GC for a particular module in C# .net ?
I want to trace out the possible memmory leaks in my module. I am getting occasional System out of memory exceptions in production.

You should use a memory profiler to profile memory allocations.
I've used JetBrains dotTrace, which has a nice mode of taking two snapshots at different times and showing which objects were allocated but not collected between those two snapshots. Allows for easy finding for memory leaks, where you keep allocating new instances and not collect them.
To view the difference between two application memory states, you can
mark the start and the end of a time interval, then capture a
difference snapshot which shows how much memory was allocated and
released during the marked time interval. The view can be filtered to
show only live, new, or dead objects, or the difference between new
and dead objects.

A memory profiler is a good idea. You can also get a rough sketch using the PerformanceCounters See the msdn. You could then gather some stats on your prod environment if it's hard to reproduce locally.

you can try .NET Memory Allocation Profiling with Visual Studio.
try this msdn article to show you how http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio-2012.aspx.
About 1/3 of the way through the article it shows how to Run the .NET memory allocation profiler.

JetBrains, as mentioned, as well as .NET Memory profiler has helped me several times. If you have a memory leakage problem in WPF there are some good advice in this post (old but a lot is still valid):
http://blogs.msdn.com/b/jgoldb/archive/2008/02/04/finding-memory-leaks-in-wpf-based-applications.aspx

Why does .NET reserve so much memory for my application?

When I run my application, in a profiler I see that is uses about 80MB of memory (total committed bytes, performance counter). But when I look at the size of the allocated memory, it is over 400MB!
So my question is, why is .NET reserving so much memory for my application? Is this normal?

you should read Memory Mystery. I had similar questions a while ago and stopped asking myself after reading this.
I read other sources, but I cant find now, use keywords "unreasonable allocation of memory windows OS". In a nutshell, OS gives more than your app require depending upon physically available memory resources
for e.g. if you are running your app on two machines with different RAM, it can be guaranteed that both these machines will have different memory allocations

As you no doubt know, there is a massive difference between actual memory used and allocated. An application's allocated memory doesn't mean that it's actually being used anywhere; all it really means is that the OS has 'marked' a zone of virtual memory (which is exactly that - virtual) ready for use by the application.
The memory isn't necessarily being used or starving other processes - it just could if the app starts to fill it.
This allocated number, also, will likely scale based on the overall memory ecosystem of the machine. If there's plenty of room when an app starts up, then it'll likely grab a larger allocation than if there's less.
That principle is the same as the one which says it's good practise to create a List<T>, say, with a reasonable initial capacity that'll mean a decent number of items can be added before resizing needs to take place. The OS takes the same approach with memory usage.

"Reserving" memory is by no means the same as "allocated" ram. Read the posts Steve and Krishna linked to.
The part your client needs to look at is Private Bytes. But even that isn't exactly a hard number as parts of your app may be swapped to the virtual disk.
In short, unless your Private Bytes section is pretty well out of control OR you have leaks (ie: undisposed unmanaged resources) you (and your client) should ignore this and let the OS manage what is allocated, what's in physical ram and what's swapped out to disk.

It's fairly common for software to issue one large memory request to the underlying operating system, then internally manage its own use of the allocated memory block. So common, in fact, that Windows' (and other operating systems') memory manager explicitly supports the concept, called "uncommitted memory" -- memory that the process has requested but hasn't made use of yet. That memory really doesn't exist as far as bits taking up space on your DRAM chips until the process actually makes use of it. The preallocation of memory effectively costs nothing.
Applications do this for many reasons -- though it's primarily done for performance reasons. An application with knowledge of its own memory usage patterns can optimize its allocator for that pattern; similarly, for address locality reasons, as successive memory requests from the OS won't always be 'next' to each other in memory, which can affect the performance of the CPU cache and could even preclude you from using some optimizations.
.NET in particular allocates space for the managed heap ahead of time, for both of the reasons listed above. In most cases, allocating memory on the managed heap merely involves incrementing a top-of-heap pointer, which is incredibly fast --- and also not possible with the standard memory allocator (which has a more general design to perform acceptably in a fragmented heap, whereas the CLR's GC uses memory compaction to sharply limit the fragmentation of the managed heap), and also not possible if the managed heap itself is fragmented across the process address space due to multiple allocations at different points in time.

.net memory measuring and profiling

I understand there are many questions related to this, so I'll be very specific.
I create Console application with two instructions. Create a List with some large capacity and fill it with sample data, and then clear that List or make it equal to null.
What I want to know is if there is a way for me to know/measure/profile while debugging or not, if the actual memory used by the application after the list was cleared and null-ed is about the same as before the list was created and populated. I know for sure that the application has disposed of the information and the GC has finished collecting, but can I know for sure how much memory my application would consume after this?
I understand that during the process of filling the list, a lot of memory is allocated and after it's been cleared that memory may become available to other process if it needs it, but is it possible to measure the real memory consumed by the application at the end?
Thanks
Edit: OK, here is my real scenario and objective. I work on a WPF application that works with large amounts of data read through USB device. At some point, the application allocates about 700+ MB of memory to store all the List data, which it parses, analyzes and then writes to the filesystem. When I write the data to the filesystem, I clear all the Lists and dispose all collections that previously held the large data, so I can do another data processing. I want to know that I won't run into performance issues or eventually use up all memory. I'm fine with my program using a lot of memory, but I'm not fine with it using it all after few USB processings.
How can I go around controlling this? Are memory or process profilers used in case like this? Simply using Task Manager, I see my application taking up 800 MB of memory, but after I clear the collections, the memory stays the same. I understand this won't go down unless windows needs it, so I was wondering if I can know for sure that the memory is cleared and free to be used (by my application or windows)

It is very hard to measure "real memory" usage on Windows if you mean physical memory. Most likley you want something else like:
Amount of memory allocated for the process (see Zooba's answer)
Amount of Managed memory allocated - CLR Profiler, or any other profiler listed in this one - Best .NET memory and performance profiler?
What Task Manager reports for your application
Note that it is not necessary that after garbage collection is finished amount of memory allocated for your process (1) changes - GC may keep allocated memory for future managed allocations (this behavior is not specific to CLR for memory allcation - most memory allocators keep free blocks for later usage unless forced to release it by some means). The http://blogs.msdn.com/b/maoni/ blog is excelent source for details on GC/memory.

Process Explorer will give you all the information you need. Specifically, you will probably be most interested in the "private bytes history" graph for your process.
Alternatively, it is possible to use Window's Performance Monitor to track your specific application. This should give identical information to Process Explorer, though it will let you write the actual numbers out to a separate file.
(A picture because I can...)

I personaly use SciTech Memory Profiler
It has a real time option that you can use to see your memory usage. It has help me find a number of problems with leaking memory.

Try ANTS Profiler. Its not free but you can try the trial version.
http://www.red-gate.com/products/dotnet-development/ants-performance-profiler/

How to detect where a Memory Leak is?

I have a large website that seems to be sucking up all the memory that is being allocated. There is nothing else on the server beside this site. Within a week it eats away the 2 gigs and requires a restart. Currently this is server 2008 32 bit using IIS 7. We are reinstalling to use 64bit and add more memory. It would be nice to be able to track down where the leaks are occurring.
So what is the best practice to tracking memory leaks?

Memory leaks in .NET are not that common, but when they happen it is most commonly due to unattached event handlers. Make sure you detach handlers, before the listeners go out of scope.
Another option is if you forget to call Dispose() on IDisposable resources. This may prevent cleanup of unmanaged resources (which are not handled by GC).
And yet another possible reason is a deadlocked finalizer. That will prevent all remaining objects in the finalizer queue from being collected.
I use WinDbg + Sos to track down leaks. The steps are as follows
Dump the heap and look for suspects
Use !gcroot to find out what is keeping the suspects alive
Repeat as necessary
Be aware that large memory usage may also be due to heap fragmentation. The regular heaps are compacted, but pinned objects can cause fragmentation. Also, the LOH is not compacted so fragmentation is not uncommon for LOH.
Excellent tutorials on WinDbg + sos here: http://blogs.msdn.com/tess/

Run a profiler on your code.
Here are two good options:
RedGate's memory profiler
Jetbrains dotTrace
I believe both products have a free trial.

Run, don't walk, over to Tess
Ferrandez's blog, If broken it is,
fix it you should, which has well
scripted labs dedicated to learning
how to diagnose and debug crash, hang
and memory issues with .NET code. She
has some of the best material I've
found to date to help get you started.
Commercial memory profilers such as ANTS and SciTech are excellent resources that will show what objects are in the heap, and how they are rooted. Most commercial memory profilers have the ability to load a memory 'snap' of a process (say from your production environment).
You can capture a memory 'snap' (see Snap v. Dump) using adplus.vbs or DebugDiag. Adplus is available as part of the Debugging Tools for Windows. DebugDiag will also have some rudimentary analysis (but seems to be more reliable on unmanaged code) automagically.
Monitor the Application
For an idea on what to monitor, see Improving .NET Performance and Scalability, specifically Chapter 15.
As to how to monitor, there are commercial tools available for that as well, however, every Windows machine also comes with Perfmon.exe, which can be used to record relevant performance counters.
Test the Application
For an idea on how to perform load, or stress, tests, check out the Patterns and Practices Performance Testing Guidance for Web Applications.
Debug the Application
Once you've identified you've got a problem (monitoring) and your able to reproduce the problem (testing) you can get down to debugging the problem. See the links for Tess - that information will carry you a long way.
Then rinse and repeat! :)
Good luck!
Z

In performance monitor, add counters for Process/Private Bytes and .NET CLR Memory/# Bytes in All Heaps. Private bytes is all memory and CLR memory is just managed. So if the CLR memory remains fairly even but the private bytes continues to grow over time, this means that the leak is in unmanaged resource. That usually means that you are not disposing of native resources properly. A good thing to look at is stuff like COM or IO (streams and files). Make sure all of that stuff gets disposed when you are done with it.

You can try using profilers such as dotTrace -- set it into a memory trace and run your application.
This should give you clues in which assemblies and areas of the application that eat up too much memory on the get go.

This is probably prevention rather then detection, but at the c# code level, you should check that any classes that use large resources such as images and other files are correctly implementing the dispose pattern. If needed you may also need to override the finalizer.
MSDN has good guidance on this subject.
If you have any classes in your application that you know make use of large resources, these are the first places to look for memory issues.

I've found that the EQATEC Profiler is pretty good, plus it's free!

Check out the Memory and Memory Leak labs on this blog post:
.NET Debugging Demos
They may be of some help. Basically, you can use WinDBG to analyze a memory dump and help determine what is eating up all of your memory.
We used a similar approach to determine that Regex was chewing up all of our memory, but only when the product was run on 64-bit machines. The learning curve is kind of steep, but WinDBG is a very powerful tool.

This new article of mine maybe useful: How to detect and avoid memory and resources leaks in .NET applications

Do you do any Office interop? If so, make sure you clean up your application objects. That's one possible culprit.
Another is global objects, such as anything that's static.

Do you have lots of dynamic pages on your site?
You can also try IBM's Purify
I suggest you try with a small set of dynamic pages disabling all others for the meantime. I hate to say this but it's very much possible that IIS 7 may also have leaks.

Look at this article on detecting .NET application memory leaks and related articles mentioned at the bottom of the page and hopefully you will find a solution or at least an idea to resolve it.
Thanks

c# windows app - profiling peak memory usage and identifying trends

I have a long running console app running through millions of iterations. I want to benchmark if memory usage increases linerally as the number of iterations increases.
What would be the best way to go about this?
I think I really only need to be concerned with Peak memory usage during a run right? I basically need to work out what is the max number of iterations I can run on this hardware given the memory on the Server.
I am going to setup a batch lot of runs and Log the results over different interation sizes and then graph the results to identify a memory usage trend which can then be extrapolated for any given hardware.
Looking for advice on the best way to implement this, what .net methods, classes to use or should I use external tools. This article http://www.itwriting.com/dotnetmem.php suggests I should profile my own app via code to factor out shared memory used by the .net runtime across other apps on the box.
Thanks

There are several ways to do this:
Perfmon UI
You can use the Performance Montior control panel applet (in Administrative Tools) that comes with Windows to monitor your application. Have a look at the .Net CLR Memory category and the counters inside it. You can also limit the monitoring to just your process. This is probably the easiest as it requires no code changes.
Perfmon API
You can programmatically use performance counters from .Net. For this, you'll need to use the PerformanceCounter class. This is just an API to the same underlying information that the above UI presents.
Memory Profiler
You can use a memory profiler to profile you application whilst it is running. The two I have used with success are the ANTS Memory Profiler from RedGate and the .Net Memory Profiler from SciTech. Again, this requires no code changes, but may cost you money (although there are free trial versions). There is also the CLR Profiler (a howto can be found here).
Roll Your Own
You can get hold of some limited memory information from the Process class. Get hold of the current process using Process.GetCurrentProcess(), and then look at the properties of it, specifically the ones relating to memory (MinWorkingSet, MaxWorkingSet, PagedMemorySize64, PeakPagedMemorySize64, PeakVirtualMemorySize64, PeakWorkingSet64, PrivateMemorySize64, VirtualMemorySize64, WorkingSet64). This is probably the worst solution as you have to do everything yourself, including data collection and reporting.
If all you are wanting to do is to verify your application doesn't linearly increase its memory usage as your the number of iterations increases, I would recommend monitoring using the Performance Monitor UI in Windows. It will show you what you need with minimal effort on your part.

Windows perfmon is great for this kind of stuff. You have counters for all the managed heaps and for private bytes, if you need to cross correlate with iteration counts you can publish your own perfmon counters from .Net.

when you want to profile the mem usage of your apps, here is a piece of code you can use.
First you have to declare an instance of the PerformanceCounter class
Assembly a = Assembly.GetExecutingAssembly();
_PerfCounter = new PerformanceCounter(".NET CLR Memory",
"# Bytes in all Heaps",
a.GetName().Name,
true);
and each time you want to log the memory consumption you can call
_PerfCounter.NextValue()
Hope it was helpful

Maybe this tool can do everything you want.

Better late than never, but in response to M3ntat's comment, you'll need to add ".vshost" onto the assembly name parameter if you're running from within Visual Studio, e.g.
a.GetName().Name + ".vshost"