We Keep Coding

Analyzing Gigantic C# Heap Dumps (Over 10 GB)

I have seen this question before but the responses haven't scaled to my needs. I am looking for a way to analyze heap dumps from a C# application that uses an extremely high amount of memory.
One issue I run into is that the tool I am using is x86. This has been the case for a few appealing tools (VS2013 Ultimate, Antz memory profiler, PerfView).
I have also invested some time trying CLR Profiler but this does not seem to work (Edit - work when attached to the application).
Right now I feel my only other option would be to jump into Windbg. Are there any other tools that could support this?
Thanks!

I have used Windbg a lot. A DumpHeap -stat will give you good and fast results to check which objects are flooding over. If you are lucky you get from the call stacks a clue what is allocating so much data.
Personally I like PerfView much better because it is much faster (it samples the heaps) and stores in the .gcDump file only a fraction of the data needed from the dump. You can take with PerfView a heap snapshot and ship this to the HQ which is much smaller than the original dump (factor 100 smaller usually). I have analyzed up to 36 GB dumps with PerfView with some issues but Vance Morrison was kind enough to help me out to fix his heap traversal code where a Stackoverflow occured.
PerfView may be x86 but the internal HeapDumper which extracts the data from the dump is 64 bit of course.
An alternative approach without any dumps is to enable in PerfView .NET Heap Sampl alloc tracing. This way you get call stacks for all allocations leading you directly to the code where allocations are happening.

How to trace memory allocation in C#

Are there any methods to trace how much memory is allocated, de-allocated and retrieved by GC for a particular module in C# .net ?
I want to trace out the possible memmory leaks in my module. I am getting occasional System out of memory exceptions in production.

You should use a memory profiler to profile memory allocations.
I've used JetBrains dotTrace, which has a nice mode of taking two snapshots at different times and showing which objects were allocated but not collected between those two snapshots. Allows for easy finding for memory leaks, where you keep allocating new instances and not collect them.
To view the difference between two application memory states, you can
mark the start and the end of a time interval, then capture a
difference snapshot which shows how much memory was allocated and
released during the marked time interval. The view can be filtered to
show only live, new, or dead objects, or the difference between new
and dead objects.

A memory profiler is a good idea. You can also get a rough sketch using the PerformanceCounters See the msdn. You could then gather some stats on your prod environment if it's hard to reproduce locally.

you can try .NET Memory Allocation Profiling with Visual Studio.
try this msdn article to show you how http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio-2012.aspx.
About 1/3 of the way through the article it shows how to Run the .NET memory allocation profiler.

JetBrains, as mentioned, as well as .NET Memory profiler has helped me several times. If you have a memory leakage problem in WPF there are some good advice in this post (old but a lot is still valid):
http://blogs.msdn.com/b/jgoldb/archive/2008/02/04/finding-memory-leaks-in-wpf-based-applications.aspx

Any way in C# / .NET app to get breakdown of memory allocated to each type in memory?

I have an app which consumes a lot of real time data, and because it's doing so much it's quite slow under the VS 2010 and this causes it to fail in various ways.
So I was wondering if there's any way other than this profiler that I can find out how much memory in bytes say is allocated to each type in memory and dump this out periodically?
It's quite a large application so adding my own counters isn't really feasible...

You need to use a memory profiler.
There are many around, some free and some commercial.
MemProfiler
ANTS memory profiler
dotTrace
clr profiler
Also see What Are Some Good .NET Profilers?

There is no easy general purpose way of saying GetBytesUsedForInstance(object), but it depends what you need the data for (unless all your types are value types, in which case it should be relatively simple).
We have an in memory cache for part of our application. We care most about relative amounts of memory used - ie the total cache size is twice what it was yesterday. For this, we serialize our object graphs to a stream and take the stream length (and then discard the stream). This is not an accurate measurement of "how much memory a type uses up" per se, but is useful for these relative comparisons.
Other than that - I think you are stuck using a profiler. I can highly recommend SciTech Memory Profiler - I use it a lot. It integrates well into Visual Studio, is fast (the latest version is anyway), and gives tremendously useful detail.

I would suggest like for getting a general information massively use Process Explorer.
One time you gfigure out you need to understand the stuff deeper (what kind of objects are on heap, for example) , the best tool I used for profiling is JetBrains Memory and Performance profiler. This one is payed only.
If you need only performance profiler, there is really good free option Equatec Performance profiler
Good luck.

C# memory leak?? Gen 0 and 1 increasing constantly in perfmon - What does it mean?

I am using C# 2.0 for a multi-threaded application that receives atleast thousand callbacks per second from an unmanaged dll and periodically send messages out of socket. GUI remains on main thread.
My application mostly creates object at the startup and periodically during the execution for a short lived period.
The problem I am experiencing is periodic latency spike (measured by time stamping a function at start and end) which I suppose happen when GC run.
I ran perfmon and here are my observations...
Gen0 Heap Size is flat with a spike every few seconds with periodic spike.
Gen1 Heap Size is always on the roll. Up and down
Gen2 Heap Size follows a cycle. It keep increasing till it becomes flat for a while and then drops.
Gen 0 and 1 Collections are always increasing in a range of 1 to 5 units.
Gen 2 collections is constant.

I recommend using a memory profiler in order to know if you have a real memory leak or not. There are many available and they will allow you to quickly isolate any issue.
The garbage collector is adaptive and will modify how often it runs in response to the way your application is using memory. Just looking at the generation heap sizes is going to tell you very little in terms of isolating the source of any problem. Second quessing how it works is a bad idea.
RedGate Ants Memory Profiler
SciTech .NET Memory Profiler
EQATEC .NET Profiler
CLR Profiler (Free)

So as #Jalf says, there's no evidence of a memory "leak" as such: what you discuss is closer to latency caused by garbage collection.
Others may disagree but I'd suggest anything above a few hundred callbacks per second is going to stretch a general purpose language like C#, especially one that manages memory on your behalf. So you're going to have to get smart with your memory allocation and give the runtime some help.
First, get a real profiler. Perfmon has its uses but even the profiler in later versions of Visual Studio can give you much more information. I like the SciTech profiler best (http://memprofiler.com/); there are others including a well respected one from RedGate reviewed here: http://devlicio.us/blogs/scott_seely/archive/2009/08/23/review-of-ants-memory-profiler.aspx
Once you know your baseline, aim to eliminate gen2 collections. They'll be the really slow ones. Work hard in any tight loops to eliminate as much memory allocation as you can -- strings are the usual offenders.
Some useful tips are in an old but still relevant MSDN article here: http://msdn.microsoft.com/en-us/library/ms973837.aspx.
It's also good to read Tess Ferrandez's (outstanding) blog series on debugging ASP.NET applications - book a day out of the office and start here: http://blogs.msdn.com/b/tess/archive/2008/02/04/net-debugging-demos-information-and-setup-instructions.aspx.

I remember reading a blog post about memory performance in .NET (specifically, XNA on the XBox 360) a while ago (unfortunately I can't find said link anymore).
The nutshell of achieving low latency memory performance was to make sure you never run gen 2 GC's at a performance critical time (although it is OK to run them when latency is not important; there are a bunch of notification callback functions on the GC class in more recent versions of the framework which may help with this). There are two ways to make sure this happens:
Don't allocate anything that escapes to gen 2. It's alarmingly easy for objects to escape to gen 2 when you don't realise it, so this often translates into: don't allocate anything in performance critical code. Because no objects escape to gen 2, the GC doesn't need to collect.
Allocate everything you need upfront and use object pooling. Your gen 2 heap will be big but because nothing is being added to it, the GC doesn't need to collect it.
It may be helpful to look into some XNA or Silverlight related performance articles because
games and resource constrained devices are often very latency sensitive. (Note that you have it easy because the XBox 360 and, until Mango, Windows Phone only had a single generation GC (mark-and-sweep collector)).

How to reduce memory consumption by an application in Task manager

I have developed a winforms application using C# in VS2008. When I run this application I observed in task manager that it shows 80 MB of memory is consumed.
How can I reduce this? Even very small applications also take 8 MB of memory...
What can I do to reduce the memory footprint?
Thank you very much in advance.

The task manager memory number is not always as sharp as it would be, however this small trick can cheat that number... Not meant for production
public static void RefreshMemory() {
try {
Process curProc = Process.GetCurrentProcess();
curProc.MaxWorkingSet = curProc.MaxWorkingSet;
} catch {
// Handle the exception
}
}
It would be helpful also to trace the memory objects usage with a tool like JetBrains dotTrace or another similar.

Your question clearly smells of premature optimization.
You should tackle memory usage only in the following situations :
You're developing for a device which hasn't a lot of memory available (with the portable .net framework for example)
You're reaching the limit of .net in term of memory size (ie around 1.3gbp)
Your customers are complaining.
Don't get me wrong, this doesn't mean you must waste memory or that you shouldn't take memory in consideration while coding. I just mean that in your case, memory usage is probably not a big deal.

Well, the .NET framework has a fairly significant overhead -- the simplest possible "hello world" console app has a 4 mb working set, as you have observed. There are a number of things you can do to reduce the memory footprint (reduce embedded resources, be sure to build in Release configuration, etc.)
But at the end of the day, .NET is designed for developer efficiency over memory/resource efficiency, so if you have an app that has to run in a very small efficient memory space you should consider writing it in C++ or some other language where you manage your own resources.

In addition to all the previous good comments, please remember that the numbers you are seeing in Task Manager are virtual memory, not physical memory. The actual amount of physical RAM used by the program is not obvious from looking at Task Manager. Also, Task Manager is showing you a series of snapshots - if you really had a problem, you'd want to look at it over time, with Perfmon or something, in addition to the excellent suggestion on profiling with JetBrains' dotTrace.
But don't do any optimization until your code works, and has high test coverage, and until you know what actually needs to be optimized. Otherwise, you run the risk of optimizing problems that don't exist, and worse, of optimizing the wrong problem, while ignoring the real problem or possibly making it worse.