We Keep Coding

Does VS's Diagnostic Tools measure the total Process Memory or the current Process Memory

Does the Visual Studio Diagnostic Tools measure the total memory used by an application's threads, or its current memory currently in use?
I've got an application that reads data in from a 34 megapixel camera at 3 frames per second. The math comes out that it processes 288 MB per second, or about 17 GB per minute. With this in mind, the application obviously consumes a lot of data, once it starts collecting the camera frames. I have had my eyes glued to the Diagnostics Tools for a bit, for reasons you can see below:
I've let the application run with the performance profiler for about 3 minutes, and it ends up reporting a total process memory of about 31 GB, as you can see below:
My laptop only has 16 GB of RAM, so at first glance I think the picture above basically answers my question. However, just past the 2:30min mark you can see a sharp decline in the Memory, which doesn't make sense (I don't believe anything changed in how the program was running). Also, when I opened up the task manager I could see my application was using about 9 GB of memory, prior to shooting down to about 3 GB of memory around the 2:30min mark.
With all of that in mind, what is the Process Memory really measuring?
Because the solution was buried in the comments in the accepted answer, I'll summarize the solution here, to my overall problem of why my program was using so much memory. It had to do with a bug where I was not disposing of Bitmaps. Bitmaps are unmanaged memory; if you fail to dispose of them before they go out of scope, they continue living in memory (outside of your Garbage Collector) until a threshold is reached in the number of unmanaged objects. Only when that threshold is hit, will the unmanaged objects get deleted. That is what was happening in my program, when it dipped from ~31 GB of memory to about 5 GB of memory.

However, just past the 2:30min mark you can see a sharp decline in the
Memory, which doesn't make sense (I don't believe anything changed in
how the program was running).
Garbage collection is a complicated process which can affect your application performance. So GC has been optimized to trigger only when there is a memory pressure over is threshold.
When there is a memory pressure, garbage collection process kicks off and clears unnecessary memory allocations. This is the Garbage collection normal behavior.
Does VS's Diagnostic Tools measure the total Process Memory or the
current Process Memory?
It measures your application's current memory usage. Here is a tutorial to understand VS memory tool.
The fact that it releases most of the memory usage after the garbage cycle means that there are no big memory leaks.
Redgate Ants memory profiler can show more details (Retensoin graphs etc). Here is a video that explains memory leaks bit clearly. https://documentation.red-gate.com/amp10/worked-example/video-tutorials
Is it possible to limit the amount of disc space available to a C#
program? I'm wondering if I could force C# to hold on to memory for a
shorter period of time.
You can call GC.Collect to force an immediate garbage collection after a memory expensive process. However, this is not recommended at all unless there is a really good reason for that. Garbage collect uses heuristic algorithms to optimize its behavior. You don't have to worry about this usually. One thing is MAKE SURE YOU DISPOSE ALL THE DISPOSABLE INSTANCES before they go out of scope. That can help to release memory with less garbage cycles.

C# Excessive Garbage Collection - Large Strings, G2 pressure?

I'm writing a high-ish volume web service in C# running in 64-bit IIS on Win 2k8 (.NET 4.5) that works with XML payloads and does a variety of operations on small and large objects (where the large objects are mainly strings, some over 85k (so going onto the LOH)). Requests are stateless, and memory usage remains steady over time. Lots of memory is being allocated and released per request, no memory appears to be being leaked.
Operating at a maximum of 25 transactions per second, with an average call lasting 5s, it's spending 40-60% of it's time in GC according to two profiling tools, and perfmon shows a steady 20 G0 and G1 collections over 5 seconds, and 15 G2 collections over 5 seconds - meaning lots of (we think) premature promtion into G2 for data that we'd expect to stay in G0. Everything I read indicates this is very excessive. We expect that the system should be able to perform at a higher throughput than 25 tps and assume the GC activity is preventing this.
The machines serving the requests have lots of memory - 16GB - and the application, under load, consumes at most 1GB when under load for an hour. I understand that a bigger heap won't necessarily make things better, but there is spare memory.
I appreciate this is light on specifics (will try to recreate the conditions with a trivial application if time permits) - but can anyone explain why we see so much G2 GC activity? Should I be focusing on the LOH? People keep telling me that the CLR's GC "adapts" to your load, but it's not changing it's behavior in this case and, unlike other runtimes, there seems to be little I can do to tune it (have tried workstation GC, but there is very little observable difference).

Microsoft decided to design the String class so that all strings are stored in memory as a monolithic sequence of characters. While this works well for some usage patterns, it works dreadfully for others.
One thing I've found very helpful is to avoid creating instances of String whenever possible. If a method will often be used to operate on part of a supplied string, and will in turn ask other methods to operate on parts of it, the methods should accept arguments specifying the range of the String upon which they should operate. This will avoid the need for callers of the first method to use Subst to construct a new String for the method to act upon, and will avoid the need to have the method call Subst to feed portions of the string to its callers. In some cases where I have used this technique, the creation of thousands of String instances--some quite large--could be replaced with zero.

CLR's GC "adapts" to your load
It can't know how much memory you are willing to tolerate as overhead. Here, you probably want to give the app like 5GB of heap so that collections are much rarer. The GC has no built-in tuning knobs for that (subjective note: that's a pitty).
You can force bigger heap sizes by using one of the low latency modes for short durations. That should cause the GC to try hard to avoid G2 collections. Monitor the RAM usage and disable low latency mode when consumption reaches 5GB.
This is a risky strategy but it's the best I think you can do.
I would not do it. You can maximally gain 2x throughput. Your CPU is maxed out, right? Workstation GC does not scale to multiple cores and leaves CPUs unused.

Gen2 collection not always collecting dead objects?

By monitoring the CLR #Bytes in all Heaps performance counter of a brand new .NET 4.5 server application over the last few days, I can notice a pattern that makes me think that Gen2 collection is not always collecting dead objects, but I am having trouble understanding what exactly is going on.
Server application is running in .NET Framework 4.5.1 using Server GC / Background.
This is a console application hosted as a Windows Service (with the help of Topshelf framework)
The server application is processing messages, and the throughput is somehow pretty constant for now.
What I can see looking at the graph of CLR #Bytes in all Heaps is that the memory started arround 18MB then growing up to 35MB on approx 20-24 hours (with between 20-30 Gen2 collections during that time frame), and then all of a sudden dropping back to nominal value of 18MB, then growing again up to ~35MB over 20-24 hours and dropping back to 18MB, and so on (I can see the pattern repeating over the last 6 days the app is now running) ... The growing of memory is not linear, it takes approx 5 hours to grow by 10MB and then 15-17 hours for the remaining 10 MB or so.
Thing is that I can see by looking at perfmon counters for #Gen0/#Gen1/#Gen2 collections that a bunch of Gen2 collections are going on during the 20-24 hours period (maybe arround 30) and none of them makes the memory drop back to nominal 18MB.
However, what is strange is by using an external tool to force a GC (Perfview in my case), then I can see #Induced GC going up by 1 (GC.Collect was called so this is normal) and immediately the memory is going back to nominal 18MB.
Which leads me into thinking that either the perfmon counter for #Gen2 collections is not right and only a single Gen2 collection happens after 20-22hours or so (meeehhh I really don't think so) or that the Gen2 collection does not always collect dead objects (seems more plausible) ... but in that case why would forcing a GC via GC.Collect do the trick, what would be the difference between explicitely calling into GC.Collect, v.s automatic triggered collections during the lifetime of the application.
I am sure there is a very good explanation but from the different source of documentation I have found about GC -too few :(- a Gen2 collection does collect dead objects in any case. So maybe docs are not up to date or I have misread ... Any explanation is welcome. Thanks !
EDIT : Please see this screenshot of the #Bytes in all heaps graph over 4 days
(Click for larger view)
this is easier than trying to graph things in your head. What you can see on the graph is what I said above... memory increasing over 20-24hours (and 20-30 Gen2 collections during that time frame) until reaching ~35MB then dropping all of a sudden. You will note at the end of the graph, the induced GC I triggered via an external tool, immediately dropping back memory to nominal.
EDIT #2 : I made a lot of cleaning in the code, mainly regarding finalizers. I had a lot of classes that were holding reference to disposable types, so I had to implement IDisposable on these types. However I was misguided by some articles into implementing the Diposable pattern with a Finalizer in any case. After reading some MSDN documentation I came to understand that a finalizer was only required when the type was holding native resources itself (and still in that case this could be avoided with SafeHandle). So I removed all finalizers from all these types. There were some other modications in the code, but mainly business logic, nothing ".NET framework" related.
Now the graph is very different, this is a flat line arround 20MB for days now ... exactly what I was expecting to see !
So the problem is now fixed, however I still have no idea what was the problem due to ... It seems like it might have been related to finalizers but still does not explain what I was noticing, even if we weren't calling Dispose(true) -suppressing finalizer-, the finalizer thread is supposed to kick in between collection and not every 20-24 hours ?!
Considering we have now moved away from the problem, it will take time to come back to the "buggy" version and reproduce it again. I may try to do it some time though and go to the bottom of it.
EDIT: Added Gen2 collection graph (Click for larger view)

From
http://msdn.microsoft.com/en-us/library/ee787088%28v=VS.110%29.aspx#workstation_and_server_garbage_collection
Conditions for a garbage collection
Garbage collection occurs when one of the following conditions is
true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously
adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs
continuously. This method is primarily used for unique situations and
testing.
It seems that you are hitting the 2nd one and 35 is the threshold. You should be able to configure the threshold to something else if 35 is to large.
There isn't anything special about gen2 collections that would cause them to deviate from these rules. (cf https://stackoverflow.com/a/8582251/215752)

Are any of your objects "large" objects? there's a separate "large object heap" which has different rules
Large Object Heap Fragmentation
It was improved in 4.5.1, though:
http://blogs.msdn.com/b/dotnet/archive/2011/10/04/large-object-heap-improvements-in-net-4-5.aspx

This could easily be explained if gcTrimCommitOnLowMemory is enabled. Normally, the GC keeps some extra memory allocated to the process. However, when the memory reaches a certain threshold, the GC will then "trim" the extra memory.
From the docs:
When the gcTrimCommitOnLowMemory setting is enabled, the garbage collector evaluates the system memory load and enters a trimming mode when the load reaches 90%. It maintains the trimming mode until the load drops under 85%.
This could easily explain your scenario - the memory reserves are being kept (and used) until your application reaches a certain point, which seems to be once every 20-24 hours, at which point the 90% load is detected, and the memory is trimmed to its minimum requirements (the 18mb).

Reading your first version I would say that is a normal behavior.
...but in that case why would forcing a GC via GC.Collect do the
trick, what would be the difference between explicitely calling into
GC.Collect, v.s automatic triggered collections during the lifetime of
the application.
There is two type of collections, a full collection and a partial collection. What the automatic triggered does is a partial collection, but when calling GC.Collect it will do a full collection.
Meanwhile, I might have the reason of it now that you told us that you were using finalizer on all of your objects. If for any reason one of those objects were promoted to #2 Gen, the finalizer would only run when doing a #2 Gen collection.
The following example will demonstrate what I just said.
public class ClassWithFinalizer
{
~ClassWithFinalizer()
{
Console.WriteLine("hello from finalizer");
//do nothing
}
}
static void Main(string[] args)
{
ClassWithFinalizer a = new ClassWithFinalizer();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
GC.Collect();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
GC.Collect();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
a = null;
Console.WriteLine("Collecting 0 Gen");
GC.Collect(0);
GC.WaitForPendingFinalizers();
Console.WriteLine("Collecting 0 and 1 Gen");
GC.Collect(1);
GC.WaitForPendingFinalizers();
Console.WriteLine("Collecting 0, 1 and 2 Gen");
GC.Collect(2);
GC.WaitForPendingFinalizers();
Console.Read();
}
The output will be:
Class a is on #0 generation
Class a is on #1 generation
Class a is on #2 generation
Collecting 0 Gen
Collecting 0 and 1 Gen
Collecting 0, 1 and 2 Gen
hello from finalizer
As you can see, only when doing a collection on the generation where the object is, the memory of the objects with finalizer will be reclaimed.

Just figure I'll throw in my 2 cents here. I'm not an expert at this but maybe this might help your investigation.
If you're using a 64-bit platform try adding this to your .config file. I read that that can be an issue.
<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>
The only other thing I would point out is that you could prove your hypothesis by troubleshooting from the inside if you are in control of the source code.
Calling something along the lines of this your app's main memory consuming class, and setting it to run on timed intervals, could shed some light onto what's really going on.
private void LogGCState() {
int gen = GC.GetGeneration(this);
//------------------------------------------
// Comment out the GC.GetTotalMemory(true) line to see what's happening
// without any interference
//------------------------------------------
StringBuilder sb = new StringBuilder();
sb.Append(DateTime.Now.ToString("G")).Append('\t');
sb.Append("MaxGens: ").Append(GC.MaxGeneration).Append('\t');
sb.Append("CurGen: ").Append(gen).Append('\t');
sb.Append("CurGenCount: ").Append(GC.CollectionCount(gen)).Append('\t');
sb.Append("TotalMemory: ").Append(GC.GetTotalMemory(false)).Append('\t');
sb.Append("AfterCollect: ").Append(GC.GetTotalMemory(true)).Append("\r\n");
File.AppendAllText(#"C:\GCLog.txt", sb.ToString());
}
Also there is a pretty good article here on using the GC.RegisterForFullGCNotification method. Obviously this would enable you to also include the time span of a full collection so that you could possibly tune performance and collection frequency to your specific needs. This method also let's you specify a heap threshold to to trigger notifications (or collections?).
There's also probably a way to set that in the apps .config file but I haven't looked. For the most part 35MB is a pretty small footprint for a Server Application these days. Heck, my web browser makes it up to 300-400MB sometimes :) So, the Framework might just see 35MB as a good default point to free up memory.
Anyhow, I can tell by the thoughtfulness of your question that I'm probably just pointing out the obvious. But, it feels worth mentioning. I wish you luck!.
On a funny note
At the top of this post I had originally written "if (you're using a 64-bit platform)". That made me crack up. Take care!

I have exactly the same situation in my WPF application. No finalizers in my code btw. However it seems that ongoing GC actually collects Gen 2 objects. I can see that GC.GetTotalMemory() results reduces up to 150mb after Gen2 collection triggered.
So I'm under impression that Gen2 heap size does not show amount of bytes that is used by live objects. It is rather just a heap size or amount of bytes that is allocated for Gen2 purposes. You may have plenty of free memory there.
Under some conditions (not on each gen 2 collection) this heap size is trimmed. And at this particular moment my application gets a huge performance hit - it may hung up to sevetal seconds. Wondering why...

C# memory leak?? Gen 0 and 1 increasing constantly in perfmon - What does it mean?

I am using C# 2.0 for a multi-threaded application that receives atleast thousand callbacks per second from an unmanaged dll and periodically send messages out of socket. GUI remains on main thread.
My application mostly creates object at the startup and periodically during the execution for a short lived period.
The problem I am experiencing is periodic latency spike (measured by time stamping a function at start and end) which I suppose happen when GC run.
I ran perfmon and here are my observations...
Gen0 Heap Size is flat with a spike every few seconds with periodic spike.
Gen1 Heap Size is always on the roll. Up and down
Gen2 Heap Size follows a cycle. It keep increasing till it becomes flat for a while and then drops.
Gen 0 and 1 Collections are always increasing in a range of 1 to 5 units.
Gen 2 collections is constant.

I recommend using a memory profiler in order to know if you have a real memory leak or not. There are many available and they will allow you to quickly isolate any issue.
The garbage collector is adaptive and will modify how often it runs in response to the way your application is using memory. Just looking at the generation heap sizes is going to tell you very little in terms of isolating the source of any problem. Second quessing how it works is a bad idea.
RedGate Ants Memory Profiler
SciTech .NET Memory Profiler
EQATEC .NET Profiler
CLR Profiler (Free)

So as #Jalf says, there's no evidence of a memory "leak" as such: what you discuss is closer to latency caused by garbage collection.
Others may disagree but I'd suggest anything above a few hundred callbacks per second is going to stretch a general purpose language like C#, especially one that manages memory on your behalf. So you're going to have to get smart with your memory allocation and give the runtime some help.
First, get a real profiler. Perfmon has its uses but even the profiler in later versions of Visual Studio can give you much more information. I like the SciTech profiler best (http://memprofiler.com/); there are others including a well respected one from RedGate reviewed here: http://devlicio.us/blogs/scott_seely/archive/2009/08/23/review-of-ants-memory-profiler.aspx
Once you know your baseline, aim to eliminate gen2 collections. They'll be the really slow ones. Work hard in any tight loops to eliminate as much memory allocation as you can -- strings are the usual offenders.
Some useful tips are in an old but still relevant MSDN article here: http://msdn.microsoft.com/en-us/library/ms973837.aspx.
It's also good to read Tess Ferrandez's (outstanding) blog series on debugging ASP.NET applications - book a day out of the office and start here: http://blogs.msdn.com/b/tess/archive/2008/02/04/net-debugging-demos-information-and-setup-instructions.aspx.

I remember reading a blog post about memory performance in .NET (specifically, XNA on the XBox 360) a while ago (unfortunately I can't find said link anymore).
The nutshell of achieving low latency memory performance was to make sure you never run gen 2 GC's at a performance critical time (although it is OK to run them when latency is not important; there are a bunch of notification callback functions on the GC class in more recent versions of the framework which may help with this). There are two ways to make sure this happens:
Don't allocate anything that escapes to gen 2. It's alarmingly easy for objects to escape to gen 2 when you don't realise it, so this often translates into: don't allocate anything in performance critical code. Because no objects escape to gen 2, the GC doesn't need to collect.
Allocate everything you need upfront and use object pooling. Your gen 2 heap will be big but because nothing is being added to it, the GC doesn't need to collect it.
It may be helpful to look into some XNA or Silverlight related performance articles because
games and resource constrained devices are often very latency sensitive. (Note that you have it easy because the XBox 360 and, until Mango, Windows Phone only had a single generation GC (mark-and-sweep collector)).

c# multithreading mem usage - App slowing down and CPU usage drops

I've got a multithreaded app that manipulates in-memory data (no database or network access). I tried this on 2 machines, one machine is Xeon dual quad core CPU, the other is twin dial-cores. 5 threads are spawned.
Then this multithreaded process starts it runs very quickly and the CPU usage is at 60% for 5 cores, the physical memory is 50% of the RAM capacity. (Info from task manager). After it's about 1/3 of the way through it starts to slow down and the CPU utilisation drops to just below 20%. By the time it gets to 2/3s of the way it's so slow that it takes 1 day to complete the last third while it takes half an hour to do the first 1/3.
The process creates many SortedLists and Lists, so I am starting to suspect that the Garbage Collector can't cope, although the Task Manager memory usage is not so bad... I want to try to force the GC to free the unused collections immediately, is this a reasonable or even doable? And why does CPU utilitsation drop?

Forcing the garbage collector to run is almost always a bad idea. (In some instances, forcing it to run early could actually promote the lifetimes of objects)
Download a tool like Memprofiler, Ants or dotTrace (they all have trial versions), to identify whether you are leaking memory. Are you allocating objects larger than 85Kb?
Also, what version of the OS and .NET Framework are you using? (there were differences in how the server and PC versions of the GC worked)
Also, be aware that insertion into a SortedList is O(N) (whereas a SortedDictionary insertion is O(logN):
The SortedList generic class is a
binary search tree with O(log n)
retrieval, where n is the number of
elements in the dictionary. In this,
it is similar to the SortedDictionary
generic class. The two classes have
similar object models, and both have
O(log n) retrieval. Where the two
classes differ is in memory use and
speed of insertion and removal:
SortedList uses less memory than
SortedDictionary.
SortedDictionary has faster insertion
and removal operations for unsorted
data, O(log n) as opposed to O(n) for
SortedList.
If the list is populated all at once
from sorted data, SortedList is faster
than SortedDictionary.
Ref.
How are you managing multithreaded access to these lists? Can you post some cut-down code?

I guess adding lots of items to a heavily loaded collection isn;t as performant as it could be. I noticed something similar with an old SQL query - 100 records in the recordset was quick, but half a million records slowed things down exponentially.
To check the GC, run up perfmon and view (or log) the performance counters for the garbage collector and memory allocations.

Sounds like a data structure locking issue. It's difficult to say without knowing exactly what you're doing.
Try using one of the lock-free non-contiguous collections such as ConcurrentDictionary or ConcurrentBag and/or a proper queue like BlockingCollection.

You are more than likely using all you physical memory up with your data and Windows starts to use virtual memory after that which is much slower. You should try a memmory profiler to see which object are takeing up all your memmory, and consider disposing of some of those objest periodically to keep from using up all your RAM.

60% CPU on 5 cores from 5 threads. I assume that is 60% on each core. This is actually very bad. You cannot drive the CPU to 100% doing memory operation alone (no Database, no network, no file IO) it means your contention on locking is huge. As the program progresses, your structures presumably grow in size (more elements in some list/dictionaries), you are holding locks for longer, and the result is less CPU and even slower performance.
Is hard to tell w/o any real performance data, but this does not look like GC related. It looks more like high contention in the data structures. You should trace your app under profiler and see where the CPU/wait times are spent most. See Pinpoint a performance issue using hotpath in Visual Studio 2008 for a quick introduction to the sampling profiler.