We Keep Coding

Condemned Generation is Not Garbage-Collected

I'm analyzing the outcome of some simple C# code, in terms of GC performance. I've used PerfView with GC Collect Only ticked to gather meaningful info.
Consider the following relevant output from PerfView's GCStats, particularly GC #1106 and #1108:
Both #1106 and #1108 are listed as gen1 Non-concurrent GCs. Both have a reason of AllocLarge.
Now consider the associated Condemned Reasons table:
In a blogpost by Maoni Stephens, the owner for the .NET GC, we know that once a GC is started, it can act against a higher gen, as follows:
Now after the GC starts, we then decide what generation we would
actually collect. It might stay as a gen0 GC, or get escalated to a
gen1 or even a gen2 GC – this is something we decide as one of the
very first things we do in a GC. And factors that would cause us to
escalate to a high generation GC are what we called “condemned
reasons” (so for a GC there’s only one trigger reason but can be
multiple condemned reasons).
Based on the Condemned Reasons table, it turns out both #1106 and #1108 "elevate" themselves from gen0 GCs to gen2 GCs due to both breaking the threshold for allocations in gen2.
But how come:
1) Both #1106 and #1108 start off as gen0 GCs, get eventually "escalated" to gen2, but show up in GCStats as gen1 GCs ? Please note that this is not some PerfView issue, as the detailed ETW events show the same exact data in terms of generations.
2) Both #1106 and #1108 get registered as gen1 GCs, but with a reason of AllocLarge ? AllocLarge refers to the large object allocation threshold exceeded, and the LOH can only be processed as part of gen2 collections.
The data above was captured on a .NET Framework 4.7.2 running on a Windows 10 x64. I'm assuming though that the concepts behind the GC decisions would be largely the same for other, relatively recent versions of .NET Framework, or even .NET Core, so this should make little difference to what's being asked here.
I've also not burdened the question with the actual code, since what's causing the GCs to occur in the form explained above is not triggered by any special instruction, but by the inner decisions the GC takes.
Update 1: I've provided more data to the initial question. I've also specifically looked at and couldn't find any other Microsoft-Windows-DotNETRuntime* ETW event around GCs #1106 and #1108 referenced that could shed more light on either question.
Update 2: I've changed the name of the question, since there are no gen2 GCs occurring after all. The CLR is performing only gen1 GCs for #1106 and #1108, as the column for the gen2 survival rate states NaN, which implies no gen2 GC has run.
Looking over the .NET Core CLR source code (for inspiration as what might happen in .NET Framework), the function that does the actual GC work gc_heap::gc1 reads early on the condemned generation info (below), and provided this is a non-concurrent GC (similar to our #1106 and #1108) it proceeds with the mark phase against the respective generation.
As for the chosen condemned generation, within gc_heap::generation_to_condemn the check for the highest gen budget allocations exceeded is done before the check for low ephemeral segment, and the latter doesn't overwrite the decision, but simply selects the max between whatever generation was deemed condemned before and its own selection. Nonetheless we know the final generation is chosen correctly, as the number of the condemned (final) generation is 2 in GCStats. But why isn't the GC acting on this decision ?

Thank you for reporting the issue to us. We have been working on the issue for a while now. The runtime support for recording the actual reasons in the runtime is already merged, and the tooling support for displaying the reasons is also mostly done, the remaining work is simply polishing the UI.
Once these changes are merged, it will be available in the next version of .NET Core and PerfView. This should make the data more accurately describe what is happen in the runtime.
For your question 1, the case for #1106 and #1108 is probably avoid_unproductive. My changes will reflect this information to the trace.
For your question 2, when the user code is trying to allocate a large object and the GC cannot fit it in the current free list, it will trigger a GC, trying to find space for the object. Note that AllocLarge is a trigger reason, and the trigger reasons are separated from the condemned reasons. Trigger reasons describe why a GC is triggered, while condemned reasons describe why the GC chose a certain generation to condemn.

I've reached out to Maoni, and she was kind enough to point out that the behavior observed in this question is due to a tuning logic that prevents consecutive gen2 GCs from occurring, instead falling back to gen1 GCs.
There's work underway to get this backing-away-from-gen2-GCs behavior logged appropriately using ETW, at least using .NET Core, and Maoni opened an issue specifically for this here.

Counting total objects queued for garbage collection

I wanted to add a small debug UI to my OpenGL game, which will be updated frequently with various debugging options/output displays. One thing I wanted was a constant counter that shows active objects in each generation of the garbage collector. I don't want names or anything, just a total count; something that I can eyeball when I do certain things within the game.
My problem, however, is that I can't seem to find a way to count the total objects currently alive in the various generations.
I even considered keeping a global static field, which would be incremented within every constructor and decremented within class finalizers. This would require hand-coding said functionality into every class though, and would not solve the problem of a "per-generation total".
Do you know how I could go about doing this?

(Question title:) "Counting total objects queued for garbage collection"
(From the question's body:) "My problem, however, is that I can't seem to find a way to count the total objects currently alive in the various generations."
Remark: Your question's title and body ask for opposite things. In the title, you're asking for the number of objects that can no longer be reached via any GC root, while in the body, you're asking for "live" objects, i.e. those that can still be reached via any GC root.
Let me start by saying that there might not be any way to do this, basically because objects in .NET are not reference-counted, so they cannot be immediately marked as "no longer needed" when the last reference to them disappears or goes out of scope. I believe .NET's mark-and-compact garbage collector only discovers which objects are alive and which can be reclaimed during an actual garbage collection (during the "mark" phase). You however seem to want this information in advance, i.e. before a GC occurs.
That being said, here are perhaps your best options:
Perhaps your best bet in .NET's managed Framework Class Library are performance counters. But it doesn't look like there are any suitable counters available: There are performance counters giving you the number of allocated bytes in the various GC generations, but AFAIK no counters for the number of live/dead objects.
You might also want to take a look at the CLR's (i.e. the runtime's) unmanaged, COM-based Debugging API. Given that you have retrieved an ICorDebugProcess5 interface, these methods might be of interest:
ICorDebugProcess5::EnumerateGCReferences method:
"Gets an enumerator for all objects that are to be garbage-collected in a process."
See also this answer to a similar question on SO.
Note that this is about objects that are to be garbage-collected, not about live objects.
ICorDebugProcess5::GetGCHeapInformation method:
"Provides general information about the garbage collection heap, including whether it is currently enumerable."
If it turns out that the managed heap is enumerable, you could use…
ICorDebugProcess5::EnumerateHeap method:
"Gets an enumerator for the objects on the managed heap."
The objects returned by this enumerator are of this type:
COR_HEAPOBJECT structure:
"Provides information about an object on the managed heap."
You might not be actually interested in these details, but just in the number of objects returned by the enumerator.
(I haven't used this API myself, perhaps there exists a better and more efficient way.)
In Sept 2015, Microsoft published a managed library called clrmd aka Microsoft.Diagnostics.Runtime on GitHub. It is based on the same foundation as the unmanaged debugging API mentioned above. The project includes documentation about enumerating objects in the GC heap.
Btw. there is an extremely informative book out there by Ben Watson, "Writing High-Performance .NET Code", which includes solid tips on how to make .NET memory allocation and GC more efficient.

Garbage Collector doesn't have to collect objects.
... that fact will be discovered when the garbage collector
runs the collector for whatever generation the object was in. (If it
runs at all, which it might not. There is no guarantee that the GC
runs.)
(C) Eric Lippert
If the application performs normally and the memory consumption is not increasing the GC can let it work without interruptions. That means that numbers will differ from run to run.
If I were you I wouldn't spend time on getting generations information, but just the size of used memory.
The simple but not very accurate way is to get it from GC.
// Determine the best available approximation of the number
// of bytes currently allocated in managed memory.
Console.WriteLine("Total Memory: {0}", GC.GetTotalMemory(false));
If you see that used memory increases and decreases often, then you can use existing profilers to figure out where are you allocating too mush, or even where the memory leak is.

.NET Free memory usage (how to prevent overallocation / release memory to the OS)

I'm currently working on a website that makes large use of cached data to avoid roundtrips.
At startup we get a "large" graph (hundreds of thouthands of different kinds of objects).
Those objects are retrieved over WCF and deserialized (we use protocol buffers for serialization)
I'm using redgate's memory profiler to debug memory issues (the memory didn't seem to fit with how much memory we should need "after" we're done initializing and end up with this report
Now what we can gather from this report is that:
1) Most of the memory .NET allocated is free (it may have been rightfully allocated during deserialisation, but now that it's free, i'd like for it to return to the OS)
2) Memory is fragmented (which is bad, as everytime i refresh the cash i need to redo the memory hungry deserialisation process and this, in turn creates large object that may throw an OutOfMemoryException due to fragmentation)
3) I have no clue why the space is fragmented, because when i look at the large object heap, there are only 30 instances, 15 object[] are directly attached to the GC and totally unrelated to me, 1 is a char array also attached directly to the GC Heap, the remaining 15 are mine but are not the cause of this as i get the same report if i comment them out in code.
So my question is, what can i do to go further with this? I'm not really sure what to look for in debugging / tools as it seems my memory is fragmented, but not by me, and huge amounts of free spaces are allocated by .net , which i can't release.
Also please make sure you understand the question well before answering, i'm not looking for a way to free memory within .net (GC.Collect), but to free memory that is already free in .net , to the system as well as to defragment said memory.
Note that a slow solution is fine, if it's possible to manually defragment the large heap i'd be all for it as i can call it at the end of RefreshCache and it's ok if it takes 1 or 2 second to run.
Thanks for your help!
A few notes i forgot:
1) The project is a .net 2.0 website, i get the same results running it in a .net 4 pool, idem if i run it in a .net 4 pool and convert it to .net 4 and recompile.
2) These are results of a release build, so debug build can not be the issue.
3) And this is probably quite important, i do not get these issues at all in the webdev server, only in IIS, in the webdev i get memory consumption rather close to my actual consumption (well more, but not 5-10X more!)

Objects allocated on the large object heap (objects >= 85,000 bytes, normally arrays) are not compacted by the garbage collector. Microsoft decided that the cost of moving those objects around would be too high.
The recommendation is to reuse large objects if possible to avoid
fragmentation on the managed heap and the VM space.
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
I'm assuming that your large objects are temporary byte arrays created by your deserialization library. If the library allows you to supply your own byte arrays, you could preallocate them at the start of the program and then reuse them.

I know this isn't the answer you'd like to hear, but you can't forcefully release the memory back to the OS. However, for what reason do you want to do so? .NET will free its heap back to the OS once you're running low on physical memory. But if there's an ample amount of free physical memory, .NET will keep its heap to make future allocation of objects faster. If you really wanted to force .NET to release its heap back to the OS, I suppose you could write a C program which just mallocs until it runs out of memory. This should cause the OS to signal .NET to free its unused portion of the heap.
It's better that unused memory be reseved for .NET so that your application will have better allocation performance (since the runtime knows what memory is free and what isn't, allocation can just use the free memory without having to syscall into the OS to get more memory).
The garbage collector is in charge of defragmenting the heap. Every so often (usually during collection runs), it will move objects around the heap if it determines this needs to be done. (This is why C++/CLI has the pin_ptr construct for "pinning" objects).
Fragmentation usually isn't a big issue though with memory, since it provides fast random access.
As for your OutOfMemoryException, I don't have a good answer for. Ordinarily I'd suspect that your old object graph isn't being collected (some object somewhere is holding a reference onto it, a "memory leak"). But since you're using a profiler, I don't know then.

As of .NET 4.5.1 you can set a one-time flag to compact LOH before issuing a call to GC collect, i.e.
Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect(); // This will cause the LOH to be compacted (once).

Some testing and some C++ later, i've found the reason why i get so much free memory, it's because of IIS instancing the CLR via VM Hoarding (providing a dll to instantiate it without VM Hoarding takes up as much initial memory, but does release most of it as time goes which is the behavior i expect).
So this does fix my reported memory issue, however i still get about 100mb free memory no matter what, and i still think this is due to fragmentation and fragments only being released at once, because the profiler still reports memory fragmentation. So not marking my own answer as an answer in hope someone can shed some light on this or direct me to tools that can either fix this or help me debug the root cause.

It's intriguing that it works differently on the WebDevServer as to IIS...
Is it possible that IIS is using the server garbage-collector, and the WebDev server the workstation garbage collector? The method of garbage collection can affect fragmentation. It'll probably be set in your aspnet.config file. See: http://support.microsoft.com/kb/911716

If you havent found your answer I think the following clues can help you :
Back to the basics : we sometimes forget that the objects can be explicitly set free, call explicitly the Dispose method of the objects (because you didnt mention it, I suppose you do an "object = null" instruction instead).
Use the inherited method, you dont need to implement one, unless your class doesnt have it, which I doubt it.
MSDN Help states about this method :
... There is no performance benefit in implementing the Dispose
method on types that use only managed resources (such as arrays)
because they are automatically reclaimed by the garbage collector. Use
the Dispose method primarily on managed objects that use native
resources and on COM objects that are exposed to the .NET
Framework. ...
Because it says that "they are automatically reclaimed by garbage collector" we can infer that when the method is called does the "releasing thing" (Again Im trying only to give you clues).
Besides I found this interesting article (I suppose ... I didn read it ...completely) : Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework (http://msdn.microsoft.com/en-us/magazine/bb985010.aspx) which states the following in the "Forcing an Object to Clean Up" section :
..., it is also recommended that you add an additional method to
the type that allows a user of the type to explicitly clean up the
object when they want. By convention, this method should be called
Close or Dispose ....
Maybe the answer lies in this article if you read it carefully or just keep investigating in this direction.

C# memory leak?? Gen 0 and 1 increasing constantly in perfmon - What does it mean?

I am using C# 2.0 for a multi-threaded application that receives atleast thousand callbacks per second from an unmanaged dll and periodically send messages out of socket. GUI remains on main thread.
My application mostly creates object at the startup and periodically during the execution for a short lived period.
The problem I am experiencing is periodic latency spike (measured by time stamping a function at start and end) which I suppose happen when GC run.
I ran perfmon and here are my observations...
Gen0 Heap Size is flat with a spike every few seconds with periodic spike.
Gen1 Heap Size is always on the roll. Up and down
Gen2 Heap Size follows a cycle. It keep increasing till it becomes flat for a while and then drops.
Gen 0 and 1 Collections are always increasing in a range of 1 to 5 units.
Gen 2 collections is constant.

I recommend using a memory profiler in order to know if you have a real memory leak or not. There are many available and they will allow you to quickly isolate any issue.
The garbage collector is adaptive and will modify how often it runs in response to the way your application is using memory. Just looking at the generation heap sizes is going to tell you very little in terms of isolating the source of any problem. Second quessing how it works is a bad idea.
RedGate Ants Memory Profiler
SciTech .NET Memory Profiler
EQATEC .NET Profiler
CLR Profiler (Free)

So as #Jalf says, there's no evidence of a memory "leak" as such: what you discuss is closer to latency caused by garbage collection.
Others may disagree but I'd suggest anything above a few hundred callbacks per second is going to stretch a general purpose language like C#, especially one that manages memory on your behalf. So you're going to have to get smart with your memory allocation and give the runtime some help.
First, get a real profiler. Perfmon has its uses but even the profiler in later versions of Visual Studio can give you much more information. I like the SciTech profiler best (http://memprofiler.com/); there are others including a well respected one from RedGate reviewed here: http://devlicio.us/blogs/scott_seely/archive/2009/08/23/review-of-ants-memory-profiler.aspx
Once you know your baseline, aim to eliminate gen2 collections. They'll be the really slow ones. Work hard in any tight loops to eliminate as much memory allocation as you can -- strings are the usual offenders.
Some useful tips are in an old but still relevant MSDN article here: http://msdn.microsoft.com/en-us/library/ms973837.aspx.
It's also good to read Tess Ferrandez's (outstanding) blog series on debugging ASP.NET applications - book a day out of the office and start here: http://blogs.msdn.com/b/tess/archive/2008/02/04/net-debugging-demos-information-and-setup-instructions.aspx.

I remember reading a blog post about memory performance in .NET (specifically, XNA on the XBox 360) a while ago (unfortunately I can't find said link anymore).
The nutshell of achieving low latency memory performance was to make sure you never run gen 2 GC's at a performance critical time (although it is OK to run them when latency is not important; there are a bunch of notification callback functions on the GC class in more recent versions of the framework which may help with this). There are two ways to make sure this happens:
Don't allocate anything that escapes to gen 2. It's alarmingly easy for objects to escape to gen 2 when you don't realise it, so this often translates into: don't allocate anything in performance critical code. Because no objects escape to gen 2, the GC doesn't need to collect.
Allocate everything you need upfront and use object pooling. Your gen 2 heap will be big but because nothing is being added to it, the GC doesn't need to collect it.
It may be helpful to look into some XNA or Silverlight related performance articles because
games and resource constrained devices are often very latency sensitive. (Note that you have it easy because the XBox 360 and, until Mango, Windows Phone only had a single generation GC (mark-and-sweep collector)).

Long running windows service with a memory leak? Or just looks like one?

This sounds a little odd, but I'm not sure if my process has a memory leak or not, and I was hoping to get some information.
I was recently assigned an investigation into why a windows service in production was consuming roughly a gig of ram (the server it is running on has 8 gigs). This is outside of my experience as a developer, but it's been a very good chance for me to read up on how garbage collection is working on msdn and other sources. But at this point I'm very confused about /when/ collection actually runs, on this point, ever article I read is vague.
1) I have found a specific operation that increases memory by ~30kb each time it is executed.
2) I have very carefully gone over the code and believe that I am properly closing everything, and removing references
3) I have used several memory profilers, all of them seem to indicate that my old objects are linked to the gc.
4) If I leave the process absolutely idle for a few days, the memory usage suddenly plummets down to ~8 megs
So based on this, I'm not even sure I have a memory leak. Given that GC is an expensive process, is it possible that I grew to 1 gig in production just because there was still free ram to be had, and acquiring it was "cheaper" than running GC? Especially given that this service is run ~6 times a second? If this is the case, what options do I have? I am of the understanding that I cannot forceably trigger GC, do I have any resort?
Thank you for any input you might have, I realize memory leaks and gc in csharp is a deep topic and if there's a particularly helpful read on the subject, I'd be happy to be pointed that way as well.

You certainly CAN force a garbage collection - just call GC.Collect. It's not that you can't but that the garbage collector usually does a better job of figuring out when it should run than you do. But here, you can call it explicitly - as a debugging tool - to see whether or not the allocated memory is eligible for collection.

Memory leak usually implies that memory is never deallocated and process eventually crashes with OutOfMemoryException. You are saying that it does get deallocated after a while
4) If I leave the process absolutely idle for a few days, the memory usage suddenly plummets down to ~8 megs
You certainly can force garbage collection using GC.Collect. But as others have said it is not a good long term solution. I highly recommend you to read about garbage collection in this book. If you still convinced that you have memory leak you can create a dump of the process in production environment using Process Explorer. And analyze it later using WinDbg. Unless you can use dotTrace or ANTS on production which would be a lot easier.