Condemned Generation is Not Garbage-Collected - c#

I'm analyzing the outcome of some simple C# code, in terms of GC performance. I've used PerfView with GC Collect Only ticked to gather meaningful info.
Consider the following relevant output from PerfView's GCStats, particularly GC #1106 and #1108:
Both #1106 and #1108 are listed as gen1 Non-concurrent GCs. Both have a reason of AllocLarge.
Now consider the associated Condemned Reasons table:
In a blogpost by Maoni Stephens, the owner for the .NET GC, we know that once a GC is started, it can act against a higher gen, as follows:
Now after the GC starts, we then decide what generation we would
actually collect. It might stay as a gen0 GC, or get escalated to a
gen1 or even a gen2 GC – this is something we decide as one of the
very first things we do in a GC. And factors that would cause us to
escalate to a high generation GC are what we called “condemned
reasons” (so for a GC there’s only one trigger reason but can be
multiple condemned reasons).
Based on the Condemned Reasons table, it turns out both #1106 and #1108 "elevate" themselves from gen0 GCs to gen2 GCs due to both breaking the threshold for allocations in gen2.
But how come:
1) Both #1106 and #1108 start off as gen0 GCs, get eventually "escalated" to gen2, but show up in GCStats as gen1 GCs ? Please note that this is not some PerfView issue, as the detailed ETW events show the same exact data in terms of generations.
2) Both #1106 and #1108 get registered as gen1 GCs, but with a reason of AllocLarge ? AllocLarge refers to the large object allocation threshold exceeded, and the LOH can only be processed as part of gen2 collections.
The data above was captured on a .NET Framework 4.7.2 running on a Windows 10 x64. I'm assuming though that the concepts behind the GC decisions would be largely the same for other, relatively recent versions of .NET Framework, or even .NET Core, so this should make little difference to what's being asked here.
I've also not burdened the question with the actual code, since what's causing the GCs to occur in the form explained above is not triggered by any special instruction, but by the inner decisions the GC takes.
Update 1: I've provided more data to the initial question. I've also specifically looked at and couldn't find any other Microsoft-Windows-DotNETRuntime* ETW event around GCs #1106 and #1108 referenced that could shed more light on either question.
Update 2: I've changed the name of the question, since there are no gen2 GCs occurring after all. The CLR is performing only gen1 GCs for #1106 and #1108, as the column for the gen2 survival rate states NaN, which implies no gen2 GC has run.
Looking over the .NET Core CLR source code (for inspiration as what might happen in .NET Framework), the function that does the actual GC work gc_heap::gc1 reads early on the condemned generation info (below), and provided this is a non-concurrent GC (similar to our #1106 and #1108) it proceeds with the mark phase against the respective generation.
As for the chosen condemned generation, within gc_heap::generation_to_condemn the check for the highest gen budget allocations exceeded is done before the check for low ephemeral segment, and the latter doesn't overwrite the decision, but simply selects the max between whatever generation was deemed condemned before and its own selection. Nonetheless we know the final generation is chosen correctly, as the number of the condemned (final) generation is 2 in GCStats. But why isn't the GC acting on this decision ?

Thank you for reporting the issue to us. We have been working on the issue for a while now. The runtime support for recording the actual reasons in the runtime is already merged, and the tooling support for displaying the reasons is also mostly done, the remaining work is simply polishing the UI.
Once these changes are merged, it will be available in the next version of .NET Core and PerfView. This should make the data more accurately describe what is happen in the runtime.
For your question 1, the case for #1106 and #1108 is probably avoid_unproductive. My changes will reflect this information to the trace.
For your question 2, when the user code is trying to allocate a large object and the GC cannot fit it in the current free list, it will trigger a GC, trying to find space for the object. Note that AllocLarge is a trigger reason, and the trigger reasons are separated from the condemned reasons. Trigger reasons describe why a GC is triggered, while condemned reasons describe why the GC chose a certain generation to condemn.

I've reached out to Maoni, and she was kind enough to point out that the behavior observed in this question is due to a tuning logic that prevents consecutive gen2 GCs from occurring, instead falling back to gen1 GCs.
There's work underway to get this backing-away-from-gen2-GCs behavior logged appropriately using ETW, at least using .NET Core, and Maoni opened an issue specifically for this here.

Related

Tracking Down a .NET Windows Service Memory Leak

Before installing my windows service in production, I was looking for reliable tests that I can perform to make sure my code doesn't contain memory leaks.
However, All what I can find on the net was using task manager to look at used memory or some paid memory profiler tools.
From my understanding, looking at the task manager is not really helpful and cannot confirm the memory leakage (in case, there is).
How to confirm whether there is a memory leak or not?
Is there any free tools to find the source of memory leaks?
Note: I'm using .Net Framework 4.6 and Visual Studio 2015 Community
Well you can use task manager.
GC apps can leak memory, and it will show there.
But...
Free tool - ".Net CLR profiler"
There is a free tool, and it's from Microsoft, and it's awesome. This is a must-use for all programs that leak references. Search MS' site.
Leaking references means you forget to set object references to null, or they never leave scope, and this is almost as likely to occur in Garbage collected languages as not - lists building up and not clearing, event handlers pointing to delegates, etc.
It's the GC equivalent of memory leaks and has the same result. This program tells you what references are taking up tons of memory - and you will know if it's supposed to be that way or not, and if not, you can go find them and fix the problem!
It even has a cool visualization of what objects allocate what memory (so you can track down mistakes). I believe there are youtubes of this if you need an explanation.
Wikipedia page with download links...
NOTE: You will likely have to run your app not as a service to use this. It starts first and then runs your app. You can do this with TopShelf or by just putting the guts in a dll that runs from an EXE that implments the service integrations (service host pattern).
Although managed code implies no direct memory management, you still have to manage your instances. Those instances 'claim' memory. And it is all about the usage of these instances, keeping them alive when you don't expect them to be.
Just one of many examples: wrong usage of disposable classes can result in a lot of instances claiming memory. For a windows service, a slow but steady increase of instances can eventually result in to much memory usage.
Yes, there is a tool to analyze memory leaks. It just isn't free. However you might be able to identify your problem within the 7 day trial.
I would suggest to take a loot at the .NET Memory Profiler.
It is great to analyze memory leaks during development. It uses the concept of snapshots to compare new instances, disposed instances etc. This is a great help to understand how your service uses its memory. You can then dig deeper into why new instances get created or are kept alive.
Yes, you can test to confirm whether memory leaks are introduced.
However, just out-of-the box this will not be very useful. This is because no one can anticipate what will happen during runtime. The tool can analyze your app for common issues, but this is not guaranteed.
However, you can use this tool to integrate memory consumption into your unit test framework like NUnit or MSTest.
Of course a memory profiler is the first kind of tool to try, but it will only tell you whether your instances keep increasing. You still want to know whether it is normal that they are increasing. Also, once you have established that some instances keep increasing for no good reason, (meaning, you have a leak,) you will want to know precisely which call trees lead to their allocation, so that you can troubleshoot the code that allocates them and fix it so that it does eventually release them.
Here is some of the knowledge I have collected over the years in dealing with such issues:
Test your service as a regular executable as much as possible. Trying to test the service as an actual service just makes things too complicated.
Get in the habit of explicitly undoing everything that you do at the end of the scope of that thing which you are doing. For example, if you register an observer to the event of some observee, there should should always be some point in time (the disposal of the observer or the observee?) that you de-register it. In theory, garbage collection should take care of that by collecting the entire graph of interconnected observers and observees, but in practice, if you don't kick the habit of forgetting to undo things that you do, you get memory leaks.
Use IDisposable as much as possible, and make your destructors report if someone forgot to invoke Dispose(). More about this method here: Mandatory disposal vs. the "Dispose-disposing" abomination Disclosure: I am the author of that article.
Have regular checkpoints in your program where you release everything that should be releasable (as if the program is performing an orderly shutdown in order to terminate) and then force a garbage collection to see whether you have any leaks.
If instances of some class appear to be leaking, use the following trick to discover the precise calling tree that caused their allocation: within the constructor of that class, allocate an exception object without throwing it, obtain the stack trace of the exception, and store it. If you discover later that this object has been leaked, you have the necessary stack trace. Just don't do this with too many objects, because allocating an exception and obtaining the stack trace from it is ridiculously slow, only Microsoft knows why.
You could try the free Memoscope memory profiler
https://github.com/fremag/MemoScope.Net
I do not agree that you can trust the Task Manager to check if you have a memory leak or not. The problem with a garbage collector is that it can decide based on heuristics to keep the memory after a memory spike and do not return it to the OS. You might have a 2 GB Commit size but 90% of them can be free.
You should use VMMAP to check during the tests what type of memory your process contains. You do not only have the managed heap, but also unmanaged heap, private bytes, stacks (thread leaks), shared files and much more which need to be tracked.
VMMap has also command line interface which makes it possible to create snapshots at regular intervals which you can examine later. If you have a memory growth you can find out which type of memory is leaked which needs depending on the leak type different debugging tooling approaches.
I would not say that the Garbage collector is infallible. There are times when it fails unknowingly and they are not so straight forward. Memory streams are a common cause of memory leaks. You can open them in one context and they may never even get closed, even though the usage is wrapped in a using statement (the definition of a disposable object that should be cleaned up immediately after its usage falls out of scope). If you are experiencing crashes due to running out of memory, Windows does create dump files that you can sift through.
enter link description here
This is by no means fun or easy and is quite tedious but it tends to be your best bet.
Common areas that are easy to create memory leaks are anything that is using the System.Drawing dll, memory streams, and if you are doing some serious multi-threading.
If you use Entity Framework and a DI pattern, perhaps using Castle Windsor, you can easily get memory leaks.
The main thing to do is use the using( ){ } statement where-ever you can to automatically mark objects as disposed.
Also, you want to turn off automatic tracking on Entity Framework where you are only reading and not writing. Best to isolate your writes, use a using() {} at this point, get a dbContext (with tracking on), write your data.
If you want to investigate what is on the heap. The best tool I've used is RedGate ANTS http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/solving-memory-problems/getting-started not cheap but it works.
However, by using the using() {} pattern where-ever you can (don't make a static or singleton DbContext and never have one context in a massive loop of updates, dispose of them as often as you can!) then you find memory isn't often an issue.
Hope this helps.
Unless you're dealing with unmanaged code, i would be so bold to say you don't have to worry about memory leaks. Any unreferenced object in managed code will be removed by the garbage collector, and the possibility in finding a memory leak within the .net framework i would say you should be considered very lucky (well, unlucky). You don't have to worry about memory leak.
However, you can still encounter ever-growing memory usage, if references to objects are never released. For example, say you keep an internal log structure, and you just keep adding entries to a log list. Then every entry still have references from the log list and therefore will never be collected.
From my experience, you can definitely use the task manager as an indicator whether your system has growing issues; if the memory usage steadily keep rising, you know you have an issue. If it grows to a point but eventually converges to a certain size, it indicates it has reached its operating threshold.
If you want a more detailed view of managed memory usage, you can download the process explorer here, developed by Microsoft. It is still quite blunt, but it gives a somewhat better statistical view than task manager.

Gen2 collection not always collecting dead objects?

By monitoring the CLR #Bytes in all Heaps performance counter of a brand new .NET 4.5 server application over the last few days, I can notice a pattern that makes me think that Gen2 collection is not always collecting dead objects, but I am having trouble understanding what exactly is going on.
Server application is running in .NET Framework 4.5.1 using Server GC / Background.
This is a console application hosted as a Windows Service (with the help of Topshelf framework)
The server application is processing messages, and the throughput is somehow pretty constant for now.
What I can see looking at the graph of CLR #Bytes in all Heaps is that the memory started arround 18MB then growing up to 35MB on approx 20-24 hours (with between 20-30 Gen2 collections during that time frame), and then all of a sudden dropping back to nominal value of 18MB, then growing again up to ~35MB over 20-24 hours and dropping back to 18MB, and so on (I can see the pattern repeating over the last 6 days the app is now running) ... The growing of memory is not linear, it takes approx 5 hours to grow by 10MB and then 15-17 hours for the remaining 10 MB or so.
Thing is that I can see by looking at perfmon counters for #Gen0/#Gen1/#Gen2 collections that a bunch of Gen2 collections are going on during the 20-24 hours period (maybe arround 30) and none of them makes the memory drop back to nominal 18MB.
However, what is strange is by using an external tool to force a GC (Perfview in my case), then I can see #Induced GC going up by 1 (GC.Collect was called so this is normal) and immediately the memory is going back to nominal 18MB.
Which leads me into thinking that either the perfmon counter for #Gen2 collections is not right and only a single Gen2 collection happens after 20-22hours or so (meeehhh I really don't think so) or that the Gen2 collection does not always collect dead objects (seems more plausible) ... but in that case why would forcing a GC via GC.Collect do the trick, what would be the difference between explicitely calling into GC.Collect, v.s automatic triggered collections during the lifetime of the application.
I am sure there is a very good explanation but from the different source of documentation I have found about GC -too few :(- a Gen2 collection does collect dead objects in any case. So maybe docs are not up to date or I have misread ... Any explanation is welcome. Thanks !
EDIT : Please see this screenshot of the #Bytes in all heaps graph over 4 days
(Click for larger view)
this is easier than trying to graph things in your head. What you can see on the graph is what I said above... memory increasing over 20-24hours (and 20-30 Gen2 collections during that time frame) until reaching ~35MB then dropping all of a sudden. You will note at the end of the graph, the induced GC I triggered via an external tool, immediately dropping back memory to nominal.
EDIT #2 : I made a lot of cleaning in the code, mainly regarding finalizers. I had a lot of classes that were holding reference to disposable types, so I had to implement IDisposable on these types. However I was misguided by some articles into implementing the Diposable pattern with a Finalizer in any case. After reading some MSDN documentation I came to understand that a finalizer was only required when the type was holding native resources itself (and still in that case this could be avoided with SafeHandle). So I removed all finalizers from all these types. There were some other modications in the code, but mainly business logic, nothing ".NET framework" related.
Now the graph is very different, this is a flat line arround 20MB for days now ... exactly what I was expecting to see !
So the problem is now fixed, however I still have no idea what was the problem due to ... It seems like it might have been related to finalizers but still does not explain what I was noticing, even if we weren't calling Dispose(true) -suppressing finalizer-, the finalizer thread is supposed to kick in between collection and not every 20-24 hours ?!
Considering we have now moved away from the problem, it will take time to come back to the "buggy" version and reproduce it again. I may try to do it some time though and go to the bottom of it.
EDIT: Added Gen2 collection graph (Click for larger view)
From
http://msdn.microsoft.com/en-us/library/ee787088%28v=VS.110%29.aspx#workstation_and_server_garbage_collection
Conditions for a garbage collection
Garbage collection occurs when one of the following conditions is
true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously
adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs
continuously. This method is primarily used for unique situations and
testing.
It seems that you are hitting the 2nd one and 35 is the threshold. You should be able to configure the threshold to something else if 35 is to large.
There isn't anything special about gen2 collections that would cause them to deviate from these rules. (cf https://stackoverflow.com/a/8582251/215752)
Are any of your objects "large" objects? there's a separate "large object heap" which has different rules
Large Object Heap Fragmentation
It was improved in 4.5.1, though:
http://blogs.msdn.com/b/dotnet/archive/2011/10/04/large-object-heap-improvements-in-net-4-5.aspx
This could easily be explained if gcTrimCommitOnLowMemory is enabled. Normally, the GC keeps some extra memory allocated to the process. However, when the memory reaches a certain threshold, the GC will then "trim" the extra memory.
From the docs:
When the gcTrimCommitOnLowMemory setting is enabled, the garbage collector evaluates the system memory load and enters a trimming mode when the load reaches 90%. It maintains the trimming mode until the load drops under 85%.
This could easily explain your scenario - the memory reserves are being kept (and used) until your application reaches a certain point, which seems to be once every 20-24 hours, at which point the 90% load is detected, and the memory is trimmed to its minimum requirements (the 18mb).
Reading your first version I would say that is a normal behavior.
...but in that case why would forcing a GC via GC.Collect do the
trick, what would be the difference between explicitely calling into
GC.Collect, v.s automatic triggered collections during the lifetime of
the application.
There is two type of collections, a full collection and a partial collection. What the automatic triggered does is a partial collection, but when calling GC.Collect it will do a full collection.
Meanwhile, I might have the reason of it now that you told us that you were using finalizer on all of your objects. If for any reason one of those objects were promoted to #2 Gen, the finalizer would only run when doing a #2 Gen collection.
The following example will demonstrate what I just said.
public class ClassWithFinalizer
{
~ClassWithFinalizer()
{
Console.WriteLine("hello from finalizer");
//do nothing
}
}
static void Main(string[] args)
{
ClassWithFinalizer a = new ClassWithFinalizer();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
GC.Collect();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
GC.Collect();
Console.WriteLine("Class a is on #{0} generation", GC.GetGeneration(a));
a = null;
Console.WriteLine("Collecting 0 Gen");
GC.Collect(0);
GC.WaitForPendingFinalizers();
Console.WriteLine("Collecting 0 and 1 Gen");
GC.Collect(1);
GC.WaitForPendingFinalizers();
Console.WriteLine("Collecting 0, 1 and 2 Gen");
GC.Collect(2);
GC.WaitForPendingFinalizers();
Console.Read();
}
The output will be:
Class a is on #0 generation
Class a is on #1 generation
Class a is on #2 generation
Collecting 0 Gen
Collecting 0 and 1 Gen
Collecting 0, 1 and 2 Gen
hello from finalizer
As you can see, only when doing a collection on the generation where the object is, the memory of the objects with finalizer will be reclaimed.
Just figure I'll throw in my 2 cents here. I'm not an expert at this but maybe this might help your investigation.
If you're using a 64-bit platform try adding this to your .config file. I read that that can be an issue.
<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>
The only other thing I would point out is that you could prove your hypothesis by troubleshooting from the inside if you are in control of the source code.
Calling something along the lines of this your app's main memory consuming class, and setting it to run on timed intervals, could shed some light onto what's really going on.
private void LogGCState() {
int gen = GC.GetGeneration(this);
//------------------------------------------
// Comment out the GC.GetTotalMemory(true) line to see what's happening
// without any interference
//------------------------------------------
StringBuilder sb = new StringBuilder();
sb.Append(DateTime.Now.ToString("G")).Append('\t');
sb.Append("MaxGens: ").Append(GC.MaxGeneration).Append('\t');
sb.Append("CurGen: ").Append(gen).Append('\t');
sb.Append("CurGenCount: ").Append(GC.CollectionCount(gen)).Append('\t');
sb.Append("TotalMemory: ").Append(GC.GetTotalMemory(false)).Append('\t');
sb.Append("AfterCollect: ").Append(GC.GetTotalMemory(true)).Append("\r\n");
File.AppendAllText(#"C:\GCLog.txt", sb.ToString());
}
Also there is a pretty good article here on using the GC.RegisterForFullGCNotification method. Obviously this would enable you to also include the time span of a full collection so that you could possibly tune performance and collection frequency to your specific needs. This method also let's you specify a heap threshold to to trigger notifications (or collections?).
There's also probably a way to set that in the apps .config file but I haven't looked. For the most part 35MB is a pretty small footprint for a Server Application these days. Heck, my web browser makes it up to 300-400MB sometimes :) So, the Framework might just see 35MB as a good default point to free up memory.
Anyhow, I can tell by the thoughtfulness of your question that I'm probably just pointing out the obvious. But, it feels worth mentioning. I wish you luck!.
On a funny note
At the top of this post I had originally written "if (you're using a 64-bit platform)". That made me crack up. Take care!
I have exactly the same situation in my WPF application. No finalizers in my code btw. However it seems that ongoing GC actually collects Gen 2 objects. I can see that GC.GetTotalMemory() results reduces up to 150mb after Gen2 collection triggered.
So I'm under impression that Gen2 heap size does not show amount of bytes that is used by live objects. It is rather just a heap size or amount of bytes that is allocated for Gen2 purposes. You may have plenty of free memory there.
Under some conditions (not on each gen 2 collection) this heap size is trimmed. And at this particular moment my application gets a huge performance hit - it may hung up to sevetal seconds. Wondering why...

How can I tell if the .Net 3.5 garbage collector has run?

I have an application that creates trees of nodes, then tosses them and makes new trees.
The application allocates about 20 MB upon startup. But I tried loading a large file of nodes many times, and the allocated memory went above 700 MB. I thought I would see memory being freed by the garbage collector on occasion.
The machine I'm on has 12 GB of RAM, so maybe it's just that such a "small" amount of memory being allocated doesn't matter to the GC.
I've found a lot of good info on how the GC works, and that it's best not to tell it what to do. But I wanted to verify that it's actually doing anything, and that I'm not somehow doing something wrong in the code that prevents my objects from being cleaned up.
The GC generally runs when either of the scenarios below occur:
You call GC.Collect (which you shouldn't)
Gen0's budget is exhausted
There are some other scenarios as well, but I'll skip those for now.
You didn't tell us how you measured the memory usage, but if you're looking at the memory usage of process itself (e.g. through task manager), then you may not see the numbers you expect. Remember that the .NET runtime essentially has its own memory manager that handles memory usage on behalf of you managed application. The runtime tries to be smart about it so it doesn't allocate and free memory to the OS all the time (those are expensive operations). This question may be relevant as well.
If you're concerned about memory leaks have a look at some of the answers here.
When does the .Net 3.5 garbage collector run?
I thought I would see memory being freed by the garbage collector on occasion.
Since the GC is non-deterministic, you won't be able to necessarily determine when it is going to issue a collection. Short answer: It will run when needed. Trying to analyze your code and predict or assume it should be running at a certain time usually ends up down a rabbit hole.
Answer to: do I leak objects or GC have not need to run yet?
Use memory profiler to see what objects are allocated. As basic step - force garbage collection (GC.Collect) and check out if allocated memory (GC.GetTotalMemory) seems to be reasonable.
If you want to make sure that you're not leaving any unwanted object behind you can use dotTrace memory profiler. It allows you to take two snapshots of objects in memory (taken some time apart) and compare them. You will can clearly see if any old nodes are still hanging around and what is keeping a reference to them and stops them from being collected.
You may have the managed equivalent of a memory leak. Are you maintaining stale references to these objects (i.e., do you have a List<T> or some other object which tracks these nodes)?
Are the subscribing to an event of an object that is not going out of scope? A reference to the subscribee of an event is maintained, so if you don't detach it will keep your objects alive.
You may also be forgetting to Dispose of objects that implement IDisposable. Can't say without seeing your code.
The exact behavior of the GC is implementation defined. You should design your application such that it does not matter. If you need deterministic memory management then you are using the wrong language. Use a tool (RedGate's ANTS profiler will do) to see if you are leaking references somewhere.
This comic is the best way to explain this topic :)
You can monitor the garbage collector activity in .NET 4.0 and beyond with GC.RegisterForFullGCNotification as described in this link: http://www.abhisheksur.com/2010/08/garbage-collection-notifications-in-net.html

C# memory leak?? Gen 0 and 1 increasing constantly in perfmon - What does it mean?

I am using C# 2.0 for a multi-threaded application that receives atleast thousand callbacks per second from an unmanaged dll and periodically send messages out of socket. GUI remains on main thread.
My application mostly creates object at the startup and periodically during the execution for a short lived period.
The problem I am experiencing is periodic latency spike (measured by time stamping a function at start and end) which I suppose happen when GC run.
I ran perfmon and here are my observations...
Gen0 Heap Size is flat with a spike every few seconds with periodic spike.
Gen1 Heap Size is always on the roll. Up and down
Gen2 Heap Size follows a cycle. It keep increasing till it becomes flat for a while and then drops.
Gen 0 and 1 Collections are always increasing in a range of 1 to 5 units.
Gen 2 collections is constant.
I recommend using a memory profiler in order to know if you have a real memory leak or not. There are many available and they will allow you to quickly isolate any issue.
The garbage collector is adaptive and will modify how often it runs in response to the way your application is using memory. Just looking at the generation heap sizes is going to tell you very little in terms of isolating the source of any problem. Second quessing how it works is a bad idea.
RedGate Ants Memory Profiler
SciTech .NET Memory Profiler
EQATEC .NET Profiler
CLR Profiler (Free)
So as #Jalf says, there's no evidence of a memory "leak" as such: what you discuss is closer to latency caused by garbage collection.
Others may disagree but I'd suggest anything above a few hundred callbacks per second is going to stretch a general purpose language like C#, especially one that manages memory on your behalf. So you're going to have to get smart with your memory allocation and give the runtime some help.
First, get a real profiler. Perfmon has its uses but even the profiler in later versions of Visual Studio can give you much more information. I like the SciTech profiler best (http://memprofiler.com/); there are others including a well respected one from RedGate reviewed here: http://devlicio.us/blogs/scott_seely/archive/2009/08/23/review-of-ants-memory-profiler.aspx
Once you know your baseline, aim to eliminate gen2 collections. They'll be the really slow ones. Work hard in any tight loops to eliminate as much memory allocation as you can -- strings are the usual offenders.
Some useful tips are in an old but still relevant MSDN article here: http://msdn.microsoft.com/en-us/library/ms973837.aspx.
It's also good to read Tess Ferrandez's (outstanding) blog series on debugging ASP.NET applications - book a day out of the office and start here: http://blogs.msdn.com/b/tess/archive/2008/02/04/net-debugging-demos-information-and-setup-instructions.aspx.
I remember reading a blog post about memory performance in .NET (specifically, XNA on the XBox 360) a while ago (unfortunately I can't find said link anymore).
The nutshell of achieving low latency memory performance was to make sure you never run gen 2 GC's at a performance critical time (although it is OK to run them when latency is not important; there are a bunch of notification callback functions on the GC class in more recent versions of the framework which may help with this). There are two ways to make sure this happens:
Don't allocate anything that escapes to gen 2. It's alarmingly easy for objects to escape to gen 2 when you don't realise it, so this often translates into: don't allocate anything in performance critical code. Because no objects escape to gen 2, the GC doesn't need to collect.
Allocate everything you need upfront and use object pooling. Your gen 2 heap will be big but because nothing is being added to it, the GC doesn't need to collect it.
It may be helpful to look into some XNA or Silverlight related performance articles because
games and resource constrained devices are often very latency sensitive. (Note that you have it easy because the XBox 360 and, until Mango, Windows Phone only had a single generation GC (mark-and-sweep collector)).

.Net 4 MemoryCache Leaks with Concurrent Garbage Collection

I'm using the new MemoryCache in .Net 4, with a max cache size limit in MB (I've tested it set between 10 and 200MB, on systems with between 1.75 and 8GB of memory). I don't set any time based expiration on the objects, as I'm using the cache simply as a high performance drive, and as long as there is space, I want it used. To my surprise, the cache refused to evict any objects, to the point that I would get SystemOutOfMemory exceptions.
I fired up perfmon, wired up my application to .Net CLR Memory\#Bytes In All Heaps, .Net Memory Cache 4.0, and Process\Private Bytes -- indeed, the memory consumption was out of control, and no cache trims were being registered.
Did some googling and stackoverflowing, downloaded and attached the CLRProfiler, and wham: evictions everywhere! The memory stayed within reasonable bounds based upon the memory size limit I had set. Ran it in debug mode again, no evictions. CLRProfiler again, evictions.
I finally noticed that the profiler forced the application to run without concurrent garbage collection (also see useful SO Concurrent Garbage Collection Question). I turned it off in my app.config, and, sure enough, evictions!
This seems like at best an outrageous lack of documentation to not say: this only works with non-concurrent garbage collection -- though I image since its ported from ASP.NET, they may not have had to worry about concurrent garbage collection.
So has anyone else seen this? I'd love to get some other experiences out there, and maybe some more educated insights.
Update 1
I've reproduced the issue within a single method: it seems that the cache must be written to in parallel for the cache evictions not to fire (in concurrent garbage collection mode). If there is some interest, I'll upload the test code to a public repo. I'm definitely getting toward the deep end of the the CLR/GC/MemoryCache pool, and I think I forgot my floaties...
Update 2
I published test code on CodePlex to reproduce the issue. Also, possibly of interest, the original production code runs in Azure, as a Worker Role. Interesting, changing the GC concurrency setting in the role's app.config has no effect. Possibly Azure overrides GC settings much like ASP.NET? Further, running the test code under WPF vs a Console application will produce slightly different eviction results.
You can "force" a garbage collection right after the problematic method and see if the problem reproduces executing:
System.Threading.Thread.Sleep(200);
GC.Collect();
GC.WaitForPendingFinalizers();
right at the end of the method (make sure that you free any handles to reference objects and null them out). If this prevents memory leakage, and then yes, there may be a runtime bug.
Stop-the-world garbage collection is based on determining whether a strong live reference to an object exists at the moment the world is stopped. Concurrent garbage collection usually determines whether a strong live reference to an object has existed since some particular time in the past. My conjecture would be that many strong references to objects held in WeakReferences are being individually created and discarded. If a stop-the-world garbage collector fires between the time a particular object is created and the time it's discarded, that particular object will be kept alive, but previously-discarded objects will not. By contrast, a concurrent garbage collector may not detect that all strong references an object have been discarded until a certain amount of time goes by without any strong references to that object being created.
I've sometimes wished that .net would offer something between a strong reference and a weak one, which would prevent an object from being wiped from memory, but would not protect it from being finalized or having weak WeakReferences to it invalidated. Such references would slightly complicate the GC process, requiring every object to have separate flags indicating whether strong and quasi-weak references exist to it, and whether it has been scanned for both strong and quasi-weak references, but such a feature could be helpful in many "weak event" scenarios.
I found this entry while searching for a similiar topic and I'm focusing on your Out of Memory exception.
If you put an object in the cache then it still may be referencing other objects and therefore these objects would not be garbage collected -- hence the out of memory exception and probably a CPU being pegged out due to Gen 2 garbage collection.
Are you putting "used" objects on the cache or clones of "used" objects on the cache? If you put a clone on the cache then the "used" object that possible references other objects could be garbage collected.
If you shut off your caching mechanism does your program still run out of memory? If it doesn't run out of memory then that would prove that the objects you would otherwise be putting on the cache are still holding references to other objects hindering garbage collection.
Forcing garbage collection is not a best practice and shouldn't have to be done. In this scenario forcing a garbage collection wouldn't dispose of referenced objects anyway.
MemoryCache definately has some issues. It ate 160Mb of memory on my asp.net server, just changed to simple list and added some extra logic to get the same functionality.

Categories