We Keep Coding

Why does this C# code note leak? [duplicate]

Consider the below code:
public class Class1
{
public static int c;
~Class1()
{
c++;
}
}
public class Class2
{
public static void Main()
{
{
var c1=new Class1();
//c1=null; // If this line is not commented out, at the Console.WriteLine call, it prints 1.
}
GC.Collect();
GC.WaitForPendingFinalizers();
Console.WriteLine(Class1.c); // prints 0
Console.Read();
}
}
Now, even though the variable c1 in the main method is out of scope and not referenced further by any other object when GC.Collect() is called, why is it not finalized there?

You are being tripped up here and drawing very wrong conclusions because you are using a debugger. You'll need to run your code the way it runs on your user's machine. Switch to the Release build first with Build + Configuration manager, change the "Active solution configuration" combo in the upper left corner to "Release". Next, go into Tools + Options, Debugging, General and untick the "Suppress JIT optimization" option.
Now run your program again and tinker with the source code. Note how the extra braces have no effect at all. And note how setting the variable to null makes no difference at all. It will always print "1". It now works the way you hope and expected it would work.
Which does leave with the task of explaining why it works so differently when you run the Debug build. That requires explaining how the garbage collector discovers local variables and how that's affected by having a debugger present.
First off, the jitter performs two important duties when it compiles the IL for a method into machine code. The first one is very visible in the debugger, you can see the machine code with the Debug + Windows + Disassembly window. The second duty is however completely invisible. It also generates a table that describes how the local variables inside the method body are used. That table has an entry for each method argument and local variable with two addresses. The address where the variable will first store an object reference. And the address of the machine code instruction where that variable is no longer used. Also whether that variable is stored on the stack frame or a cpu register.
This table is essential to the garbage collector, it needs to know where to look for object references when it performs a collection. Pretty easy to do when the reference is part of an object on the GC heap. Definitely not easy to do when the object reference is stored in a CPU register. The table says where to look.
The "no longer used" address in the table is very important. It makes the garbage collector very efficient. It can collect an object reference, even if it is used inside a method and that method hasn't finished executing yet. Which is very common, your Main() method for example will only ever stop executing just before your program terminates. Clearly you would not want any object references used inside that Main() method to live for the duration of the program, that would amount to a leak. The jitter can use the table to discover that such a local variable is no longer useful, depending on how far the program has progressed inside that Main() method before it made a call.
An almost magic method that is related to that table is GC.KeepAlive(). It is a very special method, it doesn't generate any code at all. Its only duty is to modify that table. It extends the lifetime of the local variable, preventing the reference it stores from getting garbage collected. The only time you need to use it is to stop the GC from being to over-eager with collecting a reference, that can happen in interop scenarios where a reference is passed to unmanaged code. The garbage collector cannot see such references being used by such code since it wasn't compiled by the jitter so doesn't have the table that says where to look for the reference. Passing a delegate object to an unmanaged function like EnumWindows() is the boilerplate example of when you need to use GC.KeepAlive().
So, as you can tell from your sample snippet after running it in the Release build, local variables can get collected early, before the method finished executing. Even more powerfully, an object can get collected while one of its methods runs if that method no longer refers to this. There is a problem with that, it is very awkward to debug such a method. Since you may well put the variable in the Watch window or inspect it. And it would disappear while you are debugging if a GC occurs. That would be very unpleasant, so the jitter is aware of there being a debugger attached. It then modifies the table and alters the "last used" address. And changes it from its normal value to the address of the last instruction in the method. Which keeps the variable alive as long as the method hasn't returned. Which allows you to keep watching it until the method returns.
This now also explains what you saw earlier and why you asked the question. It prints "0" because the GC.Collect call cannot collect the reference. The table says that the variable is in use past the GC.Collect() call, all the way up to the end of the method. Forced to say so by having the debugger attached and by running the Debug build.
Setting the variable to null does have an effect now because the GC will inspect the variable and will no longer see a reference. But make sure you don't fall in the trap that many C# programmers have fallen into, actually writing that code was pointless. It makes no difference whatsoever whether or not that statement is present when you run the code in the Release build. In fact, the jitter optimizer will remove that statement since it has no effect whatsoever. So be sure to not write code like that, even though it seemed to have an effect.
One final note about this topic, this is what gets programmers in trouble that write small programs to do something with an Office app. The debugger usually gets them on the Wrong Path, they want the Office program to exit on demand. The appropriate way to do that is by calling GC.Collect(). But they'll discover that it doesn't work when they debug their app, leading them into never-never land by calling Marshal.ReleaseComObject(). Manual memory management, it rarely works properly because they'll easily overlook an invisible interface reference. GC.Collect() actually works, just not when you debug the app.

[ Just wanted to add further on the Internals of Finalization process ]
You create an object and when the object is garbage collected, the object's Finalize method should be called. But there is more to finalization than this very simple assumption.
CONCEPTS:
Objects not implementing Finalize methods: their memory is reclaimed immediately, unless of course, they are not reachable by application code any more.
Objects implementing Finalize method: the concepts of Application Roots, Finalization Queue, Freachable Queue need to be understood since they are involved in the reclamation process.
Any object is considered garbage if it is not reachable by application code.
Assume: classes/objects A, B, D, G, H do not implement the Finalize method and C, E, F, I, J do implement the Finalize method.
When an application creates a new object, the new operator allocates memory from the heap. If the object's type contains a Finalize method, then a pointer to the object is placed on the finalization queue. Therefore pointers to objects C, E, F, I, J get added to the finalization queue.
The finalization queue is an internal data structure controlled by the garbage collector. Each entry in the queue points to an object that should have its Finalize method called before the object's memory can be reclaimed.
The figure below shows a heap containing several objects. Some of these objects are reachable from the application roots, and some are not. When objects C, E, F, I, and J are created, the .NET framework detects that these objects have Finalize methods and pointers to these objects are added to the finalization queue.
When a GC occurs (1st Collection), objects B, E, G, H, I, and J are determined to be garbage. A,C,D,F are still reachable by application code depicted as arrows from the yellow box above.
The garbage collector scans the finalization queue looking for pointers to these objects. When a pointer is found, the pointer is removed from the finalization queue and appended to the freachable queue ("F-reachable", i.e. finalizer reachable). The freachable queue is another internal data structure controlled by the garbage collector. Each pointer in the freachable queue identifies an object that is ready to have its Finalize method called.
After the 1st GC, the managed heap looks something similar to figure below. Explanation given below:
The memory occupied by objects B, G, and H has been reclaimed immediately because these objects did not have a finalize method that needed to be called.
However, the memory occupied by objects E, I, and J could not be reclaimed because their Finalize method has not been called yet. Calling the Finalize method is done by freachable queue.
A, C, D, F are still reachable by application code depicted as arrows from yellow box above, so they will not be collected in any case.
There is a special runtime thread dedicated to calling Finalize methods. When the freachable queue is empty (which is usually the case), this thread sleeps. But when entries appear, this thread wakes, removes each entry from the queue, and calls each object's Finalize method. The garbage collector compacts the reclaimable memory and the special runtime thread empties the freachable queue, executing each object's Finalize method. So here finally is when your Finalize method gets executed.
The next time the garbage collector is invoked (2nd GC), it sees that the finalized objects are truly garbage, since the application's roots don't point to it and the freachable queue no longer points to it (it's EMPTY too), therefore the memory for the objects E, I, J may be reclaimed from the heap. See figure below and compare it with figure just above.
The important thing to understand here is that two GCs are required to reclaim memory used by objects that require finalization. In reality, more than two collections cab be even required since these objects may get promoted to an older generation.
NOTE: The freachable queue is considered to be a root just like global and static variables are roots. Therefore, if an object is on the freachable queue, then the object is reachable and is not garbage.
As a last note, remember that debugging application is one thing, garbage collection is another thing and works differently. So far you can't feel garbage collection just by debugging applications. If you wish to further investigate memory get started here.

Why does this variable need to be set to null after the object is disposed?

The documentation on PowerShell here has the following interesting comment in it:
PowerShell powershell = PowerShell.Create();
using (powershell)
{
//...
}
// Even after disposing of the PowerShell object, we still
// need to set the powershell variable to null so that the
// garbage collector can clean it up.
powershell = null;
Why does powershell need to be set to null after being disposed?

It's not directly a PowerShell issue. When a using block terminates, the specified object(s) have their Dispose() methods called. These typically do some cleanup operations, often to avoid leaking memory and so forth. However, Dispose() doesn't delete the object. If a reference to it still exists outside the using block (as in this example), then the object itself is still in scope. It can't be garbage-collected because there's still a reference to it, so it's still taking up memory.
What they're doing in your example is dropping that reference. When powershell is set to null, the PowerShell object it was pointing to is orphaned, since there are no other variables referring to it. Once the garbage collector figures that out, it can free up the memory. This would happen at the end of the method anyway (because powershell would go out of scope), but this way you get the system resources back a little sooner.
(Edit: As Brian Rasmussen points out, the .NET runtime is extremely clever about garbage collection. Once it reaches the last reference to powershell in your code, the runtime should detect that you don't need it anymore and release it for garbage collection. So the powershell = null; line isn't actually doing anything.)
By the way, this pattern looks very strange to me. The usual approach is something like this:
using (PowerShell powershell = PowerShell.Create())
{
//...
}
This way, powershell goes out of scope at the end of the using block, right after it's disposed. It's easier to tell where the variable is relevant, and you save some code because you don't need the powershell = null line anymore. I'd even say this is better coding practice, because powershell never exists in an already-disposed state. If someone modifies your original code and tries to use powershell outside the using block, whatever happens will probably be bad.

It doesn't need to be set to null, and it really shouldn't be. The .NET garbage collector is quite capable of detecting that an object not being used after a particular instruction even if your code does not assign a null value to the corresponding variable. (See http://blogs.msdn.com/b/cbrumme/archive/2003/04/19/51365.aspx for details.) As for why an "official" example contains this extra code with misleading comments, even docs can have bugs...

The advice is incorrect, as long as its unreachable it is eligible for garbage collection. (see What is the correct way to free memory in C# for some code demonstrating this) the only reasons to null things out are
To make it easier to troubleshoot memory leaks
To remove references (e.g. public properties) that the GC may have trouble determining are unreachable
Both of which you would really only do within an objects Dispose method

This use of GC.SuppressFinalize() doesn't feel right

I have been having some issues with utilizing a vendor library where occasionally an entity calculated by the library would be null when it should always have valid data in it.
The functioning code (after debugging the issue with the vendor) is roughly as follows:
Task.Factory.StartNew(() => ValidateCalibration(pelRectRaw2Ds, crspFeatures, Calibration.Raw2DFromPhys3Ds));
.....
private void ValidateCalibration(List<Rectangle> pelRectRaw2Ds, List<List<3DCrspFeaturesCollection>> crspFeatures, List<3DCameraCalibration> getRaw2DFromPhys3Ds)
{
var calibrationValidator = new 3DCameraCalibrationValidator();
// This is required according to vendor otherwise validationResultsUsingRecomputedExtrinsics is occasionally null after preforming the validation
GC.SuppressFinalize(calibrationValidator);
3DCameraCalibrationValidationResult validationResultUsingOriginalCalibrations;
3DCameraCalibrationValidationResult validationResultsUsingRecomputedExtrinsics;
calibrationValidator.Execute(pelRectRaw2Ds, crspFeatures, getRaw2DFromPhys3Ds, out validationResultUsingOriginalCalibrations, out validationResultsUsingRecomputedExtrinsics);
Calibration.CalibrationValidations.Add(new CalibrationValidation
{
Timestamp = DateTime.Now,
UserName = Globals.InspectionSystemObject.CurrentUserName,
ValidationResultUsingOriginalCalibrations = validationResultUsingOriginalCalibrations,
ValidationResultsUsingRecomputedExtrinsics = validationResultsUsingRecomputedExtrinsics
});
}
The validation process is a fairly time consuming operation so I hand it off to a Task. The problem I had was that originally I did not have the call to GC.SuppressFinalize(calibrationValidator) and when the application was run from a Release build, then the out parameter validationResultsUsingRecomputedExtrinsics would be null. If I ran the application from a Debug build (either with or without the Debugger attached) then validationResultsUsingRecomputedExtrinsics would contain valid data.
I don't fully understand what GC.SuppressFinalize() has done in this situation, or how it has fixed the problem. Everything I can find regarding GC.SuppressFinalize() is that it is used when implementing IDisposable. I can't find any use of it in "standard" code.
How/why does the addition of the call to GC.SuppressFinalize(calibrationValidator) fix this problem?
I understand that without intimate knowledge of the internals of the vendor library, it might not be possible to know for sure, but any insight would help.
The application is compiled with VS2012, targeting .NET 4.0. That vendor library requires that the useLegacyV2RuntimeActivationPolicy="true" option is specified in app.config.
This is the justification I received from the vendor:
The SuppressFinalize command makes sure that the garbage collector will not clean something up “early”. It seems like for some reason your application was sometimes having the garbage collector get a bit zealous and clean up the object before you were truly done with it; it is almost certainly scope related and possibly due to the multi-threading causing confusion on the scope of the calibrationValidator. Below is the response I got from Engineering.
Because the variable was created in the local scope, and that function runs in the background thread, Garbage Collection runs in the main thread, and it seems that the Garbage collection is not smart enough in handling multi-thread situations. Sometimes, it just releases it too early (internal execution of validator not finished yet, and still needs this variable).

This is in all likelihood a hack to solve a premature garbage collection problem. Not uncommon with unmanaged code, typical in camera applications. It is not a healthy hack, good odds that this will cause a resource leak because the finalizer doesn't execute. Wrappers for unmanaged code almost always have something to do in the finalizer, it is very common that they need to release unmanaged memory.
At issue is that the calibrationValidator object can be garbage collected while the unmanaged code is running. Having another thread in your program makes this likely since that other thread can be allocating objects and trigger a GC. This is very easy to miss by the owner of the code while testing, either by never having tested it while using multiple threads or just not getting lucky enough to trigger a GC at the wrong time.
The proper fix on your end is to ensure that the jitter marks the object in use past the call so that the garbage collector won't collect it. You do so by adding GC.KeepAlive(calibrationValidator) after the Execute() call.

When it comes to understanding IDisposable, GC.SuppressFinalize, and finalizers in C#, I don't think a better explanation exists than the following article.
DG Update: Dispose, Finalization, and Resource Management
Alright! Here it is: the revised “Dispose, Finalization, and Resource Management” Design Guideline entry. I mentioned this work previously here and here. At ~25 printed pages, it's not what I would consider to be a minor update. Took me much longer than anticipated, but I'm happy with the result. I got to work with and received good amounts of feedback from HSutter, BrianGru, CBrumme, Jeff Richter, and a couple other folks on it... Good fun.
Key Concept for this question:
It is so obvious that GC.SuppressFinalize() should only be called on this that the article doesn't even mention that directly. It does however mention the practice of wrapping finalizable objects to isolate them from a public API in order to ensure that external code is not able to call GC.SuppressFinalize() on those resources (see the following quote). Whoever designed the library described in the original question has no grasp on the way finalization in .NET works.
Quoted from the blog article:
Even in the absence of one of the rare situations noted above, a finalizable object with a publicly accessible reference could have its finalization suppressed by any arbitrary untrusted caller. Specifically, they can call GC.SuppressFinalize on you and prevent finalization from occurring altogether, including critical finalization. A good mitigation strategy to deal with this is to wrap critical resources in a non-public instance that has a finalizer. So long as you do not leak this to callers, they will not be able to suppress finalization. If you migrate to using SafeHandle in your class and never expose it outside your class, you can guarantee finalization of your resources (with the caveats mentioned above and assuming a correct SafeHandle implementation).

There are some mentions of multithreading or native code being the cause of this issue. But the same thing can happen in a purely managed and mostly single-threaded program.
Consider the following program:
using System;
class Program
{
private static void Main()
{
var outer = new Outer();
Console.WriteLine(outer.GetValue() == null);
}
}
class Outer
{
private Inner m_inner = new Inner();
public object GetValue()
{
return m_inner.GetValue();
}
~Outer()
{
m_inner.Dispose();
}
}
class Inner
{
private object m_value = new object();
public object GetValue()
{
GC.Collect();
GC.WaitForPendingFinalizers();
return m_value;
}
public void Dispose()
{
m_value = null;
}
}
Here, while outer.GetValue() is being called, outer will be garbage collected and finalized (at least in Release mode). The finalizer nulls out the field of the Inner object, which means GetValue() will return null.
In real code, you most likely wouldn't have the GC calls there. Instead you would create some managed object, which (non-deterministically) causes the garbage collector to run.
(I said this code is mostly single-threaded. In fact, the finalizer will run on another thread, but because of the call to WaitForPendingFinalizers(), it's almost as if it ran on the main thread.)

Early finalization and memory leaks in C++/CLI library

I'm having issues with finalizers seemingly being called early in a C++/CLI (and C#) project I'm working on. This seems to be a very complex problem and I'm going to be mentioning a lot of different classes and types from the code. Fortunately it's open source, and you can follow along here: Pstsdk.Net (mercurial repository) I've also tried linking directly to the file browser where appropriate, so you can view the code as you read. Most of the code we deal with is in the pstsdk.mcpp folder of the repository.
The code right now is in a fairly hideous state (I'm working on that), and the current version of the code I'm working on is in the Finalization fixes (UNSTABLE!) branch. There are two changesets in that branch, and to understand my long-winded question, we'll need to deal with both. (changesets: ee6a002df36f and a12e9f5ea9fe)
For some background, this project is a C++/CLI wrapper of an unmanaged library written in C++. I am not the coordinator of the project, and there are several design decisions that I disagree with, as I'm sure many of you who look at the code will, but I digress. We wrap much of the layers of original library in the C++/CLI dll, but expose the easy-to-use API in the C# dll. This is done because the intention of the project is to convert the entire library to managed C# code.
If you're able to get the code to compile, you can use this test code to reproduce the problem.
The problem
The latest changeset, entitled moved resource management code to finalizers, to show bug, shows the original problem I was having. Every class in this code is uses the same pattern to free the unmanaged resources. Here is an example (C++/CLI):
DBContext::~DBContext()
{
this->!DBContext();
GC::SuppressFinalize(this);
}
DBContext::!DBContext()
{
if(_pst.get() != nullptr)
_pst.reset(); // _pst is a clr_scoped_ptr (managed type)
// that wraps a shared_ptr<T>.
}
This code has two benefits. First, when a class such as this is in a using statement, the resources are properly freed immediately. Secondly, if a dispose is forgotten by the user, when the GC finally decides to finalize the class, the unmanaged resources will be freed.
Here is the problem with this approach, that I simply cannot get my head around, is that occasionally, the GC will decide to finalize some of the classes that are used to enumerate over data in the file. This happens with many different PST files, and I've been able to determine it has something to do with the Finalize method being called, even though the class is still in use.
I can consistently get it to happen with this file (download)1. The finalizer that gets called early is in the NodeIdCollection class that is in DBAccessor.cpp file. If you are able to run the code that was linked to above (this project can be difficult to setup because of the dependencies on the boost library), the application would fail with an exception, because the _nodes list is set to null and the _db_ pointer was reset as a result of the finalizer running.
1) Are there any glaring problems with the enumeration code in the NodeIdCollection class that would cause the GC to finalize this class while it's still in use?
I've only been able to get the code to run properly with the workaround I've described below.
An unsightly workaround
Now, I was able to work around this problem by moving all of the resource management code from the each of the finalizers (!classname) to the destructors (~classname). This has solved the problem, though it hasn't solved my curiosity of why the classes are finalized early.
However, there is a problem with the approach, and I'll admit that it's more a problem with the design. Due to the heavy use of pointers in the code, nearly every class handles its own resources, and requires each class be disposed. This makes using the enumerations quite ugly (C#):
foreach (var msg in pst.Messages)
{
// If this using statement were removed, we would have
// memory leaks
using (msg)
{
// code here
}
}
The using statement acting on the item in the collection just screams wrong to me, however, with the approach it's very necessary to prevent any memory leaks. Without it, the dispose never gets called and the memory is never freed, even if the dispose method on the pst class is called.
I have every intention trying to change this design. The fundamental problem when this code was first being written, besides the fact that I knew little to nothing about C++/CLI, was that I couldn't put a native class inside of a managed one. I feel it might be possible to use scoped pointers that will free the memory automatically when the class is no longer in use, but I can't be sure if that's a valid way to go about this or if it would even work. So, my second question is:
2) What would be the best way to handle the unmanaged resources in the managed classes in a painless way?
To elaborate, could I replace a native pointer with the clr_scoped_ptr wrapper that was just recently added to the code (clr_scoped_ptr.h from this stackexchange question). Or would I need to wrap the native pointer in something like a scoped_ptr<T> or smart_ptr<T>?
Thank you for reading all of this, I know it was a lot. I hope I've been clear enough so that I might get some insight from people a little more experienced than I am. It's such a large question, I intend on adding a bounty when it allows me too. Hopefully, someone can help.
Thanks!
1This file is part of the freely available enron dataset of PST files

The clr_scoped_ptr is mine, and comes from here.
If it has any errors, please let me know.
Even if my code isn't perfect, using a smart pointer is the correct way to deal with this issue, even in managed code.
You do not need to (and should not) reset a clr_scoped_ptr in your finalizer. Each clr_scoped_ptr will itself be finalized by the runtime.
When using smart pointers, you do not need to write your own destructor or finalizer. The compiler-generated destructor will automatically call destructors on all subobjects, and every subobject finalizer will run when it is collected.
Looking closer at your code, there is indeed an error in NodeIdCollection. GetEnumerator() must return a different enumerator object each time it is called, so that each enumeration would begin at the start of the sequence. You're reusing a single enumerator, meaning that position is shared between successive calls to GetEnumerator(). That's bad.

Refreshing my memory of destructors/finalalisers, from some Microsoft documentation, you could at least simplify your code a little, I think.
Here's my version of your sequence:
DBContext::~DBContext()
{
this->!DBContext();
}
DBContext::!DBContext()
{
delete _pst;
_pst = NULL;
}
The "GC::SupressFinalize" is automatically done by C++/CLI, so no need for that. Since the _pst variable is initialised in the constructor (and deleting a null variable causes no problems anyway), I can't see any reason to complicate the code by using smart pointers.
On a debugging note, I wonder if you can help make the problem more apparent by sprinkling in a few calls to "GC::Collect". That should force finalization on dangling objects for you.
Hope this helps a little,

P-invoke call fails if too much memory is assigned beforehand

I've got a p-invoke call to an unmanaged DLL that was failing in my WPF app but not in a simple, starter WPF app. I tried to figure out what the problem was but eventually came to the conclusion that if I assign too much memory before making the call, the call fails. I had two separate blocks of code, both of which would succeed on their own, but that would cause failure if both were run. (They had nothing to do with what the p-invoke call is trying to do).
What kind of issues in the unmanaged library would cause such an issue? I thought that the managed and unmanaged heaps were supposed to be automatically separated.
The crash as far as I can tell is happening in a dynamically loaded secondary DLL from the one p-invoked into. Could that have something to do with it?

Unmanaged code is prone to corrupt the heap. The side effects of that corruption are very unpredictable, it depends on what happens afterwards with that corrupted memory. It is not uncommon that nothing bad happens if the corruption is not in a crucial location. Changing the memory allocation pattern of your program can change that outcome.
All you really know right now is that the unmanaged code can't be trusted. Doing something about it is invariably hard, especially from a managed host program. You won't get anywhere until you start writing unit tests for that unmanaged code, using unmanaged code to exercise it, and find a reproducible bomb that you could tackle with an unmanaged debugger.

A shot in the dark given there is not much info to work with.
Is it possible that the unmanaged DLL needs to be loaded at a specific base address and when you allocate too much memory or other assemblies are loaded, the DLL is not able to load at the correct address.
http://msdn.microsoft.com/en-us/library/w368ysh2.aspx