We Keep Coding

Does DangerousAddRef keep the whole SafeHandle alive or just the handle?

I am in the situation where I have a c# object containing an HGLOBAL and a List<Delegate>. The HGLOBAL holds references to the delegates and the HGLOBAL is registered in unmanaged code as a callback (specifically, a vtbl-style interface).
I'm trying to not get bogged down in the details of the overall design (I'm already expecting some screaming from those that understand what I'm doing in the previous paragraph). My question is currently one of lifetime management using functions I wish were documented differently.
I have converted the class holding the HGLOBAL and the List<Delegate> into a subclass of SafeHandle (because of CriticalFinalizerObject and because HGLOBAL is kinda a handle), and I do have the class that is instantiating this using CER-style atomic transactions to inform C# when the unmanaged code stops expecting it to stick around.
Does calling DangerousAddRef effectively bump up a reference count for the entire SafeHandle object? Will the List<Delegate> be included in this reference count? Another way of asking this is whether DangerousAddRef is effectively constructing a GCHandle to the entire SafeHandle object, or whether it is doing so simply to the SafeHandle.handle member inside?
If it is not applying to the object as a whole then the List<Delegate> is likely to be collected early and cause access violations when unmanaged code tries to call into it.
If it is applying to the object as a whole then I have just created a block of reference-counted objects within a garbage-collected heap and I had better be darned sure that I haven't created any reference loops.
I suspect it's the latter, but there are very few Google searches on this kinda stuff, and when the only symptoms you're dealing with are unmanaged access violations from the finalizer thread on process shutdown when no actual managed code is being ran then it starts to get difficult to see whether you're going in the right direction.

C# Garbage Collection -> to C++ delete

I'm converting a C# project to C++ and have a question about deleting objects after use. In C# the GC of course takes care of deleting objects, but in C++ it has to be done explicitly using the delete keyword.
My question is, is it ok to just follow each object's usage throughout a method and then delete it as soon as it goes out of scope (ie method end/re-assignment)?
I know though that the GC waits for a certain size of garbage (~1MB) before deleting; does it do this because there is an overhead when using delete?
As this is a game I am creating there will potentially be lots of objects being created and deleted every second, so would it be better to keep track of pointers that go out of scope, and once that size reachs 1MB to then delete the pointers?
(as a side note: later when the game is optimised, objects will be loaded once at startup so there is not much to delete during gameplay)

Your problem is that you are using pointers in C++.
This is a fundamental problem that you must fix, then all your problems go away. As chance would have it, I got so fed up with this general trend that I created a set of presentation slides on this issue. – (CC BY, so feel free to use them).
Have a look at the slides. While they are certainly not entirely serious, the fundamental message is still true: Don’t use pointers. But more accurately, the message should read: Don’t use delete.
In your particular situation you might find yourself with a lot of long-lived small objects. This is indeed a situation which a modern GC handles quite well, and which reference-counting smart pointers (shared_ptr) handle less efficiently. If (and only if!) this becomes a performance problem, consider switching to a small object allocator library.

You should be using RAII as much as possible in C++ so you do not have to explicitly deleteanything anytime.
Once you use RAII through smart pointers and your own resource managing classes every dynamic allocation you make will exist only till there are any possible references to it, You do not have to manage any resources explicitly.

Memory management in C# and C++ is completely different. You shouldn't try to mimic the behavior of .NET's GC in C++. In .NET allocating memory is super fast (basically moving a pointer) whereas freeing it is the heavy task. In C++ allocating memory isn't that lightweight for several reasons, mainly because a large enough chunk of memory has to be found. When memory chunks of different sizes are allocated and freed many times during the execution of the program the heap can get fragmented, containing many small "holes" of free memory. In .NET this won't happen because the GC will compact the heap. Freeing memory in C++ is quite fast, though.
Best practices in .NET don't necessarily work in C++. For example, pooling and reusing objects in .NET isn't recommended most of the time, because the objects get promoted to higher generations by the GC. The GC works best for short lived objects. On the other hand, pooling objects in C++ can be very useful to avoid heap fragmentation. Also, allocating a larger chunk of memory and using placement new can work great for many smaller objects that need to be allocated and freed frequently, as it can occur in games. Read up on general memory management techniques in C++ such as RAII or placement new.
Also, I'd recommend getting the books "Effective C++" and "More effective C++".

Well, the simplest solution might be to just use garbage collection in
C++. The Boehm collector works well, for example. Still, there are
pros and cons (but porting code originally written in C# would be a
likely candidate for a case where the pros largely outweigh the cons.)
Otherwise, if you convert the code to idiomatic C++, there shouldn't be
that many dynamically allocated objects to worry about. Unlike C#, C++
has value semantics by default, and most of your short lived objects
should be simply local variables, possibly copied if they are returned,
but not allocated dynamically. In C++, dynamic allocation is normally
only used for entity objects, whose lifetime depends on external events;
e.g. a Monster is created at some random time, with a probability
depending on the game state, and is deleted at some later time, in
reaction to events which change the game state. In this case, you
delete the object when the monster ceases to be part of the game. In
C#, you probably have a dispose function, or something similar, for
such objects, since they typically have concrete actions which must be
carried out when they cease to exist—things like deregistering as
an Observer, if that's one of the patterns you're using. In C++, this
sort of thing is typically handled by the destructor, and instead of
calling dispose, you call delete the object.

Substituting a shared_ptr in every instance that you use a reference in C# would get you the closest approximation at probably the lowest effort input when converting the code.
However you specifically mention following an objects use through a method and deleteing at the end - a better approach is not to new up the object at all but simply instantiate it inline/on the stack. In fact if you take this approach even for returned objects with the new copy semantics being introduced this becomes an efficient way to deal with returned objects also - so there is no need to use pointers in almost every scenario.

There are a lot more things to take into considerations when deallocating objects than just calling delete whenever it goes out of scope. You have to make sure that you only call delete once and only call it once all pointers to that object have gone out of scope. The garbage collector in .NET handles all of that for you.
The construct that is mostly corresponding to that in C++ is tr1::shared_ptr<> which keeps a reference counter to the object and deallocates when it drops to zero. A first approach to get things running would be to make all C# references in to C++ tr1::shared_ptr<>. Then you can go into those places where it is a performance bottleneck (only after you've verified with a profile that it is an actual bottleneck) and change to more efficient memory handling.

GC feature of c++ has been discussed a lot in SO.
Try Reading through this!!
Garbage Collection in C++

C# memory leak?

I have a C# application that loops through a datatable, and pushes these into some locations such as Sage and a SQL table.
While it used to work fine, I'm inexplicably now getting Out of Memory exceptions after an hour or so of running it. I've noticed in the task manager, the memory usage rises by anbout 1mb every second, and keeps on going!
I was under the impression garbage collection would take of anything, but to be sure I ensure I dispose any objects after using them. I know without code it's hard to diagnose, but there's a lot of it and I'm looking more for general advice.

but to be sure I ensure I dispose any objects after using them
Dispose() is not directly related to memory management or leaks.
You'll have to look for unused objects that are still 'reachable'. Use a memory-profiler to find out.
You can start with the free CLR-Profiler.

There are a couple of potential problems that spring to mind:
There is a large pool of objects that are left inelegible for garbage collection (i.e. they are still "reachable"). For example if you add an object to an list in every loop then the list will grown unboundedly and each element in the list will remain inelegible for garbage collection as long as that list is still reachable. I'm not claiming that this is what is happening, this is just an example of how memory might be allocated and then left without being collected.
For some reason the garbage collector isn't doing a collection.
The high memory use is actually due to an unmanaged component that you are using in your application (e.g. via P/Invoke or COM interop).
Without seeing any code its tricky to give specific advice on how to fix your problem however reading through Investigating Memory Issues should give you some pointers on how to diagnose the memory problem yourself. In particular my first step would probably be to examine performance counters to see if the garbage collector is actually running, and to check the various heap sizes.
Note that Dispose and the IDisposable interface is unrelated to memory use - its important to dispose of objects like database connections once you are done with them as it frees up any associated resources (e.g. handles) however disposing of objects that implement IDisposable is very unlikely to have an impact on memory use.

Garbage collection can only get rid of objects that are no longer referenced from anything else. In addition it can only get rid of managed objects - it has no control about memory created from native code you may be interfacing with. These therefore are the two root causes for memory leaks in C# code.
The first thing to look at is perfmon. Get the counters for the private bytes and the .net heap size for the process. If the heap size remains flat (or rises and drops) but private bytes keeps increasing you've got some native code allocating memory and not releasing it.
If the heap size just keeps growing then the leak is in your managed code and you'll need a profiler like ANTS, DotTrace or even WinDbg (with SOS extension) to inspect the heap and see what objects are lying about.

The most popular "memory leak" on .Net platform is forgotten collection that repeatetly added in some infinite loop.

When you new something for temporary memory use.
Always use following way, it ensures calling dispose.
using (Someclass A = new Someclass())
{
....something about A
}
Someclass is a class implemented interface IDisposable
GC won't save you if there some part of unsafe code is involved(P/Invoke, Com etc..), and if there still a reference some where exists.
If you find memory leaking, use WinDbg will see what is in the heap.
This article may give you some help.
http://www.codeproject.com/KB/dotnet/Memory_Leak_Detection.aspx

Early finalization and memory leaks in C++/CLI library

I'm having issues with finalizers seemingly being called early in a C++/CLI (and C#) project I'm working on. This seems to be a very complex problem and I'm going to be mentioning a lot of different classes and types from the code. Fortunately it's open source, and you can follow along here: Pstsdk.Net (mercurial repository) I've also tried linking directly to the file browser where appropriate, so you can view the code as you read. Most of the code we deal with is in the pstsdk.mcpp folder of the repository.
The code right now is in a fairly hideous state (I'm working on that), and the current version of the code I'm working on is in the Finalization fixes (UNSTABLE!) branch. There are two changesets in that branch, and to understand my long-winded question, we'll need to deal with both. (changesets: ee6a002df36f and a12e9f5ea9fe)
For some background, this project is a C++/CLI wrapper of an unmanaged library written in C++. I am not the coordinator of the project, and there are several design decisions that I disagree with, as I'm sure many of you who look at the code will, but I digress. We wrap much of the layers of original library in the C++/CLI dll, but expose the easy-to-use API in the C# dll. This is done because the intention of the project is to convert the entire library to managed C# code.
If you're able to get the code to compile, you can use this test code to reproduce the problem.
The problem
The latest changeset, entitled moved resource management code to finalizers, to show bug, shows the original problem I was having. Every class in this code is uses the same pattern to free the unmanaged resources. Here is an example (C++/CLI):
DBContext::~DBContext()
{
this->!DBContext();
GC::SuppressFinalize(this);
}
DBContext::!DBContext()
{
if(_pst.get() != nullptr)
_pst.reset(); // _pst is a clr_scoped_ptr (managed type)
// that wraps a shared_ptr<T>.
}
This code has two benefits. First, when a class such as this is in a using statement, the resources are properly freed immediately. Secondly, if a dispose is forgotten by the user, when the GC finally decides to finalize the class, the unmanaged resources will be freed.
Here is the problem with this approach, that I simply cannot get my head around, is that occasionally, the GC will decide to finalize some of the classes that are used to enumerate over data in the file. This happens with many different PST files, and I've been able to determine it has something to do with the Finalize method being called, even though the class is still in use.
I can consistently get it to happen with this file (download)1. The finalizer that gets called early is in the NodeIdCollection class that is in DBAccessor.cpp file. If you are able to run the code that was linked to above (this project can be difficult to setup because of the dependencies on the boost library), the application would fail with an exception, because the _nodes list is set to null and the _db_ pointer was reset as a result of the finalizer running.
1) Are there any glaring problems with the enumeration code in the NodeIdCollection class that would cause the GC to finalize this class while it's still in use?
I've only been able to get the code to run properly with the workaround I've described below.
An unsightly workaround
Now, I was able to work around this problem by moving all of the resource management code from the each of the finalizers (!classname) to the destructors (~classname). This has solved the problem, though it hasn't solved my curiosity of why the classes are finalized early.
However, there is a problem with the approach, and I'll admit that it's more a problem with the design. Due to the heavy use of pointers in the code, nearly every class handles its own resources, and requires each class be disposed. This makes using the enumerations quite ugly (C#):
foreach (var msg in pst.Messages)
{
// If this using statement were removed, we would have
// memory leaks
using (msg)
{
// code here
}
}
The using statement acting on the item in the collection just screams wrong to me, however, with the approach it's very necessary to prevent any memory leaks. Without it, the dispose never gets called and the memory is never freed, even if the dispose method on the pst class is called.
I have every intention trying to change this design. The fundamental problem when this code was first being written, besides the fact that I knew little to nothing about C++/CLI, was that I couldn't put a native class inside of a managed one. I feel it might be possible to use scoped pointers that will free the memory automatically when the class is no longer in use, but I can't be sure if that's a valid way to go about this or if it would even work. So, my second question is:
2) What would be the best way to handle the unmanaged resources in the managed classes in a painless way?
To elaborate, could I replace a native pointer with the clr_scoped_ptr wrapper that was just recently added to the code (clr_scoped_ptr.h from this stackexchange question). Or would I need to wrap the native pointer in something like a scoped_ptr<T> or smart_ptr<T>?
Thank you for reading all of this, I know it was a lot. I hope I've been clear enough so that I might get some insight from people a little more experienced than I am. It's such a large question, I intend on adding a bounty when it allows me too. Hopefully, someone can help.
Thanks!
1This file is part of the freely available enron dataset of PST files

The clr_scoped_ptr is mine, and comes from here.
If it has any errors, please let me know.
Even if my code isn't perfect, using a smart pointer is the correct way to deal with this issue, even in managed code.
You do not need to (and should not) reset a clr_scoped_ptr in your finalizer. Each clr_scoped_ptr will itself be finalized by the runtime.
When using smart pointers, you do not need to write your own destructor or finalizer. The compiler-generated destructor will automatically call destructors on all subobjects, and every subobject finalizer will run when it is collected.
Looking closer at your code, there is indeed an error in NodeIdCollection. GetEnumerator() must return a different enumerator object each time it is called, so that each enumeration would begin at the start of the sequence. You're reusing a single enumerator, meaning that position is shared between successive calls to GetEnumerator(). That's bad.

Refreshing my memory of destructors/finalalisers, from some Microsoft documentation, you could at least simplify your code a little, I think.
Here's my version of your sequence:
DBContext::~DBContext()
{
this->!DBContext();
}
DBContext::!DBContext()
{
delete _pst;
_pst = NULL;
}
The "GC::SupressFinalize" is automatically done by C++/CLI, so no need for that. Since the _pst variable is initialised in the constructor (and deleting a null variable causes no problems anyway), I can't see any reason to complicate the code by using smart pointers.
On a debugging note, I wonder if you can help make the problem more apparent by sprinkling in a few calls to "GC::Collect". That should force finalization on dangling objects for you.
Hope this helps a little,

C# to unmanaged dll data structures interop

I have a unmanaged DLL that exposes a function that takes a pointer to a data structure. I have C# code that creates the data structure and calls the dll function without any problem. At the point of the function call to the dll the pointer is correct.
My problem is that the DLL keeps the pointer to the structure and uses the data structure pointer at a later point in time. When the DLL comes to use the pointer it has become invalid (I assume the .net runtime has moved the memory somewhere else).
What are the possible solutions to this problem?
The possible solutions I can think of are:
Fix the memory location of the data structure somehow? I don't know how you would do this in C# or even if you can.
Allocate memory manually so that I have control over it e.g. using Marshal.AllocHGlobal
Change the DLL function contract to copy the structure data (this is what I'm currently doing as a short term change, but I don't want to change the dll at all if I can help it as it's not my code to begin with).
Are there any other better solutions?

You can allocate the structure using AllocHGlobal, which puts it in unmanaged memory, where the GC won't move it around or release it. You could also use a helper class like this to have the GC pin the memory, so it won't be moved or released until un-pinned.

See the fixed C# statement: http://msdn.microsoft.com/en-us/library/f58wzh21(VS.80).aspx

Allocate memory manually so that I have control over it e.g. using Marshal.AllocHGlobal
Pretty close.
In this specific case I'd P/Invoke LocalAlloc to allocate the memory block and use StructureToPtr to initialize it.
UPDATE: Since you can edit the DLL I'd change the DLL to provide AllocXXX and FreeXXX functions.

Without having the DLL on hand to try this, it's hard to say if this would work. I would try making the object "fixed" in your C# class, that way the memory hangs out for the life of you application. Then just pass the static object to the DLL.

The GCHandle class was designed to handle this exact scenario. Essentially, you box a copy of your struct onto the heap, and then call GCHandle.Alloc with GCHandleType.Pinned. When the DLL is done with the structure, call GCHandle.Free. To give the DLL function the address of the object, pass it GCHandle.AddrOfPinnedObject. As long as it is pinned, the GC won't move it.