We Keep Coding

Using COleDispatchDriver to dynamically load COM object leaking memory

I'm debugging some legacy code. It loads a user defined COM object, allows the user to call functions in it, and then releases it. However, we have found that every time we load and unload the COM object, we leak memory. As a test, we changed the code to load it and hang on to it, and keep re-using it until program exit and the leak went away.
Here are the relevant code snippets:
This C++ codeis is called to load the COM object, pszProgId is a string identifying the target DLL.
COleDispatchDriver *pDispatchDriver = NULL;
pDispatchDriver = new COleDispatchDriver();
if (!pDispatchDriver->CreateDispatch(pszProgId, &oleException))
{
throw &oleException;
}
pDispatchDriver->m_bAutoRelease = TRUE;
*ppvObject = (void *) pDispatchDriver;
void ** ppvObject is a pointer we pass around to generically hold different objects. It is part of a much larger structure.
And here is the code we call when releasing the COM object.
After we are done using the COM object, we release it as follows:
COleDispatchDriver* pDispatchDriver = (COleDispatchDriver*) (*((LONG_PTR*)(ppvObject)));
pDispatchDriver->ReleaseDispatch();
delete pDispatchDriver;
This is leaking about 1 meg every call. The target COM object is C#. Anyone have any idea what we're doing wrong or a better way to do what we're trying to do?
We are building this in VisualStudio 2015 in case that is relevant.
Re xMRi:
As already noted, we tried changing that flag to TRUE to no effect. As a sanity check, I tried doing that again after reading your post and again it did nothing to fix the memory leak. So for better clarity, I've updated my code to show it set to TRUE which is almost certainly the right value but still exhibiting the same memory leak described above.

ReleaseDispatch does nothing if you set m_bAutoRelease to FALSE. So in fact you don't free the instance of this COM object.
See the implementation:
void COleDispatchDriver::ReleaseDispatch()
{
if (m_lpDispatch != NULL)
{
if (m_bAutoRelease)
m_lpDispatch->Release();
m_lpDispatch = NULL;
}
}
So created the problem in setting m_bAutoRelease to FALSE yourself. Check the reasons why you are doing this.
You can directly get the LPDISPTACH pointer and call Release() but this is exactly what should be done when m_bAutoRelease is TRUE.

Image Resource Memory

Since I had a really nasty problem with an not too obvious unmanaged resource last month, I got a little hypernervous regarding memory leaking problems.
I was just coding on a very simple test app with a button with two different pictures on it and noticed I am not quite sure if I have a "problem" here or not...
If I have 2 picture resources Pic1 and Pic2 and an ImageButton-Object, which is just some object inherited from UserControl and with a changed OnPaint:
protected override void OnPaint(PaintEventArgs e)
{
base.OnPaint(e);
//stuff
if (this.keyStatus))
{ this.imageButton.DefaultImage = Resource1.Pic1; }
else
{ this.imageButton.DefaultImage = Resource1.Pic2; }
e.Graphics.DrawImage(this.defaultImage, this.ClientRectangle);
}
Beside OnPaint not being a good place for assigning DefaultImage (its just here to show you what I mean in a short piece of code), I am just assinging a reference to my precompiled resource here, am I? I am not creating a copy as I would if I would call it with new Bitmap(Resource1.Pic1).
So if I change keyStatus every 5 seconds, I would have a very annoying picture on my screen with a lot of changing, but no problems that the picture changing or turning it invisible from time to time leaks memory. Correct?
Thanks a lot!

How object references work
Say you have a random object. The object is a class type (not a value type) and not IDisposable. Basically that means the following:
// y is some object
var x = y;
Now, x doesn't copy all the data from y, but simply makes a new reference to the contents of y. Simple.
To ensure that there won't be memory leaks, the GC keeps track of all objects and (periodically) checks which objects are reachable. If an object is still reachable, it won't be deleted - if it's not, it will be removed.
And then there was unmanaged code
As long as you stick to managed code, everything is fine. The moment you run into unmanaged code (say: GDI+, which is the native counterpart of a lot of System.Drawing stuff) you need to do extra book-keeping to get rid of the code. After all, the .NET runtime doesn't know much about the unmanaged data - it merely knows that there is a pointer. Therefore, the GC will cleanup the pointer, but not the data -- which would result in memory leaks.
Therefore, the guys from .NET added IDisposable. By implementing IDisposable, you can implement extra (unmanaged) cleanup, such as releasing unmanaged memory, closing files, closing sockets, etc.
Now, the GC knows about finalizers, which are implemented as part of the Disposable pattern (details: https://msdn.microsoft.com/en-us/library/b1yfkh5e(v=vs.110).aspx ). However, you usually don't want to wait for a GC run to clean up unmanaged resources. So, it's generally a good idea to call Dispose() when an object can be cleaned up and has unmanaged resources.
As is the case with System.Drawing.Bitmap, which implements IDisposable.
In most cases, you can simply wrap IDisposable in a using statement, which will call 'Dispose()' for you in a nice try/finally clause. e.g.:
using (var myBitmap = new Bitmap(...))
{
// use myBitmap
}
// myBitmap including all resources are gone.
What about resource bitmaps
#HansPassant pointed out that resource bitmaps generate a new Bitmap every time you access a bitmap property. This basically means that the bitmaps are copied and need to be disposed.
In other words:
// Free the old bitmap if it exists:
if (this.imageButton.DefaultImage != null)
{
this.imageButton.DefaultImage.Dispose();
this.imageButton.DefaultImage = null;
}
// assign new imageButton.DefaultImage
So, this solves the memory leak, but will give you a lot of data that is copied around.
If you don't want to dispose
Here comes the part why I was surprised by the remark from Hans :) Basically you assign a Bitmap to a button every time, so you don't want to copy the data over and over again - that makes little sense.
Therefore, you might get the idea to wrap the resource into a 'static' container and simply don't deallocate it at all:
static Bitmap myPic1 = Resource1.Pic1;
static Bitmap myPic2 = Resource1.Pic2;
...
if (this.keyStatus))
{
this.imageButton.DefaultImage = myPic1;
}
else
{
this.imageButton.DefaultImage = myPic2;
}
This works, but will give you issues if you at some point decide to generate images as well. To illustrate, say we change the code like this::
if (this.keyStatus))
{
this.imageButton.DefaultImage = myPic1; // #1 don't dispose
}
else
{
Bitmap myPic3 = CreateFancyBitmap(); // #2 do dispose
this.imageButton.DefaultImage = myPic3;
}
Now, the issue here is with the combination. myPic1 is a static object and shouldn't be disposed. On the other hand, myPic3 is not, and should be disposed. If you do call Dispose(), you'll get a nasty exception at #1, because the data is no longer there. There's no proper way to distinguish the two.

Clear structure in c#

Say I have a structure with several members eg.
private struct MyStats
{
public int packsGood, packsBad, packsTotal;
public bool haveAcceptedStart;
// ...and a bunch of other parameters
}
If I have a member variable of type MyStats, how do I use it for a while and then quickly clear it? In Delphi or C++ I might have used the following code:
memset(&m_stats, 0, sizeof(MyStats)); // C++
ZeroMemory(#m_stats, SizeOf(MyStats)); // Delphi
but that doesn't appear possible in C#. Surely I don't have to rattle through all members each time or PInvoke an API call?

Memory Management should generally not be one of your concerns in C#, as opposed to C or C++. There is the garbage collector to take care of memory clearing. As soon as all the references to your struct have disappeared, the garbage collector will reclaim the memory somewhere in the future. When exactly is irrelevant for a developer.

If you have a field in your class just to be used for a while an clearing it then your class isn't properly designed and does not comply with OOP principles.
If you are using that field just as an auxiliary value you should just replace it with a variable in every scope it is used; in this way you'll reduce the number of daunting side effects and since it's a value type it will be allocated on the stack and cleared as soon as the control flow leaves the current block (given you don't capture it in a lambda expression).

Memory leak in unmanaged code?

I have traced a leak pretty far but I can't seem to understand (fix) it by myself. I used ANTS memory profiler first to make sure my code actually is stacking memory. It starts from using 25 MB but within an hour or so it is using over 100 MB. A friend of mine for whom I'm coding this for has actually been using this faulty program and he got it to spend his whole 18 GB of ram and got a out of memory exception.
The leaking part is not vital for the program, but it just is pretty much useless without the RefreshSessions() method.
I have been extending the project Vista Core Audio API Master Volume Control from Code Project.
This is the part which seems to leak. Tested by not using it and then it doesn't leak.
Updated:
public void RefreshSessions()
{
Marshal.ThrowExceptionForHR(_AudioSessionManager.GetSessionEnumerator(out _SessionEnum));
_Sessions.Refresh(_SessionEnum);
}
(Removed the class code from here)
I have not been coding too much so I may have missed something, but if more details are needed you can actually download the source or I can just answer to my best ability.
(Removed unnecessary code here)
The leak was tested with this simple console app:
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
MMDeviceEnumerator DevEnum = new MMDeviceEnumerator();
MMDevice device = DevEnum.GetDefaultAudioEndpoint(EDataFlow.eRender, ERole.eMultimedia);
Console.ReadKey();
int i = 0;
while (i < 10000)
{
device.AudioSessionManager.RefreshSessions();
i++;
}
Console.ReadKey();
}
}
}
Update 2
I think I got it fixed. Have to run some longer tests, but at least it seems like the memory usage has stabilized. The idea came from dialer who found a fix for the leak in c++.
public void RefreshSessions()
{
_Sessions.Release(); //added this
IAudioSessionEnumerator _SessionEnum;
Marshal.ThrowExceptionForHR(_AudioSessionManager.GetSessionEnumerator(out _SessionEnum));
_Sessions.Refresh(_SessionEnum);
}
This is the part in SessionCollection:
public void Release()
{
Marshal.ReleaseComObject(_AudioSessionEnumerator);
}
This is not exactly the code dialer suggested (which I ended up using anyways), but still.
And as he said as well this might not be the best way to achieve this but I will go with it since it does not seem to have any adverse effects on my app.

ANOTHER EDIT
public void RefreshSessions()
{
if (_SessionEnum != null)
{
Marshal.ReleaseComObject(_SessionEnum);
}
Marshal.ThrowExceptionForHR(_AudioSessionManager.GetSessionEnumerator(out _SessionEnum));
}
Above code releases the SessionEnum explicitly and also fixed the leak in C#. This should probably be taken care of in a better way though.
EDIT:
The following C++ program is equivalent to what you did in the loop test program. The Release call at the end of the for loop fixes the leak. I need to go for today, maybe you can play around a bit and try to fix it yourself. Or maybe someone else can find out and explain why the CLR garbage collector does not call the Release automatically at some point in the C# program above.
#include <stdio.h>
#include <tchar.h>
#include <audiopolicy.h>
#include <mmdeviceapi.h>
#define SAFE_RELEASE(p) { if ( (p) ) { (p)->Release(); (p) = 0; } }
#define CHECK_HR(hr) if (FAILED(hr)) { goto done; }
const CLSID CLSID_MMDeviceEnumerator = __uuidof(MMDeviceEnumerator);
const IID IID_IMMDeviceEnumerator = __uuidof(IMMDeviceEnumerator);
const IID IID_IAudioSessionManager2 = __uuidof(IAudioSessionManager2);
int _tmain(int argc, _TCHAR* argv[])
{
HRESULT hr;
CoInitialize(0);
IMMDeviceEnumerator *deviceEnum = 0;
CHECK_HR(hr = CoCreateInstance(
CLSID_MMDeviceEnumerator, NULL,
CLSCTX_ALL, IID_IMMDeviceEnumerator,
(void**)&deviceEnum));;
IMMDevice *endpoint = 0;
CHECK_HR(deviceEnum->GetDefaultAudioEndpoint(eRender, eMultimedia, &endpoint));
getchar();
// lazy initialization as found in MMDevice.AudioSessionManager..get
IAudioSessionManager2 *m = 0;
CHECK_HR(endpoint->Activate(IID_IAudioSessionManager2, CLSCTX_ALL, 0, (void **)&m));
for (int i = 0; i < 100000; i++)
{
IAudioSessionEnumerator *sessEnum = 0;
m->GetSessionEnumerator(&sessEnum);
sessEnum->Release(); // leak
}
getchar();
printf("success");
return 0;
done:
printf("failure");
return 1;
}
OLD
My guess:
_AudioSessionManager.GetSessionEnumerator(out _SessionEnum) yields an enumerator. When you call the constructor SessionCollection(_SessionEnum), then _SessionEnum is being enumerated over. Each enumeration step retrieves an actual unmanaged object.
If it's a value type, then it would actually be copied into session collection (remember that the List(IEnumerable e) constructor copies each element). The copy would then be garbage collected, but the original object was allocated from unmanaged code and procudes a leak. If this is the case, you should free the memory immediately after calling the Collection constructor using some unmanage memory free function.
If it's a reference type, it wouldn't be freed either because the actual object in the memory isn't garbage collected, since it was allocated from within unmanaged code. If this is the case, you need to free the memory of the objects with unmanaged library functions when you no longer need them.

If you have unmanaged code, when is the _Sessions memory released? If you simply reassign the private field, then the memory is never released.
Here's an example:
http://social.msdn.microsoft.com/forums/en-US/clr/thread/b2162d42-0d7a-4513-b02c-afd6cdd854bd
You need to use the dll's method for freeing up the memory (delete[] in C++)

.NET has always had the ability to easily leak memory - or rather, its doen to un-collected garbage that never gets cleaned up as the GC thinks they're in use. The most famous incident was the DARPA challenge team who believed the hype and thought the leak bug was in their C driver code, poor people.
Since those days, there have been quite a few memory leak profilers appear. I think the most famous one in Redgate's ANTS, but there are loads of others. run your app, see which objects out live their welcome, see which objects have a reference to them, put some code in unreferencing them at the right places (eg a few more Dispose and/or using statements).

Pinning a delegate within a struct before passing to unmanaged code

I'm trying to use an unmanaged C dll for loading image data into a C# application. The library has a fairly simple interface where you pass in a struct that contains three callbacks, one to receive the size of the image, one that receives each row of the pixels and finally one called when the load is completed. Like this (C# managed definition):
[System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)]
public struct st_ImageProtocol
{
public st_ImageProtocol_done Done;
public st_ImageProtocol_setSize SetSize;
public st_ImageProtocol_sendLine SendLine;
}
The types starting st_ImageProtocol are delgates:
public delegate int st_ImageProtocol_sendLine(System.IntPtr localData, int rowNumber, System.IntPtr pixelData);
With the test file that I'm using the SetSize should get called once, then the SendLine will get called 200 times (once for each row of pixels in the image), finally the Done callback gets triggered. What actually happens is that the SendLine is called 19 times and then a AccessViolationException is thrown claiming that the library tried to access protected memory.
I have access to the code of the C library (though I can't change the functionality) and during the loop where it calls the SendLine method it does not allocate or free any new memory, so my assumption is that the delegate itself is the issue and I need to pin it before I pass it in (I have no code inside the delegate itself currently, besides a counter to see how often it gets called, so I doubt I'm breaking anything on the managed side). The problem is that I don't know how to do this; the method I've been using to declare the structs in unmanaged space doesn't work with delegates (Marshal.AllocHGlobal()) and I can't find any other suitable method. The delegates themselves are static fields in the Program class so they shouldn't be being garbage collected, but I guess the runtime could be moving them.
This blog entry by Chris Brumme says that delegates don't need to be pinned before being passed into unmanaged code:
Clearly the unmanaged function pointer must refer to a fixed address. It would be a disaster if the GC were relocating that! This leads many applications to create a pinning handle for the delegate. This is completely unnecessary. The unmanaged function pointer actually refers to a native code stub that we dynamically generate to perform the transition & marshaling. This stub exists in fixed memory outside of the GC heap.
But I don't know if this holds true when the delegate is part of a struct. It does imply that it is possible to manually pin them though, and I'm interested in how to do this or any better suggestions as to why a loop would run 19 times then suddenly fail.
Thanks.
Edited to answer Johan's questions...
The code that allocates the struct is as follows:
_sendLineFunc = new st_ImageProtocol_sendLine(protocolSendLineStub);
_imageProtocol = new st_ImageProtocol()
{
//Set some other properties...
SendLine = _sendLineFunc
};
int protocolSize = Marshal.SizeOf(_imageProtocol);
_imageProtocolPtr = Marshal.AllocHGlobal(protocolSize);
Marshal.StructureToPtr(_imageProtocol, _imageProtocolPtr, true);
Where the _sendLineFunc and the _imageProtocol variables are both static fields of the Program class. If I understand the internals of this correctly, that means that I'm passing an unmanaged pointer to a copy of the _imageProtocol variable into the C library, but that copy contains a reference to the static _sendLineFunc. This should mean that the copy isn't touched by the GC - since it is unmanaged - and the delegate won't be collected since it is still in scope (static).
The struct actually gets passed to the library as a return value from another callback, but as a pointer:
private static IntPtr beginCallback(IntPtr localData, en_ImageType imageType)
{
return _imageProtocolPtr;
}
Basically there is another struct type that holds the image filename and the function pointer to this callback, the library figures out what type of image is stored in the file and uses this callback to request the correct protocol struct for the given type. My filename struct is declared and managed in the same way as the protocol one above, so probably contains the same mistakes, but since this delegate is only called once and called quickly I haven't had any problems with it yet.
Edited to update
Thanks to everybody for their responses, but after spending another couple of days on the problem and making no progress I decided to shelve it. In case anyone is interested I was attempting write a tool for users of the Lightwave 3D rendering application and a nice feature would have been the ability to view all the different image formats that Lightwave supports (some of which are fairly exotic). I thought that the best way to do this would be to write a C# wrapper for the plugin architecture that Lightwave uses for image manipulation so I could use their code to actually load the files. Unfortunately after trying a number of the plugins against my solution I had a variety of errors that I couldn't understand or fix and my guess is that Lightwave doesn't call the methods on the plugins in a standard way, probably to improve the security of running external code (wild stab in the dark, I admit). For the time being I'm going to drop the image feature and if I do decide to reinstate it I'll approach it in a different way.
Thanks again, I learnt a lot through this process even though I didn't get the result I wanted.

I had a similar problem when registering a callback delegate (it would be called, then poof!). My problem was that the object with the method being delegated was getting GC'ed. I created the object in a more global place so as to keep it from being GC'ed.
If something like that doesn't work, here are some other things to look at:
As additional info, take a look at GetFunctionPointerForDelegate from the Marshal class. That is another way you could do this. Just make sure that the delegates are not GC'ed. Then, instead of delegates in your struct, declare them as IntPtr.
That may not solve the pinning, but take a look at fixed keyword, even though that may not work for you since you are dealing with a longer lifetime than for what that is typically used.
Finally, look at stackalloc for creating non-GC memory. These methods will require the use of unsafe, and might therefore put some other constraints on your Assemblies.

It would be interesting to know a little more:
How do you create the ImageProtocol struct? Is it a local variable or a class member or do you allocate it in unmanaged memory with Marshal.AllocHGlobal?
How is it sent to the C function? Directly as stack variable or as a pointer?
A really tricky problem! It feels like the delegate data is moved around by the GC which causes the access violation. The interesting thing is that the delegate data type is a reference data type, which stores its data on the GC heap. This data contains things like the address of the function to call (function pointer) but also a reference to the object that contains the function. This should mean that even though the actual function code is stored outside of the GC heap, the data that holds the function pointer is stored in the GC heap and can hence be moved by the GC. I thought about the problem a lot last night but haven't come up with a solution....

You don't say exactly how the callback is declared in the C library. Unless it is explictly declared __stdcall you'll slowly corrupt your stack. You'll see your method get called (probably with the parameters reversed) but at some point in the future the program will crash.
So far as I know there is no way around that, other than writing another callback function in C that sits between the C# code and the library that wants a __cdecl callback.

If the c function is a __cdecl function then you have to use the Attribut
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
before the delegate declaration.