We Keep Coding

Displaying a Modal Window from a C# COM component that uses async called by a C++ MFC application

I've created a simple c# COM component that is meant to be called by a C++ MFC application. It works but has an undesired side-effect.
The C++ client calls the c# component using it's main GUI thread, so I can't block it because the C# code is making Database operations that may take longer. That's why i need async or a thread for this...
This is the C# code, simplified:
public async void ShowOverviewDialogAsync()
{
var w = new Window();
var dbOperationOk = await LongDbOperaionAsync();
w.ShowDialog();
}
Well, the C++ code goes in, and returns after the call to async which is expected. Here is the C++ calling code:
HRESULT hr = cSharpCom.CoCreateInstance(__uuidof(CSharpCom));
if (SUCCEEDED(hr))
{
cSharpCom->ShowOverviewDialogAsync();
}
// continues without waiting for the dialog close...
The thing is, after this call, the C++ code continues to do its thing and the cSharpCom object goes out of scope. So the only way to call a Release is to make the c# object a member and on destructing or creating a new one, do a release call:
if (_cSharpCom != NULL) _cSharpCom->Release();
HRESULT hr = _cSharpCom.CoCreateInstance(__uuidof(CSharpCom));
if (SUCCEEDED(hr))
{
_cSharpCom->ShowOverviewDialogAsync();
}
This is the first Drawback. The second drawback is that the modal C# window is not a real Modal window as the message pooling of the MFC(C++) continues as normal, which is actually undesired (Right?)
These are the current ideas:
return a HWND to the calling code and somehow make the modal call using the C++ Cwnd:RunModalLoop.
return some kind of event to the calling code, that can be waited upon without blocking the GUI thread.
set a callback that will make sure the COM object will be released.
override the Task class so that it returns a COM aware Interface that allows the C++ code to wait on something (see 2. and 3.)
Number 4 is what I will probably try out. The COM signature would look like:
HRESULT _stdcall ShowOverviewDialogAsync([out, retval] IUnknown** pRetVal);
Just for the lack of completeness, if the method has no async, the MFC C++ GUI hread works as expected and shows the Window in a modal way:
public void ShowOverviewDialogAsync()
{
var w = new Window();
var dbOperationOk = LongDbOperaion();
w.ShowDialog(); // C++ will wait here - non-blocking
}
If you are asking why my async method is returning void. Well, it's defined in an interface and I can't use Task as a return value, unless I go for the solution 4 above.
[ComVisible(true)]
[Guid("XXXXXX-xxxx-xxx-xxx-XXXXX")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface ICSharpCom
{
/// <summary>
/// Displays the dialog in async mode
/// </summary>
[DispId(0)]
void ShowOverviewDialogAsync();
}
Besides, COM interop does not accept generics like Task.
How to test it. Create a MFC application with one button that calls the c# com as defined.
Any ideas appreciated.
cheers,
Marco

I've managed to get it working, although I'm not very happy with the spaghetti code needed in c++.
This is how I've managed until now:
[ComVisible(true)]
[Guid("XXXXXX-xxxx-xxx-xxx-XXXXX")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface ICSharpCom
{
/// <summary>
/// Displays the dialog in async mode
/// </summary>
[DispId(0)]
void ShowOverviewDialogAsync(IntPtr eventHandle);
}
//Implementation:
...
_resetEvent = new ManualResetEvent(false);
m.SafeWaitHandle = new Microsoft.Win32.SafeHandles.SafeWaitHandle(eventHandle, ownsHandle);
// ... at some later point in the method, when the window closes
_resetEvent .Set()
Until now, looks ok.
On the C++ side it gets complicated:
auto handle = CreateEvent(NULL, TRUE, FALSE, CString("MyEvent"));
if (handle != NULL)
{
HANDLE handles[] = { handle };
hr = prescOverview->ShowOverviewDialog((long)handle);
auto handleCount = _countof(handles);
if (SUCCEEDED(hr))
{
BOOL running = TRUE;
do {
DWORD const res = ::MsgWaitForMultipleObjectsEx(
handleCount,
handles,
INFINITE,
QS_ALLINPUT,
MWMO_INPUTAVAILABLE | MWMO_ALERTABLE);
if (res == WAIT_FAILED)
{
running = FALSE;
hr = GetLastError();
break;
}
else if (res == WAIT_OBJECT_0 + 0)
{
CComBSTR err;
prescOverview->GetError(&err);
CString errContainer(err);
if (errContainer.GetLength() > 0)
{
// log
hr = E_FAIL;
}
else {
hr = S_OK;
}
running = FALSE;
break;
}
else if (res >= WAIT_OBJECT_0 && res <= WAIT_OBJECT_0 + handleCount)
{
// process messages.
MSG msg;
while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE))
{
if (msg.message == WM_QUIT)
{
PostQuitMessage(static_cast<int>(msg.wParam));
hr = ERROR_CANCELLED;
running = FALSE;
break;
}
else {
TranslateMessage(&msg);
DispatchMessage(&msg);
}
}
}
} while (running);
}
CloseHandle(handle);
}
}
It's working. However The calling client must know how to do it -> read the docs and it's not intuitive.
I'm still working on it though.
Cheers,
Marco

I've just noticed i've never updated this.
Well, the easiest way to achieve this is to use events. If you are using ATL this will help you. I've managed this by first declaring the C# COM like this:
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
[ComDefaultInterface(typeof(ICSharpCom))]
[ComSourceInterfaces(typeof(ICSharpComEventHandler))]
[Guid("XXXXXX-xxxx-xxx-xxx-XXXXX")]
public class CSharpCom : ICSharpCom
{
[ComVisible(false)]
public delegate void WorkCompleted(string result);
public event WorkCompleted OnWorkCompleted;
public int DoWork(string input)
{
Task t = ....
// do some hard work aync by usin
OnWorkCompleted?.Invoke(t.Result);
}
}
[ComVisible(true)]
[Guid("XXXXXX-xxxx-yyy-xxx-XXXXX")]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
public interface ICSharpComEventHandler
{
[DispId(1)]
void OnWorkCompleted(string result);
}
On The C++ side, it's not that easy
_ATL_FUNC_INFO OnWorkCompletedInfo = {CC_STDCALL, VT_EMPTY, 1, {VT_BSTR}};
class CEventSink : public IDispEventSimpleImpl<ONE_GOOD_ID, CEventSink, &DIID_ICSharpComEventHandler>
{
public:
BEGIN_SINK_MAP(CEventSink)
SINK_ENTRY_INFO(ONE_GOOD_ID, DIID_ICSharpComEventHandler, 1, OnWorkCompleted, &OnWorkCompletedInfo)
END_SINK_MAP()
const void __stdcall OnWorkCompleted(_bstr_t result)
{
// do something
}
CEventSink(CComPtr<ICSharpCom> psharp)
{
if (!pEps)
{
throw std::invalid_argument("psharp was null");
}
m_pSharp = psharp;
DispEventAdvise(m_pSharp);
}
void __stdcall StartListening()
{
DispEventAdvise(m_pSharp);
}
void __stdcall StopListening()
{
DispEventUnadvise(m_pSharp);
m_pSharp = nullptr;
}
private:
CComPtr<ICSharpCom> m_pSharp;
};
That's it guys.
Sorry for not posting before.
Cheers,
Marco

Cross-platform C code and preventing garbage collection

I have a set of C functions that I need to use on an ARM target, in C++ and in C#. I can successfully wrap up the C into a C++ DLL and then into a C# DLL and use all the C functions I've bound successfully. However, I have a debug function that I want to be able to print to the C# GUI and the delegate it uses is being garbage collected rather than left in place for the duration.
Managed Debugging Assistant 'CallbackOnCollectedDelegate' has detected a
problem in 'C:\utm\pc\utm_win32_app\bin\Debug\utm_win32_app.vshost.exe'.
Additional Information: A callback was made on a garbage collected delegate of
type
'utm_dll_wrapper_cs!MessageCodec.MessageCodec_dll+guiPrintToConsoleCallback::
Invoke'. This may cause application crashes, corruption and data loss. When
passing delegates to unmanaged code, they must be kept alive by the managed
application until it is guaranteed that they will never be called.
Here's the snippet of C code that uses and sets up the callback mp_guiPrintToConsole:
#ifdef WIN32
static void (* mp_guiPrintToConsole) (const char*) = NULL;
void logMsg (const char * pFormat, ...)
{
char buffer[MAX_DEBUG_MESSAGE_LEN];
va_list args;
va_start (args, pFormat);
vsnprintf (buffer, sizeof (buffer), pFormat, args);
va_end (args);
#ifdef WIN32
if (mp_guiPrintToConsole)
{
(*mp_guiPrintToConsole) (buffer);
}
#else
// Must be on ARM
printf (buffer);
#endif
}
void initDll (void (*guiPrintToConsole) (const char *))
{
#ifdef WIN32
mp_guiPrintToConsole = guiPrintToConsole;
// This is the signal to the GUI that we're done with initialisation
logMsg ("ready.\r\n");
#endif
}
Here's the C++ code, built into a DLL along with the C code, that can be called from C# and passes in the function pointer printToConsole:
void msInitDll (void (*printToConsole) (const char *))
{
initDll (printToConsole);
}
Here's the snippet code from the C# DLL that calls msInitDll(), passing in guiPrintToConsole(), and defines the delegate onConsoleTrace, which I guess is the thing that is disappearing:
[UnmanagedFunctionPointer (CallingConvention.Cdecl)]
public delegate void _msInitDll([MarshalAs (UnmanagedType.FunctionPtr)] guiPrintToConsoleCallback callbackPointer);
public _msInitDll msInitDll;
public delegate void ConsoleTrace(string data);
public event ConsoleTrace onConsoleTrace;
public void guiPrintToConsole(StringBuilder data)
{
if (onConsoleTrace != null)
{
onConsoleTrace (data.ToString ());
}
}
public void bindDll(string dllLocation)
{
IntPtr ptrDll = LoadLibrary (dllLocation);
if (ptrDll == IntPtr.Zero) throw new Exception (String.Format ("Cannot find {0}", dllLocation));
//...
// All the other DLL function bindings are here
//...
msInitDll = (_msInitDll)bindItem(ptrDll, "msInitDll", typeof(_msInitDll));
msInitDll(guiPrintToConsole);
}
I've looked at the various answers here and the most promising seemed to be to create a static variable in the C# code:
static GCHandle gch;
...and then use that to reference onConsoleTrace in the C# bindDll() function:
gch = GCHandle.Alloc(onConsoleTrace);
However, that doesn't do me any good. I've tried a few other attempts at declaring things static but nothing seems to get me where I want to be. Can anyone suggest another approach to fixing the problem? I have a bug that I need to fix and the lack of any debug is proving quite annoying.
Rob

The following line uses some syntactic sugar:
msInitDll(guiPrintToConsole);
The full syntax is:
msInitDll(new guiPrintToConsoleCallback(guiPrintToConsole));
Hopefully now you see why the delegate can get garbage-collected.
One simple workaround:
var callback = new guiPrintToConsoleCallback(guiPrintToConsole);
msInitDll(callback);
// ... some other code
GC.KeepAlive(callback);
Now the delegate is guaranteed to be alive up to the GC.KeepAlive call.
But you most probably need the delegate to stay alive for longer. As the error message says, simply keep a reference to it. If you need it for the full C# app lifetime duration, turn the callback local to a static field in your class. Static fields are treated as GC roots as their values are always reachable.

And the answer was, in the C# DLL code, add the static variable:
public static guiPrintToConsoleCallback debugCallback;
...and then, in C# bindDLL(), change:
msInitDll(guiPrintToConsole);
...to
debugCallback = new guiPrintToConsoleCallback(guiPrintToConsole);
msInitDll(debugCallback);
Simple when you know how.

Struct contains a function and object to be handled by the function in C# .Net

I am trying to create a struct that contains a function and an object to be handled by that function. For example:
public delegate void MyFunc(object o);
public struct MyData
{
public object o;
public MyFunc func;
public MyData(object o, MyFunc func)
{
this.o = o;
this.func = func;
}
public HandleData()
{
func(o);
}
}
The purpose of this struct is to use any function to handle any data.
I wrap this struct into a IntPtr data type and send to another to handle this struct
private void PrepareData(object o, MyFunc func)
{
MyData md = new MyData(o, func);
int size = Marshal.SizeOf(md);
IntPtr wParam = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(md, wParam, false);
DoJob(wParam);
}
private void DoJob(IntPtr wParam)
{
if (wParam != IntPtr.Zero)
{
Type type = typeof(MyData);
MyData md = (MyData)Marshal.PtrToStructure(p,type);
md.HandleData();
}
}
Sometimes, I receive error like:
1) "[System.RuntimeType] = {Name = Cannot evaluate expression because the code of the current method is optimized. FullName = Cannot evaluate expression because the code of the current method is optimized.}"
2) in HanldeData function, the func instance variable has been Garbage Collected, and is not able to work properly.
like:
{Method = Cannot evaluate expression because a thread is stopped at a point where garbage collection is impossible, possibly because the code is optimized.}
Managed Debugging Assistant 'CallbackOnCollectedDelegate' has detected a problem
Note: I haven't check the "optimise code" in the property of the project.

Without a complete code example and a clear description of what you are actually trying to achieve here, it's impossible to know for sure what the best answer. That said, frankly, the whole scheme seems nuts to me. IntPtr? Seriously?
I don't see anything in your question that describes a problem that can't be accomplished more easily simply by wrapping the delegate and object in a new delegate object.
E.g.:
private void PrepareData(object o, MyFunc func)
{
DoJob(() => func(o));
}
private void DoJob(Action wParam)
{
if (wParam != null)
{
wParam();
}
}
Note that in your original code, if the only references left to your object and delegate are in the unmanaged block you allocate, they may in fact be GC'ed, as they would then be unreachable via any managed reference (which is the only thing the GC cares about).
Note also that even if the objects are still reachable via a managed reference, that the GC may move the objects in memory (e.g. to compact the heap), rendering the values you've copied into your unmanaged block of memory useless.
If you stick to the use of managed code and objects in your program, you will avoid these problems.
(I also note that your code example doesn't even seem valid, as your DoJob() parameter name is wParam, but the variable you check and marshal back to a managed struct is named p).

I am not sure that the Unmanaged Code will be GC'ed. However, I follow the tutorial: link
and what I am missing in my code is just to forget free the allocated memory for the next use.
private void DoJob(IntPtr wParam)
{
if (wParam != IntPtr.Zero)
{
Type type = typeof(MyData);
MyData md = (MyData)Marshal.PtrToStructure(p,type);
md.HandleData();
// Free the unmanaged representation of MyData struct.
Marshal.FreeHGlobal(wParam);
}
}
It works very well until now after many tests.

How to instance a C# class in UNmanaged memory? (Possible?)

UPDATE: There is now an accepted answer that "works". You should never, ever, ever, ever use it. Ever.
First let me preface my question by stating that I'm a game developer. There's a legitimate - if highly unusual - performance-related reason for wanting to do this.
Say I have a C# class like this:
class Foo
{
public int a, b, c;
public void MyMethod(int d) { a = d; b = d; c = a + b; }
}
Nothing fancy. Note that it is a reference type that contains only value types.
In managed code I'd like to have something like this:
Foo foo;
foo = Voodoo.NewInUnmanagedMemory<Foo>(); // <- ???
foo.MyMethod(1);
What would the function NewInUnmanagedMemory look like? If it can't be done in C#, could it be done in IL? (Or maybe C++/CLI?)
Basically: Is there a way - no matter how hacky - to turn some totally arbitrary pointer into an object reference. And - short of making the CLR explode - damn the consequences.
(Another way to put my question is: "I want to implement a custom allocator for C#")
This leads to the follow-up question: What does the garbage collector do (implementation-specific, if need be) when faced with a reference that points outside of managed memory?
And, related to that, what would happen if Foo had a reference as a member field? What if it pointed at managed memory? What if it only ever pointed at other objects allocated in unmanaged memory?
Finally, if this is impossible: Why?
Update: Here are the "missing pieces" so far:
#1: How to convert an IntPtr to an object reference? It might be possible though unverifiable IL (see comments). So far I've had no luck with this. The framework seems to be extremely careful to prevent this from happening.
(It would also be nice to be able to get the size and layout information for non-blittable managed types at runtime. Again, the framework tries to make this impossible.)
#2: Assuming problem one can be solved - what does the GC do when it encounters an object reference that points outside of the GC heap? Does it crash? Anton Tykhyy, in his answer, guesses that it will. Given how careful the framework is to prevent #1, it does seem likely. Something that confirms this would be nice.
(Alternatively the object reference could point to pinned memory inside the GC heap. Would that make a difference?)
Based on this, I'm inclined to think that this idea for a hack is impossible - or at least not worth the effort. But I'd be interested to get an answer that goes into the technical details of #1 or #2 or both.

I have been experimenting creating classes in unmanaged memory. It is possible but has a problem I am currently unable to solve - you can't assign objects to reference-type fields -see edit at the bottom-, so you can have only structure fields in your custom class.
This is evil:
using System;
using System.Reflection.Emit;
using System.Runtime.InteropServices;
public class Voodoo<T> where T : class
{
static readonly IntPtr tptr;
static readonly int tsize;
static readonly byte[] zero;
public static T NewInUnmanagedMemory()
{
IntPtr handle = Marshal.AllocHGlobal(tsize);
Marshal.Copy(zero, 0, handle, tsize);
IntPtr ptr = handle+4;
Marshal.WriteIntPtr(ptr, tptr);
return GetO(ptr);
}
public static void FreeUnmanagedInstance(T obj)
{
IntPtr ptr = GetPtr(obj);
IntPtr handle = ptr-4;
Marshal.FreeHGlobal(handle);
}
delegate T GetO_d(IntPtr ptr);
static readonly GetO_d GetO;
delegate IntPtr GetPtr_d(T obj);
static readonly GetPtr_d GetPtr;
static Voodoo()
{
Type t = typeof(T);
tptr = t.TypeHandle.Value;
tsize = Marshal.ReadInt32(tptr, 4);
zero = new byte[tsize];
DynamicMethod m = new DynamicMethod("GetO", typeof(T), new[]{typeof(IntPtr)}, typeof(Voodoo<T>), true);
var il = m.GetILGenerator();
il.Emit(OpCodes.Ldarg_0);
il.Emit(OpCodes.Ret);
GetO = m.CreateDelegate(typeof(GetO_d)) as GetO_d;
m = new DynamicMethod("GetPtr", typeof(IntPtr), new[]{typeof(T)}, typeof(Voodoo<T>), true);
il = m.GetILGenerator();
il.Emit(OpCodes.Ldarg_0);
il.Emit(OpCodes.Ret);
GetPtr = m.CreateDelegate(typeof(GetPtr_d)) as GetPtr_d;
}
}
If you care about memory leak, you should always call FreeUnmanagedInstance when you are done with your class.
If you want more complex solution, you can try this:
using System;
using System.Reflection.Emit;
using System.Runtime.InteropServices;
public class ObjectHandle<T> : IDisposable where T : class
{
bool freed;
readonly IntPtr handle;
readonly T value;
readonly IntPtr tptr;
public ObjectHandle() : this(typeof(T))
{
}
public ObjectHandle(Type t)
{
tptr = t.TypeHandle.Value;
int size = Marshal.ReadInt32(tptr, 4);//base instance size
handle = Marshal.AllocHGlobal(size);
byte[] zero = new byte[size];
Marshal.Copy(zero, 0, handle, size);//zero memory
IntPtr ptr = handle+4;
Marshal.WriteIntPtr(ptr, tptr);//write type ptr
value = GetO(ptr);//convert to reference
}
public T Value{
get{
return value;
}
}
public bool Valid{
get{
return Marshal.ReadIntPtr(handle, 4) == tptr;
}
}
public void Dispose()
{
if(!freed)
{
Marshal.FreeHGlobal(handle);
freed = true;
GC.SuppressFinalize(this);
}
}
~ObjectHandle()
{
Dispose();
}
delegate T GetO_d(IntPtr ptr);
static readonly GetO_d GetO;
static ObjectHandle()
{
DynamicMethod m = new DynamicMethod("GetO", typeof(T), new[]{typeof(IntPtr)}, typeof(ObjectHandle<T>), true);
var il = m.GetILGenerator();
il.Emit(OpCodes.Ldarg_0);
il.Emit(OpCodes.Ret);
GetO = m.CreateDelegate(typeof(GetO_d)) as GetO_d;
}
}
/*Usage*/
using(var handle = new ObjectHandle<MyClass>())
{
//do some work
}
I hope it will help you on your path.
Edit: Found a solution to reference-type fields:
class MyClass
{
private IntPtr a_ptr;
public object a{
get{
return Voodoo<object>.GetO(a_ptr);
}
set{
a_ptr = Voodoo<object>.GetPtr(value);
}
}
public int b;
public int c;
}
Edit: Even better solution. Just use ObjectContainer<object> instead of object and so on.
public struct ObjectContainer<T> where T : class
{
private readonly T val;
public ObjectContainer(T obj)
{
val = obj;
}
public T Value{
get{
return val;
}
}
public static implicit operator T(ObjectContainer<T> #ref)
{
return #ref.val;
}
public static implicit operator ObjectContainer<T>(T obj)
{
return new ObjectContainer<T>(obj);
}
public override string ToString()
{
return val.ToString();
}
public override int GetHashCode()
{
return val.GetHashCode();
}
public override bool Equals(object obj)
{
return val.Equals(obj);
}
}

"I want to implement a custom allocator for C#"
GC is at the core of the CLR. Only Microsoft (or the Mono team in case of Mono) can replace it, at a great cost in development effort. GC being at the core of the CLR, messing around with the GC or the managed heap will crash the CLR — quickly if you're very-very lucky.
What does the garbage collector do (implementation-specific, if need be) when faced with a reference that points outside of managed memory?
It crashes in an implementation-specific way ;)

Purely C# Approach
So, there are a few options. The easiest is to use new/delete in an unsafe context for structs. The second is to use built-in Marshalling services to deal with unmanaged memory (code for this is visible below). However, both of these deal with structs (though I think the latter method is very close to what you want). My code has a limitation in that you must stick to structures throughout and use IntPtrs for references (using ChunkAllocator.ConvertPointerToStructure to get the data and ChunkAllocator.StoreStructure to store the changed data). This is obviously cumbersome, so you'd better really want the performance if you use my approach. However, if you are dealing with only value-types, this approach is sufficient.
Detour: Classes in the CLR
Classes have a 8 byte "prefix" in their allocated memory. Four bytes are for the sync index for multithreading, and four bytes are for identifying their type (basically, virtual method table and run-time reflection). This makes it hard to deal with unamanaged memory since these are CLR specific and since the sync index can change during run-time. See here for details on run-time object creation and here for an overview of memory layout for a reference type. Also check out CLR via C# for a more in-depth explanation.
A Caveat
As usual, things are rarely so simple as yes/no. The real complexity of reference types has to do with how the garbage collector compacts allocated memory during a garbage collection. If you can somehow ensure that a garbage collection doesn't happen or that it won't affect the data in question (see the fixed keyword) then you can turn an arbitrary pointer into an object reference (just offset the pointer by 8 bytes, then interpret that data as a struct with the same fields and memory layout; perhaps use StructLayoutAttribute to be sure). I would experiment with non-virtual methods to see if they work; they should (especially if you put them on the struct) but virtual methods are no-go due to the virtual method table that you'd have to discard.
One Does Not Simply Walk Into Mordor
Simply put, this means that managed reference types (classes) cannot be allocated in unmanaged memory. You could use managed reference types in C++, but those would be subject to garbage collection... and the process and code is more painful than the struct-based approach. Where does that leave us? Back where we started, of course.
There is a Secret Way
We could brave Shelob's Lair memory allocation ourselves. Unfortunately, this is where our paths must part, because I am not that knowledgeable about it. I will provide you with a link or two - perhaps three or four in actuality. This is rather complicated and begs the question: Are there other optimizations you could try? Cache coherency and superior algorithms is one approach, as is judicious application of P/Invoke for performance-critical code. You could also apply the aforementioned structures-only memory allocation for key methods/classes.
Good luck, and let us know if you find a superior alternative.
Appendix: Source Code
ChunkAllocator.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
namespace MemAllocLib
{
public sealed class ChunkAllocator : IDisposable
{
IntPtr m_chunkStart;
int m_offset;//offset from already allocated memory
readonly int m_size;
public ChunkAllocator(int memorySize = 1024)
{
if (memorySize < 1)
throw new ArgumentOutOfRangeException("memorySize must be positive");
m_size = memorySize;
m_chunkStart = Marshal.AllocHGlobal(memorySize);
}
~ChunkAllocator()
{
Dispose();
}
public IntPtr Allocate<T>() where T : struct
{
int reqBytes = Marshal.SizeOf(typeof(T));//not highly performant
return Allocate<T>(reqBytes);
}
public IntPtr Allocate<T>(int reqBytes) where T : struct
{
if (m_chunkStart == IntPtr.Zero)
throw new ObjectDisposedException("ChunkAllocator");
if (m_offset + reqBytes > m_size)
throw new OutOfMemoryException("Too many bytes allocated: " + reqBytes + " needed, but only " + (m_size - m_offset) + " bytes available");
T created = default(T);
Marshal.StructureToPtr(created, m_chunkStart + m_offset, false);
m_offset += reqBytes;
return m_chunkStart + (m_offset - reqBytes);
}
public void Dispose()
{
if (m_chunkStart != IntPtr.Zero)
{
Marshal.FreeHGlobal(m_chunkStart);
m_offset = 0;
m_chunkStart = IntPtr.Zero;
}
}
public void ReleaseAllMemory()
{
m_offset = 0;
}
public int AllocatedMemory
{
get { return m_offset; }
}
public int AvailableMemory
{
get { return m_size - m_offset; }
}
public int TotalMemory
{
get { return m_size; }
}
public static T ConvertPointerToStruct<T>(IntPtr ptr) where T : struct
{
return (T)Marshal.PtrToStructure(ptr, typeof(T));
}
public static void StoreStructure<T>(IntPtr ptr, T data) where T : struct
{
Marshal.StructureToPtr(data, ptr, false);
}
}
}
Program.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace MemoryAllocation
{
class Program
{
static void Main(string[] args)
{
using (MemAllocLib.ChunkAllocator chunk = new MemAllocLib.ChunkAllocator())
{
Console.WriteLine(">> Simple data test");
SimpleDataTest(chunk);
Console.WriteLine();
Console.WriteLine(">> Complex data test");
ComplexDataTest(chunk);
}
Console.ReadLine();
}
private static void SimpleDataTest(MemAllocLib.ChunkAllocator chunk)
{
IntPtr ptr = chunk.Allocate<System.Int32>();
Console.WriteLine(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Int32>(ptr));
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Int32>(ptr) == 0, "Data not initialized properly");
System.Diagnostics.Debug.Assert(chunk.AllocatedMemory == sizeof(Int32), "Data not allocated properly");
int data = MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Int32>(ptr);
data = 10;
MemAllocLib.ChunkAllocator.StoreStructure(ptr, data);
Console.WriteLine(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Int32>(ptr));
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Int32>(ptr) == 10, "Data not set properly");
Console.WriteLine("All tests passed");
}
private static void ComplexDataTest(MemAllocLib.ChunkAllocator chunk)
{
IntPtr ptr = chunk.Allocate<Person>();
Console.WriteLine(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr));
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr).Age == 0, "Data age not initialized properly");
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr).Name == null, "Data name not initialized properly");
System.Diagnostics.Debug.Assert(chunk.AllocatedMemory == System.Runtime.InteropServices.Marshal.SizeOf(typeof(Person)) + sizeof(Int32), "Data not allocated properly");
Person data = MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr);
data.Name = "Bob";
data.Age = 20;
MemAllocLib.ChunkAllocator.StoreStructure(ptr, data);
Console.WriteLine(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr));
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr).Age == 20, "Data age not set properly");
System.Diagnostics.Debug.Assert(MemAllocLib.ChunkAllocator.ConvertPointerToStruct<Person>(ptr).Name == "Bob", "Data name not set properly");
Console.WriteLine("All tests passed");
}
struct Person
{
public string Name;
public int Age;
public Person(string name, int age)
{
Name = name;
Age = age;
}
public override string ToString()
{
if (string.IsNullOrWhiteSpace(Name))
return "Age is " + Age;
return Name + " is " + Age + " years old";
}
}
}
}

You can write code in C++ and call it from .NET using P/Invoke or you can you can write code in managed C++ that gives you full access to the native API from inside a .NET language. However, on the managed side you can only work with managed types so you will have to encapsulate your unmanaged objects.
To give a simple example: Marshal.AllocHGlobal allows you to allocate memory on the Windows heap. The handle returned is not of much use in .NET but can be required when calling a native Windows API requiring a buffer.

This is not possible.
However you can use a managed struct and create a pointer of this struct type. This pointer can point anywhere (including to unmanaged memory).
The question is, why would you want to have a class in unmanaged memory? You wouldn't get GC features anyway. You can just use a pointer-to-struct.

Nothing like that is possible. You can access managed memory in unsafe context, but said memory is still managed and subject to GC.
Why?
Simplicity and security.
But now that I think about it, I think you can mix managed and unmanaged with C++/CLI. But I'm not sure about that, because I never worked with C++/CLI.

I don't know a way to hold a C# class instance in the unmanaged heap, not even in C++/CLI.

It's possible to design a value-type allocator entirely within .net, without using any unmanaged code, which can allocate and free an arbitrary number of value-type instances without any significant GC pressure. The trick is to create a relatively small number of arrays (possibly one for each type) to hold the instances, and then pass around "instance reference" structs which hold the array indices of the index in question.
Suppose, for example, that I want to have a "creature" class which holds XYZ positions (float), XYZ velocity (also float), roll/pitch/yaw (ditto), damage (float), and kind (enumeration). An interface "ICreatureReference" would define getters and setters for all those properties. A typical implementation would be a struct CreatureReference with a single private field int _index, and property accessors like:
float Position {
get {return Creatures[_index].Position;}
set {Creatures[_index].Position = value;}
};
The system would keep a list of which array slots are used and vacant (it could, if desired, use one of the fields within Creatures to form a linked list of vacant slots). The CreatureReference.Create method would allocate an item from the vacant-items list; the Dispose method of a CreatureReference instance would add its array slot to the vacant-items list.
This approach ends up requiring an annoying amount of boilerplate code, but it can be reasonably efficient and avoid GC pressure. The biggest problems with are probably that (1) it makes structs behave more like reference types than structs, and (2) it requires absolute discipline with calling IDispose, since non-disposed array slots will never get reclaimed. Another irksome quirk is that one will be unable to use property setters for read-only values of type CreatureReference, even though the property setters would not try to mutate any fields of the CreatureReference instance to which they are applied. Using an interface ICreatureReference may avoid this difficulty, but one must be careful to only declare storage locations of generic types constrained to ICreatureReference, rather than declaring storage locations of ICreatureReference.

Implement callback C# functions for C++ DLL

I'm writing a DLL wrapper for my C++ library, to be called from C#. This wrapper should also have callback functions called from the library and implemented in C#. These functions have for instance std::vector<unsigned char> as output parameters. I don't know how to make this. How do I pass a buffer of unknown size from C# to C++ via a callback function?
Let's take this example
CallbackFunction FunctionImplementedInCSharp;
void FunctionCalledFromLib(const std::vector<unsigned char>& input, std::vector<unsigned char>& output)
{
// Here FunctionImplementedInCSharp (C# delegate) should somehow be called
}
void RegisterFunction(CallbackFunction f)
{
FunctionImplementedInCSharp = f;
}
How should CallbackFunction be defined and what is the code inside FunctionCalledFromLib?
One of the things that dumb me is: how do I delete a buffer created by C# inside C++ code?

As of Visual Studio 2013 at least, there is a safe way to pass callbacks from C# to C++ and have C++ store them and invoke them asynchronously later from unmanaged code. What you can do is create a managed C++/CX class (e.g., named "CallbackManager") to hold the callback delegate references in a map, keyed off an enum value for each. Then your unmanaged code can retrieve a managed delegate reference from the managed C++/CX CallbackManager class via the delegate's associated enum value. That way you don't have to store raw function pointers and so you don't have to worry about the delegate being moved or garbage-collected: it stays in the managed heap throughout its lifecycle.
On the C++ side in CallbacksManager.h:
#include <unordered_map>
#include <mutex>
using namespace Platform;
namespace CPPCallbacks
{
// define callback IDs; this is what unmanaged C++ code will pass to the managed CallbacksManager class to retrieve a delegate instance
public enum class CXCallbackType
{
cbtLogMessage,
cbtGetValueForSetting
// TODO: add additional enum values as you add more callbacks
}
// defines the delegate signatures for our callbacks; these are visible to the C# side as well
public delegate void LogMessageDelegate(int level, String^ message);
public delegate bool GetValueForSettingDelegate(String^ settingName, String^* settingValueOut);
// TODO: define additional callbacks here as you need them
// Singleton WinRT class to manage C# callbacks; since this class is marked 'public' it is consumable from C# as well
public ref class CXCallbacksManager sealed
{
private:
CXCallbacksManager() { } // this is private to prevent incorrect instantiation
public:
// public methods and properties are all consumable by C# as well
virtual ~CXCallbacksManager() { }
static property CXCallbacksManager^ Instance
{
CXCallbacksManager^ get();
}
bool UnregisterCallback(CXCallbackType cbType);
void UnregisterAllCallbacks();
Delegate^ GetCallback(CXCallbackType cbType);
// define callback registration methods
RegisterLogMessageCallback(LogMessageDelegate^ cb) { RegisterCallback(CXCallbackType::cbtLogMessage, cb); }
RegisterGetValueForSettingCallback(GetValueForSettingDelegate^ cb) { RegisterCallback(CXCallbackType::GetValueForSetting, cb); }
// TODO: define additional callback registration methods as you add more callbacks
private:
void RegisterCallback(CXCallbackType cbType, Delegate^ rCallbackFunc);
typedef unordered_map<CXCallbackType, Delegate^> CALLBACK_MAP;
typedef pair<CXCallbackType, Delegate^> CBType_Delegate_Pair;
// Note: IntelliSense errors shown for static data is a Visual Studio IntellSense bug; the code below builds fine
// See http://social.msdn.microsoft.com/Forums/windowsapps/en-US/b5d43215-459a-41d6-a85e-99e3c30a162e/about-static-member-of-ref-class?forum=winappswithnativecode
static mutex s_singletonMutex;
static CXCallbacksManager^ s_rInstance;
mutex m_callbackMapMutex;
CALLBACK_MAP m_callbacksMap; // key=CallbackType, value = C# delegate (function) pointer
};
}
In CallbacksManager.cpp we implement the managed C++/CX class accessed by both C# and our unmanaged C++ code:
#include <assert.h>
#include "CXCallbacksManager.h"
using namespace Platform;
namespace CPPCallbacks
{
// define static class data
CXCallbacksManager^ CXCallbacksManager::s_rInstance;
mutex CXCallbacksManager::s_singletonMutex;
// Returns our singleton instance; this method is thread-safe
CXCallbacksManager^ CXCallbacksManager::Instance::get()
{
s_singletonMutex.lock();
if (s_rInstance == nullptr)
s_rInstance = ref new CXCallbacksManager(); // this lives until the application terminates
s_singletonMutex.unlock();
return s_rInstance;
}
// Register a C# callback; this method is thread-safe
void CXCallbacksManager::RegisterCallback(const CXCallbackType cbType, Delegate^ rCallbackFunc)
{
_ASSERTE(rCallbackFunc);
m_callbackMapMutex.lock();
m_callbacksMap.insert(CBType_Delegate_Pair(cbType, rCallbackFunc));
m_callbackMapMutex.unlock();
}
// Unregister a C# callback; this method is thread-safe
// Returns: true on success, false if no callback was registered for callbackType
bool CXCallbacksManager::UnregisterCallback(const CXCallbackType cbType)
{
m_callbackMapMutex.lock();
const bool bRemoved = (m_callbacksMap.erase(cbType) > 0);
m_callbackMapMutex.unlock();
return bRemoved;
}
// Unregister all callbacks; this method is thread-safe
void CXCallbacksManager::UnregisterAllCallbacks()
{
// must lock the map before iterating across it
// Also, we can't change the contents of the map as we iterate across it, so we have to build a vector of all callback types in the map first.
vector<CXCallbackType> allCallbacksList;
m_callbackMapMutex.lock();
for (CALLBACK_MAP::const_iterator it = m_callbacksMap.begin(); it != m_callbacksMap.end(); it++)
allCallbacksList.push_back(it->first);
for (unsigned int i = 0; i < allCallbacksList.size(); i++)
{
CALLBACK_MAP::const_iterator it = m_callbacksMap.find(allCallbacksList[i]);
if (it != m_callbacksMap.end()) // sanity check; should always succeed
UnregisterCallback(it->first);
}
m_callbackMapMutex.unlock();
}
// Retrieve a registered C# callback; returns NULL if no callback registered for type
Delegate^ CXCallbacksManager::GetCallback(const CXCallbackType cbType)
{
Delegate^ rCallbackFunc = nullptr;
m_callbackMapMutex.lock();
CALLBACK_MAP::const_iterator it = m_callbacksMap.find(cbType);
if (it != m_callbacksMap.end())
rCallbackFunc = it->second;
else
_ASSERTE(false); // should never happen! This means the caller either forgot to register a callback for this cbType or already unregistered the callback for this cbType.
m_callbackMapMutex.unlock();
return rCallbackFunc;
}
}
The delegate instances remain stored in the managed heap by our CXCallbacksManager class, so now it's easy and safe to store callbacks on the C++ side for unmanaged code to invoke later asynchronously. Here is the C# side registering two callbacks:
using CPPCallbacks;
namespace SomeAppName
{
internal static class Callbacks
{
// invoked during app startup to register callbacks for unmanaged C++ code to invoke asynchronously
internal static void RegisterCallbacks()
{
CPPCallbacks.CXCallbacksManager.Instance.RegisterLogMessageCallback(new LogMessageDelegate(LogMessageDelegateImpl));
CPPCallbacks.CXCallbacksManager.Instance.RegisterGetValueForSettingCallback(new GetValueForSettingDelegate(GetValueForSettingDelegateImpl));
// TODO: register additional callbacks as you add them
}
//-----------------------------------------------------------------
// Callback delegate implementation methods are below; these are invoked by C++
// Although these example implementations are in a static class, you could also pass delegate instances created
// from inside a non-static class, which would maintain their state just like any other instance method (i.e., they have a 'this' object).
//-----------------------------------------------------------------
private static void LogMessageDelegateImpl(int level, string message)
{
// This next line is shown for example purposes, but at this point you can do whatever you want because
// you are running in a normal C# delegate context.
Logger.WriteLine(level, message);
}
private static bool GetValueForSettingDelegateImpl(String settingName, out String settingValueOut)
{
// This next line is shown for example purposes, but at this point you can do whatever you want because
// you are running in a normal C# delegate context.
return Utils.RetrieveEncryptedSetting(settingName, out settingValueOut);
}
};
}
Lastly, here is how to invoke your registered C# callbacks from unmanaged C++ code:
#include <assert.h>
#include <atlstr.h> // for CStringW
#include "CXCallbacksManager.h"
using namespace CPPCallbacks;
// this is an unmanaged C++ function in the same project as our CXCallbacksManager class
void LogMessage(LogLevel level, const wchar_t *pMsg)
{
_ASSERTE(msg);
auto rCallback = static_cast<LogMessageDelegate^>(CXCallbacksManager::Instance->GetCallback(CXCallbackType::cbtLogMessage));
_ASSERTE(rCallback);
rCallback(level, ref new String(pMsg)); // invokes C# method
}
// this is an unmanaged C++ function in the same project as our CXCallbacksManager class
// Sets settingValue to the value retrieved from C# for pSettingName
// Returns: true if the value existed and was set, false otherwise
bool GetValueForSetting(const wchar_t *pSettingName, CStringW &settingValue)
{
bool bRetCode = false;
auto rCallback = static_cast<GetValueForSettingDelegate^>(CXCallbacksManager::Instance->GetCallback(CXCallbackType::cbtGetValueForSetting));
_ASSERTE(rCallback);
if (rCallback) // sanity check; should never be null
{
String^ settingValueOut;
bRetCode = rCallback(ref new String(pSettingName), &settingValueOut);
// store the retrieved setting value to our unmanaged C++ CStringW output parameter
settingValue = settingValueOut->Data();
}
return bRetCode;
}
This all works because although you cannot store a managed delegate reference as a member variable inside an unmanaged class, you can still retrieve and invoke a managed delegate from unmanaged code, which is what the above two native C++ methods do.

There are some things you should be aware of. The first is that if you are calling a .NET delegate from unmanaged code, then unless you follow some pretty narrow constraints, you will be in for pain.
Ideally, you can create a delegate in C# pass it into managed code, marshal it into a function pointer, hold onto it for as long as you like, then call it with no ill effects. The .NET documentation says so.
I can tell you that this is simply not true. Eventually, part of your delegate or its thunk will get garbage collected and when you call the function pointer from unmanaged code you will get sent into oblivion. I don't care what Microsoft says, I've followed their prescription to the letter and watched function pointers get turned into garbage, especially in server-side code behinds.
Given that, the most effective way to use function pointers is thus:
C# code calls unmanaged code, passing in delegate.
Unmanaged code marshals the delegate to a function pointer.
Unmanaged code does some work, possible calling the function pointer.
Unmanaged code drops all references to the function pointer.
Unmanaged code returns to managed code.
Given that, suppose we have the following in C#:
public void PerformTrick(MyManagedDelegate delegate)
{
APIGlue.CallIntoUnamangedCode(delegate);
}
and then in managed C++ (not C++/CLI):
static CallIntoUnmanagedCode(MyManagedDelegate *delegate)
{
MyManagedDelegate __pin *pinnedDelegate = delegate;
SOME_CALLBACK_PTR p = Marshal::GetFunctionPointerForDelegate(pinnedDelegate);
CallDeepIntoUnmanagedCode(p); // this will call p
}
I haven't done this recently in C++/CLI - the syntax is different - I think it ends up looking like this:
// This is declared in a class
static CallIntoUnamangedCode(MyManagedDelegate ^delegate)
{
pin_ptr<MyManagedDelegate ^> pinnedDelegate = &delegate;
SOME_CALLBACK_PTR p = Marshal::GetFunctionPointerForDelegate(pinnedDelegate);
CallDeepIntoUnmanagedCode(p); // This will call p
}
When you exit this routines, the pinning gets released.
When you really, really need to have function pointers hanging around for a while before calling, I have done the following in C++/CLI:
Made a hashtable that is a map from int -> delegate.
Made register/unregister routines that add new delegates into the hashtable, bumping up a counter for the hash int.
Made a single static unmanaged callback routine that is registered into unmanaged code with an int from the register call. When this routine is called, it calls back into managed code saying "find the delegate associated with <int> and call it on these arguments".
What happens is that the delegates don't have thunks that do transitions anymore since they're implied. They're free to hang around in limbo being moved by the GC as needed. When they get called, the delegate will get pinned by the CLR and released as needed. I have also seen this method fail, particularly in the case of code that statically registers callbacks at the beginning of time and expects them to stay around to the end of time. I've seen this fail in ASP.NET code behind as well as server side code for Silverlight working through WCF. It's rather unnerving, but the way to fix it is to refactor your API to allow late(r) binding to function calls.
To give you an example of when this will happen - suppose you have a library that includes a function like this:
typedef void * (*f_AllocPtr) (size_t nBytes);
typedef void *t_AllocCookie;
extern void RegisterAllocFunction(f_AllocPtr allocPtr, t_AllocCookie cookie);
and the expectation is that when you call an API that allocates memory, it will be vectored off into the supplied f_AllocPtr. Believe it or not, you can write this in C#. It's sweet:
public IntPtr ManagedAllocMemory(long nBytes)
{
byte[] data = new byte[nBytes];
GCHandle dataHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
unsafe {
fixed (byte *b = &data[0]) {
dataPtr = new IntPtr(b);
RegisterPointerHandleAndArray(dataPtr, dataHandle, data);
return dataPtr;
}
}
}
RegisterPointerHandleAndArray stuffs the triplet away for safe keeping. That way when the corresponding free gets called, you can do this:
public void ManagedFreeMemory(IntPtr dataPointer)
{
GCHandle dataHandle;
byte[] data;
if (TryUnregister(dataPointer, out dataHandle, out data)) {
dataHandle.Free();
// do anything with data? I dunno...
}
}
And of course this is stupid because allocated memory is now pinned in the GC heap and will fragment it to hell - but the point is that it's doable.
But again, I have personally seen this fail unless the actual pointers are short lived. This typically means wrapping your API, so that when you call into a routine that accomplishes a specific task, it registers callbacks, does the task, and then pulls the callbacks out.

As it turns out, the answer to the original question is rather simple, once you know it, and the whole callback issue was no issue. The input buffer parameter is replaced with parameter pair unsigned char *input, int input_length, and the output buffer parameter is replaced with parameter pair unsigned char **output, int *output_length. The C# delegate should be something like this
public delegate int CallbackDelegate(byte[] input, int input_length,
out byte[] output, out int output_length);
And wrapper in C++ should be something like this
void FunctionCalledFromLib(const std::vector<unsigned char>& input, std::vector<unsigned char>& output)
{
unsigned char *output_aux;
int output_length;
FunctionImplementedInCSharp(
&input[0], input.size(), &ouput_aux, &output_length);
output.assign(output_aux, output_aux + output_length);
CoTaskMemFree(output_aux); // IS THIS NECESSARY?
}
The last line is the last part of the mini-puzzle. Do I have to call CoTaskMemFree, or will the marshaller do it for me automagically?
As for the beautiful essay by plinth, I hope to bypass the whole problem by using a static function.

There is no point to using C++/cli.
And here is a real world example from my project.
public ImageSurface(byte[] pngData)
: base(ConstructImageSurfaceFromPngData(pngData), true)
{
offset = 0;
}
private static int offset;
private static IntPtr ConstructImageSurfaceFromPngData(byte[] pngData)
{
NativeMethods.cairo_read_func_t func = delegate(IntPtr closure, IntPtr out_data, int length)
{
Marshal.Copy(pngData, offset, out_data, length);
offset += length;
return Status.Success;
};
return NativeMethods.cairo_image_surface_create_from_png_stream(func, IntPtr.Zero);
}
That is used to transfer PNG data from C# to the native cairo API.
You can see how the C function pointer cairo_read_func_t is implemented in C# and then used as a callback for cairo_image_surface_create_from_png_stream.
Here is a similar example.