We Keep Coding

C# marshal unmanaged pointer return type

I have an unmanaged library which has a function like this:
type* foo();
foo basically allocates an instance of the unmanaged type on the managed heap through Marshal.AllocHGlobal.
I have a managed version of type. It's not blittable but I have MarshalAs attributes set on members so I can use Marshal.PtrToStructure to get a managed version of it. But having to wrap calls to foo with extra bookkeeping to call Marshal.PtrToStructure is a bit annoying.
I'd like to be able to do something like this on the C# side:
[DllImport("mylib", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStruct)]
type* foo();
and have C#'s marshaller handle the conversion behind the scenes, like it does for function arguments. I thought I should be able to do this because type is allocated on the managed heap. But maybe I can't? Is there any way to have C#'s inbuilt marshaller handle the unmanaged-to-managed transition on the return type for me without having to manually call Marshal.PtrToStructure?

A custom marshaler works fine if, on the .NET side, typeis declared as a class, not as a struct.
This is clearly stated in UnmanagedType enumeration:
Specifies the custom marshaler class when used with the
MarshalAsAttribute.MarshalType or MarshalAsAttribute.MarshalTypeRef
field. The MarshalAsAttribute.MarshalCookie field can be used to pass
additional information to the custom marshaler. You can use this
member on any reference type.
Here is some sample code that should work fine
[[DllImport("mylib", CallingConvention = CallingConvention.Cdecl)]
[return : MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef= typeof(typeMarshaler))]
private static extern type Foo();
private class typeMarshaler : ICustomMarshaler
{
public static readonly typeMarshaler Instance = new typeMarshaler();
public static ICustomMarshaler GetInstance(string cookie) => Instance;
public int GetNativeDataSize() => -1;
public object MarshalNativeToManaged(IntPtr nativeData) => Marshal.PtrToStructure<type>(nativeData);
// in this sample I suppose the native side uses GlobalAlloc (or LocalAlloc)
// but you can use any allocation library provided you use the same on both sides
public void CleanUpNativeData(IntPtr nativeData) => Marshal.FreeHGlobal(nativeData);
public IntPtr MarshalManagedToNative(object managedObj) => throw new NotImplementedException();
public void CleanUpManagedData(object managedObj) => throw new NotImplementedException();
}
[StructLayout(LayoutKind.Sequential)]
class type
{
/* declare fields */
};
Of course, changing unmanaged struct declarations into classes can have deep implications (that may not always raise compile-time errors), especially if you have a lot of existing code.
Another solution is to use Roslyn to parse your code, extract all Foo-like methods and generate one additional .NET method for each. I would do this.

type* foo()
This is very awkward function signature, hard to use correctly in a C or C++ program and that never gets better when you pinvoke. Memory management is the biggest problem, you want to work with the programmer that wrote this code to make it better.
Your preferred signature should resemble int foo(type* arg, size_t size). In other words, the caller supplies the memory and the native function fills it in. The size argument is required to avoid memory corruption, necessary when the version of type changes and gets larger. Often included as a field of type. The int return value is useful to return an error code so you can fail gracefully. Beyond making it safe, it is also much more efficient since no memory allocation is required at all. You can simply pass a local variable.
... allocates an instance of the unmanaged type on the managed heap through Marshal.AllocHGlobal
No, this is where memory management assumptions get very dangerous. Never the managed heap, native code has no decent way to call into the CLR. And you cannot assume that it used the equivalent of Marshal.AllocHGlobal(). The native code typically uses malloc() to allocate the storage, which heap is used to allocate from is an implementation detail of the CRT it links. Only that CRT's free() function is guaranteed to release it reliably. You cannot call free() yourself. Skip to the bottom to see why AllocHGlobal() appeared to be correct.
There are function signatures that forces the pinvoke marshaller to release the memory, it does so by calling Marshal.FreeCoTaskMem(). Note that this is not equivalent to Marshal.AllocHGlobal(), it uses a different heap. It assumes that the native code was written to support interop well and used CoTaskMemAlloc(), it uses the heap that is dedicated to COM interop.
It's not blittable but I have MarshalAs attributes set...
That is the gritty detail that explains why you have to make it awkward. The pinvoke marshaller does not want to solve this problem since it has to marshal a copy and there is too much risk automatically releasing the storage for the object and its members. Using [MarshalAs] is unnecessary and does not make the code better, simply change the return type to IntPtr. Ready to pass to Marshal.PtrToStructure() and whatever memory release function you need.
I have to talk about the reason that Marshal.AllocHGlobal() appeared to be correct. It did not used to be, but has changed in recent Windows and VS versions. There was a big design change in Win8 and VS2012. The OS no longer creates separate heaps that Marshal.AllocHGlobal and Marshal.AllocCoTaskMem allocate from. It is now a single heap, the default process heap (GetProcessHeap() returns it). And there was a corresponding change in the CRT included with VS2012, it now also uses GetProcessHeap() instead of creating its own heap with HeapCreate().
Very big change and not publicized widely. Microsoft has not released any motivation for this that I know of, I assume that the basic reason was WinRT (aka UWP), lots of memory management nastiness to get C++, C# and Javascript code to work together seamlessly. This is quite convenient to everybody that has to write interop code, you can now assume that Marshal.FreeHGlobal() gets the job done. Or Marshal.FreeCoTaskMem() like the pinvoke marshaller uses. Or free() like the native code would use, no difference anymore.
But also a significant risk, you can no longer assume that the code is bug-free when it works well on your dev machine and must re-test on Win7. You get an AccessViolationException if you guessed wrong about the release function. It is worse if you also have to support XP or Win2003, no crash at all but you'll silently leak memory. Very hard to deal with that when it happens since you can't get ahead without changing the native code. Best to get it right early.

COM interop passing wrong pointers, with only a single byte of my data

I'm having a problem passing pointers to native code using COM. I want to build up a byte array in managed (C#) code and pass that array to native (C++) code. I'm handling the managed-code side, and my colleague owns the native side. Note that I'm much stronger on the managed side, and I've pretty much been using the COM object as written.
The COM signature (simplified) looks this:
unsafe void DoSomething([In] byte* buffer, [In] uint length)
And I call it like this:
var arr = File.ReadAllBytes(#"c:\temp\foo.bar");
fixed (byte* p = arr)
{
// let's see what we're actually pointing at
IntPtr ip = new IntPtr(p);
Console.WriteLine(ip.ToInt32().ToString("x"));
interop.DoSomething(p, arr.Length);
}
Debugging on the managed side, I see the data I expect at the memory location printed out (using Visual Studio's Memory view). When I debug the unmanaged side, I see the right data at that location as well. But, the pointer doesn't point to the right place! It points to a completely different memory location. That location contains the correct first byte of my data, but garbage for the rest of it. And then, of course, lots of wonderful crashy badness happens.
So, for instance, I see:
managed side: pointer (p) is 0x1234567, Memory at 0x1234567 contains the contents of my file.
unmanaged side: pointer (buffer) is 0x5678901, first byte of memory at that location contains the first byte of my file, rest is garbage. Memory at 0x1234567 contains the contents of my file.
I've also tried automatic marshaling:
void DoSomething([In] [MarshalAs(UnmanagedType.LPArray,
ArraySubType = UnmanagedType.U1)]
byte[] buffer,
[In] uint length)
... using straight byte arrays on the managed side; the same result.
And I've tried allocating memory with Marshal.AllocHGlobal, copying data into that with Marshal.Copy and passing the resulting IntPtr, cast to a byte*. Same result.
What the heck am I missing here?
This is a 32-bit process. I've tried smallish (300B) and large (10MB) buffer sizes. C# 3.5, VS 2010.

The core problem is that your COM method is not Automation compatible. A byte* is ambiguous. It can mean either a single byte passed by reference (ref byte in C#) or it can mean an array pointer passed by value (byte[] in C#). A type library cannot express the difference. Automation requires that you pass arrays as a SAFEARRAY.
You'll get away with this when you use the method from a C++ program, you simply pass a pointer to an array. But this goes wrong after Tlbimp.exe translates the type library into an interop library, the interop stub will have the argument declared as ref byte. Which is the exact reason you see only one byte copied.
Fixing this is painful if you can't fix the COM server. You'll have to disassemble the interop library with ildasm.exe /out. Then edit the method declaration in the IL file, then put humpty-dumpty back together with ilasm.exe. Use a little test program to know the exact way the IL should be edited.

Have you tried:
void DoSomething([In] [MarshalAs(UnmanagedType.LPArray, SizeParamIndex=1)]
byte[] buffer, [In] uint length)
?

Is it safe to keep C++ pointers in C#?

I'm currently working on some C#/C++ code which makes use of invoke. In the C++ side there is a std::vector full of pointers each identified by index from the C# code, for example a function declaration would look like this:
void SetName(char* name, int idx)
But now I'm thinking, since I'm working with pointers couldn't I sent to C# the pointer address itself then in code I could do something like this:
void SetName(char*name, int ptr)
{
((TypeName*)ptr)->name = name;
}
Obviously that's a quick version of what I'm getting at (and probably won't compile).
Would the pointer address be guaranteed to stay constant in C++ such that I can safely store its address in C# or would this be too unstable or dangerous for some reason?

In C#, you don't need to use a pointer here, you can just use a plain C# string.
[DllImport(...)]
extern void SetName(string name, int id);
This works because the default behavior of strings in p/invoke is to use MarshalAs(UnmanagedType.LPStr), which converts to a C-style char*. You can mark each argument in the C# declaration explicitly if it requires some other way of being marshalled, eg, [MarshalAs(UnmanagedType.LPWStr)], for an arg that uses a 2-byte per character string.
The only reason to use pointers is if you need to retain access to the data pointed to after you've called the function. Even then, you can use out parameters most of the time.
You can p/invoke basically anything without requiring pointers at all (and thus without requiring unsafe code, which requires privileged execution in some environments).

Yes, no problem. Native memory allocations never move so storing the pointer in an IntPtr on the C# side is fine. You need some kind of pinvoked function that returns this pointer, then
[DllImport("something.dll", CharSet = CharSet.Ansi)]
void SetName(IntPtr vector, string name, int index);
Which intentionally lies about this C++ function:
void SetName(std::vector<std::string>* vect, const char* name, int index) {
std::string copy = name;
(*vect)[index] = copy;
}
Note the usage of new in the C++ code, you have to copy the string. The passed name argument points to a buffer allocated by the pinvoke marshaller and is only valid for the duration of the function body. Your original code cannot work. If you intend to return pointers to vector<> elements then be very careful. A vector re-allocates its internal array when you add elements. Such a returned pointer will then become invalid and you'll corrupt the heap when you use it later. The exact same thing happens with a C# List<> but without the risk of dangling pointers.

I think it's stable till you command C++ code and perfectly aware what he does, and other developers that work on the same code know about that danger too.
So by my opinion, it's not very secure way of architecture, and I would avoid it as much as I can.
Regards.

The C# GC moves things, but the C++ heap does not move anything- a pointer to an allocated object is guaranteed to remain valid until you delete it. The best architecture for this situation is just to send the pointer to C# as an IntPtr and then take it back in C++.
It's certainly a vastly, incredibly better idea than the incredibly BAD, HORRIFIC integer cast you've got going there.

Problem with Interop C#/C: AccessViolationException

and thanks in advice for any help.
i have this trivial function in C:
__declspec(dllexport) Point* createPoint (int x, int y) {
Point *p;
p = (Point*) malloc(sizeof(Point));
p->x = x;
p->y=y;
return p;
}
Point is a very simple struct with two int fields, x and y.
I would like calling this function from C#.
I use this code:
[DllImport("simpleC.dll", EntryPoint = "createPoint", CallingConvention = CallingConvention.Cdecl, SetLastError = true, CharSet = CharSet.Auto)]
[return: MarshalAs(UnmanagedType.LPStruct)]
public static extern Point createPoint(int x, int y);
Point p = Wrapper.createPoint(1, 2);
But at runtime I have an AccessViolationException. Watching exception in detail, I found that exception is thrown from Marshal.CoTaskMemFree(IntPtr) method.
It seems that this method is unable to free memory allocated by C malloc.
What am i doing wrong?
Really thanks.

CoTaskMemFree cannot be used to free memory allocated by malloc (because they use different allocators). According to MSDN, "The runtime always uses the CoTaskMemFree method to free memory. If the memory you are working with was not allocated with the CoTaskMemAlloc method, you must use an IntPtr and free the memory manually using the appropriate method."
Additionally, Adam Nathan notes that "UnmanagedType.LPStruct is only supported for one specific case: treating a System.Guid value type as an unmanaged GUID with an extra level of indirection. ... You should probably just stay away from UnmanagedType.LPStruct."
There are two possible solutions:
Declare the return type of the method as IntPtr and use Marshal.ReadInt32 to read the fields of the struct, or use Marshal.PtrToStructure to copy the data to a managed struct, or use unsafe code to cast the IntPtr value to a Point *. The C library will need to expose a destroyPoint(Point *) method that frees the memory.
Change the C method signature to void getPoint(int x, int y, Point *). This lets C# allocate the struct, and the C method simply fills in the data values. (Most of the Win32 APIs are defined this way).
One final note: Unless your method uses the SetLastError Win32 API, you don't need to specify SetLastError = true on your P/Invoke attribute.

Since you don't have the code that frees "p", it is hard to say. However it is likely that the way malloc() and free() work together is completely different to the way C# manages memory. Since C# has garbage collection (I believe) it is likely that it uses a completely different memory management system.
In any case, the correct solution is that if you use your library to create an object, you should also use it to destroy it. Implement a "destroyPoint" function that frees the memory in your C library, import it to the C# code, and call it from there to destroy the objects created by your C library.
As a general design/coding rule, every "create" function should have a matching "free/destroy/delete" function. Apart from nothing else, it makes it easy to ensure that all created items get properly destroyed.

How is the Point type defined on the C# side?
It has to be unsafe, or you need to return a void pointer (IntPtr). The GC is not able to count references from outside (here the allocated memory), thus your code can not expect to manage externally allocated memory via the GC. One alternative is to keep a static reference to avoid a Garbage collection, if you need to keep the object persistently during the runtime of your application.

Will struct modifications in C# affect unmanaged memory?

My gut reaction is no, because managed and unmanaged memory are distinct, but I'm not sure if the .NET Framework is doing something with Marshaling behind the scenes.
What I believe happens is:
When getting a struct from my unmanaged DLL, it is the same as making that call gets an IntPtr and then uses it and the Marshal class to copy the struct into managed memory (and changes made to the struct in managed memory do not bubble up).
I can't seem to find this documented anywhere on MSDN. Any links would be appreciated.
Here is what my code looks like:
[DllImport("mydll.dll", BestFitMapping=false, CharSet=CharSet.Ansi)]
private static extern int GetStruct(ref MyStruct s);
[StructLayout(LayoutKind.Sequential, Pack=0)]
struct MyStruct
{
public int Field1;
public IntPtr Field2;
}
public void DoSomething()
{
MyStruct s = new MyStruct();
GetStruct(ref s);
s.Field1 = 100; //does unmanaged memory now have 100 in Field1 as well?
s.Field2 = IntPtr.Zero; //does unmanaged memory now have a NULL pointer in field Field2 as well?
}

No, the P/Invoke marshaller copied the unmanaged structure member values into the managed version of the structure. In general, the managed version of a structure is not in any way compatible with the unmanaged version of it. The memory layout is not discoverable, something the CLR uses to reorder fields to make the structure smaller. Marshaling is essential, you have to create a copy.
Modifying the structure is not possible with the given function signature since you let fill in the memory that's passed to it. The function itself already copies the structure. You can however party on the Field2 value since it is a raw pointer. If that points to a structure then marshal it yourself with Marshal.PtrToStructure(). Modify the managed copy of it and copy it back to unmanaged memory with Marshal.StructureToPtr(). Or access it directly with Marshal.ReadXxx() and WriteXxx().

CSharp Language Specification.doc pg 26
Struct constructors are invoked with the new operator, but that does not imply that memory is being allocated. Instead of dynamically allocating an object and returning a reference to it, a struct constructor simply returns the struct value itself (typically in a temporary location on the stack), and this value is then copied as necessary.
Since, there is nothing special about a 'struct' backing store, so one would not expect there to be annonymous marshalling operations going on behind the member assignments.