We Keep Coding

Do I need to call Marshal.DestroyStructure after copying a struct to a byte[]?

Actually I have many questions, so I try to describe them in detail, thanks for your patience.
I want to write a some generic methods for wrapping native APIs, just like ReadProcessMemory and WriteProcessMemory.
Take WriteProcessMemory for example, I found some ways to do them:
1.GCHandle
// Write memory: Non-Array
var lengthInBytes = (IntPtr)Marshal.SizeOf<T>();
var tHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
try
{
// Call native api;
}
finally
{
tHandle.Free();
}
// Write memory: Array
var lengthInBytes = (IntPtr)(data.Length * Marshal.SizeOf<T>());
var tHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
try
{
// Call native api;
}
finally
{
tHandle.Free();
}
This way is very simple, but the only problem is the object that to be pinned must be a built-in value type ( except fixed array, but the type of the element in array is subject to the same requirements ), . So I found another way: Marshal.AllocHGlobal method.
2. The members in Marshal class
//Write memory: Non-Array
var lengthInBytes = Marshal.SizeOf<T>();
var memPtr = Marshal.AllocHGlobal(lengthInBytes);
try
{
Marshal.StructureToPtr<T>(data, memPtr, false);
// Call native API
}
finally
{
Marshal.DestroyStructure<T>(memPtr);
Marshal.FreeHGlobal(memPtr);
}
// Write memory: Array
var perSize = Marshal.SizeOf<T>();
var arrayLength = data.Length;
var lengthInBytes = arrayLength * perSize;
var memPtr = Marshal.AllocHGlobal(lengthInBytes);
var baseAdr = memPtr;
try
{
foreach (var item in data)
{
Marshal.StructureToPtr<T>(item, baseAdr, false);
baseAdr += perSize;
}
// call native API.
}
finally
{
for (int i = 0; i < arrayLength; i++, baseAdr -= perSize)
{
Marshal.DestroyStructure<T>(baseAdr);
}
Marshal.FreeHGlobal(memPtr);
}
This way will alloc some memory, copy data, and even traverse array twice, But it supports the reference types, and I was satisfied with it, But then I got a idea: how about using Byte[] instead of IntPtr?
Specifically,The extern method is:
[DllImport("Kernel32.dll", SetLastError = true, CallingConvention = CallingConvention.Winapi)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern Boolean WriteProcessMemory([In] HandleRef hProcess, [In]IntPtr lpBaseAddress, [In, MarshalAs(UnmanagedType.LPArray)] Byte[] lpBuffer, [In, MarshalAs(UnmanagedType.SysInt)]IntPtr nSize, [Out, Optional, MarshalAs(UnmanagedType.SysInt)] out IntPtr lpNumberOfBytesWritten);
And the code in method is:
var perSize = Marshal.SizeOf<T>();
var buffer = new Byte[perSize];
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
var pBuffer = handle.AddrOfPinnedObject();
try
{
Marshal.StructureToPtr<T>(data, pBuffer, false);
//Call native API.
}
finally
{
// Question!!!!!!!!!!!!!!
Marshal.DestroyStructure<T>(pBuffer);
handle.Free();
}
So there is a question: do I need to call Marshal.DestroyStructure method?
MSDN says
StructureToPtr(T, IntPtr, Boolean) copies the contents of structure
to the pre-allocated block of memory that the ptr parameter points to.
If structure contains reference types that marshal to COM interface
pointers (interfaces, classes without layout, and System.Object), the
managed objects are kept alive with reference counts. All other
reference types (for example, strings and arrays) are marshaled to
copies. To release these managed or unmanaged objects, you must call
the DestroyStructure(IntPtr) method before you free the memory
block.
And this method is "Marshals data from a managed object of a specified type to an unmanaged block of memory."
But variable buffer is a Byte[](In managed memory), so if I wont call DestroyStructure, can GC release the reference types in T? ( By the way, in my test, this way is much faster than second way, even though Marshal.DestroyStructure method has also been called too... )
And more questions is, will there be more ways to get the pointer of generic T? And if there are any mistake in the solution above?

Pass a large buffer using P-Invoke

I need to pass a buffer containing audio stream from C# to native dll. Buffer resides in struct. It's desirable that buffer will pass via interface & not via disk path. I've seen this method:
// native
struct MyStruct
{
short* buffer
}
void Foo(MyStruct *myStruct);
// managed
[StructLayout(LayoutKind.Sequential)]
public struct MyStruct
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst=1000)]
public short[] buffer;
}
[DllImport("My.dll")]
public static extern void Foo(ref MyStruct myStruct);
First question is if this code is correct for short* buffer of size up to 1K shorts?
Second, size is unkown in advance: do I need to set maximum size in SizeConst (could be several MB)?

First of all, the two structures in your question don't match. The C# structure would match
struct MyStruct
{
short arr[1000];
};
That's what ByValArray means – an array allocated inline in the structure.
If the size is dynamic (that is only known at runtime), then you probably should not expect to get the marshaller to handle this for you. You certainly don't want to force marshalling to a constant sized buffer every time because that will be inefficient. In fact you really want to avoid copying the buffer around at all. And the p/invoke marshaler has an upper limit on the size of objects that it is prepared to marshal.
It is going to be far cleaner and efficient to pin the array manually, and pass its address. And you should also pass the length of the array so that the C++ code knows how much it is expected to read.
On the C++ side:
struct BufferStruct
{
int len;
short* arr;
};
void Foo(const BufferStruct buffer);
On the C# side:
[StructLayout(LayoutKind.Sequential)]
public struct BufferStruct
{
public int len;
public IntPtr arr;
}
[DllImport("My.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void Foo(BufferStruct buffer);
Then you call the function like this:
short[] arr = ...;
GCHandle gch = GCHandle.Alloc(arr, GCHandleType.Pinned);
try
{
BufferStruct buffer;
buffer.len = buffer.Length;
buffer.arr = gch.AddrOfPinnedObject();
Foo(buffer);
}
finally
{
gch.Free();
}
All of this would be easier still if you did not force the array into a struct. If you passed the length and array as parameters, then the marshaller would pin the array for you and make the code even simpler.

Marshal a std::vector<uint64_t> from C++ to C#

no matter what I try. I appear to get garbage results when I marshal the data across! The data after the marshal copy just contains an array of what looks like uninitialized data. Just pure garbage.
Thanks for your help in advance!
C++
typedef uint64_t TDOHandle;
extern "C" DATAACCESSLAYERDLL_API const TDOHandle * __stdcall DB_GetRecords()
{
const Database::TDBRecordVector vec = Database::g_Database.GetRecords();
if (vec.size() > 0)
{
return &vec[0];
}
return nullptr;
}
C#
The declaration
[System.Security.SuppressUnmanagedCodeSecurity()]
[DllImport("DataCore.dll")]
static private extern IntPtr DB_GetRecords();
//The marshalling process
IntPtr ptr_records = DB_GetRecords();
if (ptr_records != null)
{
Byte[] recordHandles = new Byte[DB_GetRecordCount()*sizeof (UInt64)];
Marshal.Copy(ptr_records, recordHandles, 0, recordHandles.Length);
Int64[] int64Array = new Int64[DB_GetRecordCount()];
Buffer.BlockCopy(recordHandles, 0, int64Array, 0, recordHandles.Length);
}

You are returning the address of memory owned by a local variable. When the function returns, the local variable is destroyed. Hence the address you returned is now meaningless.
You need to allocate dynamic memory and return that. For instance, allocate it with CoTaskMemAlloc. Then the consuming C# can deallocate it with a call to Marshal.FreeCoTaskMem.
Or allocate the memory using new, but also export a function from your unamanaged code that can deallocate the memory.
For example:
if (vec.size() > 0)
{
TDOHandle* records = new TDOHandle[vec.size()];
// code to copy content of vec to records
return records;
}
return nullptr;
And then you would export another function that exposed the deallocator:
extern "C" DATAACCESSLAYERDLL_API void __stdcall DB_DeleteRecords(
const TDOHandle * records)
{
if (records)
delete[] record;
}
All that said, it seems that you can obtain the array length before you call the function to populate the array. You do that with DB_GetRecordCount(). In that case you should create an array in your managed code, and pass that to the unmanaged code for it to populate. That side steps all the issues of memory management.

I'll add that there is another way to do it:
public sealed class ULongArrayWithAllocator
{
// Not necessary, default
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate IntPtr AllocatorDelegate(IntPtr size);
private GCHandle Handle;
private ulong[] allocated { get; set; }
public ulong[] Allocated
{
get
{
// We free the handle the first time the property is
// accessed (we are already C#-side when it is accessed)
if (Handle.IsAllocated)
{
Handle.Free();
}
return allocated;
}
}
// We could/should implement a full IDisposable interface, but
// the point of this class is that you use it when you want
// to let C++ allocate some memory and you want to retrieve it,
// so you'll access LastAllocated and free the handle
~ULongArrayWithAllocator()
{
if (Handle.IsAllocated)
{
Handle.Free();
}
}
// I'm using IntPtr for size because normally
// sizeof(IntPtr) == sizeof(size_t) and vector<>.size()
// returns a size_t
public IntPtr Allocate(IntPtr size)
{
if (allocated != null)
{
throw new NotSupportedException();
}
allocated = new ulong[(long)size];
Handle = GCHandle.Alloc(allocated, GCHandleType.Pinned);
return Handle.AddrOfPinnedObject();
}
}
[DllImport("DataCore.dll", CallingConvention = CallingConvention.StdCall)]
static private extern IntPtr DB_GetRecords(ULongArrayWithAllocator.AllocatorDelegate allocator);
and to use it:
var allocator = new ULongArrayWithAllocator();
DB_GetRecords(allocator.Allocate);
// Here the Handle is freed
ulong[] allocated = allocator.Allocated;
and C++ side
extern "C" DATAACCESSLAYERDLL_API void __stdcall DB_GetRecords(TDOHandle* (__stdcall *allocator)(size_t)) {
...
// This is a ulong[vec.size()] array, that you can
// fill C++-side and can retrieve C#-side
TDOHandle* records = (*allocator)(vec.size());
...
}
or something similar :-) You pass a delegate to the C++ function that can allocate memory C#-side :-) And then C# side you can retrieve the last memory that was allocated. It is important that you don't make more than one allocation C++-side in this way in a single call, because you are saving a single LastAllocated reference, that is "protecting" the allocated memory from the GC (so don't do (*allocator)(vec.size());(*allocator)(vec.size());)
Note that it took me 1 hour to write correctly the calling conventions of the function pointers, so this isn't for the faint of heart :-)

Calling a C DLL from a C# Program

I need to pass a pointer to a structure to my DLL, any ideas how would I go about doing that?
In my C DLL:
typedef struct
{
int length;
unsigned char *value;
} Sample;
__declspec(dllexport) void __stdcall helloWorld( Sample *sample );
In my C# code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
namespace CSharpConsole
{
class Program
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct Sample
{
public Int32 length;
// What Should I Declare Here?
}
[DllImport("C:\\CTestDLL.dll")]
private static extern void helloWorld( Sample sample ); // How would I make this a pointer?
void HelloWorld()
{
Sample sample = new Sample();
sample .length = 20;
// How can I fill up the values of value?
helloWorld( sample ); // How should I pass it inside here
return;
}
static void Main(string[] args)
{
Program program = new Program();
program.HelloWorld();
}
}
}

To pass a pointer to a value type into a P/Invoke function just declare the parameter as a ref or out. This implicitly takes a reference to the parameter and passes that:
[DllImport("C:\\CTestDLL.dll")]
private static extern void helloWorld(ref Sample sample);
Since your structure has an array in it, you'll have to take care to declare it properly for this to work. I strongly recommend, if possible, that you turn it into a fixed-length array, as it will make the marshaler much, much happier:
typedef struct
{
int length;
unsigned char value[MAX_LENGTH];
} Sample;
This becomes:
public struct Sample
{
int length;
[MarshalAs(UnmanagedType.LPArray, SizeConst = MAX_LENGTH)] byte[] value;
}
If that is not possible, then the runtime has no way of knowing how much data to marshal back; in that case, you probably will have to resort to manual marshaling of your data:
public struct Sample
{
int length;
IntPtr value;
}
var sample = new Sample();
helloWorld(ref sample);
byte[] value = new byte[sample.length];
Marshal.Copy(sample.value, value, 0, sample.Length);
However, based on your comment to another answer, it looks like you just need to get a block of bytes out of the C DLL into C#. For that you don't really need a structure at all, and eliminating it would simplify things a lot. If you just want to pass in an array and have it filled in and returned to you, try something like this:
(This assumes you have control over both C and C# code bases; if not then the ref suggestion is the way to accomplish what you need.)
// In C# code:
[DllImport(#"C:\CTestDll.dll")]
private static extern void helloWorld(
int length,
[MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 0)] byte[] buffer);
byte[] buffer = new byte[1024 * 8];
helloWorld(1024 * 8, buffer);
// In C:
__declspec(dllexport) void __stdcall helloWorld(int, unsigned char *);
void helloWorld(int cb, unsigned char *buf)
{
memcpy(buf, DATASRC, cb);
}
In C, an unsigned char is, by definition, the same size as a C# byte - 8 bits, no sign. In C#, a char is actually two bytes. The runtime will automatically "convert" an unsigned char * to a byte[] for you. (There's not really a conversion at all; they are just different names for the same type.)

Default Marshaling for Value Types - gives some information on marshalling structs.
Calling Win32 DLLs in C# with P/Invoke - a little over half way down the page there's a table showing the type conversions between the standard unmanaged and managed types.
There are a few ways to pass around and convert the types.
As Michael Edenfield pointed out you can generally just use the ref keyword to indicate that a parameter should by passed by reference which is basically a pointer.
However sometimes things don't cooperate, particularly when it comes to strings or complex data types so this is where the IntPtr type comes in.
You have a couple of options for using IntPtrs.
You can create an IntPtr to a block of unmanaged memory of a specified size using:
IntPtr pointer = Marshal.AllocHGlobal(sizeOfBufferInBytes);
//Do stuff with the pointer
Marshal.FreeHGlobal(pointer); //Don't forget to release the memory
This is obviously a bit dangerous because you're manually allocating unmanaged memory.
You'll need to Marshal the data from the buffer back into a managed type, using something like Marshal.Copy(), Marshal.PtrToStructure(), Buffer.BlockCopy(), etc.
Alternatively you can create a managed object and pin it in memory while you need to, get a pointer to it and pass that to your method.
MyObject instance = new MyObject();
GCHandle gch = GCHandle.Alloc(instance, GCHandleType.Pinned);
importedMethod(gch.AddrOfPinnedObject()); //AddrOfPinnedObject() gives you an IntPtr
gch.Free(); //Release the pinned memory so the garbage collector can deal with it
This avoids the necessity for manually marshalling back to the correct data type but this is not always an option depending on the type of MyObject and whether it's blittable.

This works for me:
DLL:
typedef struct
{
int length;
unsigned char *value;
} Sample;
extern "C" __declspec(dllexport) void __stdcall helloWorld( Sample *sample )
{
MessageBoxA(NULL, (LPCSTR) sample->value, (LPCSTR) sample->value, 0);
}
C#:
class Program
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private class Sample
{
public Int32 length;
public String value;
}
[DllImport("C:\\Users\\Kep\\Documents\\Visual Studio 2010\\Projects\\SODLL\\Debug\\DLL.dll")]
private static extern void helloWorld(Sample sample);
static void Main(string[] args)
{
Sample s = new Sample();
s.length = 10;
s.value = "Huhu";
helloWorld(s);
}
}
Important thing is to mark it as a class, not a struct in C#.
With this, you could also use constructors etc. in C#:
class Program
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private class Sample
{
public Int32 length;
public String value;
public Sample(String s)
{
length = s.Length;
value = s;
}
}
[DllImport("C:\\Users\\Kep\\Documents\\Visual Studio 2010\\Projects\\SODLL\\Debug\\DLL.dll")]
private static extern void helloWorld(Sample sample);
static void Main(string[] args)
{
Sample s = new Sample("Huhu");
helloWorld(s);
}
}

C#: Object with custom marshaller not containing data after PInvoke call

I am having a problem with PInvoking some WinAPI functions that accept WAVEFORMATEX structures as parameters. Since the length of the WAVEFORMATEX structure can vary, I implemented a WaveFormatEX class that is marshalled by a custom marshaller class (which implements ICustmoMarshaller). This is following an example provided by Aaron Lerch in his Blog (Part 1, Part 2), but with a few modifications from my side.
When I call the API function from my code, the methods MarshalManagedToNative and MarshalNativeToManaged of the custom marshaller are called, and at the end of MarshalNativeToManaged, the managed object contains the correct values. But when the execution returns to my calling code, the WaveFormatEx object does not contain the values read during the API call.
So the question is: Why does the data that is correctly marshalled back from native to managed not show up in my WaveFormatEx object after the native API call? What am I doing wrong here?
Edit:
To clarify, the function call succeeds, so does the marshalling of the WaveFormatEx object back to managed code. Just when the execution returns from the marshalling method to the scope from where the method was called, the WaveFormatEx object that was declared in that calling scope does not contain the result data.
Here are the function prototype and the WaveFormatEx class:
[DllImport("avifil32.dll")]
public static extern int AVIStreamReadFormat(
int Stream,
int Position,
[In, Out, MarshalAs(UnmanagedType.CustomMarshaler,
MarshalTypeRef = typeof(WaveFormatExMarshaler))]
WaveFormatEx Format,
ref int Size
);
[StructLayout(LayoutKind.Sequential)]
public class WaveFormatEx
{
public int FormatTag;
public short Channels;
public int SamplesPerSec;
public int AvgBytesPerSec;
public short BlockAlign;
public short BitsPerSample;
public short Size;
public byte[] AdditionalData;
public WaveFormatEx(short AdditionalDataSize)
{
WaveFormat.Size = AdditionalDataSize;
AdditionalData = new byte[AdditionalDataSize];
}
}
The marshalling methods look like this:
public object MarshalNativeToManaged(System.IntPtr NativeData)
{
WaveFormatEx ManagedObject = new WaveFormatEx(0);
ManagedObject = (WaveFormatEx)Marshal.PtrToStructure(
NativeData, typeof(WaveFormatEx));
ManagedObject.AdditionalData = new byte[ManagedObject.Size];
// If there is extra data, marshal it
if (ManagedObject.WaveFormat.Size > 0)
{
NativeData = new IntPtr(
NativeData.ToInt32() +
Marshal.SizeOf(typeof(WaveFormatEx)));
ManagedObject.AdditionalData = new byte[ManagedObject.WaveFormat.Size];
Marshal.Copy(NativeData, ManagedObject.AdditionalData, 0,
ManagedObject.WaveFormat.Size);
}
return ManagedObject;
}
public System.IntPtr MarshalManagedToNative(object Object)
{
WaveFormatEx ManagedObject = (WaveFormatEx)Object;
IntPtr NativeStructure = Marshal.AllocHGlobal(
GetNativeDataSize(ManagedObject) + ManagedObject.WaveFormat.Size);
Marshal.StructureToPtr(ManagedObject, NativeStructure, false);
// Marshal extra data
if (ManagedObject.WaveFormat.Size > 0)
{
IntPtr dataPtr = new IntPtr(NativeStructure.ToInt32()
+ Marshal.SizeOf(typeof(WaveFormatEx)));
Marshal.Copy(ManagedObject.AdditionalData, 0, dataPtr, Math.Min(
ManagedObject.WaveFormat.Size,
ManagedObject.AdditionalData.Length));
}
return NativeStructure;
}
And this is my calling code:
WaveFormatEx test = new WaveFormatEx(100);
int Size = System.Runtime.InteropServices.Marshal.SizeOf(test);
// After this call, test.FormatTag should be set to 1 (PCM audio),
// but it is still 0, as well as all the other members
int Result = Avi.AVIStreamReadFormat(AudioStream, 0, test, ref Size);

There are several mistakes in the code and the declarations that prevents this code from working on a 64-bit operating system. Be sure to set the Platform Target to x86.
Are you sure the native function actually returns data? What is the Result return value? A non-zero value indicates failure.
The proper way to call this function is to call it twice. First with the lpFormat argument set to null (IntPtr.Zero) so it tells you how large a buffer it needs (returned by lpbcFormat). Then you create the buffer and call it again.
Instead of a custom marshaller, I would just create the buffer with Marshal.AllocHGobal after the first call and pass the IntPtr it returns as the lpFormat argument in the second call. Then, iff you get a success return code, use Marshal.PtrToStructure to write the WaveFormatEx. And Marshal.Copy to get the additional data.
Fwiw, using ref causes the P/Invoke marshaller to pass a WaveFormatEx** to the function but it expects a WaveFormatEx*. Which will cause it to overwrite data in the garbage collected heap, destroying its internal format. A kaboom is next when the CLR notices this.
Check out the NAudio project as a good alternative for doing this yourself.