We Keep Coding

Having trouble marshalling an array of structs from c to c#

I'm trying to marshal an array of c structs into C# (Using Unity) but, no matter the method I use, I always get an exception or a crash.
I'm loading dlls (libretro cores) that conform (or should...) to the Libretro API, the c/c++ side is not available to me (more precisely, not allowed to be modified by me), which means I have to handle the data I get back from that dll no matter how it is laid out.
The C API structs are defined as follow (RETRO_NUM_CORE_OPTION_VALUES_MAX is a constant with a value of 128):
struct retro_core_option_value
{
const char *value;
const char *label;
};
struct retro_core_option_definition
{
const char *key;
const char *desc;
const char *info;
struct retro_core_option_value values[RETRO_NUM_CORE_OPTION_VALUES_MAX];
const char *default_value;
};
struct retro_core_options_intl
{
struct retro_core_option_definition *us;
struct retro_core_option_definition *local;
};
My C# mappings look like this at the moment:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct retro_core_option_value
{
public char* value;
public char* label;
}
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct retro_core_option_definition
{
public char* key;
public char* desc;
public char* info;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = RETRO_NUM_CORE_OPTION_VALUES_MAX)]
public retro_core_option_value[] values;
public char* default_value;
}
[StructLayout(LayoutKind.Sequential)]
public struct retro_core_options_intl
{
public IntPtr us;
public IntPtr local;
}
The callback function has the following signature on the C# side:
public unsafe bool Callback(retro_environment cmd, void* data)
retro_environment is an unsigned int converted to an enum, a switch is performed on it and then dictates how to handle the void* data pointer appropriately. Here data is a retro_core_options_intl*.
I'm able to do the void* conversion in 2 ways:
retro_core_options_intl intl = Marshal.PtrToStructure<retro_core_options_intl>((IntPtr)data);
or
retro_core_options_intl* intl = (retro_core_options_intl*)data;
I get a readable address with both approaches (intl.us for the first and intl->us for the second), the "local" part is empty in my particular case but the "us" part is defined as mandatory by the API. intl->us points to an array of retro_core_option_definition of variable length.
The issue I'm having is trying to read the values inside of this mandatory construct.
The array I'm trying to load right now can be seen here: https://github.com/visualboyadvance-m/visualboyadvance-m/blob/master/src/libretro/libretro_core_options.h at line 51.
The API defines a fixed size for the "struct retro_core_option_value values[RETRO_NUM_CORE_OPTION_VALUES_MAX]" struct member, but code that comes in is almost always defined as an array where the last element is "{ NULL, NULL }" to indicate the end, so they don't always (almost never) contain 128 values.
I tried:
retro_core_options_intl intl = Marshal.PtrToStructure<retro_core_options_intl>((IntPtr)data);
retro_core_option_definition us = Marshal.PtrToStructure<retro_core_option_definition>(intl.us);
This gives a NullReferenceException.
retro_core_options_intl intl = Marshal.PtrToStructure<retro_core_options_intl>((IntPtr)data);
retro_core_option_definition[] us = Marshal.PtrToStructure<retro_core_option_definition[]>(intl.us);
This gives a retro_core_option_definition array of 0 length.
retro_core_options_intl intl = Marshal.PtrToStructure<retro_core_options_intl>((IntPtr)data);
retro_core_option_definition us = new retro_core_option_definition();
Marshal.PtrToStructure(intl.us, us);
This gives a "destination is a boxed value".
That's basically where I'm at...
Any help would be much appreciated :)
The entire codebase can be found here: https://github.com/Skurdt/LibretroUnityFE

First thing I see is that you either need to use wchar_t types instead of char types in you C code, or you can use byte instead of char in C#. System.Char in C# is two bytes. char in C code is 1 byte.
You can also use System.String in the C# code and annotate it with a MarshalAs attribute to tell it what type of char data is coming in, such as Ansi or Unicode C strings.

interop with nim return Struct Array containing a string /char* member

interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.

The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.

Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}

Marshalling Complex structures between C#/C++

I'm trying to populate an array of structures from C++ and pass the result back to C#.
I thought maybe creating a struct with an array of structures maybe the way forward as most examples I have come across use structures(but passing basic types). I have tried the following but no luck so far.
Found an example at: http://limbioliong.wordpress.com/2011/08/20/passing-a-pointer-to-a-structure-from-c-to-c-part-2/?relatedposts_exclude=542
I have the following in C#
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
namespace CSharpDLLCall
{
class Program
{
[StructLayout(LayoutKind.Sequential,Pack=8)]
public struct Struct1
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public double[] d1;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public double[] d2;
}
[StructLayout(LayoutKind.Sequential)]
public struct TestStructOuter
{
public int length;
public IntPtr embedded;
}
static void Main(string[] args)
{
Program program = new Program();
program.demoArrayOfStructs();
}
public void demoArrayOfStructs()
{
TestStructOuter outer = new TestStructOuter();
testStructOuter.embedded = new Struct1[10];
outer.length = 10;
outer.embedded = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(Struct1)) * 10);
Marshal.StructureToPtr(outer, outer.embedded, false);
testAPI2(ref outer);
outer = (TestStructOuter)(Marshal.PtrToStructure(outer.embedded, typeof(TestStructOuter)));
Marshal.FreeHGlobal(outer.embedded);
}
[DllImport(#"C:\CPP_Projects\DLL\DLLSample\Release\DLLSample.dll")]
static extern void testAPI2(IntPtr pTestStructOuter);
}
}
In C++ in the header i have
#ifdef DLLSAMPLE_EXPORTS
#define DLLSAMPLE_API __declspec(dllexport)
#else
#define DLLSAMPLE_API __declspec(dllimport)
#endif
#include <iostream>
using namespace std;
#pragma pack(1)
struct struct1
{
double d1[];
double d2[];
};
struct TestStructOuter
{
struct1* embedded;
};
extern "C"
{
DLLSAMPLE_API void __stdcall testAPI2(TestStructOuter* pTestStructOuter);
}
In the cpp I have:
#include "stdafx.h"
#include "DLLSample.h"
__declspec(dllexport) void __stdcall testAPI2(TestStructOuter* pTestStructOuter)
{
// not sure that this is necessary
// for (int i = 0; i < 10 ; ++i)
// {
// pTestStructOuter->embedded = new struct1;
// }
for (int i = 0; i < 10 ; ++i)
{
struct1 s1;
for (int idx = 0; idx < 10; ++idx)
{
s1.d1[i] = i+0.5;
s1.d2[i] = i+0.5;
}
pTestStructOuter->embedded[0] = s1;
}
}
This doesn't seem to work the error i get:
The parameter is incorrect.(Exception from HRESULT:0x80070057 (E_INVALIDARG))
Which probably means that its not recognizing the array of structures. Any ideas how I can do this? Thanks.

Okay, I have a working sample. I'm posting this as another answer because it's a very different approach.
So, on the C++ side, I've got this header file:
struct Struct1
{
int d1[10];
int d2[10];
};
extern "C" __declspec(dllexport) void __stdcall
TestApi2(int* pLength, Struct1 **pStructures);
And the following code:
__declspec(dllexport) void __stdcall
TestApi2(int* pLength, Struct1 **pStructures)
{
int len = 10;
*pLength = len;
*pStructures = (Struct1*)LocalAlloc(0, len * sizeof(Struct1));
Struct1 *pCur = *pStructures;
for (int i = 0; i < len; i++)
{
for (int idx = 0; idx < 10; ++idx)
{
pCur->d1[idx] = i + idx;
pCur->d2[idx] = i + idx;
}
pCur ++;
}
}
Now, the important bit is LocalAlloc. That's actually the place where I've had all the issues, since I allocated the memory wrong. LocalAlloc is the method .NET calls when it does Marshal.AllocHGlobal, which is very handy, since that means we can use the memory in the caller and dispose of it as needed.
Now, this method allows you to return an arbitrary length array of structures. The same approach can be used to eg. return a structure of an array of structures, you're just going deeper. The key is the LocalAlloc - that's memory you can easily access using the Marshal class, and it's memory that isn't thrown away.
The reason you have to also return the length of the array is because there's no way to know how much data you're "returning" otherwise. This is a common "problem" in unmanaged code, and if you've ever done any P/Invoking, you know everything about this.
And now, the C# side. I've tried hard to do this in a nice way, but the problem is that arrays of structures simply aren't blittable, which means you can't simply use MarshalAs(UnmanagedType.LPArray, ...). So, we have to go the IntPtr way.
The definitions look like this:
[StructLayout(LayoutKind.Sequential)]
public class Struct1
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public int[] d1;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public int[] d2;
}
[DllImport(#"Win32Project.dll", CallingConvention = CallingConvention.StdCall)]
static extern void TestApi2(out int length, out IntPtr structs);
Basically, we get a pointer to the length of the "array", and the pointer to the pointer to the first element of the array. That's all we need to read the data.
The code follows:
int length;
IntPtr pStructs;
TestApi2(out length, out pStructs);
// Prepare the C#-side array to copy the data to
Struct1[] structs = new Struct1[length];
IntPtr current = pStructs;
for (int i = 0; i < length; i++)
{
// Create a new struct and copy the unmanaged one to it
structs[i] = new Struct1();
Marshal.PtrToStructure(current, structs[i]);
// Clean-up
Marshal.DestroyStructure(current, typeof(Struct1));
// And move to the next structure in the array
current = (IntPtr)((long)current + Marshal.SizeOf(structs[i]));
}
// And finally, dispose of the whole block of unmanaged memory.
Marshal.FreeHGlobal(pStructs);
The only thing that changes if you want to really return a structure of an array of structures is that the method parameters move into the "wrapping" structure. The handy thing is that .NET can automatically handle the marshalling of the wrapper, the less-handy thing is that it can't handle the inner array, so you again have to use length + IntPtr to manage this manually.

First, don't use unknown-length arrays at all. You're killing any possible use of arrays between managed and unmanaged code - you can't tell the required size (Marshal.SizeOf is useless), you have to manually pass the array length etc. If it's not a fixed length array, use IntPtr:
[StructLayout(LayoutKind.Sequential,Pack=8)]
public struct Struct1
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public double[] d1;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public double[] d2;
}
[StructLayout(LayoutKind.Sequential)]
public struct TestStructOuter
{
public int length;
public IntPtr embedded;
}
(if your array of embedded is fixed length, feel free to ignore this, but do include the ByValArray, SizeConst bit. You also have to add ArraySubType=System.Runtime.InteropServices.UnmanagedType.Struct to the attribute).
As you've probably noticed, the TestStructOuter is pretty much useless in this case, it only adds complexity (and do note that marshalling is one of the most expensive operations compared to native languages, so if you're calling this often, it's probably a good idea to keep things simple).
Now, you're only allocating memory for the outer struct. Even in your code, Marshal.SizeOf has no idea how big the struct should be; with IntPtr, this is even more so (or, to be more precise, you're only requesting a single IntPtr, ie. 4-8 bytes or so + the length). Most often, you'd want to pass the IntPtr directly, rather than doing this kind of wrapping (using the same thing in C++/CLI is a different thing entirely, although for your case this is unnecessary).
The signature for your method will be something like this:
[DllImport(#"C:\CPP_Projects\DLL\DLLSample\Release\DLLSample.dll",
CallingConvention=System.Runtime.InteropServices.CallingConvention.StdCall)]
public static extern void testAPI2(ref TestStructOuter pTestStructOuter) ;
Now, if you're going with the IntPtr approach, you want to keep allocating memory manually, only you have to do it properly, eg.:
TestStructOuter outer = new TestStructOuter();
testStructOuter.length = 1;
testStructOuter.embedded = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(Struct1)));
Marshal.StructureToPtr(embedded, testStructOuter.embedded, false);
And finally, you call the method:
// The marshalling is done automatically
testAPI2(ref outer);
Don't forget to release the memory, ideally in the finally clause of a try around everything since the memory allocation:
Marshal.FreeHGlobal(outer.embedded);
Now, this is overly complicated and not exactly optimal. Leaving out the TestStructOuter is one possibility, then you can simply pass the length and pointer to the embedded array directly, avoiding one unnecessary marshalling. Another option would be to use a fixed size array (if it needs to be an array at all :)) in TestStructOuter, which will cure a lot of your headaches and eliminate any need for manual marshalling. In other words, if TestStructOuter is defined as I've noted before:
[StructLayout(LayoutKind.Sequential)]
public struct TestStructOuter
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10,
ArraySubType=UnmanagedType.Struct)]
public Struct1[] embedded;
}
Suddenly, your whole call and everything becomes as simple as
testAPI2(ref outer);
The whole marshalling is done automatically, no need for manual allocations or conversions.
Hope this helps :)

Hint: Leadn C++/CLI. I had to deal with complex interop two times in my life - once with ISDN (AVM devkits make it a lot easier - sadly C++, I needed C#) and now with Nanex (great real time complex and full market ata feeds, sadl complex C++, I need C#)
Both cases I make my own wrapper in C++/CLI, talking down to C++ and exposing a real object model in C#. Allows me to make things a lot nicer and handle a lot of edge cases that our friendly Marshaller can not handle efficiently.

Pointer declaration syntax using & and Intptr. Are both identical?

Case 1:
int i;
int* pi = &i;
Case 2:
int i;
IntPtr pi = &i;
Are both cases identical?
My purpose is that:-
I am copying a value to string variable.
Converting string to bytes array
using Marshal.Alloc(sizeofBytesArray) to get IntPtr ptrToArray
marshal.copy(array,0,ptrToArray,sizeofBytesArray)
Sending this ptrToArray to a vb6 application by using a structure and passing structure via SendMessage win32 api.
On VB6 app:-
I am picking up the value from the structure that gives me the address of the array.
using CopyMemory to copy the bytesarray data into a string..
More Code:
string aString = text;
byte[] theBytes = System.Text.Encoding.Default.GetBytes(aString);
// Marshal the managed struct to a native block of memory.
int myStructSize = theBytes.Length;
IntPtr pMyStruct = Marshal.AllocHGlobal(myStructSize); //int* or IntPtr is good?
try
{
Marshal.Copy(theBytes, 0, pMyStruct, myStructSize);
...............

According to the msdn page:-
The IntPtr type can be used by languages that support pointers, and as a common means of referring to data between languages that do and do not support pointers.
IntPtr objects can also be used to hold handles. For example, instances of IntPtr are used extensively in the System.IO.FileStream class to hold file handles.
Are both cases indentical?
No, they are not identical,
The difference is in the underlying implementation of IntPtr
forex
int i = 10 ;
int *pi = i ;
int c = *pi ; // would work
but to do the same thing , you have to cast IntPtr to int*
int i =10 ;
IntPtr pi = &i ;
int *tempPtr = (int*)pi ;
int c = *tempPtr ;
The internal representation of IntPtr is like void* but it is exposed like an integer. You can use it whenever you need to store an unmanaged pointer and don't want to use unsafe code.
It has two limitations:
It cannot be dereferenced directly (you have to cast it as a true pointer).
It doesn't know the type of the data that it points to. (because of void*)

How to map a C array to C#?

My question has to do with trying to call a function written in C from C#. I've looked in a header file that came with the C library to understand the functions as they exist in the C dll. Here's what I see:
C code (for a function called "LocGetLocations"):
typedef enum {
eLocNoError,
eLocInvalidCriteria,
eLocNoMatch,
eLocNoMoreLocations,
eLocConnectionError,
eLocContextError,
eLocMemoryError
} tLocGetStatus;
typedef void *tLocFindCtx;
typedef void *tLocation;
PREFIX unsigned int POSTFIX LocGetLocations
(
tLocFindCtx pCtx,
tLocation *pLoc,
unsigned int pNumLocations,
tLocGetStatus *pStatus
);
In C#, I have this:
[DllImport(#"VertexNative\Location.dll")]
public static extern uint LocGetLocations(IntPtr findContext, out byte[] locations, uint numberLocations, out int status);
The problem is that I don't quite know how to handle the pLoc parameter in C#. I'm bringing it over as a byte array, although I'm not sure if that is correct. The C library's documentation says that that parameter is a pointer to an array of handles.
How can I get an array back on the C# side and access its data?
The example I was given in C, looks like this:
tLocation lLocation[20];
// other stuff
LocGetLocations(lCtx, lLocation, 20, &lStatus)
Any help would be much appreciated!

Generally, the only thing that matters is the size of the parameters. As I recall enums are integers in C, so you can simply use that. Or better, recreate the same enum in C#, I think it would work. One thing to remember is that when dealing with complex structs, one needs to use attributes to tell the framework about the desired alignment of members.

In the end, this signature works:
[DllImport(#"VertexNative\Location.dll")]
public static extern uint LocGetLocations(IntPtr findContext, [Out] IntPtr[] locations, uint numberLocations, out int status);
And I can call it like this (some refactoring needed):
IntPtr[] locations = new IntPtr[20];
int status;
// findContext is gotten from another method invocation
uint result = GeoCodesNative.LocGetLocations(findContext, locations, 20, out status);
Thanks for the help!