We Keep Coding

interop with nim return Struct Array containing a string /char* member

interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.

The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.

Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}

How use pinvoke for C struct array pointer to C#

I am trying to use pinvoke to marshall an array of structures inside another structure from C to C#. AFAIK, no can do.
So instead, in the C structure I declare a ptr to my array, and malloc. Problems: 1) How do I declare the equivalent on the C# side? 2) How do I allocate and use the equivalent on the C# side?
//The C code
typedef struct {
int a;
int b; } A;
typedef struct {
int c;
// A myStruct[100]; // can't do this, so:
A *myStruct; } B;
//The c# code:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public class A{
int a;
int b;
}
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public class B{
int c;
// can't declare array of [100] A structures...
?
}
[EDIT]: Somehow I misinterpreted what I read elsewhere about fixed array of objects on the c# side.
And I can fix the array size in C So compiled ok, but then I get "object reference not set to an instance of an object" when using:
data.B[3].a = 4567; So, reading elsewhere about what this error might be, I added this method:
public void initA()
{
for (int i = 0; i < 100; i++) { B[i] = new A(); }
}
Again, compiled OK, but same error msg.

To marshal "complex" structures like this between C and C#, you have a couple of options.
In this case, I would highly recommend that you try to embed a fixed array into your C-side structure, as it will simplify the C# side a lot. You can use the MarshalAs attribute to tell C# how much space it needs to allocate in the array at run-time:
// In C:
typedef struct
{
int a;
int b;
} A;
typedef struct
{
int c;
A data[100];
} B;
// In C#:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct A
{
int a;
int b;
}
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct B
{
int c;
[MarshalAs(UnmanagedType.LPArray, SizeConst=100)]
A[] data = new data[100];
}
If you don't know, or can't specify, a fixed size for your array, then you'll need to do what you did and declare it as a pointer in C. In this case, you cannot tell C# how much memory the array is going to use at run-time, so you're pretty much stuck with doing all of the marshalling by hand. This question has a good rundown of how that works, but the basic idea is:
You should add a field to your structure that includes a count of the array elements (this will make your life much easier)
Declare the field in C# as: IntPtr data; with no attributes.
Use Marshal.SizeOf(typeof(A)) to get the size of the struct in unmanaged memory.
Use Marshal.PtrToStructure to convert a single unmanaged structure to C#
Use IntPtr.Add(ptr, sizeofA) to move to the next structure in the array
Loop until you run out.

Marshalling C struct containing arrays to C#

With great help of the stackoverflow community, I've managed to call a native DLL function. However, I can't modify the values of ID or intersects array. No matter what I do with it on the DLL side, the old value remains. It seems read-only.
Here are some code fragments:
C++ struct:
typedef struct _Face {
int ID;
int intersects[625];
} Face;
C# mapping:
[StructLayout(LayoutKind.Sequential)]
public struct Face {
public int ID;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 625)]
public int[] intersects;
}
C++ method (type set to DLL in VS2010):
extern "C" int __declspec(dllexport) __stdcall
solve(Face *faces, int n){
for(int i =0; i<n; i++){
for(int r=0; r<625; r++){
faces[i].intersects[r] = 333;
faces[i].ID = 666;
}
}
C# method signature:
[DllImport("lib.dll", CharSet = CharSet.Ansi, CallingConvention = CallingConvention.StdCall)]
public static extern int solve(Face[] faces, int len);
C# method invocation:
Face[] faces = new Face[10];
faces[0].intersects = new int[625];
faces[0].ID = -1; //.. and add 9 more ..
solve(faces, faces.Length);
// faces[0].ID still equals -1 and not 666
Kindest regards,
e.

You have to tell the pinvoke marshaller explicitly that the array needs to be marshaled back. You do this with the [In] and [Out] attributes. Like this:
[DllImport("...")]
public static extern int solve([In, Out] Face[] faces, int len);

This is an output field only? To get to the bottom of this, I'd try substituting your Face[] parameter with a large-enough a byte[] and see if the byte array gets filled with anything (you'll have to change your [DllExport] a bit).
Also, one other thing I used to experience when doing this with char*'s is that I had to pre-allocate the buffer in C#. For example:
StringBuilder theString=new StringBuilder();
MyUnmanagedFunction(theString);
would not work. But assuming that returned theString was a max 256 characters, I would do this:
StringBuilder theString=new StringBuilder(256);
MyUnmanagedFunction(theString);
And I'd be in business. I'd recommend trying the byte[] substitution, if that doesn't work, try the pre-allocated byte array. Once you are seeing the byte array actually get changed by your C++ code, then you can figure out how to marshal that thing into your C# struct.

Some P/Invoke C# to C marshalling questions working with booleans in structs

I have some problems working with boolean types and marshalling this in a struct back and forth between C# and C. I am very rusty in C but hopefully there's nothing crucially wrong in that part.
As far as I've read/seen, .NET Boolean and C# bool type is 4 bytes long whilst the C type bool is only 1 byte. For memory footprint reasons, I do not which to use the defined BOOL 4 bytes version in the C code.
Here is some simple test code that hopefully will make my questions clear:
C code:
typedef struct
{
double SomeDouble1;
double SomeDouble2;
int SomeInteger;
bool SomeBool1;
bool SomeBool2;
} TestStruct;
extern "C" __declspec(dllexport) TestStruct* __stdcall TestGetBackStruct(TestStruct* structs);
__declspec(dllexport) TestStruct* __stdcall TestGetBackStruct(TestStruct* structs)
{
return structs;
}
I call this code in C# using the following definitions:
[StructLayout(LayoutKind.Explicit)]
public struct TestStruct
{
[FieldOffset(0)]
public double SomeDouble1;
[FieldOffset(8)]
public double SomeDouble2;
[FieldOffset(16)]
public int SomeInteger;
[FieldOffset(17), MarshalAs(UnmanagedType.I1)]
public bool SomeBool1;
[FieldOffset(18), MarshalAs(UnmanagedType.I1)]
public bool SomeBool2;
};
[DllImport("Front.dll", CharSet = CharSet.Auto, SetLastError = true, CallingConvention = CallingConvention.StdCall)]
public static extern IntPtr TestGetBackStruct([MarshalAs(UnmanagedType.LPArray, SizeConst = 1)] TestStruct[] structs);
and here is the actual test function in C#:
[Test]
public void Test_CheckStructParsing()
{
var theStruct = new TestStruct();
theStruct.SomeDouble1 = 1.1;
theStruct.SomeDouble2 = 1.2;
theStruct.SomeInteger = 1;
theStruct.SomeBool1 = true;
theStruct.SomeBool2 = false;
var structs = new TestStruct[] { theStruct };
IntPtr ptr = TestGetBackStruct(structs);
var resultStruct = (TestStruct)Marshal.PtrToStructure(ptr, typeof(TestStruct));
}
This works in the sense that I do get a struct back (using the debugger to inspect it), but with totally wrong values. I.e. the marshalling does not work at all. I've tried different version of the C# struct without success. So here are my questions (1 & 2 most important):
Is the C function correct for this purpose?
How is the struct to be written correctly in order to get me the correct values in the struct back to C#? (Is it even necessary to define the struct with the StructLayout(LayoutKind.Explicit) attribute using the FieldOffset values or can I use StructLayout(LayoutKind.Sequential) instead)?
Since I am returning a pointer to the TestStruct in C, I guess it should be possible to get back an array of TestStructs in C#. But this does not seem to be possible using the Marshal.PtrToStructure function. Would it be possible in some other way?
Apparantly it is possible to use something called unions in C by having multiple struct fields point to the same memory allocation using the same FieldOffset attribute value. I understand this, but I still don't get yet when such scenario would be useful. Please enlighten me.
Can someone recommend a good book on C# P/Invoke to C/C++? I am getting a bit tired of getting pieces of information here and there on the web.
Much obliged for help with these questions. I hope they were not too many.

Stop using LayoutKind.Explicit and get rid of the FieldOffset attributes and your code will work. Your offsets were not correctly aligning the fields.
public struct TestStruct
{
public double SomeDouble1;
public double SomeDouble2;
public int SomeInteger;
[MarshalAs(UnmanagedType.I1)]
public bool SomeBool1;
[MarshalAs(UnmanagedType.I1)]
public bool SomeBool2;
};
Declare the function in C# like this:
public static extern void TestGetBackStruct(TestStruct[] structs);
The default marshalling will match your C++ declaration (your code is C++ rather than C in fact) but you must make sure that your allocate the TestStruct[] parameter in the C# code before calling the function. Normally you would also pass the length of the array as a parameter so that the C++ code knows how many structs there are.
Please don't try to return the array of structures from the function. Use the structs parameter as an in/out parameter.
I know of no book with an emphasis on P/Invoke. It appears to be something of a black art!

Marshalling an array of structures from C++ to C#?

In my C# code I'm trying to fetch an array of structures from a legacy C++ DLL (the code I cannot change).
In that C++ code, the structure is defined like this:
struct MyStruct
{
char* id;
char* description;
};
The method that I'm calling (get_my_structures) returns a pointer to an array of MyStruct structures:
MyStruct* get_my_structures()
{
...
}
There is another method that returns the number of stuctures so I do know how many structures get returned.
In my C# code, I have defined MyStruct like this:
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPStr)] // <-- also tried without this
private string _id;
[MarshalAsAttribute(UnmanagedType.LPStr)]
private string _description;
}
The interop signature looks like this:
[DllImport("legacy.dll", EntryPoint="get_my_structures")]
public static extern IntPtr GetMyStructures();
Finally, the code that fetches the array of MyStruct structures looks like this:
int structuresCount = ...;
IntPtr myStructs = GetMyStructures();
int structSize = Marshal.SizeOf(typeof(MyStruct)); // <- returns 8 in my case
for (int i = 0; i < structuresCount; i++)
{
IntPtr data = new IntPtr(myStructs.ToInt64() + structSize * i);
MyStruct ms = (MyStruct) Marshal.PtrToStructure(data, typeof(MyStruct));
...
}
The trouble is, only the very first structure (one at the offset zero) gets marshaled correctly. Subsequent ones have bogus values in _id and _description members. The values are not completely trashed, or so it seems: they are strings from some other memory locations. The code itself does not crash.
I have verified that the C++ code in get_my_structures() does return correct data. The data is not accidentally deleted or modified during or after the call.
Viewed in a debugger, C++ memory layout of the returned data looks like this:
0: id (char*) <---- [MyStruct 1]
4: description (char*)
8: id (char*) <---- [MyStruct 2]
12: description (char*)
16: id (char*) <---- [MyStruct 3]
...
[Update 18/11/2009]
Here is how the C++ code prepares these structures (the actual code is much uglier, but this is a close enough approximation):
static char buffer[12345] = {0};
MyStruct* myStructs = (MyStruct*) &buffer;
for (int i = 0; i < structuresCount; i++)
{
MyStruct* ms = <some other permanent address where the struct is>;
myStructs[i].id = (char*) ms->id;
myStructs[i].description = (char*) ms->description;
}
return myStructs;
Admittedly, the code above does some ugly casting and copies raw pointers around, but it still does seem to do that correctly. At least that's what I see in the debugger: the above (static) buffer does contain all these naked char* pointers stored one after another, and they point to valid (non-local) locations in memory.
Pavel's example shows that this is really the only place where things can go wrong. I will try to analyze what happens with those 'end' locations where the strings really are, not the locations where the pointers get stored.

I cannot reproduce your problem, which leads me to suspect that it's really on C++ side of things. Here's the complete source code for my attempt.
dll.cpp - compile with cl.exe /LD:
extern "C" {
struct MyStruct
{
char* id;
char* description;
};
__declspec(dllexport)
MyStruct* __stdcall get_my_structures()
{
static MyStruct a[] =
{
{ "id1", "desc1" },
{ "id2", "desc2" },
{ "id3", "desc3" }
};
return a;
}
}
test.cs - compile with csc.exe /platform:x86:
using System;
using System.Runtime.InteropServices;
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPStr)]
public string _id;
[MarshalAsAttribute(UnmanagedType.LPStr)]
public string _description;
}
class Program
{
[DllImport("dll")]
static extern IntPtr get_my_structures();
static void Main()
{
int structSize = Marshal.SizeOf(typeof(MyStruct));
Console.WriteLine(structSize);
IntPtr myStructs = get_my_structures();
for (int i = 0; i < 3; ++i)
{
IntPtr data = new IntPtr(myStructs.ToInt64() + structSize * i);
MyStruct ms = (MyStruct) Marshal.PtrToStructure(data, typeof(MyStruct));
Console.WriteLine();
Console.WriteLine(ms._id);
Console.WriteLine(ms._description);
}
}
}
This correctly prints out all 3 structs.
Can you show your C++ code that fills the structs? The fact that you can call it from C++ directly and get correct results does not necessarily mean it's correct. For example, you could be returning a pointer to a stack-allocated struct. When doing a direct call, then, you'd get a technically invalid pointer, but the data would likely remain preserved. When doing P/Invoke marshalling, the stack could be overwritten by P/Invoke data structures by the point it tries to read values from there.

I would change the structure. Instead of strings etc. , use IntPtr:
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
private IntPtr _id;
private IntPtr _description;
}
Then each value of the C# array could be manually marshalled to string using Marshal.PtrToString taking into account charset etc.

I usually end up working these things out by trial and error. Make sure you have the CharSet property set on your StructLayout, and I would try UnmanagedType.LPTStr, seems to work better for char *, even though I am not sure why.
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Auto)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private string _id;
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private string _description;
}

I think, also, in addition to the answers given, that you need to supply the length as well, ie
[MarshalAsAttribute(UnmanagedType.LPTStr), SizeConst = , ArraySubType = System.Runtime.InteropServices.UnmanagedType.AnsiBStr)]
This is a trial and error to get this right, also, another thing to consider, in some WinAPI calls that expect a string parameter, usually a ref parameter, it might be worth your while to try the StringBuilder class also...Nothing else comes to mind on this other than the points I have mentioned here... Hope this helps, Tom

You have to use UnmanagedType.LPTStr for char*. Also a StringBuilder is recommended for a non const char*:
And a CharSet specification:
[StructLayout(LayoutKind.Sequential, Charset = CharSet.Auto)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private StringBuilder _id;
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private StringBuilder _description;
}
As for the DllImport declaration, have you tried
[DllImport("legacy.dll", EntryPoint="get_my_structures")]
public static extern MarshalAs(UnmanagedType.LPArray) MyStruct[] GetMyStructures();
?
Also, if the previous doesn't work, leave it at IntPtr and try to Mashal the returned structs like this:
for (int i = 0; i < structuresCount; i++)
{
MyStruct ms = (MyStruct) Marshal.PtrToStructure(myStructs, typeof(MyStruct));
...
myStructs += Marshal.SizeOf(ms);
}