We Keep Coding

interop with nim return Struct Array containing a string /char* member

interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.

The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.

Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}

Get fixed array values from unsafe struct

I'm using an external C library that comes with a C# wrapper file containing a struct like (among many other things):
[StructLayout(LayoutKind.Sequential, Pack = 8)]
unsafe public struct Data
{
public fixed double Values[3];
};
I can get the values by adding an additional method to that struct:
public double[] CopyFixedDoubleArray()
{
double[] result = new double[3];
fixed (double* pDst = result)
{
fixed (double* pSrc = Values)
{
double* pd = pDst;
double* ps = pSrc;
for (int i = 0; i < result.Length; i++)
{
*pd = *ps;
pd++; ps++;
}
}
}
return result;
}
But this is not what I want because I don't want to touch the wrapper file (to avoid having to keep the external file in our SCM repository). Whatever I try to do to get the values from outside that struct results in the following error:
Fixed size buffers can only be accessed through locals or fields
What I've tried:
double a = data.Values[2];
double a; unsafe { a = data.Values[2]; }
double a; fixed (double* p = data.Values) { a = p[2]; }
fixed (double a = data.Values[2]) { /* use a */ }
Is there any way to get around this?

Suddenly the word 'extension method' crossed my mind. And yes, I have it working:
public static double[] CopyFixedDoubleArray(this Data data)
{
unsafe
{
return new[] { data.Values[0], data.Values[1], data.Values[2] };
}
}

Rather than patch your code up, have you considered shoving this over to the marshaller? That's its entire purpose: to assist with native interop while letting you write pure managed code.
You'd have to declare your struct like this:
[StructLayout(LayoutKind.Sequential, Pack = 8)]
public struct Data
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public double[] Values;
};
When this structure is filled by the marshaller (through either a direct function call through a pointer, or from mapping the struct on top of a byte buffer) it will allocate the proper array and copy its contents over. This makes your entire function obsolete, but if you really wanted to you could simply use Array.Copy (or the Linq equivalent).
Notice there's no more unsafe modifier. You can run this in a partially trusted environment with no issues!

Getting Array of struct from IntPtr

I have some struct like this
struct MyStruct
{
public int field1;
public int field2;
public int field3;
}
and I have pointer to array of this struct.
So, I need to get array from this pointer.
I'm tried to using Marshal.PtrToStructure, but i had memory reading error.
This is my methode:
public MyStruct[] GetArrayOfStruct(IntPtr pointerToStruct, int length)
{
var sizeInBytes = Marshal.SizeOf(typeof(TCnt));
MyStruct[] output = new MyStruct[length];
for (int i = 0; i < length; i++)
{
IntPtr p = new IntPtr((pointerToStruct.ToInt32() + i * sizeInBytes));
output[i] = (MyStruct)System.Runtime.InteropServices.Marshal.PtrToStructure(p, typeof(MyStruct));
}
return output;
}
So, what am i doing wrong ?

This function worked for me, assuming that the size of the struct is fixed:
public static void MarshalUnmananagedArray2Struct<T>(IntPtr unmanagedArray, int length, out T[] mangagedArray)
{
var size = Marshal.SizeOf(typeof(T));
mangagedArray = new T[length];
for (int i = 0; i < length; i++)
{
IntPtr ins = new IntPtr(unmanagedArray.ToInt64() + i * size);
mangagedArray[i] = Marshal.PtrToStructure<T>(ins);
}
}

Two problems. You use TCnt instead of MyStruct in the Marshal.SizeOf() call. Your IntPtr arithmetic cannot work on a 64-bit machine, you must use IntPtr.ToInt64() or cast to (long).
Just getting the wrong IntPtr or length is certainly a possibility too of course. Use Debug + Windows + Memory + Memory 1 and put "pointerToStruct" in the Address box for basic verification.

Structs in C and C# are not the same thing. One of the differences is that in C# you have to explicitly demand that your struct should be sequentially laid out. If you didn't write
[StructLayout(LayoutKind.Sequential)] or [StructLayout(LayoutKind.Explicit)] attribute to your structure I don't believe that you can manage it in this way. Microsoft states that PtrToStructure is to be used to convert structures from unmanaged to managed memory
You should test if adding this attributes to your struct helps, If it doesn't yet help try allocating memory with Marshal.AllocHGlobal(IntPtr) and use Marshal.Copy to init your structure and then try using PtrToStructure. If this works then you can't use PtrToStructure with managed memory

How to pinvoke a variable-length array of structs from GetTokenInformation() safely for 32-bit and 64-bit? C#

I'm following the pinvoke code provided here but am slightly scared by the marshalling of the variable-length array as size=1 and then stepping through it by calculating an offset instead of indexing into an array. Isn't there a better way? And if not, how should I do this to make it safe for 32-bit and 64-bit?
[StructLayout(LayoutKind.Sequential)]
public struct SID_AND_ATTRIBUTES
{
public IntPtr Sid;
public uint Attributes;
}
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_GROUPS
{
public int GroupCount;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
public SID_AND_ATTRIBUTES[] Groups;
};
public void SomeMethod()
{
IntPtr tokenInformation;
// ...
string retVal = string.Empty;
TOKEN_GROUPS groups = (TOKEN_GROUPS)Marshal.PtrToStructure(tokenInformation, typeof(TOKEN_GROUPS));
int sidAndAttrSize = Marshal.SizeOf(new SID_AND_ATTRIBUTES());
for (int i = 0; i < groups.GroupCount; i++)
{
// *** Scary line here:
SID_AND_ATTRIBUTES sidAndAttributes = (SID_AND_ATTRIBUTES)Marshal.PtrToStructure(
new IntPtr(tokenInformation.ToInt64() + i * sidAndAttrSize + IntPtr.Size),
typeof(SID_AND_ATTRIBUTES));
// ...
}
I see here another approach of declaring the length of the array as much bigger than it's likely to be, but that seemed to have its own problems.
As a side question: When I step through the above code in the debugger I'm not able to evaluate tokenInformation.ToInt64() or ToInt32(). I get an ArgumentOutOfRangeException. But the line of code executes just fine!? What's going on here?

Instead of guessing what the offset, is its generally better to use Marshal.OffsetOf(typeof(TOKEN_GROUPS), "Groups") to get the correct offset to the start of the array.

I think it looks okay -- as okay as any poking about in unmanaged land is, anyway.
However, I wonder why the start is tokenInformation.ToInt64() + IntPtr.Size and not tokenInformation.ToInt64() + 4 (as the GroupCount field type is an int and not IntPtr). Is this for packing/alignment of the structure or just something fishy? I do not know here.
Using tokenInformation.ToInt64() is important because on a 64-bit machine will explode (OverflowException) if the IntPtr value is larger than what an int can store. However, the CLR will handle a long just fine on both architectures and it doesn't change the actual value extracted from the IntPtr (and thus put back into the new IntPtr(...)).
Imagine this (untested) function as a convenience wrapper:
// unpacks an array of structures from unmanaged memory
// arr.Length is the number of items to unpack. don't overrun.
void PtrToStructureArray<T>(T[] arr, IntPtr start, int stride) {
long ptr = start.ToInt64();
for (int i = 0; i < arr.Length; i++, ptr += stride) {
arr[i] = (T)Marshal.PtrToStructure(new IntPtr(ptr), typeof(T));
}
}
var attributes = new SID_AND_ATTRIBUTES[groups.GroupCount];
PtrToStructureArray(attributes, new IntPtr(tokenInformation.ToInt64() + IntPtr.Size), sidAndAttrSize);
Happy coding.

Marshalling an array of structures from C++ to C#?

In my C# code I'm trying to fetch an array of structures from a legacy C++ DLL (the code I cannot change).
In that C++ code, the structure is defined like this:
struct MyStruct
{
char* id;
char* description;
};
The method that I'm calling (get_my_structures) returns a pointer to an array of MyStruct structures:
MyStruct* get_my_structures()
{
...
}
There is another method that returns the number of stuctures so I do know how many structures get returned.
In my C# code, I have defined MyStruct like this:
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPStr)] // <-- also tried without this
private string _id;
[MarshalAsAttribute(UnmanagedType.LPStr)]
private string _description;
}
The interop signature looks like this:
[DllImport("legacy.dll", EntryPoint="get_my_structures")]
public static extern IntPtr GetMyStructures();
Finally, the code that fetches the array of MyStruct structures looks like this:
int structuresCount = ...;
IntPtr myStructs = GetMyStructures();
int structSize = Marshal.SizeOf(typeof(MyStruct)); // <- returns 8 in my case
for (int i = 0; i < structuresCount; i++)
{
IntPtr data = new IntPtr(myStructs.ToInt64() + structSize * i);
MyStruct ms = (MyStruct) Marshal.PtrToStructure(data, typeof(MyStruct));
...
}
The trouble is, only the very first structure (one at the offset zero) gets marshaled correctly. Subsequent ones have bogus values in _id and _description members. The values are not completely trashed, or so it seems: they are strings from some other memory locations. The code itself does not crash.
I have verified that the C++ code in get_my_structures() does return correct data. The data is not accidentally deleted or modified during or after the call.
Viewed in a debugger, C++ memory layout of the returned data looks like this:
0: id (char*) <---- [MyStruct 1]
4: description (char*)
8: id (char*) <---- [MyStruct 2]
12: description (char*)
16: id (char*) <---- [MyStruct 3]
...
[Update 18/11/2009]
Here is how the C++ code prepares these structures (the actual code is much uglier, but this is a close enough approximation):
static char buffer[12345] = {0};
MyStruct* myStructs = (MyStruct*) &buffer;
for (int i = 0; i < structuresCount; i++)
{
MyStruct* ms = <some other permanent address where the struct is>;
myStructs[i].id = (char*) ms->id;
myStructs[i].description = (char*) ms->description;
}
return myStructs;
Admittedly, the code above does some ugly casting and copies raw pointers around, but it still does seem to do that correctly. At least that's what I see in the debugger: the above (static) buffer does contain all these naked char* pointers stored one after another, and they point to valid (non-local) locations in memory.
Pavel's example shows that this is really the only place where things can go wrong. I will try to analyze what happens with those 'end' locations where the strings really are, not the locations where the pointers get stored.

I cannot reproduce your problem, which leads me to suspect that it's really on C++ side of things. Here's the complete source code for my attempt.
dll.cpp - compile with cl.exe /LD:
extern "C" {
struct MyStruct
{
char* id;
char* description;
};
__declspec(dllexport)
MyStruct* __stdcall get_my_structures()
{
static MyStruct a[] =
{
{ "id1", "desc1" },
{ "id2", "desc2" },
{ "id3", "desc3" }
};
return a;
}
}
test.cs - compile with csc.exe /platform:x86:
using System;
using System.Runtime.InteropServices;
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPStr)]
public string _id;
[MarshalAsAttribute(UnmanagedType.LPStr)]
public string _description;
}
class Program
{
[DllImport("dll")]
static extern IntPtr get_my_structures();
static void Main()
{
int structSize = Marshal.SizeOf(typeof(MyStruct));
Console.WriteLine(structSize);
IntPtr myStructs = get_my_structures();
for (int i = 0; i < 3; ++i)
{
IntPtr data = new IntPtr(myStructs.ToInt64() + structSize * i);
MyStruct ms = (MyStruct) Marshal.PtrToStructure(data, typeof(MyStruct));
Console.WriteLine();
Console.WriteLine(ms._id);
Console.WriteLine(ms._description);
}
}
}
This correctly prints out all 3 structs.
Can you show your C++ code that fills the structs? The fact that you can call it from C++ directly and get correct results does not necessarily mean it's correct. For example, you could be returning a pointer to a stack-allocated struct. When doing a direct call, then, you'd get a technically invalid pointer, but the data would likely remain preserved. When doing P/Invoke marshalling, the stack could be overwritten by P/Invoke data structures by the point it tries to read values from there.

I would change the structure. Instead of strings etc. , use IntPtr:
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
private IntPtr _id;
private IntPtr _description;
}
Then each value of the C# array could be manually marshalled to string using Marshal.PtrToString taking into account charset etc.

I usually end up working these things out by trial and error. Make sure you have the CharSet property set on your StructLayout, and I would try UnmanagedType.LPTStr, seems to work better for char *, even though I am not sure why.
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Auto)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private string _id;
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private string _description;
}

I think, also, in addition to the answers given, that you need to supply the length as well, ie
[MarshalAsAttribute(UnmanagedType.LPTStr), SizeConst = , ArraySubType = System.Runtime.InteropServices.UnmanagedType.AnsiBStr)]
This is a trial and error to get this right, also, another thing to consider, in some WinAPI calls that expect a string parameter, usually a ref parameter, it might be worth your while to try the StringBuilder class also...Nothing else comes to mind on this other than the points I have mentioned here... Hope this helps, Tom

You have to use UnmanagedType.LPTStr for char*. Also a StringBuilder is recommended for a non const char*:
And a CharSet specification:
[StructLayout(LayoutKind.Sequential, Charset = CharSet.Auto)]
public class MyStruct
{
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private StringBuilder _id;
[MarshalAsAttribute(UnmanagedType.LPTStr)]
private StringBuilder _description;
}
As for the DllImport declaration, have you tried
[DllImport("legacy.dll", EntryPoint="get_my_structures")]
public static extern MarshalAs(UnmanagedType.LPArray) MyStruct[] GetMyStructures();
?
Also, if the previous doesn't work, leave it at IntPtr and try to Mashal the returned structs like this:
for (int i = 0; i < structuresCount; i++)
{
MyStruct ms = (MyStruct) Marshal.PtrToStructure(myStructs, typeof(MyStruct));
...
myStructs += Marshal.SizeOf(ms);
}