We Keep Coding

Why can't I marshal UCS-4 strings properly in C#?

I am trying to marshal a hid_device_info struct in C#, but I can't figure out how to translate the wchar_t* strings to managed C# strings. I have tried all possible values in the MarshalAs attribute, but all of them returned the first character only and nothing else.
I have tried replacing all the wide strings with pointers so I can manually look at them, this is the struct that I have so far:
public struct HidDeviceInfo
{
public IntPtr path; // This one marshals fine because it's just a regular char_t*
public ushort vendor_id;
public ushort product_id;
public IntPtr serial_number; // wchar_t*
public ushort release_number;
public IntPtr manufacturer_string; // wchar_t*
public IntPtr product_string; // wchar_t*
public ushort usage_page;
public ushort usage;
public int interface_number;
public IntPtr next;
}
When I manually iterate through one of the pointers (serial_number for example), I can see that all the characters have 4 bytes (1 ascii byte followed by 3 zeros). I have tried all the possible Marshal.PtrToString... methods, but none of them are able to retrieve the full string.
I have a suspicion that the strings are being treated as 2 byte characters since I can't specify the character width anywhere in C#, and this is why it stops after the first character. Of course, by knowing this, I could easily write my own string marshaler, but I feel like there must be an existing solution and I'm overlooking something obvious.
This struct is coming from a P/Invoked function and Marshal.PtrToStructure:
[DllImport(LibUsbName, CharSet = CharSet.Unicode)]
public static extern IntPtr hid_enumerate(ushort vendorId, ushort productId);
I've also tried all the possible CharSet values.
This can't be a character type mismatch, as it was in this question, because I've tried all possible combinations of different character types.

I ended up writing this method that works fine for me, but only if all character are ASCII and the char width is guaranteed to be 4 bytes.
private static string ToUcs4String(this IntPtr ptr)
{
var builder = new StringBuilder();
var buffer = new byte[4];
while (true)
{
Marshal.Copy(ptr, buffer, 0, 4);
if (buffer[0] == 0)
break;
builder.Append((char) buffer[0]);
ptr += 4;
}
return builder.ToString();
}

Passing array/list of object/struct from C# Library to C++ MFC application

I'm trying to pass an array of object or struct from a C# library (.net 4.5) called by a C++ MFC application but I can't get correct data. An additional difficulty is that C++ application don't know the object count.
I have successfully passed simple type (int, string) and also a struct with the following code (using UnmanagedExports 1.2.7):
C#
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 4)]
public struct ItemA
{
// Size: 4
public int Id;
// Size: 100
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 50)]
public string Name;
// Size: 2(4)
public bool IsEnabled;
}
[DllExport(CallingConvention = CallingConvention.Cdecl)]
static public IntPtr GetItemA(int itemId)
{
// Check structure size
Debug.Assert(Marshal.SizeOf(typeof(ItemA)) == 108);
// Get ItemA from a WS
var itemA = client.GetItemsA().First(i => i.Id == itemId);
// Convert ItemA to structure pointer
IntPtr buf = Marshal.AllocHGlobal(Marshal.SizeOf(itemA));
Marshal.StructureToPtr(itemA, buf, false);
return buf;
}
C++
#pragma pack(4)
typedef struct
{
// Size: 4
int Id;
// Size: 100
TCHAR Name[50];
// Size: 2(4)
bool IsEnabled;
} ItemA;
extern "C"
{
ItemA* GetItemA(int itemId);
}
// (...)
void CMyAppDlg::OnBnClickedButtonGetItemA()
{
// Check structure size
static_assert(sizeof ItemA == 108, "Size does not match C# struct size");
ItemA* structItemA = GetItemA(1);
}
I have searched everywhere but I don't find any functionnal response to my specific problem. Can you help to write both C# and C++ codes for GetItemAList which can return an array of struct or another type to C++ application?
Thanks in advance!
Edit1 : I changed my code to solve packing/size struct issue (64 bits compilation).
Edit2 : I replace manual alignment by #pragma pack.
Edit3 : I'm always stuck with GetItemAList. Here is my actual c#/c++ codes with an MemoryException after calling c#. Can you help me with this sh**? Thanks in advance.
C#
[DllExport(CallingConvention = CallingConvention.Cdecl)]
static public void GetItemAList([In, Out, MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.Struct, SizeParamIndex = 1)] ref ItemA[] myArray, ref int length)
{
// Get ItemAList from a WS
var itemAArray = client.GetItemsAList().ToArray();
// Set myArray values
length = itemAArray.Length;
for (int i = 0; i < length; i++)
{
myArray[i] = itemAArray[i];
}
}
C++
extern "C"
{
void GetItemAList(ItemA*& myArray, int& length);
}
// (...)
void CMyAppDlg::OnBnClickedButtonGetItemA()
{
ItemA* myArray = new ItemA[100];
int length = 100;
GetItemAList(myArray, length);
}

I have not done any checking, but your C++ version is probably compiled with 4 bytes packing (assuming your are compiling in 32 bits).
In that case, you would have on C++ side:
// Field, Field size, Total size
// -------------------------------
// Id, 0, 4
// Name, 50, 54
// IsEnabled, 1, 55
// (padding) 1, 56
On that side, you should add a static_assert validating that you got the expected size.
static_check(sizeof ItemA == 56, "Size does not match C# struct size")
On the hand on C# side, you would have:
// Field, Field size, Total size
// -------------------------------
// Id, 0, 4
// Name, 50, 54
// IsEnabled, 4, 58
// (no padding) 0, 58
And if bool was 1 byte, you would have a size of 55 instead. In both case, it would not match the size on C++ side.
As it is somewhat hard to know all rules and not make any mistake and also to ensure nothing is broken in future maintenance, you should always add validation in your code.
On C# side, typical validation would be something like: Debug.Assert(Marshal.Sizeof(typeof(ItemA)) == 58.
You could also validate the offset or size of some fields but generally if the size you compute by hand do match on both sides, you probably got it right particularily if you use 1 byte packing.

interop with nim return Struct Array containing a string /char* member

interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.

The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.

Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}

How do I marshal a pointer to a series of null-terminated strings in C#?

I need some help with the following. I've got a c++ API (no access to source) and I'm struggling with the methods returning char* attributes, or returned structures containing char* attributes. According to the API's documentation the return value is as follows:
Return Values
If the function succeeds, the return value is a pointer to a series of null-terminated strings, one for each project on the host system, ending with a second null character. The following example shows the buffer contents with <null> representing the terminating null character:
project1<null>project2<null>project3<null><null>
If the function fails, the return value is NULL
The problem I'm having is that the returned pointer in C# only contains the first value... project1 in this case. How can I get the full list to be able to loop through them on the managed side?
Here's the c# code:
[DllImport("vmdsapi.dll", EntryPoint = "DSGetProjectList", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr DSGetProjectList();
Calling method:
IntPtr ptrProjectList = DSAPI.DSGetProjectList();
string strProjectList = Marshal.PtrToStringAnsi(ptrProjectList).ToString();
strProjectList only contains the first item.
Here's the info from the API's header file...
DllImport char *DSGetProjectList dsproto((void));
Here's some sample code from a c++ console app which I've used for testing purposes...
char *a;
a = DSGetProjectList( );
while( *a ) {
printf("a=%s\n", a);
a += 1 + strlen(a);
}
Each iteration correctly displays every project in the list.

The problem is that when converting the C++ char* to a C# string using Marshal.PtrToStringAnsi, it stops at the first null character.
You shouldn't convert directly the char* to a string.
You could copy the char* represented by an IntPtr to a byte[] using Marshal.Copy and then extract as many string as necessary (see Matthew Watson's answer for extracting strings from a managed array), but you'll need to get the multi-string size first.
As leppie suggest you can also extract the first string using Marshal.PtrToStringAnsi then increment the pointer by this string size and extract the next string and so on. You stops when is extracts an empty string (from the last NULL character).
Something like :
IntPtr ptrProjectList = DSAPI.DSGetProjectList();
List<string> data;
string buffer;
do {
buffer = Marshal.PtrToStringAnsi(ptrProjectList);
ptrProjectList += buffer.size() + 1;
data.Add(buffer);
}while(buffer.size() > 0)

This kind of string is called a Multi-String, and it's quite common in the Windows API.
Marshaling them is fiddly. What you have to do is marshal it as a char[] array, rather than a string, and then convert the char[] array to a set of strings.
See here for an example solution. I've copied the relevant code into this answer, but it is copied from the link I gave:
static List<string> MultiStringToList(char[] multistring)
{
var stringList = new List<string>();
int i = 0;
while (i < multistring.Length)
{
int j = i;
if (multistring[j++] == '\0')
break;
while (j < multistring.Length)
{
if (multistring[j++] == '\0')
{
stringList.Add(new string(multistring, i, j - i - 1));
i = j;
break;
}
}
}
return stringList;
}

How to pinvoke a variable-length array of structs from GetTokenInformation() safely for 32-bit and 64-bit? C#

I'm following the pinvoke code provided here but am slightly scared by the marshalling of the variable-length array as size=1 and then stepping through it by calculating an offset instead of indexing into an array. Isn't there a better way? And if not, how should I do this to make it safe for 32-bit and 64-bit?
[StructLayout(LayoutKind.Sequential)]
public struct SID_AND_ATTRIBUTES
{
public IntPtr Sid;
public uint Attributes;
}
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_GROUPS
{
public int GroupCount;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
public SID_AND_ATTRIBUTES[] Groups;
};
public void SomeMethod()
{
IntPtr tokenInformation;
// ...
string retVal = string.Empty;
TOKEN_GROUPS groups = (TOKEN_GROUPS)Marshal.PtrToStructure(tokenInformation, typeof(TOKEN_GROUPS));
int sidAndAttrSize = Marshal.SizeOf(new SID_AND_ATTRIBUTES());
for (int i = 0; i < groups.GroupCount; i++)
{
// *** Scary line here:
SID_AND_ATTRIBUTES sidAndAttributes = (SID_AND_ATTRIBUTES)Marshal.PtrToStructure(
new IntPtr(tokenInformation.ToInt64() + i * sidAndAttrSize + IntPtr.Size),
typeof(SID_AND_ATTRIBUTES));
// ...
}
I see here another approach of declaring the length of the array as much bigger than it's likely to be, but that seemed to have its own problems.
As a side question: When I step through the above code in the debugger I'm not able to evaluate tokenInformation.ToInt64() or ToInt32(). I get an ArgumentOutOfRangeException. But the line of code executes just fine!? What's going on here?

Instead of guessing what the offset, is its generally better to use Marshal.OffsetOf(typeof(TOKEN_GROUPS), "Groups") to get the correct offset to the start of the array.

I think it looks okay -- as okay as any poking about in unmanaged land is, anyway.
However, I wonder why the start is tokenInformation.ToInt64() + IntPtr.Size and not tokenInformation.ToInt64() + 4 (as the GroupCount field type is an int and not IntPtr). Is this for packing/alignment of the structure or just something fishy? I do not know here.
Using tokenInformation.ToInt64() is important because on a 64-bit machine will explode (OverflowException) if the IntPtr value is larger than what an int can store. However, the CLR will handle a long just fine on both architectures and it doesn't change the actual value extracted from the IntPtr (and thus put back into the new IntPtr(...)).
Imagine this (untested) function as a convenience wrapper:
// unpacks an array of structures from unmanaged memory
// arr.Length is the number of items to unpack. don't overrun.
void PtrToStructureArray<T>(T[] arr, IntPtr start, int stride) {
long ptr = start.ToInt64();
for (int i = 0; i < arr.Length; i++, ptr += stride) {
arr[i] = (T)Marshal.PtrToStructure(new IntPtr(ptr), typeof(T));
}
}
var attributes = new SID_AND_ATTRIBUTES[groups.GroupCount];
PtrToStructureArray(attributes, new IntPtr(tokenInformation.ToInt64() + IntPtr.Size), sidAndAttrSize);
Happy coding.