We Keep Coding

PInvoke struct with nested struct array

I'm trying to PInvoke a method which has a struct parameter with nested struct array pointer. The c declaration looks like this:
duckdb_state duckdb_query(duckdb_connection connection, const char *query, duckdb_result *out_result);
typedef struct {
void *data;
bool *nullmask;
duckdb_type type;
char *name;
} duckdb_column;
typedef struct {
idx_t column_count;
idx_t row_count;
duckdb_column *columns;
char *error_message;
} duckdb_result;
I declared them in C# like this:
[DllImport("duckdb.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint = "duckdb_query")]
public static extern DuckdbState DuckdbQuery(IntPtr connection, string query, out DuckdbResult result);
[StructLayout(LayoutKind.Sequential)]
public struct DuckdbColumn
{
IntPtr data;
bool nullmask;
DuckdbType type;
string name;
}
[StructLayout(LayoutKind.Sequential)]
public struct DuckdbResult
{
public long column_count;
public long row_count;
public IntPtr columns;
public string error_message;
}
But when I try to execute the query and read the columns I don't get any meaningful data:
result = DuckdbQuery(connection, "SELECT * FROM integers", out queryResult);
DuckdbColumn[] columns = new DuckdbColumn[queryResult.column_count];
var queryResultColumns = queryResult.columns;
var columnPointer = Marshal.ReadIntPtr(queryResultColumns);
var ptrToStructure = (DuckdbColumn)Marshal.PtrToStructure(columnPointer, typeof(DuckdbColumn));
How should I change the PInvoke declarations so that I can read the columns after executign the query?
There is example c code at: DuckDB c example
Update 1
I can get column names with the following code:
for (int i = 0; i < queryResult.column_count; i++)
{
var column = (DuckdbColumn)Marshal.PtrToStructure(queryResult.columns + 8 + (Marshal.SizeOf<DuckdbColumn>() + 8) * i, typeof(DuckdbColumn));
columns[i] = column;
}
but type field still says DUCKDB_TYPE_INVALID
Update 2
As suggested by David in his answer I changed bool nullmask;to IntPtr nullmask; and I can now read column information like this:
for (int i = 0; i < queryResult.column_count; i++)
{
var column = (DuckdbColumn)Marshal.PtrToStructure(queryResult.columns + Marshal.SizeOf<DuckdbColumn>() * i, typeof(DuckdbColumn));
columns[i] = column;
}

You have translated this field incorrectly
bool *nullmask
This is not a bool it's a pointer. Declare it as
IntPtr nullmask;
There could be other errors because we can't see all the translations. Additionally, the +8 in your array access pointer arithmetic looks suspicious.

faster way to return data as an array interoping c++

this is a very clean and nice solution to marsahall a struct array from unmanaged C++ code.
it is allmost perfect solution when it comes to simplicity, it took me a while to get to that level of understanding the concept, so that in few lines of code, as you can see C# Main(), i have a populated array of struct ready to be 'harvested'..
typedef struct {
int Id;
BSTR StrVal;
}Package;
extern "C" __declspec(dllexport) void dodata(int requestedLength,int StringSize, Package **Packs){
int count;
count=0;
*Packs = (Package*)LocalAlloc(0, requestedLength * sizeof(Package));
Package *Cur = *Packs;
while(count!= requestedLength)
{
Cur[count].StrVal = NULL;
Cur[count].Id = count;
Cur[count].StrVal=SysAllocString(L"abcdefghij");
Cur[count].StrVal[StringSize-1]=count+'0';
++count;
}
}
C#
[DllImport(#"ExportStructArr.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
public static extern void dodata(int requestedLength, int StringSize, out IntPtr csPkPtr);
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct csPk
{
public int V;
[MarshalAsAttribute(UnmanagedType.BStr, SizeConst = 10)]
public string testStr;
}
static void Main(string[] args){
int ArrL = 16000;
csPk[] Cpk = new csPk[ArrL];
IntPtr CpkPtr = IntPtr.Zero;
int szPk = Marshal.SizeOf(typeof(csPk));
dodata(ArrL, 10, out CpkPtr);
}
now all i have to do is :
for (int i = 0; i < Cpk.Length; i++)
{
Cpk[i] = (csPk)Marshal.PtrToStructure(new IntPtr(CpkPtr.ToInt32() + (szPk * i)), typeof(csPk));
}
the solution is quite easy as you can see
the question is using unsafe or any kind of transformation to the data, even going down to bytes...
how could i optimize it to perform better returning the data ?
Edit:
links i have tried to learn from other answers here in SO:
answer from Hans Passant
answer from AbdElRaheim
answer from supercat
also tried google : wikipedia , a github post by stephentoub

this is a complete blazing fast solution to populate a list of objects, i did my best
and i will be happy to have comments and suggestions.
c++
typedef struct _DataPacket
{
BSTR buffer;
UINT size;
} DataPacket;
extern "C" __declspec(dllexport) void GetPacksUnsafe( int size, DataPacket** DpArray )
{
int szr = size;int count=0;
*DpArray = (DataPacket*)CoTaskMemAlloc( szr * sizeof( DataPacket ));
if ( DpArray != NULL )
{
DataPacket* CurPack = *DpArray;
for ( int i = 0; i < szr; i++, CurPack++ )
{
CurPack->size = i;
CurPack->buffer = SysAllocString(L"SomeText00");
CurPack->buffer[9]=i+'0';
}
}
}
C#
[DllImport(#"ExportStructArr.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
public static extern void GetPacksUnsafe(int size, PackU** outPackUArr);
[StructLayout(LayoutKind.Sequential)]
public unsafe struct PackU
{
public char* StrVal;
public int IntVal;
}
public static unsafe List<PackU> PopulateLstPackU(int ArrL)
{
PackU* PackUArrOut;
List<PackU> RtLstPackU = new List<PackU>(ArrL);
GetPacksUnsafe(ArrL, &PackUArrOut);
PackU* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new PackU(){ StrVal = CurrentPack->StrVal, IntVal=CurrentPack->IntVal});
}
Marshal.FreeCoTaskMem((IntPtr)PackUArrOut);
for (int i = 0; i < 10; i++)
{
Console.WriteLine("{0}", new string(RtLstPackU[i].StrVal));
}
return RtLstPackU;
}
using the code is as simple as it could possibly be
static unsafe void Main(string[] args)
{
int ArrL = 100000;
List<PackU> LstPackU;
LstPackU = PopulateLstPackU(ArrL);
}
there you have a list of custom data as fast as a bullet..
EDIT
using pointers instead of strings :
typedef struct _DataPackCharPnt
{
char* buffer;
UINT IntVal;
} DataPackCharPnt;
extern "C" __declspec(dllexport) void GetPacksPnt( int size, DataPackCharPnt** DpArrPnt )
{
int count = 0;
int TmpStrSize = 10;
*DpArrPnt = (DataPackCharPnt*)CoTaskMemAlloc( size * sizeof( DataPackCharPnt ));
DataPackCharPnt* CurPackPnt = *DpArrPnt;
char dummyStringDataObject[]= "abcdefgHi";
for ( int i = 0; i < size; i++,CurPackPnt++ )
{
dummyStringDataObject[9] = i+'0';
CurPackPnt->IntVal=i;
CurPackPnt->buffer = (char*)malloc(sizeof(char)*TmpStrSize);
strcpy(CurPackPnt->buffer, dummyStringDataObject);
}
}
reduced the time taken from 11 to 7 ms populating 100k elements
is there any part of creating the buffer i could omit ?
the duty of dummyStringDataObject is to simulate work, say getting a file name then set the buffer with its value, so except for this extra time which is the whole purpose of this function, to return some unknown values and lengths of the strings...
could you optimize it even further ?

How to create an array of structures in C# using DLL (written in C)?

I want to fill a structure with data so I can pass it to my DLL functions. The structures are similar in both C# and C implementations. For instance something like (in C):
typedef struct{
apple[] apples;
leaf[] leaves;
} tree;
typedef struct{
int taste;
} apple;
typedef struct{
int color;
} leaf;
I want to create a tree structure on C#, fill it with apples and leaves and then send it to a function I have on a DLL.
How?

First of all, I think you should change the C struct a little bit:
typedef struct{
UINT cApples;
const apple *pApples;
UINT cLeaves;
const leaf *pLeaves;
} tree;
On the C# side:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct Tree
{
internal uint cApples;
internal IntPtr pApples;
internal uint cLeaves;
internal IntPtr pLeaves;
}
You can define Leaf and Apple similarly. Then you can populate them on C# side as follows:
private Tree CreateTree(Leaf[] leaves, Apple[] apples)
{
var result = new Tree();
result.cLeaves = (uint)leaves.Length;
result.cApples = (uint)apples.Length;
result.pLeaves = AllocateNativeMemory(leaves);
result.pApples = AllocateNativeMemory(apples);
return result;
}
private IntPtr AllocateNativeMemory<T>(T[] elements) where T : struct
{
int elementSize = Marshal.SizeOf(typeof(T));
IntPtr result = Marshal.AllocHGlobal(elements.Length * elementSize);
for (int i = 0; i < elements.Length; i++)
{
Marshal.StructureToPtr(
elements[i], new IntPtr((long)result + i * elementSize), false);
}
return result;
}
Now you can pass the result of the CreateTree method to the extern method that calls the C side.
Remark: The allocated memory should be freed; otherwise, your app will leak. If you decide that the C# side is responsible to free the allocated memory, you should do it at the end as follows:
private static void FreeTree(Tree tree)
{
FreeNativeMemory<Leaf>(tree.pLeaves, tree.cLeaves);
FreeNativeMemory<Apple>(tree.pApples, tree.cApples);
}
private static void FreeNativeMemory<T>(IntPtr arrayPtr, uint arrayLen) where T : struct
{
int elementSize = Marshal.SizeOf(typeof(T));
for (uint i = 0; i < arrayLen; i++)
{
Marshal.DestroyStructure(new IntPtr((long)arrayPtr + i * elementSize), typeof(T));
}
Marshal.FreeHGlobal(arrayPtr);
}

Updating an C dll struct array and its elements values from C# code using dllimport

I have C code which will be build as a dynamic library (DLL) , which i would like
to call C function from C# using dLL created from the C code
C code :
struct data
{
char data_val1[100];
float data_val2;
float data_val3[50];
};
typedef struct data data;
#ifdef __cplusplus
extern "C" __declspec(dllexport) void cfun_call(data *pdata,long count);
#endif
#ifdef __cplusplus
extern "C"
{
#endif
__declspec(dllexport) void cfun_call(data *pdata,long count)
{
int x = 0;
for(x=0;x<count;x++)
{
data[x].data_val2 = (pdata->data_val3[49] + pdata->data_val3[48]) / 2.0;
}
}
#ifdef __cplusplus
}
#endif
Here i wanted to import the function "cfun_call" in C# code, and pass values to the fucntion call
and manipulate the passed values in C function from the dll and wanted to display the updated values
back to the C# code and display it, Since my expertise in C# is limited i need some help to solve this issue
C# code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
class Program
{
public class data
{
public char[] data_val1 = new char[100];
public float data_val2;
public float[] data_val3 = new float[50];
};
[DllImport("mycdll.dll", EntryPoint = "cfun_call", CallingConvention = CallingConvention.Cdecl, ExactSpelling = false)]
// void cfun_call(data *pdata,long count); //C function for reference
public static extern void cfun_call([In, Out] data[] ouputdata, long count);
static void Main(string[] args)
{
data[] objData = new data[10];
for (int i = 0; i < 10; i++)
{
//Fill the data in objitemData
objData[i] = new objData();
for (int j = 0; j < 100; j++)
{
objData[i].data_val1[j] = '\0';
}
for (int k = 0; k < 50; k++)
{
objData[i].data_val3[k] = 20.00;
}
objData[i].data_val2 = 0.00;
}
cfun_call(objData,10); //Making call to C dll function
for (int i = 0; i < 10; i++)
Console.WriteLine("{0} ", objData[i].data_val2);
Console.WriteLine("");//new line
Console.ReadLine();
}
}
Here the values (objData) passed from C# function is not updated by using the C dll fucntion , I am not sure why.
Can anyone point me to right direction ?
Edit 1:
I have updated code as suggested ,
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct data
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 100)]
public char[] data_val1;
public float data_val2;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 50)]
public float[] data_val3;
};
Initialized struct elements like below ,
data[] objData = new data[10];
for (int i = 0; i < 10; i++)
{
//Fill the data in objitemData
objData[i] = new objData();
for (int j = 0; j < 100; j++)
{
objData[i].data_val1[j] = '\0'; //I am getting exception here
}
for (int k = 0; k < 50; k++)
{
objData[i].data_val3[k] = 20.00;
}
objData[i].data_val2 = 0.00;
}
Runtime i am getting null ptr exception , like
An unhandled exception of type 'System.NullReferenceException' occurred in mybinary.exe
Additional information: Object reference not set to an instance of an object.
How to initialize the struct array elements properly in manged code ?
Edit 2:
Hi one more question , when i add , objData[i].data_val3[k] = randomData; //randomvalues, It is not updated when making cfun_call while using contnt value it is updated why ?

Your translation of the struct is incorrect. You need it to be like so:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct data
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 100)]
public char[] data_val1;
public float data_val2;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 50)]
public float[] data_val3;
};
You have to make this a struct since you need to pass an array of values. Your declaration using class leads to you passing an array of references.
You will need to initialize the arrays explicitly now. That might look like so:
data[] objData = new data[10];
for (int i = 0; i < 10; i++)
{
objData[i].data_val1 = new char[100];
objData[i].data_val2 = 0.00;
objData[i].data_val3 = new float[50];
for (int k = 0; k < 50; k++)
{
objData[i].data_val3[k] = 20.0f;
}
}
Further, C++ long is 32 bits wide, but C# long is 64 bits wide. You therefore have a mismatch. Your p/invoke should be:
[DllImport("mycdll.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void cfun_call(
[In, Out] data[] ouputItem_data,
int count
);

Marshaling structs with strings and arrays from managed to unmanaged code

I have a simple C# data structure with a string, an int and a vector of ints:
class MyManagedClass
{
public string m_Str;
int m_Int;
int[] m_IntArray;
}
The equivalent in C++ is:
struct myUnmanagedStruct
{
char* m_Str;
UINT m_Int;
UINT* m_IntArray;
}
I have an unmanaged function that creates an array of myUnmanagedStruct structs. What is the best way to write a managed wrapper that correctly marshals the data such that what is created on the unmanaged side is correctly passed back to the managed side? (i.e. I want to make an array of MyManagedClass objects from an array of MyUnmanagedStructs)
Note:
a) A string is created in the unmanaged struct
b) A vector of ints is created in the unmanaged struct
My best attempt so far is:
On the managed side:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack=1)]
public class MyManagedClass
{
public MyManagedClass()
{
m_IntArray = new int[4];
}
public String m_Str;
public int m_Int;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public int[] m_IntArray;
}
[StructLayout(LayoutKind.Sequential)]
public struct UnsafeLCEArray
{
public int m_Int;
public IntPtr m_CharBuf;
public IntPtr m_IntBuf;
}
public unsafe class LibWrap
{
// Declares managed prototypes for the unmanaged function.
[DllImport("mydll.dll", EntryPoint = "GetUnmanagedStructs")]
public static extern voidGetUnmanagedStructs(out int size, out IntPtr outArray);
}
On the unmanaged side:
typedef struct _MyUnmanagedStruct
{
char* m_Str;
UINT m_Int;
UINT* m_IntArray;
} MyUnmanagedStruct;
typedef struct _UNSAFELCEARRAY
{
char* strBuf;
UINT intBuf;
UINT* intArrayBuf;
} UNSAFELCEARRAY;
extern "C" __declspec(dllexport) void GetUnmanagedStructs( int* pSize, UNSAFELCEARRAY** ppStruct )
{
const int cArraySize = 5;
*pSize = cArraySize;
int numBytes = cArraySize * sizeof( MyUnmanagedStruct);
*ppStruct = (UNSAFELCEARRAY*)CoTaskMemAlloc(numBytes);
UNSAFELCEARRAY* pCurStruct = *ppStruct;
char* typenamebuffer;
char* numBuffer;
int var = 999;
for( int i = 0; i < cArraySize; i++, pCurStruct++ )
{
pCurStruct->intBuf = i+1;
typenamebuffer = (char*)CoTaskMemAlloc( 8 );
memcpy_s(typenamebuffer, 8, "bufABCD", 8);
pCurStruct->strBuf = typenamebuffer;
numBuffer = (char*)CoTaskMemAlloc( 16 );
++var;
memcpy_s(numBuffer, 4, &var, 4);
++var;
memcpy_s(numBuffer+4, 4, &var, 4);
++var;
memcpy_s(numBuffer+8, 4, &var, 4);
++var;
memcpy_s(numBuffer+12, 4, &var, 4);
pCurStruct->intArrayBuf = (UINT*)numBuffer;
}
}
Everything works if I remove the vector of ints from the managed and unmanaged objects, but with the code above the array of ints is uninitialized on return. I use the function below to generate MyManagedClasses from MyUnmanagedStructs
int size;
IntPtr outArray;
LibWrap.GetUnmanagedStructs(out size, out outArray);
manArray = new MyManagedClass[size];
IntPtr current = outArray;
for (int i = 0; i < size; i++)
{
manArray[i] = new MyManagedClass();
Marshal.PtrToStructure(current, manArray[i]);
Marshal.DestroyStructure(current, typeof(sb_LCE));
int numBytes = Marshal.SizeOf(manArray[i]);
current = (IntPtr)((long)current + numBytes);
}
Marshal.FreeCoTaskMem(outArray);
Forgive the lengthy explanation and the fact the unmanaged strut is being filled with dummy values. This is just for illustration.