I am seeing a pretty bizarre issue while trying to pass an array from C++ to C#. I am using Marshal.Copy (specifically: https://msdn.microsoft.com/en-us/library/a53bd6cz(v=vs.110).aspx).
Problem: float array from C++ to C# is yielding a few NaN's in the resulting array.
(Note: I am working in the context of the Unity game engine)
Code
Example C++ code:
extern "C" bool UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API getSomeFloats(float** points, int* count) {
std::vector<float> results;
std::vector<SOME_TYPE> key_points = <SOME_POINTS>
for (auto iter = key_points.begin(); iter < key_points.end(); iter++) {
results.push_back(static_cast<float>(iter->pt.x));
results.push_back(static_cast<float>(iter->pt.y));
}
*points = results.data();
*count = results.size();
//<Print results to csv here>
return true;
}
Example C# code:
[DllImport("NativePlugin")]
private static extern bool getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
private static float[] getFloatArrayFromNative() {
IntPtr ptrResultItems = IntPtr.Zero;
int resultItemsLength = 0;
bool success = getSomeFloats (ref ptrResultItems, ref resultItemsLength);
float[] resultItems = null;
if (success) {
// Load the results into a managed array.
resultItems = new float[resultItemsLength];
Marshal.Copy (ptrResultItems
, resultItems
, 0
, resultItemsLength);
// <PRINT out resultItems to csv here>
return resultItems;
} else {
Debug.Log ("Something went wrong getting some floats");
return new float[] { -1, -2 };
}
}
Example Ouput:
Take the following example:
C++ output (print_out.csv):
123, 456, 789
C# output (print_out_cs.csv):
123, NaN, 789
I'm completely stumped on this one. I just don't understand why only some (roughly 7/100) floats are returning NaN. Does anyone have any advice/insight that might help?
Thanks!
Found few problems in your code:
1. std::vector<float> results; is declared on the stack. It will be gone by the time the function has returned. Declare it as a pointer
std::vector<float> *results = new std::vector<float>(10);
but make sure to also declare a function that will free it on the C++ side.
2.The function parameter do not match.
Your C++:
getSomeFloats(float** points, int* count, CameraPose* pose)
Your C#:
getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
You either have to remove CameraPose* pose from the C++ side or add IntPtr pose to the C# side.
3. The use of UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API.
You don't need that. This is used when you want to use Unity's built in functions such as GL.IssuePluginEvent. You are not using it in this case.
This should do it:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
}
}
4.C# has a garbage collector that moves variables around in the memory. You must pin C# array if you want to modify it from C++ side. You only need to pin C# array. Another option is to allocate the array on C++ side, return it to C# copy it to a temporary variable on the C# side then delete it on the C++ side.
5.Copy the result back to the array instead of assigning it.
Recommended method:
There are just many ways to do this and some of them are extremely slow. If you want to use Marshal.Copy, you have to allocate the array on the C++ side or else you will run into some undefined behavior.
The fastest and the most efficient way to do this is to allocate the array on the C# side as a global variable. Pass the array and its length to the native side. Also pass a third parameter which C++ can use to tell C# the amount of index that has been updated or written to.
This is much more better than creating new array, copying it to C# variable then destroying it each time the function is called.
This is what you should be using:
C++:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
std::vector<float> results;
for (int i = 0; i < count; i++)
{
//Fill the array
data[i] = results[i];
}
*outValue = results.size();
}
}
You can also use: std::copy ( data, data+count, results.begin() ); instead of loop to copy the data too.
C#:
[DllImport("NativePlugin", CallingConvention = CallingConvention.Cdecl)]
private static extern void fillArrayNative(IntPtr data, int count, out int outValue);
public unsafe void getFillArrayNative(float[] outArray, int count, out int outValue)
{
//Pin Memory
fixed (float* p = outArray)
{
fillArrayNative((IntPtr)p, count, out outValue);
}
}
Usage:
const int arraySize = 44500;
float[] arrayToFill = new float[arraySize];
void Start()
{
int length = arrayToFill.Length;
int filledAmount = 0;
getFillArrayNative(arrayToFill, length, out filledAmount);
//You can then loop through it with with the returned filledAmount
for (int i = 0; i < filledAmount; i++)
{
//Do something with arrayToFill[i]
}
}
This is just an example and it is faster than all other methods I've used before. Avoid doing it the way you are currently doing it with Marshal.Copy. If you still want to do it your way or use Marshal.Copy then here is the appropriate way to do it which requires allocation, copying data and de-allocating memory in each call.
The pointer you return in getSomeFloats is owned by results. Before getSomeFloats returns, the vector destructor for results will free that memory. When the C# code tries to use the pointer, you are accessing unallocated memory, which results in Undefined Behavior. In your case most of the data hasn't been changed yet, but some of it has. Potentially any or all of the data could have been changed (if the memory has been reused), or even a program crash (if the freed memory has been returned to the OS).
Related
I am seeing a pretty bizarre issue while trying to pass an array from C++ to C#. I am using Marshal.Copy (specifically: https://msdn.microsoft.com/en-us/library/a53bd6cz(v=vs.110).aspx).
Problem: float array from C++ to C# is yielding a few NaN's in the resulting array.
(Note: I am working in the context of the Unity game engine)
Code
Example C++ code:
extern "C" bool UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API getSomeFloats(float** points, int* count) {
std::vector<float> results;
std::vector<SOME_TYPE> key_points = <SOME_POINTS>
for (auto iter = key_points.begin(); iter < key_points.end(); iter++) {
results.push_back(static_cast<float>(iter->pt.x));
results.push_back(static_cast<float>(iter->pt.y));
}
*points = results.data();
*count = results.size();
//<Print results to csv here>
return true;
}
Example C# code:
[DllImport("NativePlugin")]
private static extern bool getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
private static float[] getFloatArrayFromNative() {
IntPtr ptrResultItems = IntPtr.Zero;
int resultItemsLength = 0;
bool success = getSomeFloats (ref ptrResultItems, ref resultItemsLength);
float[] resultItems = null;
if (success) {
// Load the results into a managed array.
resultItems = new float[resultItemsLength];
Marshal.Copy (ptrResultItems
, resultItems
, 0
, resultItemsLength);
// <PRINT out resultItems to csv here>
return resultItems;
} else {
Debug.Log ("Something went wrong getting some floats");
return new float[] { -1, -2 };
}
}
Example Ouput:
Take the following example:
C++ output (print_out.csv):
123, 456, 789
C# output (print_out_cs.csv):
123, NaN, 789
I'm completely stumped on this one. I just don't understand why only some (roughly 7/100) floats are returning NaN. Does anyone have any advice/insight that might help?
Thanks!
Found few problems in your code:
1. std::vector<float> results; is declared on the stack. It will be gone by the time the function has returned. Declare it as a pointer
std::vector<float> *results = new std::vector<float>(10);
but make sure to also declare a function that will free it on the C++ side.
2.The function parameter do not match.
Your C++:
getSomeFloats(float** points, int* count, CameraPose* pose)
Your C#:
getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
You either have to remove CameraPose* pose from the C++ side or add IntPtr pose to the C# side.
3. The use of UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API.
You don't need that. This is used when you want to use Unity's built in functions such as GL.IssuePluginEvent. You are not using it in this case.
This should do it:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
}
}
4.C# has a garbage collector that moves variables around in the memory. You must pin C# array if you want to modify it from C++ side. You only need to pin C# array. Another option is to allocate the array on C++ side, return it to C# copy it to a temporary variable on the C# side then delete it on the C++ side.
5.Copy the result back to the array instead of assigning it.
Recommended method:
There are just many ways to do this and some of them are extremely slow. If you want to use Marshal.Copy, you have to allocate the array on the C++ side or else you will run into some undefined behavior.
The fastest and the most efficient way to do this is to allocate the array on the C# side as a global variable. Pass the array and its length to the native side. Also pass a third parameter which C++ can use to tell C# the amount of index that has been updated or written to.
This is much more better than creating new array, copying it to C# variable then destroying it each time the function is called.
This is what you should be using:
C++:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
std::vector<float> results;
for (int i = 0; i < count; i++)
{
//Fill the array
data[i] = results[i];
}
*outValue = results.size();
}
}
You can also use: std::copy ( data, data+count, results.begin() ); instead of loop to copy the data too.
C#:
[DllImport("NativePlugin", CallingConvention = CallingConvention.Cdecl)]
private static extern void fillArrayNative(IntPtr data, int count, out int outValue);
public unsafe void getFillArrayNative(float[] outArray, int count, out int outValue)
{
//Pin Memory
fixed (float* p = outArray)
{
fillArrayNative((IntPtr)p, count, out outValue);
}
}
Usage:
const int arraySize = 44500;
float[] arrayToFill = new float[arraySize];
void Start()
{
int length = arrayToFill.Length;
int filledAmount = 0;
getFillArrayNative(arrayToFill, length, out filledAmount);
//You can then loop through it with with the returned filledAmount
for (int i = 0; i < filledAmount; i++)
{
//Do something with arrayToFill[i]
}
}
This is just an example and it is faster than all other methods I've used before. Avoid doing it the way you are currently doing it with Marshal.Copy. If you still want to do it your way or use Marshal.Copy then here is the appropriate way to do it which requires allocation, copying data and de-allocating memory in each call.
The pointer you return in getSomeFloats is owned by results. Before getSomeFloats returns, the vector destructor for results will free that memory. When the C# code tries to use the pointer, you are accessing unallocated memory, which results in Undefined Behavior. In your case most of the data hasn't been changed yet, but some of it has. Potentially any or all of the data could have been changed (if the memory has been reused), or even a program crash (if the freed memory has been returned to the OS).
I am wanting to update an array that was created inside C#, and then pass a pointer to that array over to C++ and let C++ populate the indexes, to be used back in C#. Right now I am Using a Marshal.Copy() to accomplish this task, but I would like to avoid the potentially unnecessary copy, and call back to c++ to release the array. Is this even possible?
These array are floats and ints, for geometric mesh data.
My current usage (working and not what I want to use)
C#
IntPtr intptr=new IntPtr();
int count = 0;
PopulateArray(ref intptr, ref count);
float[] resultVertices = new float[count];
Marshal.Copy(intptr, resultVertices, 0, count);
C++
extern "C" __declspec(dllexport) bool PopulateArray(float** resultVerts, int* resultVertLength){
*resultVerts = new float[5]{0.123f, 3.141529f, 127.001f, 42.42f, 0};
int myX = 5;
*resultVertLength = myX;
return true;
}
The only safe way to have C++ code update a managed C# array is to pin the array. Otherwise, it's possible for the garbage collector to try to move the array while the native code is running. You can do this with a GCHandle object.
int count = 5;
float[] resultVertices = new float[count];
GCHandle handle = GCHandle.Alloc(resultVertices, GCHandleType.Pinned);
IntPtr address = handle.AddrOfPinnedObject();
PopulateArray(address, count);
handle.Free();
It can also be done with unsafe code, which is somewhat more intuitive to read and remember:
int count = 5;
float[] resultVertices = new float[count];
unsafe
{
fixed(float* ptr = resultVertices)
{
PopulateArray(ptr, count);
}
}
Another alternative is to have C# allocate an unmanaged chunk of memory and pass that to the C++ method. This is better than what you're doing because you are not placing the responsibility of allocation/deallocation in the C++ library code and instead keeping that all in your C#. I know you want to avoid the coy but sometimes doing the copy is more performant than pinning objects, but it depends on how large they are. I recommend you do performance testing to determine which is best for your situation.
int count = 5;
float[] resultVertices = new float[count];
IntPtr unmanagedMemory = Marshal.AllocHGlobal(count * Marshal.SizeOf(typeof(float)));
PopulateArray(unmanagedMemory, count);
Marshal.Copy(unmanagedMemory, resultVertices, 0, count);
In all these scenarios you should set your C++ code to operate like this:
extern "C" __declspec(dllexport) bool PopulateArray(float* resultVerts, int vertLength)
{
resultVerts[0] = 0.123f;
// fill out the rest of them any way you like.
return true;
}
If the array size is variable, then I recommend having a separate C++ method that calculates the size and returns it rather than having the C++ method allocate the memory.
If you are willing to allow C# to allocate the array (probably a safer alternative) then you could do this behavior with standard PInvoke attributes.
Change your C++ declaration to:
extern "C" __declspec(dllexport) bool PopulateArray(float resultVerts[], int resultVertLength)
and your C# declaration to:
[DllImport("Win32Library.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern bool PopulateArray([MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] float[] resultVerts, int resultVertLength);
Your usage from the C# side would then change to:
var resultVertices = new float[5];
PopulateArray(resultVertices, resultVertices.Length);
interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.
The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.
Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}
From safe, managed code in C#, I would like to call a function in a C API that receives an array of pointers (void**).
I have the corresponding managed array of IntPtr objects, but the Marshal methods advertised in the documentation at MSDN do not seem sufficient to provide an IntPtr to an unmanaged block of memory with the correct content.
I had hoped to obtain an IntPtr with 'Marshal.AllocHGlobal' and then assign the correct content using 'Marshal.Copy', but it seems the function has not been overloaded for an array of IntPtr.
Any thoughts on the best way to do this?
Thanks in advance.
The P/Invoke marshaller already does this, you don't have to help. Just declare the function argument as an array:
[DllImport("blah.dll")]
private static extern void SomeFunction(IntPtr[] array);
Just in case: although you don't have to use the unsafe keyword here, there isn't anything safe about it. The C code can easily corrupt the heap when it writes past the end of the block you allocated.
Pass the array as an IntPtr[], IntPtr are by default marshaled as void*. No
need for unsafe.
[DllImport("containingFoo.dll")]
public static extern void foo( IntPtr[] ptr);
...
// some floats
float[] fpa = {7.2F, 2.3F, 3.3F, 4.5F, 6.5F};
// allocate unmanaged for float[] fpa and int (length of array)
IntPtr fptr = Marshal.AllocHGlobal(fpa.Length *
Marshal.SizeOf(typeof(float)));
IntPtr iptr = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(int)));
// set length of array
Marshal.WriteInt32(iptr, fpa.Length);
// copy the array
Marshal.Copy(fpa, 0, fptr, fpa.Length);
// strore both pointers in IntPtr[]
IntPtr[] pptr = {fptr, iptr};
// call foo passing the IntPtr[] to C
foo(pptr);
//C/C++
// note that stdcall is the default calling convention when using
PInvoke!!!!
void __stdcall foo(void** data)
{
float * fa = (float*)*data; // first element points to float array
int *ip = (int*)data + 1; // pointer to next element in void array
int *pp = (int*)*ip; // get pointer to int
for (int i = 0; i < *pp ; i++)
{
printf("\t: %f\n", *fa++);
}
}
I have the following C++ method :
__declspec(dllexport) void __stdcall getDoubles(int *count, double **values);
the method allocates and fills an array of double and sets *count to the size of the array.
The only way i managed to get this to work via pinvoke is :
[System.Runtime.InteropServices.DllImportAttribute("xx.dll")]
public static extern void getDoubles(ref int count, ref System.IntPtr values);
and usage is :
int count = 0;
IntPtr doubles = new IntPtr();
Nappy.getDoubles(ref count, ref doubles);
double[] dvs = new double[count];
for(int i = 0;i < count;++{
dvs[i] = (double)Marshal.PtrToStructure(doubles, typeof(System.Double));
doubles = new IntPtr(doubles.ToInt64()+Marshal.SizeOf(typeof(System.Double)));
}
the values end up in the dvs array.
Is there a better way ti do this not invloving pointer arithmetic in a managed language...
I think you can use
Marshal.Copy( source, destination, 0, size );
You'll need to change the unmanaged method signature so it reads like this:
__declspec(dllexport) void __stdcall getDoubles(SAFEARRAY *array);
Then you should be able to use the following managed version of the function:
[System.Runtime.InteropServices.DllImportAttribute("xx.dll")]
public static extern void getDoubles(
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VT_R8)]
double[] array
);
Of course that you'll also have to rewrite your unmanaged code to work with SAFEARRAYs.
More about this topic can be found at MSDN.
One question though, I recall working with ZLib in C# which is able, without any wrapper, to work with byte[] while the unmanaged counterpart is BYTE*, have you tried working directly with double* / double[] ?