I create a SAFEARRAY storing VARIANTs that are BYTEs in C++.
When this structure is marshaled to C#, a weird thing happens.
If I print the content of this structure in C# to a WinForms ListBox, e.g.:
byte data[]
TestSafeArray(out data);
lstOutput.Items.Clear();
foreach (byte x in data)
{
lstOutput.Items.Add(x); // Strange numbers
}
I get some numbers that seem unrelated to the original ones. Moreover, each time I run the C# client for a new test, I get a different set of numbers.
Note that if I inspect the content of that data array with the Visual Studio debugger, I get the correct numbers, as the following screenshot shows:
However, if I CopyTo the marshaled data array to a new one, I get the correct numbers:
byte[] data;
TestSafeArray(out data);
// Copy to a new byte array
byte[] byteData = new byte[data.Length];
data.CopyTo(byteData, 0);
lstOutput.Items.Clear();
foreach (byte x in byteData)
{
lstOutput.Items.Add(x); // ** WORKS! **
}
This is the C++ repro code I use to build the SAFEARRAY (this function is exported from a native DLL):
extern "C" HRESULT __stdcall TestSafeArray(/* [out] */ SAFEARRAY** ppsa)
{
HRESULT hr = S_OK;
try
{
const std::vector<BYTE> v{ 11, 22, 33, 44 };
const int count = static_cast<int>(v.size());
CComSafeArray<VARIANT> sa(count);
for (int i = 0; i < count; i++)
{
CComVariant var(v[i]);
hr = sa.SetAt(i, var);
if (FAILED(hr))
{
return hr;
}
}
*ppsa = sa.Detach();
}
catch (const CAtlException& e)
{
hr = e;
}
return hr;
}
And this is the C# P/Invoke I used:
[DllImport("NativeDll.dll", PreserveSig = false)]
private static extern void TestSafeArray(
[Out, MarshalAs(UnmanagedType.SafeArray,
SafeArraySubType = VarEnum.VT_VARIANT)]
out byte[] result);
Note that if in C++ I create a SAFEARRAY storing BYTEs directly (instead of a SAFEARRAY(VARIANT)), I get the correct values immediately in C#, without the intermediate CopyTo operation.
[Out, MarshalAs(UnmanagedType.SafeArray,
SafeArraySubType = VarEnum.VT_VARIANT)]
out byte[] result);
You fibbed. You told the marshaller that you wanted an array of variants, indeed compatible with what the C++ compiler produced. It will dutifully produce an object[], object is the standard marshaling for a VARIANT. The elements of the array are boxed bytes.
That did not fool the debugger, it ignored the program declaration and looked at the array type and discovered object[], readily visible in your screenshot. So it properly accessed the boxed bytes. And it did not fool Array.CopyTo(), it takes an Array argument so is forced to look at the element type. And properly converted the boxed bytes to bytes, it knows how to do that.
But the C# compiler is fooled badly. It has no idea that it needs to emit an unbox instruction. Not actually sure what goes wrong, you are probably getting the low byte of the object address. Pretty random indeed :)
Fibbing in pinvoke declarations can be very useful. Works just fine in this particular case if the array contains actual objects, like strings, arrays or interface pointers. The probable reason the marshaller doesn't scream bloody murder. Not here, the boxing trips it up bad. You'll have to fix the declaration, use object[].
Related
I am seeing a pretty bizarre issue while trying to pass an array from C++ to C#. I am using Marshal.Copy (specifically: https://msdn.microsoft.com/en-us/library/a53bd6cz(v=vs.110).aspx).
Problem: float array from C++ to C# is yielding a few NaN's in the resulting array.
(Note: I am working in the context of the Unity game engine)
Code
Example C++ code:
extern "C" bool UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API getSomeFloats(float** points, int* count) {
std::vector<float> results;
std::vector<SOME_TYPE> key_points = <SOME_POINTS>
for (auto iter = key_points.begin(); iter < key_points.end(); iter++) {
results.push_back(static_cast<float>(iter->pt.x));
results.push_back(static_cast<float>(iter->pt.y));
}
*points = results.data();
*count = results.size();
//<Print results to csv here>
return true;
}
Example C# code:
[DllImport("NativePlugin")]
private static extern bool getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
private static float[] getFloatArrayFromNative() {
IntPtr ptrResultItems = IntPtr.Zero;
int resultItemsLength = 0;
bool success = getSomeFloats (ref ptrResultItems, ref resultItemsLength);
float[] resultItems = null;
if (success) {
// Load the results into a managed array.
resultItems = new float[resultItemsLength];
Marshal.Copy (ptrResultItems
, resultItems
, 0
, resultItemsLength);
// <PRINT out resultItems to csv here>
return resultItems;
} else {
Debug.Log ("Something went wrong getting some floats");
return new float[] { -1, -2 };
}
}
Example Ouput:
Take the following example:
C++ output (print_out.csv):
123, 456, 789
C# output (print_out_cs.csv):
123, NaN, 789
I'm completely stumped on this one. I just don't understand why only some (roughly 7/100) floats are returning NaN. Does anyone have any advice/insight that might help?
Thanks!
Found few problems in your code:
1. std::vector<float> results; is declared on the stack. It will be gone by the time the function has returned. Declare it as a pointer
std::vector<float> *results = new std::vector<float>(10);
but make sure to also declare a function that will free it on the C++ side.
2.The function parameter do not match.
Your C++:
getSomeFloats(float** points, int* count, CameraPose* pose)
Your C#:
getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
You either have to remove CameraPose* pose from the C++ side or add IntPtr pose to the C# side.
3. The use of UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API.
You don't need that. This is used when you want to use Unity's built in functions such as GL.IssuePluginEvent. You are not using it in this case.
This should do it:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
}
}
4.C# has a garbage collector that moves variables around in the memory. You must pin C# array if you want to modify it from C++ side. You only need to pin C# array. Another option is to allocate the array on C++ side, return it to C# copy it to a temporary variable on the C# side then delete it on the C++ side.
5.Copy the result back to the array instead of assigning it.
Recommended method:
There are just many ways to do this and some of them are extremely slow. If you want to use Marshal.Copy, you have to allocate the array on the C++ side or else you will run into some undefined behavior.
The fastest and the most efficient way to do this is to allocate the array on the C# side as a global variable. Pass the array and its length to the native side. Also pass a third parameter which C++ can use to tell C# the amount of index that has been updated or written to.
This is much more better than creating new array, copying it to C# variable then destroying it each time the function is called.
This is what you should be using:
C++:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
std::vector<float> results;
for (int i = 0; i < count; i++)
{
//Fill the array
data[i] = results[i];
}
*outValue = results.size();
}
}
You can also use: std::copy ( data, data+count, results.begin() ); instead of loop to copy the data too.
C#:
[DllImport("NativePlugin", CallingConvention = CallingConvention.Cdecl)]
private static extern void fillArrayNative(IntPtr data, int count, out int outValue);
public unsafe void getFillArrayNative(float[] outArray, int count, out int outValue)
{
//Pin Memory
fixed (float* p = outArray)
{
fillArrayNative((IntPtr)p, count, out outValue);
}
}
Usage:
const int arraySize = 44500;
float[] arrayToFill = new float[arraySize];
void Start()
{
int length = arrayToFill.Length;
int filledAmount = 0;
getFillArrayNative(arrayToFill, length, out filledAmount);
//You can then loop through it with with the returned filledAmount
for (int i = 0; i < filledAmount; i++)
{
//Do something with arrayToFill[i]
}
}
This is just an example and it is faster than all other methods I've used before. Avoid doing it the way you are currently doing it with Marshal.Copy. If you still want to do it your way or use Marshal.Copy then here is the appropriate way to do it which requires allocation, copying data and de-allocating memory in each call.
The pointer you return in getSomeFloats is owned by results. Before getSomeFloats returns, the vector destructor for results will free that memory. When the C# code tries to use the pointer, you are accessing unallocated memory, which results in Undefined Behavior. In your case most of the data hasn't been changed yet, but some of it has. Potentially any or all of the data could have been changed (if the memory has been reused), or even a program crash (if the freed memory has been returned to the OS).
I am seeing a pretty bizarre issue while trying to pass an array from C++ to C#. I am using Marshal.Copy (specifically: https://msdn.microsoft.com/en-us/library/a53bd6cz(v=vs.110).aspx).
Problem: float array from C++ to C# is yielding a few NaN's in the resulting array.
(Note: I am working in the context of the Unity game engine)
Code
Example C++ code:
extern "C" bool UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API getSomeFloats(float** points, int* count) {
std::vector<float> results;
std::vector<SOME_TYPE> key_points = <SOME_POINTS>
for (auto iter = key_points.begin(); iter < key_points.end(); iter++) {
results.push_back(static_cast<float>(iter->pt.x));
results.push_back(static_cast<float>(iter->pt.y));
}
*points = results.data();
*count = results.size();
//<Print results to csv here>
return true;
}
Example C# code:
[DllImport("NativePlugin")]
private static extern bool getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
private static float[] getFloatArrayFromNative() {
IntPtr ptrResultItems = IntPtr.Zero;
int resultItemsLength = 0;
bool success = getSomeFloats (ref ptrResultItems, ref resultItemsLength);
float[] resultItems = null;
if (success) {
// Load the results into a managed array.
resultItems = new float[resultItemsLength];
Marshal.Copy (ptrResultItems
, resultItems
, 0
, resultItemsLength);
// <PRINT out resultItems to csv here>
return resultItems;
} else {
Debug.Log ("Something went wrong getting some floats");
return new float[] { -1, -2 };
}
}
Example Ouput:
Take the following example:
C++ output (print_out.csv):
123, 456, 789
C# output (print_out_cs.csv):
123, NaN, 789
I'm completely stumped on this one. I just don't understand why only some (roughly 7/100) floats are returning NaN. Does anyone have any advice/insight that might help?
Thanks!
Found few problems in your code:
1. std::vector<float> results; is declared on the stack. It will be gone by the time the function has returned. Declare it as a pointer
std::vector<float> *results = new std::vector<float>(10);
but make sure to also declare a function that will free it on the C++ side.
2.The function parameter do not match.
Your C++:
getSomeFloats(float** points, int* count, CameraPose* pose)
Your C#:
getSomeFloats (ref IntPtr ptrResultItems, ref int resultItemsLength);
You either have to remove CameraPose* pose from the C++ side or add IntPtr pose to the C# side.
3. The use of UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API.
You don't need that. This is used when you want to use Unity's built in functions such as GL.IssuePluginEvent. You are not using it in this case.
This should do it:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
}
}
4.C# has a garbage collector that moves variables around in the memory. You must pin C# array if you want to modify it from C++ side. You only need to pin C# array. Another option is to allocate the array on C++ side, return it to C# copy it to a temporary variable on the C# side then delete it on the C++ side.
5.Copy the result back to the array instead of assigning it.
Recommended method:
There are just many ways to do this and some of them are extremely slow. If you want to use Marshal.Copy, you have to allocate the array on the C++ side or else you will run into some undefined behavior.
The fastest and the most efficient way to do this is to allocate the array on the C# side as a global variable. Pass the array and its length to the native side. Also pass a third parameter which C++ can use to tell C# the amount of index that has been updated or written to.
This is much more better than creating new array, copying it to C# variable then destroying it each time the function is called.
This is what you should be using:
C++:
#define DLLExport __declspec(dllexport)
extern "C"
{
DLLExport void fillArrayNative(float* data, int count, int* outValue)
{
std::vector<float> results;
for (int i = 0; i < count; i++)
{
//Fill the array
data[i] = results[i];
}
*outValue = results.size();
}
}
You can also use: std::copy ( data, data+count, results.begin() ); instead of loop to copy the data too.
C#:
[DllImport("NativePlugin", CallingConvention = CallingConvention.Cdecl)]
private static extern void fillArrayNative(IntPtr data, int count, out int outValue);
public unsafe void getFillArrayNative(float[] outArray, int count, out int outValue)
{
//Pin Memory
fixed (float* p = outArray)
{
fillArrayNative((IntPtr)p, count, out outValue);
}
}
Usage:
const int arraySize = 44500;
float[] arrayToFill = new float[arraySize];
void Start()
{
int length = arrayToFill.Length;
int filledAmount = 0;
getFillArrayNative(arrayToFill, length, out filledAmount);
//You can then loop through it with with the returned filledAmount
for (int i = 0; i < filledAmount; i++)
{
//Do something with arrayToFill[i]
}
}
This is just an example and it is faster than all other methods I've used before. Avoid doing it the way you are currently doing it with Marshal.Copy. If you still want to do it your way or use Marshal.Copy then here is the appropriate way to do it which requires allocation, copying data and de-allocating memory in each call.
The pointer you return in getSomeFloats is owned by results. Before getSomeFloats returns, the vector destructor for results will free that memory. When the C# code tries to use the pointer, you are accessing unallocated memory, which results in Undefined Behavior. In your case most of the data hasn't been changed yet, but some of it has. Potentially any or all of the data could have been changed (if the memory has been reused), or even a program crash (if the freed memory has been returned to the OS).
interoping nim dll from c# i could call and execute the code below
if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly
but i could not transfer the data as it is, i tried everything but i could not accomplish the task
proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
PackArrINOUT.newSeq(parSze)
var dummyStr = "abcdefghij"
for i, curDataPack in PackArrINOUT.mpairs:
dummyStr[9] = char(i + int8'0')
curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)
type
DataPackArr = seq[DataPack]
DataPack = object
buffer: string
intVal: uint32
when i do same in c/c++ the type i am using is either an IntPtr or char*
that is happy to contain returned buffer member
EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
unsigned int dumln, Index;dataPack* CurDp = {NULL};
char dummy[STRMAX];
*DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
CurDp = *DpArr;
strncpy(dummy, "abcdefgHij", STRMAX);
dumln = sizeof(dummy);
for ( Index = 0; Index < size; Index++,CurDp++)
{
CurDp->IVal = Index;
dummy[dumln-1] = '0' + Index % (126 - '0');
CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
strcpy(CurDp->Sval, dummy);
}
}
c# signature for c code above
[DllImport(#"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);
C# Struct
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
public IntPtr StrVal;
}
}
finally calling the function like so:
public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
{
DataPackg.TestC* PackUArrOut;
List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
c_returnDataPack((uint)ArrL, &PackUArrOut);
DataPackg.TestC* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
}
//Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
return RtLstPackU;
}
how could i produce similar c code as above from Nim ?
it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not
Edit:
this is what i tried to make things simple
struct array of int members
Update:
it seem that the problem is to do with my settings of nim in my windows OS.
i will be updating as soon as i discover what exactly is wrong.
The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:
NI length; # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated
Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.
With this in mind, you have several options:
1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here:
https://gist.github.com/zah/fe8f5956684abee6bec9
2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.
EDIT:
Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.
Try to change your struct to:
public unsafe static class DataPackg
{
[StructLayout(LayoutKind.Sequential)]
public struct TestC
{
public uint Id;
[MarshalAs(UnmanagedType.LPStr)]
public String StrVal;
}
}
I receive IntPtr value from C++ library method and i need to get byte[] array from this received IntPtr. When I try do this:
byte[] myArray = new byte[Marshal.SizeOf(myReceivedIntPtr)];
Marshal.Copy(myReceivedIntPtr, myArray, 0, Marshal.SizeOf(myReceivedIntPtr));
I receive an exception: AccessViolationException.
What I missed?
EDIT:
Here is Method name and params in header of C++ library:
int MSR_DecodeTrack(char *AscBuff, unsigned char *BinBuff, unsigned char bpc, unsigned char parity, unsigned char ss, unsigned char es);
And I call it from C# this way:
UInt32 result;
IntPtr myReceivedIntPtr;
IntPtr BinBuff;
byte BPC1 = 7, Parity1 = 1, SS1 = 0x05, ES1 = 0x1F;
result = MSR_DecodeTrack(ref myReceivedIntPtr, ref BinBuff, BPC1, Parity1, SS1, ES1);
And here is else code:
byte[] myArray = new byte[Marshal.SizeOf(myReceivedIntPtr)];
Marshal.Copy(myReceivedIntPtr, myArray, 0, Marshal.SizeOf(myReceivedIntPtr));
I'm pretty sure Marshal.SizeOf(myReceivedIntPtr) will return the size of the IntPtr object, not the size of the unmanaged array. You will need to get the size of the unmanaged array from the c++ library.
I haven't had to do this in a while so I'm a bit rusty but from looking at the c++ headers I think it is expecting a pointer to a buffer. You are giving it a pointer to a pointer (intptr is a pointer and passing it by ref means you are passing the c++ code a pointer to that pointer). Firstly, don't use ref. Secondly, I suspect you are supposed to be passing it an already sized buffer - is there any documentation describing what you are supposed to pass to the function? What is the return code from the function? You are checking it for success aren't you? ;)
The method named MSR_DecodeTrack should also return the length of each array it allocates.
As far as I know, when you allocate an array in C/C++ environment, you get a pointer to the zero element of that array, nothing more. For that reason, you should also somehow return the length of the allocated array back to the caller. It can be done by an additional "ref" parameter.
HTH.
I have an unmanaged C++ function which is calling a managed C# method in a DLL. The purpose of the C# method is to take an array of bytes (allocated by the C++ caller), populate the array, and return it. I can get the array INTO the C# method, but the populated data are lost when they get back to the C++ function. Right now, this is my test code to debug the process:
C# DLL Method:
// Take an array of bytes and modify it
public ushort GetBytesFromBlaster([MarshalAs(UnmanagedType.LPArray)] byte[] dataBytes)
{
dataBytes[0] = (byte)'a';
dataBytes[1] = (byte)'b';
dataBytes[2] = (byte)'c';
return 3;
}
C++ function which calls the DLL:
// bytes[] has been already allocated by its caller
short int SimGetBytesP2P(unsigned char bytes[])
{
unsigned short int numBytes = 0;
bytes[0] = 'x';
bytes[1] = 'y';
bytes[2] = 'z';
// bytes[] are {'x', 'y', 'z'} here
guiPtr->GetBytesFromBlaster(bytes, &numBytes);
// bytes[] SHOULD be {'a', 'b', 'c'} here, but they are still {'x', 'y', 'z'}
return(numBytes);
}
I believe it has something to do with C# turning the C++ pointer into a new managed array, but modifying the original one. I have tried several variations using the "ref" modifyer, etc., but no luck. Also, these data are NOT null-terminated strings; the date bytes are raw 1-byte values, not null-terminated.
Can anyone please shed some light on this? Thanks!
Stuart
You could do the marshaling yourself. Have the C# function accept a parameter by value of type IntPtr. Also a second parameter indicating array length. No special marshaling attributes are needed or wanted.
Then, use Marshal.Copy and copy the array from the unmanaged pointer to a managed byte[] array that you allocated. Do your thing, and then when you're done, use Marshal.Copy to copy it back to the C++ unmanaged array.
These particular overloads should get you started:
http://msdn.microsoft.com/en-us/library/ms146625.aspx
http://msdn.microsoft.com/en-us/library/ms146631.aspx
For example:
public ushort GetBytesFromBlaster(IntPtr dataBytes, int arraySize)
{
byte[] managed = new byte[arraySize];
Marshal.Copy(dataBytes, managed, 0, arraySize);
managed[0] = (byte)'a';
managed[1] = (byte)'b';
managed[2] = (byte)'c';
Marshal.Copy(managed, 0, dataBytes, arraySize);
return 3;
}
Alternatively you could implement a custom marshaller as described in http://msdn.microsoft.com/en-us/library/w22x2hw6.aspx if the default one isn't doing what you need it to. But that looks like more work.
I believe that you just need to add a SizeConst attribute:
public ushort GetBytesFromBlaster(
[MarshalAs(UnmanagedType.LPArray, SizeConst=3)]
byte[] dataBytes
)
and the default marshaller should do the rest for you.