We Keep Coding

How to P/Invoke a function with an unknown struct in C#?

I know this sounds really strange, but I don't know how to even ask this properly. I've been trying to P/Invoke into NVidia's NVML library with limited success: I've managed to call a few of the APIs exported by that library
Now I am trying to call nvmlDeviceGetHandleByIndex_v2 but I've been stuck for a long while on this one. It takes in a nvmlDevice_t pointer, but I've found nothing on what nvmlDevice_t actually is beyond this header definition:
typedef struct nvmlDevice_st* nvmlDevice_t;
The problem is that the header file does not make any other reference to nvmlDevice_st so I don't know how much heap space to allocate for it, if any. I've found this official C++ example that calls that same function like this:
nvmlDevice_t device;
CheckNVMLErrors(nvmlDeviceGetHandleByIndex(device_index, &device));
My main problem is that I'm not familiar enough with C/C++ to understand the implicit mechanics/memory allocation done by the device declaration, and the nvml.h header does not define what nvmlDevice_st actually is.
I tried calling it with a ref int parameter (with an initial 0 value) and apparently it does work but I want to understand why, if possible. For reference, the value of that ref int parameter after the call was 1460391512, in case something can be gleamed off that.

If you look at the source, that is just an internal pointer used by the SDK. The value it points to has no meaning to you. You use it to identify a device you are working with.
Think Handle or HWND in Windows. You call something like FindWindow(), it returns what seems to be a random value back to you. You don't care what that value holds, you just use that value to identify that window when you call GetWindowText() or any other windowing methods.
So, you are on the right track with using ref int, but what you want is a pointer. So you should use out IntPtr to get the value.

What are differences between fixed buffers, and using the `MarshalAs` attr, with `UnmanagedType.ByValArray`?

Well, there are some obvious differences:
fixed buffer:
they must be declared in unsafe block (implying that the whole project must be compiled with -unsafe switch).
for using the fixed buffer, the containing object must be fixed,
using MarshalAs attribute:
although the size is given to the marshaller, there is no guaranty that the actual array would have sufficient element count, nor that it's not null.
the array could be used simply and intuitively as any other array.
But what I cannot find answer for, is why fixed buffer are needed in the first place?
When one must use them?
Why one would want to use it, assuming one can validate the size of regular managed array?
I can think of performance constraints, that might make one choose fixed buffer over regular arrays...
Is that all?
Thanks in advance.

Yes, efficiency is certainly the primary reason. When you apply UnmanagedType.ByValArray then the struct must always be marshaled. In other words, the CLR is forced to create a new copy of the struct and initialize it with the values from the managed struct since the unmanaged layout of the struct is different. That can be avoided when you use a fixed buffer, provided the other members of the struct are blittable as well. In which case the CLR can simply pass a pointer to the struct. Much faster of course.
There are a few interop scenarios where you must use a fixed size buffer. Typically when the array member is misaligned, that violates the atomicity guarantee of the .NET memory model. Or you declare a union (fields overlap each other) and the CLR objects against overlapping a field of a reference type with a field of a value type. That's incompatible with the garbage collector, it cannot reliably detect the object pointer. You'll get a TypeLoadException at runtime when that's the case.
Both scenarios are fundamentally unverifiable. It is always unsafe if the native code writes back to the struct, memory corruption occurs when it writes past the end of the array. Extremely hard to diagnose. The need to explicitly use the unsafe keyword when you use a fixed size buffer only applies to the lack of index checking when you access the fixed size buffer in your C# code.

Non-blittable error on a blittable type

I have this struct and this code:
[StructLayout(LayoutKind.Sequential, Pack = 8)]
private class xvid_image_t
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public int[] stride;
// [MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
// public IntPtr[] plane;
}
public int decore()
{
xvid_image_t myStruct = new xvid_image_t();
myStruct.stride = new int[4]; // can be commented out - same result
GCHandle.Alloc(myStruct, GCHandleType.Pinned);
// ...
}
When I try to run it I get an ArgumentException saying:
Object contains non-primitive or non-blittable data
After reading this MSDN page saying
The following complex types are also blittable types:
One-dimensional arrays of blittable types, such as an array of integers. However, a type that contains a variable array of blittable types is not itself blittable.
Formatted value types that contain only blittable types (and classes if they are marshaled as formatted types). For more information about formatted value types, see Default Marshaling for Value Types.
I don't understand what I am doing wrong.
I don't just want to use Marshal, but to understand this too.
So what I actually want is to know:
Why?
How can I resolve this?
Will the solution you provide also work with the commented line in the struct?
I am using .Net 4.5 but a solution for .Net 2.0 is also needed.

Object contains non-primitive or non-blittable data
That's the exception message you get. You are focusing on the "non-blittable" part of the message, but that's not the problem. It is the "non-primitive" part that's the issue. An array is a non-primitive data type.
The CLR is trying to keep you out of trouble here. You could pin the object but then you still have a problem, the array won't be pinned. An object isn't truly pinned when it has fields that need to be pinned as well.
And you have a bigger problem with the UnmanagedType.ByValArray, that requires a structural conversion. In other words, the layout that you need is completely different from the layout of the managed class object. Only the pinvoke marshaller can make this conversion.
You can get what you want without using the pinvoke marshaller by using fixed size buffers, using the fixed keyword. This requires using the unsafe keyword. Make it look like this:
[StructLayout(LayoutKind.Sequential)]
unsafe private struct xvid_image_t {
public fixed int stride[4];
}
Note that you have to change the declaration to a struct type. It is now a value type, you no longer need to use GCHandle to pin the value when you make it a local variable. Do make sure that whatever unmanaged code takes the structure value, usually by reference, does not store a pointer to the struct. That's going to blow up badly and utterly undiagnosably. The unsafe keyword is appropriate here. If it does store the pointer then you really do have to byte the bullet and use Marshal.AllocHGlobal() and Marshal.StructureToPtr() to ensure the pointer stays valid while the unmanaged code is using it.

An annoying limitation of .NET is that the only array-ish things it recognizes are a stand-alone System.Array object and a System.String, both of which are reference types. It's possible for code written in C# to use a fixed array (as noted by Hans Passant), but such a type is not recognized by .NET itself, and code which uses fixed arrays is not verifiable. Additionally, fixed arrays are limited to holding primitives, and cannot be accessed by other languages such as vb.net.
Two alternatives to using a fixed array are to
replace the fixed array with some combination of fields which together total the proper size (using N variables in most cases, but perhaps replacing e.g. a char[4] with a UInt32, or a char[8] with a UInt64). If the array is not too large, one might define (either via cut/paste or Reflection) a set of static methods which take a struct by ref and read/write the proper element, and then create an array of delegates to call such methods.
replace the entire structure with an array, and then pass the first element of that array as a ref parameter. This may be even more "dangerous" than using a fixed array within a structure, but is the only way I know of in vb.net to get "pass-by-ref" semantics with a structure that contains something that really needs to be accessed as an array.
While I can understand that value-type arrays might have been considered "confusing" (especially if they were auto-boxed) there are places where they would have been the semantically-correct approach for array storage, both from the standpoint of allowing pass-by-ref semantics for COM interop and also from the standpoint of methods that are supposed to return a small number of values. For example, in System.Drawing2d, there is a method which returns the current graphics transform as a float[6]; other than by experimentation, there would be no clear way of knowing whether changes to that array after it is returned would affect, might affect, or are guaranteed not to affect anything else. If the method returned a value-type array, it would be clear that changes to the returned array cannot affect anything else. Nonetheless, whether or not value-type arrays would have been a useful part of the Framework, the fact remains that whether for good or bad reasons no such thing exists.

I took the below answer from this link (here)
SItuLongEmailMsg msg = newSItuLongEmailMsg();
// set members
msg.text = new byte[2048];
// assign to msg.text
int msgSize = Marshal.SizeOf(msg);
IntPtr ptr = Marshal.AllocHGlobal(msgSize);
Marshal.StructureToPtr(msg, ptr, true);
byte[] dataOut = new byte[msgSize];
Marshal.Copy(ptr, dataOut, 0, msgSize);

Is it safe to keep C++ pointers in C#?

I'm currently working on some C#/C++ code which makes use of invoke. In the C++ side there is a std::vector full of pointers each identified by index from the C# code, for example a function declaration would look like this:
void SetName(char* name, int idx)
But now I'm thinking, since I'm working with pointers couldn't I sent to C# the pointer address itself then in code I could do something like this:
void SetName(char*name, int ptr)
{
((TypeName*)ptr)->name = name;
}
Obviously that's a quick version of what I'm getting at (and probably won't compile).
Would the pointer address be guaranteed to stay constant in C++ such that I can safely store its address in C# or would this be too unstable or dangerous for some reason?

In C#, you don't need to use a pointer here, you can just use a plain C# string.
[DllImport(...)]
extern void SetName(string name, int id);
This works because the default behavior of strings in p/invoke is to use MarshalAs(UnmanagedType.LPStr), which converts to a C-style char*. You can mark each argument in the C# declaration explicitly if it requires some other way of being marshalled, eg, [MarshalAs(UnmanagedType.LPWStr)], for an arg that uses a 2-byte per character string.
The only reason to use pointers is if you need to retain access to the data pointed to after you've called the function. Even then, you can use out parameters most of the time.
You can p/invoke basically anything without requiring pointers at all (and thus without requiring unsafe code, which requires privileged execution in some environments).

Yes, no problem. Native memory allocations never move so storing the pointer in an IntPtr on the C# side is fine. You need some kind of pinvoked function that returns this pointer, then
[DllImport("something.dll", CharSet = CharSet.Ansi)]
void SetName(IntPtr vector, string name, int index);
Which intentionally lies about this C++ function:
void SetName(std::vector<std::string>* vect, const char* name, int index) {
std::string copy = name;
(*vect)[index] = copy;
}
Note the usage of new in the C++ code, you have to copy the string. The passed name argument points to a buffer allocated by the pinvoke marshaller and is only valid for the duration of the function body. Your original code cannot work. If you intend to return pointers to vector<> elements then be very careful. A vector re-allocates its internal array when you add elements. Such a returned pointer will then become invalid and you'll corrupt the heap when you use it later. The exact same thing happens with a C# List<> but without the risk of dangling pointers.

I think it's stable till you command C++ code and perfectly aware what he does, and other developers that work on the same code know about that danger too.
So by my opinion, it's not very secure way of architecture, and I would avoid it as much as I can.
Regards.

The C# GC moves things, but the C++ heap does not move anything- a pointer to an allocated object is guaranteed to remain valid until you delete it. The best architecture for this situation is just to send the pointer to C# as an IntPtr and then take it back in C++.
It's certainly a vastly, incredibly better idea than the incredibly BAD, HORRIFIC integer cast you've got going there.

Are ref and out in C# the same a pointers in C++?

I just made a Swap routine in C# like this:
static void Swap(ref int x, ref int y)
{
int temp = x;
x = y;
y = temp;
}
It does the same thing that this C++ code does:
void swap(int *d1, int *d2)
{
int temp=*d1;
*d1=*d2;
*d2=temp;
}
So are the ref and out keywords like pointers for C# without using unsafe code?

They're more limited. You can say ++ on a pointer, but not on a ref or out.
EDIT Some confusion in the comments, so to be absolutely clear: the point here is to compare with the capabilities of pointers. You can't perform the same operation as ptr++ on a ref/out, i.e. make it address an adjacent location in memory. It's true (but irrelevant here) that you can perform the equivalent of (*ptr)++, but that would be to compare it with the capabilities of values, not pointers.
It's a safe bet that they are internally just pointers, because the stack doesn't get moved and C# is carefully organised so that ref and out always refer to an active region of the stack.
EDIT To be absolutely clear again (if it wasn't already clear from the example below), the point here is not that ref/out can only point to the stack. It's that when it points to the stack, it is guaranteed by the language rules not to become a dangling pointer. This guarantee is necessary (and relevant/interesting here) because the stack just discards information in accordance with method call exits, with no checks to ensure that any referrers still exist.
Conversely when ref/out refers to objects in the GC heap it's no surprise that those objects are able to be kept alive as long as necessary: the GC heap is designed precisely for the purpose of retaining objects for any length of time required by their referrers, and provides pinning (see example below) to support situations where the object must not be moved by GC compacting.
If you ever play with interop in unsafe code, you will find that ref is very closely related to pointers. For example, if a COM interface is declared like this:
HRESULT Write(BYTE *pBuffer, UINT size);
The interop assembly will turn it into this:
void Write(ref byte pBuffer, uint size);
And you can do this to call it (I believe the COM interop stuff takes care of pinning the array):
byte[] b = new byte[1000];
obj.Write(ref b[0], b.Length);
In other words, ref to the first byte gets you access to all of it; it's apparently a pointer to the first byte.

Reference parameters in C# can be used to replace one use of pointers, yes. But not all.
Another common use for pointers is as a means for iterating over an array. Out/ref parameters can not do that, so no, they are not "the same as pointers".

ref and out are only used with function arguments to signify that the argument is to be passed by reference instead of value. In this sense, yes, they are somewhat like pointers in C++ (more like references actually). Read more about it in this article.

The nice thing about using out is that you're guaranteed that the item will be assigned a value -- you will get a compile error if not.

Actually, I'd compare them to C++ references rather than pointers. Pointers, in C++ and C, are a more general concept, and references will do what you want.
All of these are undoubtedly pointers under the covers, of course.

While comparisons are in the eye of the beholder...I say no. 'ref' changes the calling convention but not the type of the parameters. In your C++ example, d1 and d2 are of type int*. In C# they are still Int32's, they just happen to be passed by reference instead of by value.
By the way, your C++ code doesn't really swap its inputs in the traditional sense. Generalizing it like so:
template<typename T>
void swap(T *d1, T *d2)
{
T temp = *d1;
*d1 = *d2;
*d2 = temp;
}
...won't work unless all types T have copy constructors, and even then will be much more inefficient than swapping pointers.

The short answer is Yes (similar functionality, but not exactly the same mechanism).
As a side note, if you use FxCop to analyse your code, using out and ref will result in a "Microsoft.Design" error of "CA1045:DoNotPassTypesByReference."