We Keep Coding

How to P/Invoke a function with an unknown struct in C#?

I know this sounds really strange, but I don't know how to even ask this properly. I've been trying to P/Invoke into NVidia's NVML library with limited success: I've managed to call a few of the APIs exported by that library
Now I am trying to call nvmlDeviceGetHandleByIndex_v2 but I've been stuck for a long while on this one. It takes in a nvmlDevice_t pointer, but I've found nothing on what nvmlDevice_t actually is beyond this header definition:
typedef struct nvmlDevice_st* nvmlDevice_t;
The problem is that the header file does not make any other reference to nvmlDevice_st so I don't know how much heap space to allocate for it, if any. I've found this official C++ example that calls that same function like this:
nvmlDevice_t device;
CheckNVMLErrors(nvmlDeviceGetHandleByIndex(device_index, &device));
My main problem is that I'm not familiar enough with C/C++ to understand the implicit mechanics/memory allocation done by the device declaration, and the nvml.h header does not define what nvmlDevice_st actually is.
I tried calling it with a ref int parameter (with an initial 0 value) and apparently it does work but I want to understand why, if possible. For reference, the value of that ref int parameter after the call was 1460391512, in case something can be gleamed off that.

If you look at the source, that is just an internal pointer used by the SDK. The value it points to has no meaning to you. You use it to identify a device you are working with.
Think Handle or HWND in Windows. You call something like FindWindow(), it returns what seems to be a random value back to you. You don't care what that value holds, you just use that value to identify that window when you call GetWindowText() or any other windowing methods.
So, you are on the right track with using ref int, but what you want is a pointer. So you should use out IntPtr to get the value.

Legality of changing P/Invoke signature for convenience

RegSetValueEx has the following P/Invoke signature:
[DllImport("advapi32.dll", SetLastError = true)]
static extern uint RegSetValueEx(
UIntPtr hKey,
[MarshalAs(UnmanagedType.LPStr)]
string lpValueName,
int Reserved,
RegistryValueKind dwType,
IntPtr lpData,
int cbData);
The IntPtr lpData parameter can be a pain to work with. To pass a string value, I need to call one of the Marshall.StringToHGlobal functions and then release it when I'm done. My code can be simpler if I change the type of the lpData parameter to [MarshalAs(UnmanagedType.LPStr)]String lpData. This seems to work and I can imagine that behind the scenes the marshalling code is doing exactly what I would do to get an IntPtr. I haven't been able to find any definitive statements if this is legal or not. Can anyone provide one?
[Yes, I know there is managed code to interface with the registry. In my particular case I can't use it, and even if I could I would still be interested in the answer in general]

Yes, this is a perfectly normal and acceptable technique in pinvoke. Just about any winapi function argument of type BYTE*, PVOID, LPARAM lends itself to a customized [DllImport] declaration. A string, array or struct is the usual equivalent C# parameter type. Particularly useful when you declare SendMessage().
Not where it ends, you can also readily take advantage of the C# language allowing method overloads. In other words, you can write any number of RegSetValueEx() declarations with different types for the lpData argument. They all map to the same native function so nothing else you have to do. A practical example is here.
You just have to make sure that the value you pass after marshalling has the expected byte values. Beware that mistakes tend to be hard to debug, you might still want to marshal manually if the call doesn't pan out. One quirk is certainly notable for this question, you always want to use CharSet = CharSet.Unicode in the declaration, calling the ANSI version (RegSetValueExA) is not useful. Now you can also add ExactSpelling = true.

P/Invoke AccessViolationException

I have an unmanaged function call that is throwing this exception when I try to pass it the path to a file name.
I've read that this is likely caused by the DLL itself but I don't think that is the case since the DLL is used in another application, so the problem is likely in my method calling the function.
The specification:
libvlc_media_new_path (libvlc_instance_t *p_instance, const char *path)
Description:
p_instance the instance
path local filesystem path
And my method:
[DllImport("libvlc", EntryPoint = "libvlc_media_new_path")]
public static extern IntPtr NewMedia(IntPtr instance,
[MarshalAs(UnmanagedType.LPStr)] string path);
I think I'm missing the convention call but what would that likely be? Or would it be something else causing this exception?
EDIT: Based on some comments I did some poking around and found... well, nothing. The struct for the instance is opaque, which means I have no idea in Laymans terms. My guess is that it means you don't need to reconstruct it in the application that is using it?
In a blind guess based on this, I replaced the return value that I had been using with the function responsible for setting the *p_instance value to a long instead of an IntPtr since when it was an IntPtr it was returning 0, and with a long I was seeing a value. Again, what an IntPtr is I don't really know. I was pretty happy to see something not 0 in the instance variable but when I ran it past that, it errored out again.
EDIT: I've expanded the question to here.

Based on the exception you're seeing and the declaration you've provided for the native function,
libvlc_media_new_path (libvlc_instance_t *p_instance, const char *path)
your p/invoke declaration is incorrect. You've mismatched the calling conventions. The default for the .NET p/invoke system is stdcall (to match the Windows API functions), but the default for C and C++ code is cdecl. You have to tell .NET explicitly that your function uses the cdecl calling convention.
So change it to look like this:
[DllImport("libvlc", EntryPoint = "libvlc_media_new_path", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr NewMedia(IntPtr instance,
[MarshalAs(UnmanagedType.LPStr)] string path);
Of course, I'm guessing that you're right about the return value being a pointer. The native function declaration you've shown is missing the return type.
As for your question about the instance parameter, and whether you are correctly using the IntPtr type: The parameter is a pointer to a libvlc_instance_t, so you have two basic ways of making that work using p/invoke. First is to declare the parameter as an IntPtr, which gets it marshalled like a raw pointer value. This is not particularly useful for cases where the pointer needs to be anything other than opaque (i.e. retrieved from one native function, stored, and then passed to another native function). Second is to declare a managed structure that mirrors the native structure, and then write the p/invoke declaration to use this structure so that the marshaller will handle things automatically. This is most useful if you actually need to interact with the values stored in the structure pointed to by the pointer.
In this case, after a Google search, it looks like you're using one of the VLC APIs. Specifically this one. That also tells us what an libvlc_instance_t is: it is an opaque structure that represents a libvlc instance. So declaring a managed structure is not an option here, because the structure is treated as opaque even by the native code. All you really need is the pointer, passed back and forth; a perfect case for the first method I talked about above. So the declaration shown above is your winner.
The only battle now is obtaining a valid pointer to a libvlc instance that you can pass to the function whenever you call it. Chances are good that will come from a prior call to a function like libvlc_new, which is documented as creating and intializing a new libvlc instance. Its return value is exactly what you need here. So unless you've already created a libvlc instance (in which case, use that pointer), you will also need to call this function and store its result.
If the documentation is correct about the required values for the libvlc_new function's parameters, you can declare it very simply:
[DllImport("libvlc", EntryPoint = "libvlc_new", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr NewCore(int argc, IntPtr argv);
And call it thus:
IntPtr pLibVlc = NewCore(0, IntPtr.Zero);
// now pLibVlc should be non-zero
And of course, I know nothing about the VLC APIs, but my general knowledge of API design tells me that you will probably need to call the libvlc_release function with that same pointer to an instance once you're finished with it.

Try without the [MarshalAs(UnmanagedType.LPStr)], usually works for me.

CLS compliant types in P/Invoke helper assembly

Having a separate helper assembly containing only P/Invoke declarations for legacy 3rd party components, I wonder which of these two ways is The Better One™ if the assembly must be marked CLS compliant:
Use Int32 in a public P/Invoke declaration where the unmanaged declaration has unsigned int.
Use UInt32 in an internal P/Invoke declaration where the unmanaged declaration has unsigned int, and wrap it in a public method that takes an Int32 and converts it to UInt32 when calling the internal method.
What are the up- and downsides of these?

The P/Invoke marshaller isn't going to complain when the uint gets too big, you'll just end up with a negative int. The extra layer does allow you to use the checked keyword to generate an OverflowException. Which is fairly desirable.
Whether it is worth the hassle is a secondary question. Lots of APIs, like Win32, use unsigned as a logical constraint. Like the length of the string or the size of a block of memory, it can never be negative. In practice, such a number can never overflow. Because it isn't possible to allocate that much memory. I can't remember running once in an API where it was a slam-dunk that uint should be used. As such, I think you're fine just using a straight pinvoke declaration with ints.

I don't think you'd get correct behavior if you went with option 1. Int32 can only go as high as 2,147,483,647. Whereas the unsigned int goes up to 4,294,967,295. As long as you KNOW you don't need any values above 2 billion, it doesn't really matter. But to be technically correct, the public interface should expose a larger type and perform bounds checking to make sure it fits in an unsigned int and throw an exception if it doesn't. An Int64 will do (9,223,372,036,854,775,807).

IntPtr vs UIntPtr

This should be simple: I see everywhere people use IntPtr, is there any reason I should use UIntPtr instead?

Doesn't seem like there would be a good reason to. From the docs:
Remarks
...
The IntPtr type is CLS-compliant,
while the UIntPtr type is not. Only
the IntPtr type is used in the common
language runtime. The UIntPtr type is
provided mostly to maintain
architectural symmetry with the IntPtr
type.

Well according to Microsoft UIntPtr is provided mostly for architectural symmetry. Another interesting point is that UIntPtr is not CLS compliant.
On the surface it would seem that IntPtr is the preferred option. However, if you know you will be doing pointer arithmetic then UIntPtr may be a better choice since in the off chance there were some obscure problem dealing with negative values.

There is a big difference.. For example, on a 32-bit application, new UIntPtr(0xC01E0001) results in a pointer at 3223191553, but IntPtr(0xC01E0001) results in a System.OverflowException: "Arithmetic operation resulted in an overflow". That is, because 0xC01E0001 is greater than MaxValue for Int32.
So if one still wants such a pointer, a negative integer needs to be used as internal value for IntPtr, like
unchecked((IntPtr)(int)0xC01E0001)
To answer your question: For values greater than Int32.MaxValue it is easier and somehow cleaner to use UIntPtr. It is probably always the best solution, since pointers are always considered unsigned.