We Keep Coding

How to marshal arbitrary structures and unions between C# and C?

I have a C program compiled to a DLL which I load into C# (using kernel32.dll LoadLibrary and FreeLibrary). Using GetProcAddress I can get a pointer to some data in memory, and I want to convert this data to an equivalent structure in C#.
My first idea was to create structs in C# that mapped directly to the C structs (using the StructLayout and MarshalAs attributes), and then just use the Marshal class (in System.Runtime.InteropServices). This works well for "simpler" structs, but does not work for multi-dimensional arrays and unions of arrays.
The data structures in C looks like the following (for example):
struct MyStruct
{
union MyUnion my_data[16][16];
}
union MyUnion
{
uint32_t ints[2];
uint8_t bytes[8];
}
I.e, it contains arbitrarily nested structs and unions, and multidimensional arrays of these.
Is there a built in way in C# to handle marshalling of these types of data structures?
Edit:
My requirements are:
The C# unions must work the same way as the C ones, i.e writing to the bytes field should also change the ints field (afaik, using explicit struct layout you can only make unions of built in value types in C#). (This is more a C# issue than a marshalling issue)
The C# structures must be marshallable to and from C/C++
As Selvin noted in the comments it's possible to flatten multi-dimensional arrays into a single dimension, and then access by [x + y*16]. However I prefer if the original dimensions were preserved, and the method of accessing is the same as in C.
My answer below does what I want, but since marshalling between C# and C++ is so common it feels like there should be a simpler way to achieve that result.

My current solution is to create a wrapper class around the raw data byte[], with accessors that imitate the structure of the underlying data.
For example, for a simple structure in C:
struct MySimpleStruct
{
uint32_t my_ints[16];
}
I create a wrapper:
public class MySimpleStruct
{
private byte[] _underlying;
public UintAccessor[] MyInts { get; }
public MySimpleStruct(byte[] data)
{
_underlying = data;
MyInts = new UintAccessor[16];
// for-loop to initialize each accessor with the right index
}
}
public class UintAccessor
{
public uint Value
{
get => BitConverter.ToUint32(_underlying, _idx);
set => BitConverter.GetBytes(value).CopyTo(_underlying, _idx);
}
public UintAccessor(byte[] data, int index) { /* ... */ }
}
and the wrapper can be used like so
var myStruct = new MySimpleStruct(theData);
myStruct.MyInts[5].Value = 123;
These wrapper classes can become quite complex, however we automatically generate the C structures and we will generate these C# wrappers at the same time, so the complexity is only an issue when generating the code.

What's the size and alignment of C# fixed bool array in struct?

When doing P/Invoke, it is important to make the data layout match.
We can control the layout of struct by using some attribute.
For example:
struct MyStruct
{
public bool f;
}
gives a size of 4. While we can tell compiler to make it a 1 byte bool to match C++ type of bool:
struct MyStruct
{
[MarshalAs(UnmanagedType.I1)]
public bool f;
}
gives a size of 1.
These make sense. But when I test fixed bool array, I was confused.
unsafe struct MyStruct
{
public fixed bool fs[1];
}
gives a size of 4 bytes. and
unsafe struct MyStruct
{
public fixed bool fs[4];
}
still gives a size of 4 bytes. but
unsafe struct MyStruct
{
public fixed bool fs[5];
}
gives a size of 8.
It looks like in fixed bool array, the size of bool element is still 1 byte, but the alignment is 4 bytes. This doesn't match C++ bool array, which is 1 byte size and alignment.
Can someone explain me on this?
Update : I finally find out, the reason is, bool type in a struct, then that struct will NEVER be blittable! So don't expect a struct which has bool type inside to be same layout as in C.
Regards,
Xiang.

A bool is rather special, it goes back to Dennis Ritchie's decision to not give the C language a bool type. That caused plenty of mayhem, language and operating system designers added it themselves and made incompatible choices.
It was added to the Winapi as the BOOL typedef. That's the default marshaling if you don't force another type. Typedef-ed as int to keep it compatible with C, takes 4 bytes as you found out. And aligns to 4, as you found out, like any int does.
It was added to C++. Without a size specification, most C++ compiler implementations chose a single byte for storage. Most notably the Microsoft C++ compiler did, the most likely implementation you'll interop with.
It was added to COM Automation as VARIANT_BOOL. Originally targeted as the new extension model for Visual Basic to get rid of the VBX restrictions, it became wildly popular and just about any language runtime on Windows now supports it. VB back then was heavily affected by 16-bit operating system sensibilities, a VARIANT_BOOL takes 2 bytes.
All three native runtime environments are likely targets for interop in a C# program. Clearly the CLR designers had a very difficult choice to make, having to pick between 1, 2 and 4 bytes. There is no way to win, while the CLR does have a shot at guessing at COM interop, it cannot know whether you try to interop with a C-based api or a C++ program. So they made the only logical choice: none of them.
A struct or class type that contains a bool is never blittable. Not even when you apply [MarshalAs(UnmanagedType.U1)], the one that would make it compatible with the CLR type. Not so sure that was a good decision, it however was the one they made so we'll have to deal with it.
Getting a blittable struct is highly desirable, it avoids copying. It allows native code to directly access the managed heap and stack. Pretty dangerous and many a broken pinvoke declaration has corrupted the GC heap without the usual benefit of the unsafe keyword alert. But impossible to beat for speed.
You get a blittable struct by not using bool. Use byte instead. You can still get the bool back by wrapping the struct member with a property. Don't use an auto-implemented property, you must care about the position of the byte. Thus:
struct MyStruct
{
private byte _f;
public bool f {
get { return _f != 0; }
set { _f = value ? 1 : 0; }
}
}
Native code is oblivious to the property. Don't fret about runtime overhead for the getter and setter, the jitter optimizer makes them disappear and they turn into a single CPU instruction each.

Should work:
[StructLayout(LayoutKind.Sequential)]
unsafe struct MyStruct
{
public fixed bool fs[5];
}

User Defined Types (UDT) in C#?

I know in VB you can define a UDT like:
Public Type Buffer1
ProductCode As String
SerialNumber As String
Date As Date
End Type
Is it possible to create a User Defined Type in C#? I know that they are used to implement data structures. I have researched and cannot seem to find anything.

The .NET equivalent of the VB6 Type is a struct with public fields. If what you need is the equivalent of the VB6 type, you should the ignore people who are telling you not to use a mutable struct.
Some people believe all data types should behave like class objects, and since structs with exposed fields behave in ways that class objects cannot they are evil.
In reality, there are times when the equivalent of a VB6 Type can be very useful, and any attempt to achieve such functionality with anything other than an exposed-field or otherwise "mutable" struct will be more cumbersome and yield less performant code than would using a type whose natural semantics precisely match what one is trying to achieve. Note that struct fields should generally be exposed as fields rather than properties, since the whole purpose of a VB6 Type is to serve as an aggregation of data types, rather than an ecapsulation of protected state.
There is, alas, one problem with .NET structure types, which is that unless you declare a structure as an unsafe type, only usable within unsafe code (and not usable in contexts which require being able to run in a Partial Trust environment), arrays within structures cannot achieve the semantics of arrays nor fixed-length strings in a VB6 Type. C# will allow one to declare fixed-sized arrays within a struct using the fixed keyword, but as noted only in conjunction with unsafe. Any other array declarations within a struct will cause a struct to hold an array reference rather than an array. Thus:
struct MyStruct {int n; int[] Arr; }
MyStruct a,b;
a.Arr = new int[4];
a.Arr[0] = 3;
a.n = 1;
b=a; // Copies the value of n, and the array reference in Arr, but not the array contents
b.n = 2;
b.Arr[1] = 2;
After the above code runs, b has its own copy of variable n, so the write to b.n will not affect a.n. Field b.Arr, however, holds a reference to the same array as a.Arr. An assignment directly to field Arr of b could make it point to a different array from a.Arr, but the assignment to b.Arr[1] doesn't actually write to field Arr but instead writes to the array object identified by that field, which is the same object identified by field a.Arr.
In some cases, you can replace what in VB6 would be a fixed-sized array with a simple list of discrete variables. Icky, but workable. Alternatively, if the restrictions associated with fixed arrays aren't a problem, you could use those. Beyond that, there isn't any good way to include arrays within a structure.

C# have classes and structs:
public class Buffer1
{
string ProductCode;
string SerialNumber;
DateTime Date;
}
But if you don't know classes, you must previously read some articles or books about C# (e.g. CLR via C#)

A "User Defined Data Type" from Visual Basic is called a Structure in VB.Net, and a struct in C#.
public struct Buffer1
{
public string ProductCode;
public string SerialNumber;
public DateTime Date;
}
Note that unlike the VB6 Type, which always makes all fields public, fields in all .NET types, including structures, default to private.

Non-blittable error on a blittable type

I have this struct and this code:
[StructLayout(LayoutKind.Sequential, Pack = 8)]
private class xvid_image_t
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public int[] stride;
// [MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
// public IntPtr[] plane;
}
public int decore()
{
xvid_image_t myStruct = new xvid_image_t();
myStruct.stride = new int[4]; // can be commented out - same result
GCHandle.Alloc(myStruct, GCHandleType.Pinned);
// ...
}
When I try to run it I get an ArgumentException saying:
Object contains non-primitive or non-blittable data
After reading this MSDN page saying
The following complex types are also blittable types:
One-dimensional arrays of blittable types, such as an array of integers. However, a type that contains a variable array of blittable types is not itself blittable.
Formatted value types that contain only blittable types (and classes if they are marshaled as formatted types). For more information about formatted value types, see Default Marshaling for Value Types.
I don't understand what I am doing wrong.
I don't just want to use Marshal, but to understand this too.
So what I actually want is to know:
Why?
How can I resolve this?
Will the solution you provide also work with the commented line in the struct?
I am using .Net 4.5 but a solution for .Net 2.0 is also needed.

Object contains non-primitive or non-blittable data
That's the exception message you get. You are focusing on the "non-blittable" part of the message, but that's not the problem. It is the "non-primitive" part that's the issue. An array is a non-primitive data type.
The CLR is trying to keep you out of trouble here. You could pin the object but then you still have a problem, the array won't be pinned. An object isn't truly pinned when it has fields that need to be pinned as well.
And you have a bigger problem with the UnmanagedType.ByValArray, that requires a structural conversion. In other words, the layout that you need is completely different from the layout of the managed class object. Only the pinvoke marshaller can make this conversion.
You can get what you want without using the pinvoke marshaller by using fixed size buffers, using the fixed keyword. This requires using the unsafe keyword. Make it look like this:
[StructLayout(LayoutKind.Sequential)]
unsafe private struct xvid_image_t {
public fixed int stride[4];
}
Note that you have to change the declaration to a struct type. It is now a value type, you no longer need to use GCHandle to pin the value when you make it a local variable. Do make sure that whatever unmanaged code takes the structure value, usually by reference, does not store a pointer to the struct. That's going to blow up badly and utterly undiagnosably. The unsafe keyword is appropriate here. If it does store the pointer then you really do have to byte the bullet and use Marshal.AllocHGlobal() and Marshal.StructureToPtr() to ensure the pointer stays valid while the unmanaged code is using it.

An annoying limitation of .NET is that the only array-ish things it recognizes are a stand-alone System.Array object and a System.String, both of which are reference types. It's possible for code written in C# to use a fixed array (as noted by Hans Passant), but such a type is not recognized by .NET itself, and code which uses fixed arrays is not verifiable. Additionally, fixed arrays are limited to holding primitives, and cannot be accessed by other languages such as vb.net.
Two alternatives to using a fixed array are to
replace the fixed array with some combination of fields which together total the proper size (using N variables in most cases, but perhaps replacing e.g. a char[4] with a UInt32, or a char[8] with a UInt64). If the array is not too large, one might define (either via cut/paste or Reflection) a set of static methods which take a struct by ref and read/write the proper element, and then create an array of delegates to call such methods.
replace the entire structure with an array, and then pass the first element of that array as a ref parameter. This may be even more "dangerous" than using a fixed array within a structure, but is the only way I know of in vb.net to get "pass-by-ref" semantics with a structure that contains something that really needs to be accessed as an array.
While I can understand that value-type arrays might have been considered "confusing" (especially if they were auto-boxed) there are places where they would have been the semantically-correct approach for array storage, both from the standpoint of allowing pass-by-ref semantics for COM interop and also from the standpoint of methods that are supposed to return a small number of values. For example, in System.Drawing2d, there is a method which returns the current graphics transform as a float[6]; other than by experimentation, there would be no clear way of knowing whether changes to that array after it is returned would affect, might affect, or are guaranteed not to affect anything else. If the method returned a value-type array, it would be clear that changes to the returned array cannot affect anything else. Nonetheless, whether or not value-type arrays would have been a useful part of the Framework, the fact remains that whether for good or bad reasons no such thing exists.

I took the below answer from this link (here)
SItuLongEmailMsg msg = newSItuLongEmailMsg();
// set members
msg.text = new byte[2048];
// assign to msg.text
int msgSize = Marshal.SizeOf(msg);
IntPtr ptr = Marshal.AllocHGlobal(msgSize);
Marshal.StructureToPtr(msg, ptr, true);
byte[] dataOut = new byte[msgSize];
Marshal.Copy(ptr, dataOut, 0, msgSize);

How to marshal struct containing pointer to first element of a C-style array of unknown type

I'm trying to marshal from C++ to C# a struct that looks something like this:
typedef struct FooStruct {
Uint8 bytesPerThingie;
void *arrayOfThingies;
// other members ...
}
So, in this case there are two unknowns:
The number of elements in the array.
The size (in bytes) of each element.
I had successfully marshaled the struct itself previously, with a definition like this:
[StructLayout(LayoutKind.Sequential)]
public struct FooStruct {
public byte bytesPerThingie;
public IntPtr arrayOfThingies;
// other members...
}
but now I need to inspect and modify the embedded array.
I understand that
By itself, an array of blittable elements of a blittable type is
itself blittable, but not when it is used as a field within a
structure.
When marshaling from unmanaged code to the .NET Framework, the array length is
determined from the SizeConst argument, optionally followed by the unmanaged type
of the array elements, if they aren’t blittable.
Even assuming that the elements in the array in this case are of a blittable type, how can I set SizeConst, a compile-time argument, if I can't know the size of the array until runtime?

Long story short, you can't. The SizeConst field on the MarshalAsAttribute class is compiled into metadata on the field and cannot be changed at runtime (at least, not in a way that would benefit you).
That said, you have the following options:
Marshal the contents manually as you've been doing using the methods on the Marshal class.
Use unsafe to access the pointer directly (and change your type to use pointers). This requires the /unsafe compiler option which may or may not be an option for you.
Use C++/CLI to create a managed wrapper in C++ that will export .NET types, but handle the marshaling in C++ (which might be easier, depending on your comfort level and the complexity of the API you are trying to access).
Note that in all of the cases above, you still have to know the length of the array that is returned (it's probably in the structure along with the pointer and the type).