We Keep Coding

Are record structs passed by value or by reference and can they be blittable or not

Working with a very large number structs as well as Arrays and Lists of structs I'm always annoyed by the the usual hurdles to reduce the number of copies of said structs when passing them to methods or working with them in diverse ways.
In addition I am forced to use structs as I need to use blittable types.
I also know about things like ErrorProne.NET which helps a lot but I'm still not fine with the Solution I currently have.
Now, with the latest C# and .NET-Versions we have the record-type which can be applied to both, structs and classes or used standalone but I can't find any clear information regarding copy-when-passing beheaviour and similar stuff and about how a record struct behaves compared to a normal struct and whether record structs can be blittable or not.
Following an Example of the structs (here record structs) I'm talking about. In the optimal case, I would even like to use them in combination with a Custom ObjectPool allowing me to re-use them without my program having to allocate the memory for new of these structs again and again while GC-ing old unused ones.
[StructLayout(LayoutKind.Sequential)]
public record struct PagedSecondaryIndexKey
{
[MarshalAs(UnmanagedType.U1)]
public SecondaryIndexType IndexType;
[MarshalAs(UnmanagedType.U8)]
public ulong Page;
[MarshalAs(UnmanagedType.Struct)]
public IndexIdentifier IndexIdentifier;
public PagedSecondaryIndexKey(ulong page, SecondaryIndexType indexType, ulong identifier1, ulong identifier2, ulong identifier3)
{
IndexType = indexType;
Page = page;
IndexIdentifier.Reset(identifier1, identifier2, identifier3);
}
}
[StructLayout(LayoutKind.Sequential)]
public record struct IndexIdentifier
{
[MarshalAs(UnmanagedType.U8)]
private ulong IndexIdentifier1;
[MarshalAs(UnmanagedType.U8)]
private ulong IndexIdentifier2;
[MarshalAs(UnmanagedType.U8)]
private ulong IndexIdentifier3;
public const int Size = 3 * sizeof(ulong);
public IndexIdentifier(ulong identifier1, ulong identifier2, ulong identifier3)
{
IndexIdentifier1 = identifier1;
IndexIdentifier2 = identifier2;
IndexIdentifier3 = identifier3;
}
public IndexIdentifier(IndexIdentifier indexKeyIndexIdentifier)
{
indexKeyIndexIdentifier.IndexIdentifier1 = IndexIdentifier1;
indexKeyIndexIdentifier.IndexIdentifier2 = IndexIdentifier2;
indexKeyIndexIdentifier.IndexIdentifier3 = IndexIdentifier3;
}
}
Tried diverse things to solve the Problems mentions, used diverse tools, did a lot of research but still havn't found the answer I'm looking for.

Basically, records just add a number of default methods to the class (ToString, Equals, etc.) but otherwise a record class behaves like a class (it's a reference type) and record struct behaves like a struct (it's a value type).
That also means that a record struct is blittable if and only if an equivalent struct would be blittable. Which basically means that a struct/record struct must not contain references to be blittable.

C# interop: bad interaction between fixed and MarshalAs

I need to marshal some nested structures in C# 4.0 into binary blobs to pass to a C++ framework.
I have so far had a lot of success using unsafe/fixed to handle fixed length arrays of primitive types. Now I need to handle a structure that contains nested fixed length arrays of other structures.
I was using complicated workarounds flattening the structures but then I came across an example of the MarshalAs attribute which looked like it could save me a great deal of problems.
Unfortunately whilst it gives me the correct amount of data it seems to also stop the fixed arrays from being marshalled properly, as the output of this program demonstrates. You can confirm the failure by putting a breakpoint on the last line and examining the memory at each pointer.
using System;
using System.Threading;
using System.Runtime.InteropServices;
namespace MarshalNested
{
public unsafe struct a_struct_test1
{
public fixed sbyte a_string[3];
public fixed sbyte some_data[12];
}
public struct a_struct_test2
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public sbyte[] a_string;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public a_nested[] some_data;
}
public unsafe struct a_struct_test3
{
public fixed sbyte a_string[3];
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public a_nested[] some_data;
}
public unsafe struct a_nested
{
public fixed sbyte a_notherstring[3];
}
class Program
{
static unsafe void Main(string[] args)
{
a_struct_test1 lStruct1 = new a_struct_test1();
lStruct1.a_string[0] = (sbyte)'a';
lStruct1.a_string[1] = (sbyte)'b';
lStruct1.a_string[2] = (sbyte)'c';
a_struct_test2 lStruct2 = new a_struct_test2();
lStruct2.a_string = new sbyte[3];
lStruct2.a_string[0] = (sbyte)'a';
lStruct2.a_string[1] = (sbyte)'b';
lStruct2.a_string[2] = (sbyte)'c';
a_struct_test3 lStruct3 = new a_struct_test3();
lStruct3.a_string[0] = (sbyte)'a';
lStruct3.a_string[1] = (sbyte)'b';
lStruct3.a_string[2] = (sbyte)'c';
IntPtr lPtr1 = Marshal.AllocHGlobal(15);
Marshal.StructureToPtr(lStruct1, lPtr1, false);
IntPtr lPtr2 = Marshal.AllocHGlobal(15);
Marshal.StructureToPtr(lStruct2, lPtr2, false);
IntPtr lPtr3 = Marshal.AllocHGlobal(15);
Marshal.StructureToPtr(lStruct3, lPtr3, false);
string s1 = "";
string s2 = "";
string s3 = "";
for (int x = 0; x < 3; x++)
{
s1 += (char) Marshal.ReadByte(lPtr1+x);
s2 += (char) Marshal.ReadByte(lPtr2+x);
s3 += (char) Marshal.ReadByte(lPtr3+x);
}
Console.WriteLine("Ptr1 (size " + Marshal.SizeOf(lStruct1) + ") says " + s1);
Console.WriteLine("Ptr2 (size " + Marshal.SizeOf(lStruct2) + ") says " + s2);
Console.WriteLine("Ptr3 (size " + Marshal.SizeOf(lStruct3) + ") says " + s3);
Thread.Sleep(10000);
}
}
}
Output:
Ptr1 (size 15) says abc
Ptr2 (size 15) says abc
Ptr3 (size 15) says a
So for some reason it is only marshalling the first character of my fixed ANSI strings. Is there any way around this, or have I done something stupid unrelated to the marshalling?

This is a case of a missing diagnostic. Somebody should have spoken up and tell you that your declaration is not supported. Where that somebody is either the C# compiler, producing a compile error, or the CLR field marshaller, producing a runtime exception.
It's not like you can't get a diagnostic. You'll certainly get one when you actually start using the struct as intended:
a_struct_test3 lStruct3 = new a_struct_test3();
lStruct3.some_data = new a_nested[4];
lStruct3.some_data[0] = new a_nested();
lStruct3.some_data[0].a_notherstring[0] = (sbyte)'a'; // Eek!
Which elicits CS1666, "You cannot use fixed size buffers contained in unfixed expressions. Try using the fixed statement". Not that "try this" advice is all that helpful:
fixed (sbyte* p = &lStruct3.some_data[0].a_notherstring[0]) // Eek!
{
*p = (sbyte)'a';
}
Exact same CS1666 error. Next thing you'd try is put an attribute on the fixed buffer:
public unsafe struct a_struct_test3 {
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public fixed sbyte a_string[3];
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public a_nested[] some_data;
}
//...
a_struct_test3 lStruct3 = new a_struct_test3();
lStruct3.some_data = new a_nested[4];
IntPtr lPtr3 = Marshal.AllocHGlobal(15);
Marshal.StructureToPtr(lStruct3, lPtr3, false); // Eek!
Keeps the C# compiler happy but now the CLR speaks up and you get a TypeLoadException at runtime: "Additional information: Cannot marshal field 'a_string' of type 'MarshalNested.a_struct_test3': Invalid managed/unmanaged type combination (this value type must be paired with Struct)."
So, in a nutshell you should have gotten either CS1666 or TypeLoadException on your original attempt as well. That did not happen because the C# compiler was not forced to look at the bad part, it only generates CS1666 on a statement that accesses the array. And it did not happen at runtime because the field marshaller in the CLR did not attempt to marshal the array because it is null. You can file a bug feedback report at connect.microsoft.com but I'd be greatly surprised if they won't close it with "by design".
In general, an obscure detail matters a great deal to the field marshaller in the CLR, the chunk of code that converts struct values and class objects from their managed layout to their unmanaged layout. It is poorly documented, Microsoft does not want to nail down the exact implementation details. Mostly because they depend too much on the target architecture.
What matters a great deal is whether or not a value or object is blittable. It is blittable when the managed and unmanaged layout is identical. Which only happens when every member of the type has the exact same size and alignment in both layouts. That normally only happens when the fields are of a very simple value type (like byte or int) or a struct that itself is blittable. Notoriously not when it is bool, too many conflicting unmanaged bool types. A field of an array type is never blittable, managed arrays don't look anything like C arrays since they have an object header and a Length member.
Having a blittable value or object is highly desirable, it avoids the field marshaller from having to create a copy. The native code gets a simple pointer to managed memory and all that is needed is to pin the memory. Very fast. It is also very dangerous, if the declaration does not match then the native code can easily color outside the lines and corrupt the GC heap or stack frame. A very common reason for a program that use pinvoke to bomb randomly with ExecutionEngineException, excessively difficult to diagnose. Such a declaration really deserves the unsafe keyword but the C# compiler does not insist on it. Nor can it, compilers are not allowed to make any assumptions about managed object layout. You keep it safe by using Debug.Assert() on the return value of Marshal.SizeOf<T>, it must be an exact match with the value of sizeof(T) in a C program.
As noted, arrays are an obstacle to getting a blittable value or object. The fixed keyword is intended as a workaround for this. The CLR treats it like an opaque value type with no members, just a blob of bytes. No object header and no Length member, as close as you could get to a C array. And used in C# code like you'd use an array in a C program, you must use a pointer to address the array elements and check three times that you don't color outside of the lines. Sometimes you must use a fixed array, happens when you declare a union (overlapping fields) and you overlap an array with a value. Poison to the garbage collector, it can no longer figure out if the field stores an object root. Not detected by the C# compiler but reliably trips a TypeLoadException at runtime.
Long story short, use fixed only for a blittable type. Mixing fields of a fixed size buffer type with fields that must be marshaled cannot work. And isn't useful, the object or value gets copied anyway so you might as well use the friendly array type.

Using reflection to determine how a .Net type is layed out in memory

I'm experimenting with optimizing parser combinators in C#. One possible optimization, when the serialized format matches the in-memory format, is to just do an (unsafe) memcpy of the data to be parsed over an instance or even many instances of the type.
I want to write code that determines if the in-memory format matches the serialized format, in order to dynamically determine if the optimization can be applied. (Obviously this is an unsafe optimization and might not work for a whole bunch of subtle reasons. I'm just experimenting, not planning to use this in production code.)
I use the attribute [StructLayout(LayoutKind.Sequential, Pack = 1)] to force no padding and to force the in-memory order to match declaration order. I check for that attribute with reflection, but really all this confirms is "no padding". I also need the order of the fields. (I would strongly prefer to not have to manually specified FieldOffset attributes for every field, since that would be very error prone.)
I assumed I could use the order of fields returned by GetFields, but the documentation explicitly calls out that the order is unspecified.
Given that I am forcing the order of fields with the StructLayout attribute, is there a way to reflect on that ordering?
edit I'm fine with the restriction that all of the fields must be blittable.

This is unnecessary if using LayoutKind.Sequential with blittable types
You don't need to use reflection or any other mechanism to find out the order of struct fields in memory, as long as all the fields are blittable.
The blittable fields for a struct declared with LayoutKind.Sequential will be in memory in the order in which the fields are declared. That's what LayoutKind.Sequential means!
From this documentation:
For blittable types, LayoutKind.Sequential controls both the layout in managed memory and the layout in unmanaged memory. For non-blittable types, it controls the layout when the class or structure is marshaled to unmanaged code, but does not control the layout in managed memory.
Note that this doesn't tell you how much padding each field is using. To find that out, see below.
To determine the field order when using LayoutKind.Auto, or the field offsets when using any layout
It's fairly easy to find the struct field offsets if you're happy to use unsafe code, and to not use reflection.
You just need to take the address of each field of the struct and calculate its offset from the start of the struct. Knowing the offsets of each field, you can calculate their order (and any padding bytes between them). To calculate the padding bytes used for the last field (if any) you will also need to get the total size of the struct using sizeof(StructType).
The following example works for 32-bit and 64-bit. Note that you don't need to use fixed keyword because the struct is already fixed due to it being on the stack (you'll get a compile error if you try to use fixed with it):
using System;
using System.Runtime.InteropServices;
namespace Demo
{
[StructLayout(LayoutKind.Auto, Pack = 1)]
public struct TestStruct
{
public int I;
public double D;
public short S;
public byte B;
public long L;
}
class Program
{
void run()
{
var t = new TestStruct();
unsafe
{
IntPtr p = new IntPtr(&t);
IntPtr pI = new IntPtr(&t.I);
IntPtr pD = new IntPtr(&t.D);
IntPtr pS = new IntPtr(&t.S);
IntPtr pB = new IntPtr(&t.B);
IntPtr pL = new IntPtr(&t.L);
Console.WriteLine("I offset = " + ptrDiff(p, pI));
Console.WriteLine("D offset = " + ptrDiff(p, pD));
Console.WriteLine("S offset = " + ptrDiff(p, pS));
Console.WriteLine("B offset = " + ptrDiff(p, pB));
Console.WriteLine("L offset = " + ptrDiff(p, pL));
Console.WriteLine("Total struct size = " + sizeof(TestStruct));
}
}
long ptrDiff(IntPtr p1, IntPtr p2)
{
return p2.ToInt64() - p1.ToInt64();
}
static void Main()
{
new Program().run();
}
}
}
To determine the field offsets when using LayoutKind.Sequential
If your struct uses LayoutKind.Sequential then you can use Marshal.OffsetOf() to get the offset directly, but this does not work with LayoutKind.Auto:
foreach (var field in typeof(TestStruct).GetFields())
{
var offset = Marshal.OffsetOf(typeof (TestStruct), field.Name);
Console.WriteLine("Offset of " + field.Name + " = " + offset);
}
This is clearly a better way to do it if you are using LayoutKind.Sequential since it doesn't require unsafe code, and it's much shorter - and you don't need to know the names of the fields in advance. As I said above, it is not needed to determine the order of the fields in memory - but this might be useful if you need to find out about how much padding is used.

As a reference for those who want to know the order and the kind of layout. For example if a type contains non-blittable types.
var fields = typeof(T).GetFields(BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance);
fields.SortByFieldOffset();
var isExplicit = typeof(T).IsExplicitLayout;
var isSequential = typeof(T).IsLayoutSequential;
It uses an extension method that I wrote:
public static void SortByFieldOffset(this FieldInfo[] fields) {
Array.Sort(fields, (a, b) => OffsetOf(a).CompareTo(OffsetOf(b)) );
}
private static int OffsetOf(FieldInfo field) {
return Marshal.OffsetOf(field.DeclaringType, field.Name).ToInt32();
}
MSDN contains useful info on IsLayoutSequential.

C# equivalent to VB6 'Type'

I am trying to port a rather large source from VB6 to C#. This is no easy task - especially for me being fairly new to C#.net. This source uses numerous Windows APIs as well as numerous Types. I know that there is no equivalent to the VB6 Type in C# but I'm sure there is a way to reach the same outcome. I will post some code below to further explain my request.
VB6:
Private Type ICONDIRENTRY
bWidth As Byte
bHeight As Byte
bColorCount As Byte
bReserved As Byte
wPlanes As Integer
wBitCount As Integer
dwBytesInRes As Long
dwImageOffset As Long
End Type
Dim tICONDIRENTRY() As ICONDIRENTRY
ReDim tICONDIRENTRY(tICONDIR.idCount - 1)
For i = 0 To tICONDIR.idCount - 1
Call ReadFile(lFile, tICONDIRENTRY(i), Len(tICONDIRENTRY(i)), lRet, ByVal 0&)
Next i
I have tried using structs and classes - but no luck so far.
I would like to see a conversion of this Type structure, but if someone had any clue as to how to convert the entire thing it would be unbelievably helpful. I have spent countless hours on this small project already.
If it makes any difference, this is strictly for educational purposes only.
Thank you for any help in advance,
Evan

struct is the equivalent. You'd express it like this:
struct IconDirEntry {
public byte Width;
public byte Height;
public byte ColorCount;
public byte Reserved;
public int Planes;
public int BitCount;
public long BytesInRes;
public long ImageOffset;
}
You declare a variable like this:
IconDirEntry entry;
Generally, in C#, type prefixes are not used, nor are all caps, except possibly for constants. structs are value types in C#, so that means that they are always passed by value. It looks like you're passing them in to a method that's populating them. If you want that usage, you'll have to use classes.

I'm not exactly sure what your issue is but this is a small ex of how to use a struct.
struct aStrt
{
public int A;
public int B;
}
static void Main(string[] args)
{
aStrt saStrt;
saStrt.A = 5;
}

Your question is not clear ..
What issues are you facing when you are using either struct or class and define those field members? Are you not able to access those members using an instance created for that class ??
Else, declare the class as static and make all the members inside the class also as static , so that u can access them without any instance being created!!

Maybe you trying to get something like this?
struct IconDirEntry
{
public byte Width;
// etc...
}
IconDirEntry[] tICONDIRENTRY = new IconDireEntry[tICONDIR.idCount - 1];

qt to c# what does : operator mean within struct

I have a block of code that i'm trying to covert from an old qt file into C# but i'm a little unclear what is going on in the struct within the union below. I'm not sure what the ':' does... i'm guessing it sets the size but could not find any documentation on this. Also since C# does not have unions what is the best practice for converting something like this. Thank you
union uAWord
{
uAWord()
: m_AWord(0) {}
struct sBcdAWord
{
quint32 m_O :8;
quint32 m_S :2;
quint32 m_D :18;
quint32 m_SS :3;
quint32 m_P :1;
}
sBcdAWord m_F;
quint32 m_AWord;
}

This is what is called BitFields. the portion sBcdWord is a 32 bit word, and each field is a portion of that word taking respectively 8,2,18,3,1 BIT:
So the word layout is as below:
Bit0-Bit7 m_0
Bit8-Bit9 m_S
Bit10-Bit27 m_D
Bit28-Bit30 m_ss
Bit31 m_P
How to port this in C# depends if you are convettually porting the code, or if you need to PInvoke. In the case of PInvoke the best solution is probably to map sBcdAWord as an Unit32, and create some accessor strategy to mask on reading writing. If it is a code port, use separeted properties would be good unless there is special needing in memory usage saving.

That syntax is used to declare bitfields. The number is the number of bits for that value. See for example http://publib.boulder.ibm.com/infocenter/macxhelp/v6v81/index.jsp?topic=%2Fcom.ibm.vacpp6m.doc%2Flanguage%2Fref%2Fclrc03defbitf.htm
A good conversion to C# depends on the case I guess. As long as you are not too space-conscious, I'd just keep all needed values in parallel in a class.

That initializes m_aWord to 0.

To answer your other question, in C# you'd likely want a struct, and you'd need to use attributes to get union-like behavior out of it.
This particular example could be somewhat like:
[StructLayout(LayoutKind.Explicit)]
struct uAWord {
[FieldOffset(0)]
private uint theWord = 0;
[FieldOffset(0)]
public int m_P;
[FieldOffset(1)]
public int m_S;
[FieldOffset(3)]
public int m_SS;
[FieldOffset(7)]
public int m_O;
[FieldOffset(18)]
public int m_D;
public uAWord(uint theWord){
this.theWord = theWord;
}
}
The LayoutKind.Explicit indicates you will tell it where in the memory to map each field and the FieldOffset(int) tells which bit to start each field on. See this for more details. You'd assign this struct by setting the uint theWord in the constructor, then each of the other properties would access a chunk starting at a different memory address.
Unfortunately, that actually isn't correct. You'll need to use properties and do some bitmasking/shifting to get it right. Like this:
struct uAWord {
private uint theWord = 0;
public int m_P {get {return (theWord & 0x01);}}
public int m_S {get {return (theWord & 0x02) << 2;}}
public int m_SS {get {return (theWord & 0x04) << 3;}}
public int m_0 {get {return (theWord & 0x18) << 6;}}
}