We Keep Coding

how to transform byte[] to struct (contains a byte[] member and length member)

i have a struct define as:
[StructLayout(LayoutKind.Sequential,CharSet = CharSet.Ansi,Pack = 1)]
internal struct Message
{
[MarshalAs(UnmanagedType.U1, SizeConst = 1)]
public byte age;
[MarshalAs(UnmanagedType.U2, SizeConst = 2)]
public ushort length;
[MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.U1,SizeConst = 502)]
public byte[] data;
}
payload received via udp，when received byte[],need to be converted to struct.
data length specified as 502,but actually it should be the length member value indicating the data length,if remove the SizeConst attr, the code will throw Marshal exception at Marshal.SizeOf(typeof(T)).
public static T ToStruct<T>(this byte[] buf)
{
var lth = Marshal.SizeOf(typeof(T));
if (lth > buf.Length) return default(T);
var ptr = Marshal.AllocHGlobal(lth);
Marshal.Copy(buf, 0, ptr, lth);
var structure = Marshal.PtrToStructure(ptr, typeof(T));
Marshal.FreeHGlobal(ptr);
return (T)structure;
}
exception info:
System.ArgumentException: Type 'Itms.Agent.IotBox.TieTa.Entity.Message' cannot be marshaled as an unmanaged structure; no meaningful size or offset can be computed.
i know this is very simple to handle in c/c++, but C# doesn't have a similar approach.
any help?

when received byte[],need to be converted to struct. data length specified as 502,but actually it should be the length member value indicating the data length
that's too complex for simple marshal operations. You'll probably have to serialize/deserialize manually, i.e. something like:
byte[] payload = ...
var age = payload[0];
var len = (payload[1] << 8) | payload[2]; // invert depending on endianness
byte[] data = len == 0 ? Array.Empty<byte>() : new byte[len];
Buffer.BlockCopy(payload, 3, data, len);
var msg = new Message(age, len, data);
This would mean you could remove all the attributes, as you're not using any marshalling features.
Also... the length field seems kinda redundant, since it is simply duplicating the array's length.

Marshal.SizeOf Giving The Wrong Size

The code below is giving the wrong size when using Marshal.SizeOf, but I'm not sure why.
Here is the Struct I'm trying to get the size of:
//[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
//[Serializable]
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct BLEGenericMsg
{
public BLEMessageHdr msg_hdr;
public byte[] msg_body;
public BLEGenericMsg(int messageSize)
{
msg_hdr = new BLEMessageHdr();
msg_body = new byte[messageSize];
}
};
Here is the code that populates the struct and calls the serialize function:
BLEGenericMsg hostKeyMsg = new BLEGenericMsg(serializedPublicBytes.Length);
hostKeyMsg.msg_hdr.msg_id = MESSAGE_BASE_EVENT + EVENT_HOST_PUBLIC_KEY;
hostKeyMsg.msg_body = serializedPublicBytes;
//Only get the size of the body for the entire message, not counter or header
hostKeyMsg.msg_hdr.msg_body_sz = (uint)hostKeyMsg.msg_body.Length;
BluetoothLEHardwareInterface.Log("public Key Size: " + hostKeyMsg.msg_hdr.msg_body_sz + "\n");
byte[] temp = Serialize(hostKeyMsg);
BluetoothLEHardwareInterface.Log("temp Size: " + (uint)temp.Length + "\n");
Here is the serialize function that is getting the size of the struct:
public static byte[] Serialize<T>(T s)
where T : struct
{
var size = Marshal.SizeOf(typeof(T));
BluetoothLEHardwareInterface.Log("BLEGenericMsg Size: " + size + "\n");
var array = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(s, ptr, true);
Marshal.Copy(ptr, array, 0, size);
Marshal.FreeHGlobal(ptr);
return array;
}
The size of serializedPublicBytes is 91 bytes,
the rest of the struct is 6 bytes.
So I'm expecting the Marshal.SizeOf to be 97 bytes,
but instead it is showing only about 14 or 16 bytes.
I tried giving the size of msg_body at instantiation, but that didn't make a difference.
What am I missing?
**edit Here is here is the BLEMessageHdr struct:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct BLEMessageHdr
{
public ushort msg_id;
public uint msg_body_sz;
};

The Marshal.SizeOf() method is not returning the wrong size. In the structure you define:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct BLEGenericMsg
{
public BLEMessageHdr msg_hdr;
public byte[] msg_body;
public BLEGenericMsg(int messageSize)
{
msg_hdr = new BLEMessageHdr();
msg_body = new byte[messageSize];
}
};
the msg_body member is known as a "Flexible Array Member" (FAM) in C. It is an illegal construct in C++. Because it is illegal in C++, and because of the inherent uncertainties in the C standard (§ 6.7.2.1) with regard to the instantiation of a struct that contains a FAM, the Marshal class simply does not accept them for interop with unmanaged code.
The way array members are usually marshalled is with the MarshalAsAttribute, like so:
[MarshalAs(UnmanagedType.ByValArray, SizeConst=N)]
public byte[] msg_body;
where "N" represents the explicitly declared size of the array. Without this attribute, the msg_body member is treated as a pointer by the Marshal class. So, the size that Marshal.SizeOf() is returning is correct. Your generic Serialize() method won't work for structs that have a FAM.
You could modify it to copy over the contents of the FAM manually after the rest has been copied by the Marshal class, but this seems like a rather awkward approach to binary serialization for a managed struct.
// specify the name of the FAM and use reflection to get the value
// THIS ASSUMES that the FAM is always a byte[]
public static byte[] Serialize<T>(T s, string fam) where T : struct
{
Type tt = typeof(T);
// Reflection will get you the bytes in the FAM
FieldInfo fi = tt.GetField(fam);
byte[] famBytes = (byte[])fi.GetValue(s);
// Get the field offset that corresponds to the unmanaged layout for
// the FAM, according to the marshaller
int offset = (int)Marshal.OffsetOf(tt, fam);
var size = Marshal.SizeOf(tt) + famBytes.Length;
BluetoothLEHardwareInterface.Log("BLEGenericMsg Size: " + size + "\n");
var array = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(s, ptr, true);
Marshal.Copy(ptr, array, 0, size);
Marshal.FreeHGlobal(ptr);
// Now you're done with the marshalling, just copy over the contents of the
// byte[] to your resulting array, starting at the correct offset
Array.Copy(famBytes, 0, array, offset, famBytes.Length);
return array;
}
Naturally, you will have to likewise modify the Deserialize() method to deal with structs that have a FAM.
AGAIN, this seems like an awkward approach to this problem. You may want to really reconsider this approach.

Write contiguous structured data to a binary file using C#

I am doing a project in which I need to write structured data into a binary file. First I need to write a header, then fetch data from somewhere, populate and write the structured data blocks to the said file. I am porting C structs to C# as follows:
C header struct:
typedef struct
{
DWORD uSignature;
DWORD uRecordLength;
} Header;
C data struct:
typedef struct
{
DWORD uCode; // a two character identifier
char uLabel[10];
int uDate;
float uData[37];
} MyData;
Here is the C# header struct:
struct Header
{
public uint uSignature;
public uint uRecordLength;
}
and here is the C# data struct:
struct MyData
{
public MyData (int Count) : this ()
{
uData = new Single[Count];
}
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public byte[] uCode;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public byte[] uLabel;
public int uDate;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 37)]
public Single [] uData;
}
This file will be read by another application which can read from the binary file if data correctly formatted. I printed out the size of the two struct types and they look good. However, the output file can not be read by the said application.
So I have two questions:
Are the data types and Marshals I used correctly in C to C# conversion?
I use FileStream and BinaryWriter to write to the binary file. All data (header and subsequent data) must be in sequence (contiguous) . As I create and write data structs on the fly, I am not sure how to allocate continuous memory using something like:
public static byte[] GetData(object obj)
{
var size = Marshal.SizeOf(obj.GetType());
var data = new byte[size];
IntPtr pnt = Marshal.AllocHGlobal(size);
try
{
Marshal.StructureToPtr(obj, pnt, true);
// Copy the array to unmanaged memory.
Marshal.Copy(pnt, data, 0, size);
return data;
}
finally
{
// Free the unmanaged memory.
Marshal.FreeHGlobal(pnt);
}
}
Any help would be greatly appreciated!
[EDIT]
I added two methods to convert specific struct data into byte array, but the file is still unreadable:
private byte[] DataToByteArray(MyData data)
{
int len = 0;
var size = Marshal.SizeOf(data.GetType());
var barray = new byte[size];
data.uCode.CopyTo(barray, 0);
len += data.uCode.Length;
data.uLabel.CopyTo(barray, len);
len += data.uLabel.Length;
BitConverter.GetBytes(0).CopyTo(barray, len);
len += data.uData.Length;
Buffer.BlockCopy(data.uData, 0, barray, len, data.uData.Length);
return barray;
}
private byte[] HeadToByteArray(Header data)
{
var size = Marshal.SizeOf(data.GetType());
var barray = new byte[size];
BitConverter.GetBytes(data.uSignature).CopyTo(barray, 0);
BitConverter.GetBytes(data.uRecordLength).CopyTo(barray, 4);
return barray;
}
【EDIT2】
Here is how it works in C:
#define NQ_EX 'QN'
FILE *fout;
fopen_s(&fout, "path_to_the_file", "wb");
Header head = { val1, sizeof(MyData) };
fwrite(&head, sizeof(Header), 1, fout);
while (!stop && data_is_coming)
{
MyData data;
memset(&data, 0, sizeof(data));
data.uCode = NQ_EX;
sprintf_s(data.uLabel, "%s", getVal("field1"));
data.uData[0] = getVal("field2");
data.uData[1] = getVal("field3");
....
fwrite(&data, sizeof(MyData), 1, fout);
}

The endianness seems to be fine. After some changes and testing with the help of Jeroen and others, I am able to make it work. The problem was due to the Block.copy method. I change it to Array.copy as follows:
private byte[] DataToByteArray(MyData data)
{
int len = 0;
var size = Marshal.SizeOf(data.GetType());
var barray = new byte[size];
data.uCode.CopyTo(barray, 0);
len += data.uCode.Length;
data.uLabel.CopyTo(barray, len);
len += data.uLabel.Length;
BitConverter.GetBytes(0).CopyTo(barray, len);
len += data.uData.Length;
for (int i = 0; i < data.uData.Length; i++)
Array.Copy(BitConverter.GetBytes(data.uData[i]), 0, barray, len+i * 4, 4);
return barray;
}

How to do memcpy in C# .Net CF with the following task

Hi I am trying to convert the C/C++ Strcut to C# and how to fill the structure member with address of another structure in C#?
C/C++ Struct looks like:
typedef struct _NDISUIO_QUERY_OID
{
NDIS_OID Oid;
PTCHAR ptcDeviceName;
UCHAR Data[sizeof(ULONG)];
} NDISUIO_QUERY_OID, *PNDISUIO_QUERY_OID;
typedef struct My_Struct
{
//les have 2 variables...
UINT a;
UINT b;
}My_STATS, *PMy_STATS;
PNDISUIO_QUERY_OID pQueryOid = NULL;
pQueryOid = (PNDISUIO_QUERY_OID)malloc(sizeof(NDISUIO_QUERY_OID)+ sizeof(My_STATS)) ;
PMy_STATS Statistics;
pQueryOid->Oid = ulOIDCode;//Required OID
pQueryOid->ptcDeviceName = AUB_NAME;//REquired STRING
memcpy(pQueryOid->Data, Statistics, sizeof(My_STATS));
My C# Struct is:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = sizeof(uint))]
public string Data;
};
Problem: How to copy the Statistics structure to Data array in C#??
Thanks :)

Here's my implementation (FYI, the SDF contains all of this code and a lot more)
internal class NDISQueryOid
{
protected const int NDISUIO_QUERY_OID_SIZE = 12;
protected byte[] m_data;
public int Size { get; private set; }
public NDISQueryOid(byte[] data)
{
int extrasize = data.Length;
Size = 8 + extrasize;
m_data = new byte[Size];
Buffer.BlockCopy(data, 0, m_data, DataOffset, data.Length);
}
public NDISQueryOid(int extrasize)
{
Size = NDISUIO_QUERY_OID_SIZE + extrasize;
m_data = new byte[Size];
}
protected const int OidOffset = 0;
public uint Oid
{
get { return BitConverter.ToUInt32(m_data, OidOffset); }
set
{
byte[] bytes = BitConverter.GetBytes(value);
Buffer.BlockCopy(bytes, 0, m_data, OidOffset, 4);
}
}
protected const int ptcDeviceNameOffset = OidOffset + 4;
public unsafe byte* ptcDeviceName
{
get
{
return (byte*)BitConverter.ToUInt32(m_data, ptcDeviceNameOffset);
}
set
{
byte[] bytes = BitConverter.GetBytes((UInt32)value);
Buffer.BlockCopy(bytes, 0, m_data, ptcDeviceNameOffset, 4);
}
}
protected const int DataOffset = ptcDeviceNameOffset + 4;
public byte[] Data
{
get
{
byte[] b = new byte[Size - DataOffset];
Array.Copy(m_data, DataOffset, b, 0, Size - DataOffset);
return b;
}
set
{
Size = 8 + value.Length;
m_data = new byte[Size];
Buffer.BlockCopy(value, 0, m_data, DataOffset, value.Length);
}
}
public byte[] getBytes()
{
return m_data;
}
public static implicit operator byte[](NDISQueryOid qoid)
{
return qoid.m_data;
}
}
Note that in my usage, the NDIS IOCT takes in a pointer (most of my NDIS work is all done as unsafe) so you'd have to do some adjustment there.
So if, for example, you're querying the BSSID, I know the BSSID data is 36 bytes, so I'd create something like this:
var queryOID = new NDISQueryOid(36);
then allocate the name and call NDIS (the production code has a lot more checking than this):
byte[] nameBytes = System.Text.Encoding.Unicode.GetBytes(adapterName + '\0');
fixed (byte* pName = &nameBytes[0])
{
queryOID.ptcDeviceName = pName;
queryOID.Oid = (uint)oid;
var bytes = queryOID.getBytes();
ndis.DeviceIoControl(IOCTL_NDISUIO_QUERY_OID_VALUE, bytes, bytes);
var result = new byte[queryOID.Data.Length];
Buffer.BlockCopy(queryOID.Data, 0, result, 0, result.Length);
}
EDIT
So the result member above is a byte array of the "result" of the query. What it means and how you interpret it depends on what the OID you queried was. For example, if you were querying the currently connected SSID (i.e. NDIS_OID.SSID), then that comes back as a 4-byte length followed by the ASCII-encoded name, so you'd decipher it like this:
int len = BitConverter.ToInt32(data, 0);
if (len > 0)
{
ssid = System.Text.Encoding.ASCII.GetString(data, 4, len);
}
But again, this is only for one specific OID. You have to handle every return case for every incoming OID you decide to support.

First you have the wrong translation of your C++ code: the C# equivalent of a C++ char[] is not a string, it's a byte[]. Once you have that, you just need to know, in general, how to copy a structure into a byte array. Here's a compilable example:
using System;
using System.Runtime.InteropServices;
struct Dest
{
public byte[] Data;
}
struct Src
{
public GCHandle StringHandle;
public long A;
public long B;
}
class Program
{
static void Main()
{
Copy();
}
static void Copy()
{
var str = "Hello";
var src = new Src {
A = 3,
B = 4,
StringHandle = GCHandle.Alloc(str, GCHandleType.Normal)
};
var dst = new Dest();
unsafe
{
Src* srcPtr = &src;
dst.Data = new byte[sizeof(Src)];
Marshal.Copy((IntPtr)srcPtr, dst.Data, 0, sizeof(Src));
}
// When you're sure no one can reference the string anymore
// (Including by accessing the data you put in dst.Data!)
src.StringHandle.Free();
}
EDIT: added example of how to deal with reference types such as strings.

Safely, you can't. .NET enforces type safety, which means that you simply can't force a string to be a structure. However, you can look at the data instead of doing unsafe type casts (why are you storing two uints in a string in the first place? And marshalling it as unicode?
First, you'll have to make Data a byte array. It might be possible to do this with a string as well, but that's just adding encoding issues to the mix; if you can, use byte[] instead. Also, if you don't need to have different kinds of data inside (it seems so), you could simply put the two uint fields right inside the struct and it should work just fine:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
public uint DataA;
public uint DataB;
};
The second approach would use a const-sized byte array, long enough to hold the two uints:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = sizeof(ulong))]
public byte[] Data;
};
The first four bytes will be the first uint, the next will be the second.
And of course, you could also use a .NET struct the same way as in the original code - just make sure you use the correct datatype in _NDISUIO_QUERY_OID and it should work automagically.
One point to note though, it seems that the data returned isn't actually necessarily fixed-length. That is quite tricky and it basically means you'd have to deserialize the structure manually based on the pointer and length you get.

Memory alignment of struct variables for p/invoke - Strings lose the last character

I am trying to read records from a Btrieve (v6.15) database by using btrieve API from C# code via P/Invoke.
I have managed to read records, however last character of strings are cropped while reading. If I increase the string size in my data struct then the string is read properly but this time the next variable is not read correctly.
What might be wrong here?
Btrieve function declaration:
[DllImport("WBTRV32.dll", CharSet = CharSet.Ansi)]
static extern short BTRCALL(ushort operation,
[MarshalAs(UnmanagedType.LPArray, SizeConst = 128)] byte[] posBlk,
[MarshalAs(UnmanagedType.Struct, SizeConst = 255)]
ref RecordBuffer databuffer,
ref int dataLength,
[MarshalAs(UnmanagedType.LPArray, SizeConst = 255)] char[] keyBffer,
ushort keyLength, ushort keyNum);
My structure definition:
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct RecordBuffer
{
public short docType;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 15)]
public string docDescPlural;
public short sorting;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 15)]
public string docDescSingle;
public short copyOtherThanSrc;
public double defaultNotebookNo;
}
These are the sizes for the columns, from Btreive database manager:
signed int, 2 bytes
string, 15 bytes
signed int, 2 bytes
string, 15 bytes
signed int, 2 bytes
float, 8 bytes
The code:
private void PopulateAllRecords(string fileName)
{
byte[] positionBlock = new byte[128];
char[] fileNameArray = fileName.ToCharArray();
// Open file
RecordBuffer dataBuffer = new RecordBuffer();
int bufferLength = System.Runtime.InteropServices.Marshal.SizeOf(dataBuffer);
BReturnCodes status = (BReturnCodes) BTRCALL(
BOPEN, positionBlock, ref dataBuffer, ref bufferLength, fileNameArray, 0, 0);
if (status == BReturnCodes.NO_ERROR)
{
// Get first record
dataBuffer = new RecordBuffer();
status = (BReturnCodes) BTRCALL(
BGETFIRST, positionBlock, ref dataBuffer, ref bufferLength, fileNameArray, 0, 0);
if (status == BReturnCodes.NO_ERROR)
{
AddListViewItem(dataBuffer);
}
// Get subsequent records
while (status == BReturnCodes.NO_ERROR) // BReturnCodes.END_OF_FILE or an error will occur
{
dataBuffer = new RecordBuffer();
status = (BReturnCodes)BTRCALL(
BGETNEXT, positionBlock, ref dataBuffer, ref bufferLength, fileNameArray, 0, 0);
if (status == BReturnCodes.NO_ERROR)
{
AddListViewItem(dataBuffer);
}
}
}
else
{
MessageBox.Show("Error occured while opening file: " + status.ToString());
}
}
private void AddListViewItem(RecordBuffer buffer)
{
ListViewItem item = new ListViewItem(buffer.docType.ToString());
item.SubItems.Add(buffer.docDescPlural);
item.SubItems.Add(buffer.sorting.ToString());
item.SubItems.Add(buffer.docDescSingle);
item.SubItems.Add(buffer.copyOtherThanSrc.ToString());
item.SubItems.Add(buffer.defaultNotebookNo.ToString());
listView.Items.Add(item);
}
Edit: Changing the string to char array (thanks to weismat's answer) resolved the problem. Happy!
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct RecordBuffer
{
public short docType;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 15)]
public char[] docDescPlural;
public short sorting;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 15)]
public char[] docDescSingle;
public short copyOtherThanSrc;
public double defaultNotebookNo;
}

I would asume that the issue comes from the \0 which is expected when using a C# string with p/invoke.
I have no experience with Btrieve, but it is pretty likely there is no \0.
Could you post the original c structure? Otherwise you need to use a byte array with a fixed length of 15 and convert it into a string afterwards.
Furthermore I would use the sizeof operator to compare the size of the structure on both ends.