We Keep Coding

C#/Flatbuffers - How to make a ByteBuffer into byte[] and prefix it with ushort

So basically here is how I do it with C++:
enum ServerOpcode : uint16_t{
SERVER_AUTH_CONNECTION_RESPONSE = 0x001,
SERVER_LOGIN_REQUEST_RESPONSE = 0x002,
SERVER_NUM_MSG_TYPES = 0x003,
};
uint8_t* Vibranium::Packet::PreparePacket(ServerOpcode& serverOpcode, flatbuffers::FlatBufferBuilder& builder) {
size_t size = builder.GetSize();
uint8_t *buf = builder.GetBufferPointer();
uint8_t *actualBuffer = new uint8_t[size + 2];
actualBuffer[1] = (serverOpcode >> 8);
actualBuffer[0] = (serverOpcode&0xFF);
memcpy(actualBuffer + 2, buf, size);
return actualBuffer;
}
I know that uint16_t is exactly 2 bytes and that is why i add +2.
Can someone give example in C# of how can I cast the ByteBuffer to byte[] and than prefix it with:
public enum ServerOpcode : ushort
{
SERVER_AUTH_CONNECTION_RESPONSE = 0x001,
SERVER_LOGIN_REQUEST_RESPONSE = 0x002,
SERVER_NUM_MSG_TYPES = 0x003,
}
In C# I found out that the equivalent of uint16_t is ushort.
So my question is how can I convert ByteBuffer into byte[] and prefix it with ushort?
Can anyone make an answer showing and equivalent of PreparePacket in C# ?
P.S.
Note that I am familiar with the file_identifier but I would like to do that manually. Hope someone can provide an example in C#

Following is the solution:
public static Byte[] PrependServerOpcode(ByteBuffer byteBuffer, ServerOpcode code)
{
var originalArray = byteBuffer.ToArray(0, byteBuffer.Length);
byte[] buffer = new byte[originalArray.Length + 2];
buffer[0] = (byte)((ushort)code / 0x0100);
buffer[1] = (byte)code;
Array.Copy(originalArray, 0, buffer, 2, originalArray.Length);
return buffer;
}
public enum ServerOpcode : ushort
{
SERVER_AUTH_CONNECTION_RESPONSE = 0x001,
SERVER_LOGIN_REQUEST_RESPONSE = 0x002,
SERVER_NUM_MSG_TYPES = 0x003
}
Or alternative:
public static ByteBuffer PrependServerOpcode(ByteBuffer byteBuffer, ServerOpcode code)
{
var originalArray = byteBuffer.ToArray(0, byteBuffer.Length);
byte[] buffer = new byte[originalArray.Length + 2];
buffer[0] = (byte)((ushort)code / 0x0100);
buffer[1] = (byte)code;
Array.Copy(originalArray, 0, buffer, 2, originalArray.Length);
return new ByteBuffer(buffer);
}
Usage:
static void Main(string[] args)
{
var bb = new ByteBuffer(new byte[] { 0x01 });
var result = PrependServerOpcode(bb, ServerOpcode.SERVER_NUM_MSG_TYPES);
}

c# endianness and structure

im trying to get some packet from microcontroller to c# program:
i defined a structure in both places. i calculate crc32 of (sizeof(packet)-4bytes) and place it in mydata_t.crc32 ...
struct mydata_t
{
public byte cmd;
public UInt32 param;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
public Char[] str_buf;
public UInt32 crc32;
}
private byte[] getBytes(mydata_t str)
{
int size = Marshal.SizeOf(str);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(str, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
private mydata_t fromBytes(byte[] arr)
{
mydata_t str = new mydata_t();
str.str_buf = new char[10];
int size = Marshal.SizeOf(str);
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(arr, 0, ptr, size);
str = (mydata_t)Marshal.PtrToStructure(ptr, str.GetType());
Marshal.FreeHGlobal(ptr);
return str;
}
public static UInt32 xcrc32(byte[] buf, int len)
{
UInt32 crc = DefaultSeed;
UInt32 counter = 0;
while (len-- > 0)
{
crc = (crc << 8) ^ defaultTable[((crc >> 24) ^ buf[counter]) & 255];
counter++;
}
return crc;
}
my CRC function takes bytes and length .. so when i recieve data.. "in bytes" i convert them to structure by using (private mydata_t fromBytes(byte[] arr))
The problem:
when i convert to byte array from the structure, calculating CRC is wrong "i believe its because of the endianness of the data types in c#? how do i solve this issue ?
here what i send from microcontroller:
mydata_t datablockTX;
datablockTX.cmd = 2;
datablockTX.param = 0x98765432;
memcpy(datablockTX.str_buf,"hellohell",10);
datablockTX.crc32 = xcrc32((char*) &datablockTX,sizeof(datablockTX) - sizeof(uint32_t));
usb_write_buf((uint8_t*) &datablockTX,sizeof(datablockTX));
here what i recieved and printed:
Data Received:
CMD: 2
param: 2557891634
str: hellohell
crc: 658480750
and then the problem:
public bool ValidateCRC()
{
bool myvalidate;
UInt32 mycrc_val;
byte[] mybytes = getBytes(myblock);
mycrc_val = Crc32.Crc32Algorithm.xcrc32(mybytes, mybytes.Length- sizeof(UInt32));
//mycrc_val = Crc32.Crc32Algorithm.xcrc32(mybytes, 1);
myvalidate = (mycrc_val == myblock.crc32);
Console.WriteLine("c#:" + mycrc_val + " - MCU:" + myblock.crc32 + " - bool:" + myvalidate);
return myvalidate;
}
this is what it prints in console:
c#:667986744 - SAM:658480750 - bool:False
i tried this in MCU:
mydata_t datablockTX;
datablockTX.cmd = 2;
datablockTX.param = 0x98765432;
memcpy(datablockTX.str_buf,"hellohello",10);
//datablockTX.crc32 = xcrc32((char*) &datablockTX,sizeof(datablockTX) - sizeof(uint32_t));
datablockTX.crc32 = xcrc32((char*) &datablockTX, 5);
usb_write_buf((uint8_t*) &datablockTX,sizeof(datablockTX));
here it what i recieved:
CMD: 2
param: 2557891634
str: hellohello
crc: 1993296691
in c#:
byte[] mybytes = getBytes(myblock);
//mycrc_val = Crc32.Crc32Algorithm.xcrc32(mybytes, mybytes.Length- sizeof(UInt32));
mycrc_val = Crc32.Crc32Algorithm.xcrc32(mybytes, 5);
myvalidate = (mycrc_val == myblock.crc32);
Console.WriteLine("c#:" + mycrc_val + " - MCU:" + myblock.crc32 + " - bool:" + myvalidate);
Console:
c#:146416248 - MCU:1993296691 - bool:False

Change
public static UInt32 xcrc32(byte[] buf, int len)
{
UInt32 crc = DefaultSeed;
to
UInt32 crc = 0xff1fff1f;
The DefaultSeed you are using is wrong. (if you want to know, I bruteforced it... Tried all the 4 billion possible seeds). Works for both crcs you gave.

How to read an integer from a byte[]

I have a byte array, and I would like to read an integer from this array. How can I do it?
Something like this:
int i;
tab = new byte[32];
i = readint(tab,0,3); // i = int from tab[0] to tab[3] (int = 4 bytes?)
i = readint(tab,4,7);
etc...

byte[] bytes = { 0, 0, 0, 25 };
// If the system architecture is little-endian (that is, little end first),
// reverse the byte array.
if (BitConverter.IsLittleEndian)
Array.Reverse(bytes);
int i = BitConverter.ToInt32(bytes, 0);
Console.WriteLine("int: {0}", i);
Ref: How to: Convert a byte Array to an int

Also, there is a class called Endian in Jon Skeet's miscutil library which implements conversion methods between a byte array and various primitive types, taking endianness into account.
For your question, usage would be something like:
// Input data
byte[] tab = new byte[32];
// Pick the appropriate endianness
Endian endian = Endian.Little;
// Use the appropriate endian to convert
int a = endian.ToInt32(tab, 0);
int b = endian.ToInt32(tab, 4);
int c = endian.ToInt32(tab, 8);
int d = endian.ToInt32(tab, 16);
...
A simplified version of the Endian class would be something like:
public abstract class Endian
{
public short ToInt16(byte[] value, int startIndex)
{
return unchecked((short)FromBytes(value, startIndex, 2));
}
public int ToInt32(byte[] value, int startIndex)
{
return unchecked((int)FromBytes(value, startIndex, 4));
}
public long ToInt64(byte[] value, int startIndex)
{
return FromBytes(value, startIndex, 8);
}
// This same method can be used by int16, int32 and int64.
protected virtual long FromBytes(byte[] buffer, int startIndex, int len);
}
And then the FromBytes abstract method is implemented differently for each endian type.
public class BigEndian : Endian
{
protected override long FromBytes(byte[] buffer, int startIndex, int len)
{
long ret = 0;
for (int i=0; i < len; i++)
{
ret = unchecked((ret << 8) | buffer[startIndex+i]);
}
return ret;
}
}
public class LittleEndian : Endian
{
protected override long FromBytes(byte[] buffer, int startIndex, int len)
{
long ret = 0;
for (int i=0; i < len; i++)
{
ret = unchecked((ret << 8) | buffer[startIndex+len-1-i]);
}
return ret;
}
}

You could use BitConverter.ToInt32. Have a look at this.

If you wanted to do it manually, something like that should do the trick!
byte[] data = ...;
int startIndex = 0;
int value = data[startIndex];
for (int i=1;i<4;i++)
{
value <<= 8;
value |= data[i+startIndex];
}

Read or convert to Int32 from TWO byte arrays

I have the 4 bytes that represent an integer stored in 2 separate byte arrays. I would like to convert these into an Int32 WITHOUT copying to a third byte array and reading that using memorystream.
The reason the data is split across two byte arrays is because this is a simplified example of my issue which involves huge amounts of data that cannot fit into a single bytearray.
Is there any way to achieve this? I do not wish to concatenate the two byte arrays into a thrid because of the performance implications which are critical to me.
Moon

You can use a struct layout like this
[StructLayout(LayoutKind.Explicit, Size=4)]
struct UnionInt32Value
{
[FieldOffset(0)] public byte byte1;
[FieldOffset(1)] public byte byte2;
[FieldOffset(2)] public byte byte3;
[FieldOffset(3)] public byte byte4;
[FieldOffset(0)] public Int32 iVal;
}
Assign your bytes in the correct order then read your Int32 from iVal;
EDIT: Sample code
using System;
using System.Runtime.InteropServices;
namespace Test
{
class Program
{
[StructLayout(LayoutKind.Explicit, Size=4)]
struct UnionInt32Value
{
[FieldOffset(0)] public byte byte1;
[FieldOffset(1)] public byte byte2;
[FieldOffset(2)] public byte byte3;
[FieldOffset(3)] public byte byte4;
[FieldOffset(0)] public Int32 iVal;
}
public static void Main(string[] args)
{
UnionInt32Value v = new UnionInt32Value();
v.byte1=1;
v.byte2=0;
v.byte3=0;
v.byte4=0;
Console.WriteLine("this is one " + v.iVal);
v.byte1=0xff;
v.byte2=0xff;
v.byte3=0xff;
v.byte4=0xff;
Console.WriteLine("this is minus one " + v.iVal);
Console.Write("Press any key to continue . . . ");
Console.ReadKey(true);
}
}
}

Something like this?
int x = (array1[index] << 16) + array2[index];
Of course, you didn't specify a language, but that's the gist of it.

The BitConverter class is intended for this:
byte[] parts = { byte1, byte2, byte3, byte4 };
int value = BitConverter.ToInt32(parts, 0);

You can use BitConverter twice, like:
byte[] bytes0 = new byte[] { 255, 255 };
byte[] bytes1 = new byte[] { 0, 0 };
int res = BitConverter.ToInt16(bytes0, 0) << 16;
res |= BitConverter.ToUInt16(bytes1, 0);
Which yields -65536 (0b11111111 11111111 00000000 00000000)
If your integer parts isn't at position 0 in the array, you just replace the 0 in ToUint16 to change the position.
Little extension method:
public static class BitConverterExt
{
public static int ToInt32(byte[] arr0, int index0, byte[] arr1, int index1)
{
int partRes = BitConverter.ToInt16(arr1, index1) << 16;
return partRes | BitConverter.ToUInt16(arr0, index0);
}
}
Usage:
byte[] bytes0 = new byte[] { 0x0, 0xA };
byte[] bytes1 = new byte[] { 0x64, 0xFF };
int res = BitConverterExt.ToInt32(bytes0, 0, bytes1, 0);
//Res -10221056 (0xFF640A00)

If I understand correctly, you are having a problem whilst reading across the boundary of the two arrays. If that is so, this routine will read an integer anywhere in the two arrays, even if it is across the two of them.
int ReadInteger(byte[] array1, byte[] array2, int offset)
{
if (offset < 0 || (offset + 4) > (array1.Length + array2.Length))
throw new ArgumentOutOfRangeException();
if (offset <= (array1.Length - 4))
return BitConverter.ToInt32(array1, offset);
else if (offset >= array1.Length)
return BitConverter.ToInt32(array2, offset - array1.Length);
else
{
var buffer = new byte[4];
var numFirst = array1.Length - offset;
Array.Copy(array1, offset, buffer, 0, numFirst);
Array.Copy(array2, 0, buffer, numFirst, 4 - numFirst);
return BitConverter.ToInt32(buffer, 0);
}
}
Note: depending on how your integers are stored, you might want to change the order in which bytes are copied.

C# unsafe value type array to byte array conversions

I use an extension method to convert float arrays into byte arrays:
public static unsafe byte[] ToByteArray(this float[] floatArray, int count)
{
int arrayLength = floatArray.Length > count ? count : floatArray.Length;
byte[] byteArray = new byte[4 * arrayLength];
fixed (float* floatPointer = floatArray)
{
fixed (byte* bytePointer = byteArray)
{
float* read = floatPointer;
float* write = (float*)bytePointer;
for (int i = 0; i < arrayLength; i++)
{
*write++ = *read++;
}
}
}
return byteArray;
}
I understand that an array is a pointer to memory associated with information on the type and number of elements. Also, it seems to me that there is no way of doing a conversion from and to a byte array without copying the data as above.
Have I understood this? Would it even be impossible to write IL to create an array from a pointer, type and length without copying data?
EDIT: Thanks for the answers, I learned some fundamentals and got to try out new tricks!
After initially accepting Davy Landman's answer I found out that while his brilliant StructLayout hack does convert byte arrays into float arrays, it does not work the other way around. To demonstrate:
[StructLayout(LayoutKind.Explicit)]
struct UnionArray
{
[FieldOffset(0)]
public Byte[] Bytes;
[FieldOffset(0)]
public float[] Floats;
}
static void Main(string[] args)
{
// From bytes to floats - works
byte[] bytes = { 0, 1, 2, 4, 8, 16, 32, 64 };
UnionArray arry = new UnionArray { Bytes = bytes };
for (int i = 0; i < arry.Bytes.Length / 4; i++)
Console.WriteLine(arry.Floats[i]);
// From floats to bytes - index out of range
float[] floats = { 0.1f, 0.2f, 0.3f };
arry = new UnionArray { Floats = floats };
for (int i = 0; i < arry.Floats.Length * 4; i++)
Console.WriteLine(arry.Bytes[i]);
}
It seems that the CLR sees both arrays as having the same length. If the struct is created from float data, the byte array's length is just too short.

You can use a really ugly hack to temporary change your array to byte[] using memory manipulation.
This is really fast and efficient as it doesn't require cloning the data and iterating on it.
I tested this hack in both 32 & 64 bit OS, so it should be portable.
The source + sample usage is maintained at https://gist.github.com/1050703 , but for your convenience I'll paste it here as well:
public static unsafe class FastArraySerializer
{
[StructLayout(LayoutKind.Explicit)]
private struct Union
{
[FieldOffset(0)] public byte[] bytes;
[FieldOffset(0)] public float[] floats;
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct ArrayHeader
{
public UIntPtr type;
public UIntPtr length;
}
private static readonly UIntPtr BYTE_ARRAY_TYPE;
private static readonly UIntPtr FLOAT_ARRAY_TYPE;
static FastArraySerializer()
{
fixed (void* pBytes = new byte[1])
fixed (void* pFloats = new float[1])
{
BYTE_ARRAY_TYPE = getHeader(pBytes)->type;
FLOAT_ARRAY_TYPE = getHeader(pFloats)->type;
}
}
public static void AsByteArray(this float[] floats, Action<byte[]> action)
{
if (floats.handleNullOrEmptyArray(action))
return;
var union = new Union {floats = floats};
union.floats.toByteArray();
try
{
action(union.bytes);
}
finally
{
union.bytes.toFloatArray();
}
}
public static void AsFloatArray(this byte[] bytes, Action<float[]> action)
{
if (bytes.handleNullOrEmptyArray(action))
return;
var union = new Union {bytes = bytes};
union.bytes.toFloatArray();
try
{
action(union.floats);
}
finally
{
union.floats.toByteArray();
}
}
public static bool handleNullOrEmptyArray<TSrc,TDst>(this TSrc[] array, Action<TDst[]> action)
{
if (array == null)
{
action(null);
return true;
}
if (array.Length == 0)
{
action(new TDst[0]);
return true;
}
return false;
}
private static ArrayHeader* getHeader(void* pBytes)
{
return (ArrayHeader*)pBytes - 1;
}
private static void toFloatArray(this byte[] bytes)
{
fixed (void* pArray = bytes)
{
var pHeader = getHeader(pArray);
pHeader->type = FLOAT_ARRAY_TYPE;
pHeader->length = (UIntPtr)(bytes.Length / sizeof(float));
}
}
private static void toByteArray(this float[] floats)
{
fixed(void* pArray = floats)
{
var pHeader = getHeader(pArray);
pHeader->type = BYTE_ARRAY_TYPE;
pHeader->length = (UIntPtr)(floats.Length * sizeof(float));
}
}
}
And the usage is:
var floats = new float[] {0, 1, 0, 1};
floats.AsByteArray(bytes =>
{
foreach (var b in bytes)
{
Console.WriteLine(b);
}
});

Yes, the type information and data is in the same memory block, so that is impossible unless you overwrite the type information in a float array to fool the system that it's byte array. That would be a really ugly hack, and could easily blow up...
Here's how you can convert the floats without unsafe code if you like:
public static byte[] ToByteArray(this float[] floatArray) {
int len = floatArray.Length * 4;
byte[] byteArray = new byte[len];
int pos = 0;
foreach (float f in floatArray) {
byte[] data = BitConverter.GetBytes(f);
Array.Copy(data, 0, byteArray, pos, 4);
pos += 4;
}
return byteArray;
}

This question is the reverse of What is the fastest way to convert a float[] to a byte[]?.
I've answered with a union kind of hack to skip the whole copying of the data. You could easily reverse this (length = length *sizeof(Double).

I've written something similar for quick conversion between arrays. It's basically an ugly proof-of-concept more than a handsome solution. ;)
public static TDest[] ConvertArray<TSource, TDest>(TSource[] source)
where TSource : struct
where TDest : struct {
if (source == null)
throw new ArgumentNullException("source");
var sourceType = typeof(TSource);
var destType = typeof(TDest);
if (sourceType == typeof(char) || destType == typeof(char))
throw new NotSupportedException(
"Can not convert from/to a char array. Char is special " +
"in a somewhat unknown way (like enums can't be based on " +
"char either), and Marshal.SizeOf returns 1 even when the " +
"values held by a char can be above 255."
);
var sourceByteSize = Buffer.ByteLength(source);
var destTypeSize = Marshal.SizeOf(destType);
if (sourceByteSize % destTypeSize != 0)
throw new Exception(
"The source array is " + sourceByteSize + " bytes, which can " +
"not be transfered to chunks of " + destTypeSize + ", the size " +
"of type " + typeof(TDest).Name + ". Change destination type or " +
"pad the source array with additional values."
);
var destCount = sourceByteSize / destTypeSize;
var destArray = new TDest[destCount];
Buffer.BlockCopy(source, 0, destArray, 0, sourceByteSize);
return destArray;
}
}

public byte[] ToByteArray(object o)
{
int size = Marshal.SizeOf(o);
byte[] buffer = new byte[size];
IntPtr p = Marshal.AllocHGlobal(size);
try
{
Marshal.StructureToPtr(o, p, false);
Marshal.Copy(p, buffer, 0, size);
}
finally
{
Marshal.FreeHGlobal(p);
}
return buffer;
}
this may help you to convert an object to a byte array.

You should check my answer to a similar question: What is the fastest way to convert a float[] to a byte[]?.
In it you'll find portable code (32/64 bit compatible) to let you view a float array as a byte array or vice-versa, without copying the data. It's the fastest way that I know of to do such thing.
If you're just interested in the code, it's maintained at https://gist.github.com/1050703 .

Well - if you still interested in that hack - check out this modified code - it works like a charm and costs ~0 time, but it may not work in future since it's a hack allowing to gain full access to the whole process address space without trust requirements and unsafe marks.
[StructLayout(LayoutKind.Explicit)]
struct ArrayConvert
{
public static byte[] GetBytes(float[] floats)
{
ArrayConvert ar = new ArrayConvert();
ar.floats = floats;
ar.length.val = floats.Length * 4;
return ar.bytes;
}
public static float[] GetFloats(byte[] bytes)
{
ArrayConvert ar = new ArrayConvert();
ar.bytes = bytes;
ar.length.val = bytes.Length / 4;
return ar.floats;
}
public static byte[] GetTop4BytesFrom(object obj)
{
ArrayConvert ar = new ArrayConvert();
ar.obj = obj;
return new byte[]
{
ar.top4bytes.b0,
ar.top4bytes.b1,
ar.top4bytes.b2,
ar.top4bytes.b3
};
}
public static byte[] GetBytesFrom(object obj, int size)
{
ArrayConvert ar = new ArrayConvert();
ar.obj = obj;
ar.length.val = size;
return ar.bytes;
}
class ArrayLength
{
public int val;
}
class Top4Bytes
{
public byte b0;
public byte b1;
public byte b2;
public byte b3;
}
[FieldOffset(0)]
private Byte[] bytes;
[FieldOffset(0)]
private object obj;
[FieldOffset(0)]
private float[] floats;
[FieldOffset(0)]
private ArrayLength length;
[FieldOffset(0)]
private Top4Bytes top4bytes;
}