I have a simple object that looks like this:
public class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
}
I tried this code I found somewhere on the net:
public byte[] ObjectToByteArray(Object obj)
{
MemoryStream fs = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, obj);
byte[] rval = fs.ToArray();
fs.Close();
return rval;
}
But somehow the returning byte array has a size of 248 bytes.
I would expect it to be 4 bytes x 4 fields = 16 bytes.
QUESTION:
What's the cleanest way to convert a fixed object into a byte array?
And should the resulting array be 16 bytes in size in this case?
BinaryFormatter saves a lot of type information to be able to deserialize properly. If you want compact serialization or to comunicate via some strict protocol, you will have to do it explictly like this:
public byte[] ToByteArray()
{
List<byte> result = new List<byte>();
result.AddRange(BitConverter.GetBytes(One));
result.AddRange(BitConverter.GetBytes(Two));
result.AddRange(BitConverter.GetBytes(Three));
result.AddRange(BitConverter.GetBytes(Four));
return result.ToArray();
}
here I convert each of your UInt32 into byte array and store it in resulting array.
Edit
Turns out there is another way by using struct and Marshal
First you make struct and mark it with attributes like that:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct MyStruct
{
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 50)]
public string StringField;
public int IntField;
}
Here LayoutKind.Sequential tells clr to keep fields in memory in same order as declaration. Without Pack = 1 structures can take more memory than required. Like struct with one short field and one byte needs only 3 bytes, but by default it's size will most likely be 4 (processor has instructions for manipulating single byte, 2 bytes and 4 bytes, clr sacrifices one byte per instance of struct to reduce amount of instructions of machine code by half). Now you can use Marshal to copy bytes:
public static byte[] GetBytes<T>(T str)
{
int size = Marshal.SizeOf(str);
var bytes = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
try
{
Marshal.StructureToPtr(str, ptr, true);
Marshal.Copy(ptr, bytes, 0, size);
return bytes;
}
finally
{
Marshal.FreeHGlobal(ptr);
}
}
Everything works fine with simple types. For complex types such as string you'll have to use MarshalAs attribute and it's using is a bit more complicated (In example I told clr to marshal string as array of fixed 50 bytes size).
Here's another way...which way is best is a matter of opinion probably. I like Atomosk's answer. Combine that answer with cast operator overloads and you've got a pretty elegant solution.
class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
static public implicit operator byte[](Foo f)
{
MemoryStream m = new MemoryStream(16);
BinaryWriter w = new BinaryWriter(m);
w.Write(f.One);
w.Write(f.Two);
w.Write(f.Three);
w.Write(f.Four);
w.Close();
m.Close();
return m.ToArray();
}
static public explicit operator Foo(byte[] b)
{
Foo f = new Foo();
MemoryStream m = new MemoryStream(b);
BinaryReader r = new BinaryReader(m);
f.One = r.ReadUInt32();
f.Two = r.ReadUInt32();
f.Three = r.ReadUInt32();
f.Four = r.ReadUInt32();
r.Close();
m.Close();
return f;
}
}
class Program
{
static void Main(string[] args)
{
Foo f = new Foo() { One = 1, Two = 2, Three = 3, Four = 4 };
byte[] b = (byte[])f;
Console.WriteLine(b.Length);
f = (Foo)b;
Console.WriteLine("{0} {1} {2} {3}", f.One, f.Two, f.Three, f.Four);
Console.ReadKey(true);
}
}
Keep in mind that when you serialise an object with the BinaryFormatter, you're including things like type information. That results in the overhead you're seeing.
If you want to serialise an object into (in this case) just 16 bytes then you'll need to do it manually with something like a StreamWriter.
If size isn't an issue then a Binaryformatter in this case isn't wrong and doesn't take much code to do it, but it's not the most memory efficient way of doing it.
Edit: Here's how you'd do it with a StreamWriter
System.IO.MemoryStream stream = new System.IO.MemoryStream();
StreamWriter writer = new StreamWriter(stream);
writer.Write(myObject.One); // here's where we actually write the data to the stream
writer.Write(myObject.Two);
writer.Write(myObject.Three);
writer.Write(myObject.Four);
writer.Flush(); // make sure all the data in the stream writer ends up in the
// underlying stream
byte[] result = stream.ToArray(); // here's your resulting byte array
stream.Dispose(); // don't forget to dispose of the stream!
Here is one way to do it manually which will guarantee the 16-bytes.
using System;
using System.IO;
using System.Linq;
public class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
public Foo() {}
public Foo(byte[] array)
{
One = BitConverter.ToUInt32(array,0);
Two = BitConverter.ToUInt32(array,4);
Three = BitConverter.ToUInt32(array,8);
Four = BitConverter.ToUInt32(array,12);
}
public byte[] toByteArray()
{
byte[] retVal = new byte[16];
byte[] b = BitConverter.GetBytes(One);
Array.Copy(b, 0, retVal, 0, 4);
b = BitConverter.GetBytes(Two);
Array.Copy(b, 0, retVal, 4, 4);
b = BitConverter.GetBytes(Three);
Array.Copy(b, 0, retVal, 8, 4);
b = BitConverter.GetBytes(Four);
Array.Copy(b, 0, retVal, 12, 4);
return retVal;
}
}
public class P{
public static void Main(string[] args) {
Foo foo = new Foo();
foo.One = 1;
foo.Two = 2;
foo.Three = 3;
foo.Four = 4;
byte[] arr = foo.toByteArray();
Console.WriteLine(arr.Length);
Foo bar = new Foo(arr);
Console.WriteLine(string.Format("{0} {1} {2} {3}", bar.One, bar.Two, bar.Three, bar.Four));
}
}
Output:
16
1 2 3 4
Related
I'm trying to marshal a class instance from and to byte array using these methods:
Marshal to byte array:
public static byte[] MakeArrayFromCat(Cat cat)
{
var size = Marshal.SizeOf(typeof(Cat));
var bytes = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(cat, ptr, false);
Marshal.Copy(ptr, bytes, 0, size);
Marshal.FreeHGlobal(ptr);
return bytes;
}
Marshal to class instance:
public static Cat MakeCatFromArray(byte[] b)
{
var size = b.Length;
var bytes = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(bytes, 0, ptr, size);
var cat = (Cat)Marshal.PtrToStructure(ptr, typeof(Cat));
Marshal.FreeHGlobal(ptr);
return cat;
}
Here's my class:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public class Cat
{
public string Name { get; set; }
public int Weight { get; set; }
public int Age { get; set; }
public Cat (string name, int age, int weight)
{
Name = name;
Age = age;
Weight = weight;
}
}
So when I run the presented methods the program throws MissingMethodException meaning that there's no constructor that takes no arguements. I get it on this line (in MakeCatFromArray method):
var cat = (Cat)Marshal.PtrToStructure(ptr, typeof(Cat));
I've tried to add a default constructor, but what I get from it doesn't live up to what I expect: It gives me an instance with all its properties empty, although I marshalled an instance with non-default values. Thus I've got two questions:
1.How do I do marshalling properly in this case?
2.Is there any better way of converting an instance of a class into byte array without marking it as [Serializable] ?
The problem is with this:
Marshal.Copy(bytes, 0, ptr, size);
You are trying to create Cat from empty bytes array.
There should be b instead of bytes:
Marshal.Copy(b, 0, ptr, size);
I am parsing a byte array containing different type values stored in a fixed format. For example first 4 bytes could be an int containing size of an array -
let's say the array of doubles so next 8 bytes represent a double - the first element of the array etc. It could in theory contain values of other types, but let's say we can only have
bool,int,uint,short,ushort,long,ulong,float,double and arrays of each of these. Simple approach:
public class FixedFormatParser
{
private byte[] _Contents;
private int _CurrentPos = 0;
public FixedFormatParser(byte[] contents)
{
_Contents = contents;
}
bool ReadBool()
{
bool res = BitConverter.ToBoolean(_Contents, _CurrentPos);
_CurrentPos += sizeof(bool);
return res;
}
int ReadInt()
{
int res = BitConverter.ToInt32(_Contents, _CurrentPos);
_CurrentPos += sizeof(int);
return res;
}
// etc. for uint, short, ushort, long, ulong, float, double
int[] ReadIntArray()
{
int size = ReadInt();
if (size == 0)
return null;
int[] res = new int[size];
for (int i = 0; i < size; i++)
res[i] = ReadInt();
return res;
}
// etc. for bool, uint, short, ushort, long, ulong, float, double
}
I can obviously write 18 methods to cover each case, but seems like there should be a way to generalize this.
bool val = Read<bool>();
long[] arr = ReadArray<long>(); // or ReadArray(Read<long>);
Obviously I don't mean write 2 wrappers in addition to the 18 methods to allow for this syntax. The syntax is not important, the code duplication is the issue. Another consideration is the performance. Ideally there would not be any (or much) of a performance hit. Thanks.
Update:
Regarding other questions that are supposedly duplicates. I disagree as none of them addressed the particular generalization I am after, but one came pretty close:
First answer in
C# Reading Byte Array
described wrapping BinaryReader. This would cover 9 of the 18 methods. So half of the problem is addressed. I still would need to write all of the various array reads.
public class FixedFormatParser2 : BinaryReader
{
public FixedFormatParser2(byte[] input) : base(new MemoryStream(input))
{
}
public override string ReadString()
{
//
}
public double[] ReadDoubleArray()
{
int size = ReadInt32();
if (size == 0)
return null;
double[] res = new double[size];
for (int i = 0; i < size; i++)
res[i] = ReadDouble();
return res;
}
}
How do I not write a separate ReadXXXArray for each of the types?
Nearest I got to it:
public void WriteCountedArray(dynamic[] input)
{
if (input == null || input.Length == 0)
Write((int)0);
else
{
Write(input.Length);
foreach (dynamic val in input)
Write(val);
}
}
This compiles but calling it is cumbersome and expensive :
using (FixedFormatWriter writer = new FixedFormatWriter())
{
double[] array = new double[3];
// ... assign values
writer.WriteCountedArray(array.Select(x=>(dynamic)x).ToArray());
I like doing like this
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data;
using System.Xml;
using System.Xml.Serialization;
using System.IO;
namespace ConsoleApplication50
{
class Program
{
static void Main(string[] args)
{
new Format();
}
}
public class Format
{
public enum TYPES
{
INT,
INT16,
LONG
}
public static List<Format> format = new List<Format>() {
new Format() { name = "AccountNumber", _type = TYPES.INT ,numberOfBytes = 4},
new Format() { name = "Age", _type = TYPES.INT16 ,numberOfBytes = 2},
new Format() { name = "AccountNumber", _type = TYPES.LONG ,numberOfBytes = 8}
};
public Dictionary<string, object> dict = new Dictionary<string, object>();
public string name { get; set; }
public TYPES _type { get; set; }
public int numberOfBytes { get; set; }
public Format() { }
public Format(byte[] contents)
{
MemoryStream stream = new MemoryStream(contents);
BinaryReader reader = new BinaryReader(stream);
foreach (Format item in format)
{
switch (item._type)
{
case TYPES.INT16 :
dict.Add(item.name, reader.ReadInt16());
break;
case TYPES.INT:
dict.Add(item.name, reader.ReadInt32());
break;
case TYPES.LONG:
dict.Add(item.name, reader.ReadInt64());
break;
}
}
}
}
}
Ok, In C++ I could create a struct object that contains a bunch of value types and when needed write it out to a binary file in a single write statement like this:
struct DataFileHeader {
unsigned short Id;
unsigned short Type;
unsigned int Count;
unsigned int Offset;
};
...
struct DataFileHeader dataFileHeader;
...
rc = _write(fileHandle, &dataFileHeader, 12);
Is there any way to do that in c#? I've converted my structs to classes in c# and am working with the BinaryFormatter and trying to serialize the object but that adds a bunch of text and other stuff to the stream. I just want to write out the all the value fields in the object. Is this possible or do I have to write out each field on the object separatly?
void Main()
{
var fileHeader = new DataFileHeader()
{
Count = 10,
Id = 10,
Offset = 10,
Type = 10
};
Serialize(fileHeader).Dump();
}
// Define other methods and classes here
public byte[] Serialize<TObject>(TObject obj)
{
BinaryFormatter binaryFormatter = new BinaryFormatter();
byte[] ret = null;
using (MemoryStream ms = new MemoryStream())
{
binaryFormatter.Serialize(ms, obj);
ret = ms.ToArray();
}
return ret;
}
[Serializable]
public struct DataFileHeader
{
public short Id;
public short Type;
public int Count;
public int Offset;
};
If that doesn't work I think doing it individually might be the only way.
BitConverter.GetBytes(fileHeader.Count).Concat(BitConverter.GetBytes(fileHeader.Count)).Concat(BitConverter.GetBytes(fileHeader.Count)).Concat(BitConverter.GetBytes(fileHeader.Count)).Dump();
I'm using ProtoBuf.NET to serialize/deserialize some classes. I'm finding that on deserializing, I'm getting a corrupt byte[] (extra 0's). Before you ask, yes, I need the *WithLengthPrefix() versions of the ProtoBuf API since the ProtoBuf portion is at the starts of a custom stream :)
Anyway, I see
Original object is (JSON depiction):
{"ByteArray":"M+B6q+PXNuF8P5hl","ByteArray2":"DmErxVQ2y87IypSRcfxcWA==","K":2,"V
":1.0}
Protobuf: Raw Hex (42 bytes):
29-2A-20-0A-0C-33-E0-7A-AB-E3-D7-36-E1-7C-3F-98-65-12-10-0E-61-2B-C5-54-36-CB-CE
-C8-CA-94-91-71-FC-5C-58-08-02-15-00-00-80-3F
Regenerated object is (JSON depiction):
{"ByteArray":"AAAAAAAAAAAAAAAAM+B6q+PXNuF8P5hl","ByteArray2":"DmErxVQ2y87IypSRcf
xcWA==","K":2,"V":1.0}
The extra AAA*A in the ByteArray member are basically hex 0x00's in base64.
The app logic is similar to
static void Main(string[] args)
{
var parent = new Parent();
parent.Init();
Console.WriteLine("\nOriginal object is (JSON depiction):");
Console.WriteLine(JsonConvert.SerializeObject(parent));
using (var ms = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(ms, parent, PrefixStyle.Base128);
byte[] bytes2 = ms.ToArray();
var hex2 = BitConverter.ToString(bytes2);
Console.WriteLine("\nProtobuf: Hex ({0} bytes):\n{1}", bytes2.Length, hex2);
ms.Seek(0, SeekOrigin.Begin);
var backFirst = Serializer.DeserializeWithLengthPrefix<Parent>(ms,PrefixStyle.Base128);
Console.WriteLine("\nRegenerated object is (JSON depiction):");
Console.WriteLine(JsonConvert.SerializeObject(backFirst));
}
}
The DTO classes are
[DataContract]
[ProtoContract]
internal class Parent : Child
{
[DataMember(Name = "ByteArray", Order = 10)]
[ProtoMember(1)]
public byte[] ByteArray { get; set; }
[DataMember(Name = "ByteArray2", Order = 30, EmitDefaultValue = false)]
[ProtoMember(2)]
public byte[] ByteArray2 { get; set; }
public Parent()
{
ByteArray = new byte[12];
}
internal void Init(bool bindRow = false)
{
base.Init();
var rng = new RNGCryptoServiceProvider();
rng.GetBytes(ByteArray);
ByteArray2 = new byte[16];
rng.GetBytes(ByteArray2);
}
}
[DataContract]
[ProtoContract]
[ProtoInclude(5, typeof(Parent))]
public class Child
{
[DataMember(Name = "K", Order = 100)]
[ProtoMember(1)]
public Int32 K { get; set; }
[DataMember(Name = "V", Order = 110)]
[ProtoMember(2)]
public float V { get; set; }
internal void Init()
{
K = 2;
V = 1.0f;
}
}
I do see that when I move ByteArray = new byte[12] out form the Parent constructor into it's Init() method, ProtoBuf works fine. However we have app logic that prevents that in the real version (vs the SO trim down code you see above).
Are we doing something wrong or is this a bug in ProtoBuf?
Here we go:
public Parent()
{
ByteArray = new byte[12];
}
Note: protobuf is designed (by google) to be both appendable and mergeable. Where append / merge is synonymous (for lists / arrays etc) with "append".
Two options (both possible via attributes):
disable the constructor: [ProtoContract(SkipConstructor = true)]
disable the append: [ProtoMember(1, OverwriteList = true)]
There are other options too, but those are the ones I'd lean towards.
You remark that the array initialisation is different in the real code, but: I can't comment on code I can't see.
I was facing the same issue and my Bytestring data was actually an XML data I get from the server, so in my application I was already having an XML Serialaizer, thus I decided to use that instead of introducing a new serializer for Photobuf and decorate all of my models, I found this task is very time consuming.
Here is my function which will deserialize my Bytestring.
public SceneDTO ParseSceneAsyncFromByteString(ByteString byteStringData) {
var serializer = new CustomXmlSerializer(typeof(SceneDTO), _timingManager);
serializer.UnknownNode += OnUnknownNode;
serializer.UnknownAttribute += OnUnknownAttribute;
var ms = new MemoryStream(byteStringData.ToByteArray());
var scene = (SceneDTO) serializer.Deserialize(ms);
serializer.UnknownNode -= OnUnknownNode;
serializer.UnknownAttribute -= OnUnknownAttribute;
return scene;
}
The other reason why I was following this is I was getting the below error after used the Photobuf serializer.
Unexpected end-group in source data; this usually means the source
data is corrupt photobuf deserialzer
And even after fixing this, my model data was always null, and I decided not to spend many hours in fixing that issue as I already had a working solution.
This is the C++ Message that i need to pass from C#:
struct LoginMessage:public NMessage
{
char szUser[ 16 ];
char szPass[ 16 ];
LoginMessage()
{
msgId = CTRL_SESSION_LOGIN;
msgSize = sizeof( LoginMessage );
}
};
struct NMessage
{
DWORD msgSize;
union
{
DWORD msgId;
struct
{
BYTE msgId0;
BYTE msgId1;
BYTE msgId2;
BYTE msgId3;
};
};
NMessage()
{
}
BOOL IsLegal()
{
return msgSize>=sizeof(NMessage) && msgSize
What is the equivalent of this in C# so that C++ can understand this message?
Sample code is greatly appreciated.
To answer my own question..
[StructLayout(LayoutKind.Sequential)]
public struct LoginMessage
{
public int msgSize;
public int msgId;
[MarshalAsAttribute(UnmanagedType.ByValTStr, SizeConst = 16)]
public string szUser;
[MarshalAsAttribute(UnmanagedType.ByValTStr, SizeConst = 16)]
public string szPass;
}
It is important to take note of the sequence of the class properties so that the byte array you send to c++ is exactly the same as what it expects.
Have a look at this. it's a nice marshaling guid.
Take a look at the StructLayout attribute on MSDN. There is a good example on that page on how to create a struct that can be marshalled correctly. You can also look at http://www.pinvoke.net to see different examples of how the Windows API structs are defined to be marshaled. Once you have the struct defined right, you should be able to use Marshal.StructureToPtr or Marshal.PtrToStructure. Here's a short article on it: http://geekswithblogs.net/taylorrich/archive/2006/08/21/88665.aspx
I usually manually serialize data using something like this.
class NMessage {
byte[] buffer;
//ctor to create a new message for sending.
public NMessage(int nSize) {
buffer = new byte[nSize];
Buffer.BlockCopy(BitConverter.GetBytes(nSize), 0, buffer, 0, sizeof(UInt32));
}
//ctor to create msg from received data.
publix NMessage(byte[] buffer) {
this.buffer = buffer;
}
public UInt32 MessageId {
get { return BitConverter.ToUInt32(buffer, 4);
set { Buffer.BlockCopy(BitConverter.GetBytes(value), 0, buffer, 4, sizeof(UInt32)); }
}
...
public Byte MessageId2 {
get { return buffer[6]; }
set { buffer[6] = value; }
}
...
public UInt32 Size {
get { return BitConverter.ToUInt32(buffer, 0) }
}
public Byte[] Buffer {
get { return buffer; }
}
}
class LoginMessage : NMessage {
Encoding encoding = new ASCIIEncoding(); //or whatever encoding you need.
public LoginMessage() : base(16 + 16) {
this.MessageId = CTRL_SESSION_LOGIN;
}
public LoginMessage(NMessage message) : base(message.Buffer) {
}
public string User {
get { return encoding.GetString(buffer, 8, 16); }
set { Buffer.BlockCopy(encoding.GetBytes(value), 0, buffer, 8, 16);
}
public string Pass {
get { return encoding.GetString(buffer, 24, 16); }
set { Buffer.BlockCopy(encoding.GetBytes(value), 0, buffer, 24, 16);
}
}
So you simply create you new message, set it's data, then send the Buffer property.
Send((new LoginMessage {
User = "user",
Pass = "pass",
}).Buffer);
On the receiving end, if you have enough data, you can build the message from the byte[] you receive.
byte[] recvBuf = new byte[MAX_RECV];
int recvSize = Receive(recvBuf);
...
var message = new NMessage(recvBuf);
if (message.MessageId == CTRL_SESSION_LOGIN)
var login = new LoginMessage(message);
var user = login.User;
...
That's just a rough overview anyway, the code obviously needs a cleanup as I reduced it to only demonstrate the idea.