We Keep Coding

How to convert IStream(C++) to System::IO::Stream(c#) and vice versa

I have 2 libraries one is c++ and the other one c#.
Name of C++ library->LibA
Name of C#->LibB
In LibA, 2 main APIs will be there:
Serialize-> Serialize API will generate IStream as output with the given inputs.
Deserialize-> Deserialize API will take IStream as input and deserializes the stream and gets actual data from it.
#pragma once
struct GPosition
{
double x;
double y;
double z;
};
struct GUnitVector
{
double x;
double y;
double z;
};
struct GLine
{
GPosition m_rootPoint; /* Point on the line, should be in region of interest */
GUnitVector m_direction; /* Gradient */
double m_start; /* Parameter of start point, must be <= m_end */
double m_end; /* Parameter of end point */
};
class GraphicSerializer
{
public:
GUID objectID;
uint32_t aspectID;
uint32_t controlFlag;
vector<const GLine *> geomVector;
void Serialize(IStream &pStream);
void Deserialize(IStream* pStream);
};
In LibB, 4 APIs will be there:
Object GetObjectFromStream(Stream s)-> Takes stream as input and deserializes it and returns Objects
PutToDB(Object A)-> persists the given object to DB
Stream GetStreamFromObject(Object a)-> Takes object as input serializes it and returns it.
Object GetFromDB(objectID)-> Gets the object from DB based on id
public class CommonBase
{
public Guid id { get; set; };
public byte[] Bytes { get; set; } //contains aspect, control flag, vec<GLine>
};
public interface IGraphicRepository
{
CommonBase Get(Guid guid);
bool Put(CommonBase graphic);
}
public static class GraphicStreamUtility
{
public static CommonBase GetCommonBaseFromStream(Stream stream);
public static void SerializeCommonBaseToStream(CommonBase obj, Stream stream);
}
Now I'm writing C++/CLI to use stream generated by libA in libB and vice versa. So that I can persist and retrieve the objects to and from DB.
Can anyone please let me know how to convert IStream to .Net Stream and .Net stream to IStream.

Stream^ CGDMStreamCLI::ConvertIStreamToStream(IStream *pStream)
{
ULARGE_INTEGER ulSize{};
IStream_Size(pStream, &ulSize);
size_t size = ulSize.QuadPart;
auto buffer = std::make_unique<char[]>(size);
ULONG numRead{};
HRESULT hr = pStream->Read(buffer.get(), size, &numRead);
if (FAILED(hr) || size != numRead)
{
throw std::exception("Failed to read the stream");
}
cli::array<Byte>^ byteArray = gcnew cli::array<Byte>(numRead+2);
// convert native pointer to System::IntPtr with C-Style cast
Marshal::Copy((IntPtr)buffer.get(), byteArray, 0, numRead);
return gcnew System::IO::MemoryStream(byteArray);
}

How to use UnmanagedMemoryAccessor.ReadArray<T> (Int64, T[], Int32, Int32)

I want to read/write data to Memory Mapped File .How do I use UnmanagedMemoryAccessor.ReadArray (Int64, T[], Int32, Int32)
struct Data{public int a; public int b; public byte[] ;}
static Data _sdata = new Data();
static Data _mydata = new Data();
_mdata.byte = _sdata.byte = new byte[2] ;
_mmf = MemoryMappedFile.CreateNew( "test", 10);
var ired = _mmf.CreateViewAccessor();
ired.Read( 0, out mdata );here has Error
So, what is the T struct should correct ??

The array cannot be a reference it has to be contained within the struct. You have to enable unsafe code for your assembly and use the following struct definition :
[StructLayout(LayoutKind.Sequential, Pack = 1)]
unsafe struct Data {
public int a;
public int b;
public fixed byte bytes[100];
}

Generalize deserializing from byte array

I am parsing a byte array containing different type values stored in a fixed format. For example first 4 bytes could be an int containing size of an array -
let's say the array of doubles so next 8 bytes represent a double - the first element of the array etc. It could in theory contain values of other types, but let's say we can only have
bool,int,uint,short,ushort,long,ulong,float,double and arrays of each of these. Simple approach:
public class FixedFormatParser
{
private byte[] _Contents;
private int _CurrentPos = 0;
public FixedFormatParser(byte[] contents)
{
_Contents = contents;
}
bool ReadBool()
{
bool res = BitConverter.ToBoolean(_Contents, _CurrentPos);
_CurrentPos += sizeof(bool);
return res;
}
int ReadInt()
{
int res = BitConverter.ToInt32(_Contents, _CurrentPos);
_CurrentPos += sizeof(int);
return res;
}
// etc. for uint, short, ushort, long, ulong, float, double
int[] ReadIntArray()
{
int size = ReadInt();
if (size == 0)
return null;
int[] res = new int[size];
for (int i = 0; i < size; i++)
res[i] = ReadInt();
return res;
}
// etc. for bool, uint, short, ushort, long, ulong, float, double
}
I can obviously write 18 methods to cover each case, but seems like there should be a way to generalize this.
bool val = Read<bool>();
long[] arr = ReadArray<long>(); // or ReadArray(Read<long>);
Obviously I don't mean write 2 wrappers in addition to the 18 methods to allow for this syntax. The syntax is not important, the code duplication is the issue. Another consideration is the performance. Ideally there would not be any (or much) of a performance hit. Thanks.
Update:
Regarding other questions that are supposedly duplicates. I disagree as none of them addressed the particular generalization I am after, but one came pretty close:
First answer in
C# Reading Byte Array
described wrapping BinaryReader. This would cover 9 of the 18 methods. So half of the problem is addressed. I still would need to write all of the various array reads.
public class FixedFormatParser2 : BinaryReader
{
public FixedFormatParser2(byte[] input) : base(new MemoryStream(input))
{
}
public override string ReadString()
{
//
}
public double[] ReadDoubleArray()
{
int size = ReadInt32();
if (size == 0)
return null;
double[] res = new double[size];
for (int i = 0; i < size; i++)
res[i] = ReadDouble();
return res;
}
}
How do I not write a separate ReadXXXArray for each of the types?
Nearest I got to it:
public void WriteCountedArray(dynamic[] input)
{
if (input == null || input.Length == 0)
Write((int)0);
else
{
Write(input.Length);
foreach (dynamic val in input)
Write(val);
}
}
This compiles but calling it is cumbersome and expensive :
using (FixedFormatWriter writer = new FixedFormatWriter())
{
double[] array = new double[3];
// ... assign values
writer.WriteCountedArray(array.Select(x=>(dynamic)x).ToArray());

I like doing like this
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data;
using System.Xml;
using System.Xml.Serialization;
using System.IO;
namespace ConsoleApplication50
{
class Program
{
static void Main(string[] args)
{
new Format();
}
}
public class Format
{
public enum TYPES
{
INT,
INT16,
LONG
}
public static List<Format> format = new List<Format>() {
new Format() { name = "AccountNumber", _type = TYPES.INT ,numberOfBytes = 4},
new Format() { name = "Age", _type = TYPES.INT16 ,numberOfBytes = 2},
new Format() { name = "AccountNumber", _type = TYPES.LONG ,numberOfBytes = 8}
};
public Dictionary<string, object> dict = new Dictionary<string, object>();
public string name { get; set; }
public TYPES _type { get; set; }
public int numberOfBytes { get; set; }
public Format() { }
public Format(byte[] contents)
{
MemoryStream stream = new MemoryStream(contents);
BinaryReader reader = new BinaryReader(stream);
foreach (Format item in format)
{
switch (item._type)
{
case TYPES.INT16 :
dict.Add(item.name, reader.ReadInt16());
break;
case TYPES.INT:
dict.Add(item.name, reader.ReadInt32());
break;
case TYPES.LONG:
dict.Add(item.name, reader.ReadInt64());
break;
}
}
}
}
}

Fixed Object to Byte Array

I have a simple object that looks like this:
public class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
}
I tried this code I found somewhere on the net:
public byte[] ObjectToByteArray(Object obj)
{
MemoryStream fs = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, obj);
byte[] rval = fs.ToArray();
fs.Close();
return rval;
}
But somehow the returning byte array has a size of 248 bytes.
I would expect it to be 4 bytes x 4 fields = 16 bytes.
QUESTION:
What's the cleanest way to convert a fixed object into a byte array?
And should the resulting array be 16 bytes in size in this case?

BinaryFormatter saves a lot of type information to be able to deserialize properly. If you want compact serialization or to comunicate via some strict protocol, you will have to do it explictly like this:
public byte[] ToByteArray()
{
List<byte> result = new List<byte>();
result.AddRange(BitConverter.GetBytes(One));
result.AddRange(BitConverter.GetBytes(Two));
result.AddRange(BitConverter.GetBytes(Three));
result.AddRange(BitConverter.GetBytes(Four));
return result.ToArray();
}
here I convert each of your UInt32 into byte array and store it in resulting array.
Edit
Turns out there is another way by using struct and Marshal
First you make struct and mark it with attributes like that:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct MyStruct
{
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 50)]
public string StringField;
public int IntField;
}
Here LayoutKind.Sequential tells clr to keep fields in memory in same order as declaration. Without Pack = 1 structures can take more memory than required. Like struct with one short field and one byte needs only 3 bytes, but by default it's size will most likely be 4 (processor has instructions for manipulating single byte, 2 bytes and 4 bytes, clr sacrifices one byte per instance of struct to reduce amount of instructions of machine code by half). Now you can use Marshal to copy bytes:
public static byte[] GetBytes<T>(T str)
{
int size = Marshal.SizeOf(str);
var bytes = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
try
{
Marshal.StructureToPtr(str, ptr, true);
Marshal.Copy(ptr, bytes, 0, size);
return bytes;
}
finally
{
Marshal.FreeHGlobal(ptr);
}
}
Everything works fine with simple types. For complex types such as string you'll have to use MarshalAs attribute and it's using is a bit more complicated (In example I told clr to marshal string as array of fixed 50 bytes size).

Here's another way...which way is best is a matter of opinion probably. I like Atomosk's answer. Combine that answer with cast operator overloads and you've got a pretty elegant solution.
class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
static public implicit operator byte[](Foo f)
{
MemoryStream m = new MemoryStream(16);
BinaryWriter w = new BinaryWriter(m);
w.Write(f.One);
w.Write(f.Two);
w.Write(f.Three);
w.Write(f.Four);
w.Close();
m.Close();
return m.ToArray();
}
static public explicit operator Foo(byte[] b)
{
Foo f = new Foo();
MemoryStream m = new MemoryStream(b);
BinaryReader r = new BinaryReader(m);
f.One = r.ReadUInt32();
f.Two = r.ReadUInt32();
f.Three = r.ReadUInt32();
f.Four = r.ReadUInt32();
r.Close();
m.Close();
return f;
}
}
class Program
{
static void Main(string[] args)
{
Foo f = new Foo() { One = 1, Two = 2, Three = 3, Four = 4 };
byte[] b = (byte[])f;
Console.WriteLine(b.Length);
f = (Foo)b;
Console.WriteLine("{0} {1} {2} {3}", f.One, f.Two, f.Three, f.Four);
Console.ReadKey(true);
}
}

Keep in mind that when you serialise an object with the BinaryFormatter, you're including things like type information. That results in the overhead you're seeing.
If you want to serialise an object into (in this case) just 16 bytes then you'll need to do it manually with something like a StreamWriter.
If size isn't an issue then a Binaryformatter in this case isn't wrong and doesn't take much code to do it, but it's not the most memory efficient way of doing it.
Edit: Here's how you'd do it with a StreamWriter
System.IO.MemoryStream stream = new System.IO.MemoryStream();
StreamWriter writer = new StreamWriter(stream);
writer.Write(myObject.One); // here's where we actually write the data to the stream
writer.Write(myObject.Two);
writer.Write(myObject.Three);
writer.Write(myObject.Four);
writer.Flush(); // make sure all the data in the stream writer ends up in the
// underlying stream
byte[] result = stream.ToArray(); // here's your resulting byte array
stream.Dispose(); // don't forget to dispose of the stream!

Here is one way to do it manually which will guarantee the 16-bytes.
using System;
using System.IO;
using System.Linq;
public class Foo
{
public UInt32 One { get; set; }
public UInt32 Two { get; set; }
public UInt32 Three { get; set; }
public UInt32 Four { get; set; }
public Foo() {}
public Foo(byte[] array)
{
One = BitConverter.ToUInt32(array,0);
Two = BitConverter.ToUInt32(array,4);
Three = BitConverter.ToUInt32(array,8);
Four = BitConverter.ToUInt32(array,12);
}
public byte[] toByteArray()
{
byte[] retVal = new byte[16];
byte[] b = BitConverter.GetBytes(One);
Array.Copy(b, 0, retVal, 0, 4);
b = BitConverter.GetBytes(Two);
Array.Copy(b, 0, retVal, 4, 4);
b = BitConverter.GetBytes(Three);
Array.Copy(b, 0, retVal, 8, 4);
b = BitConverter.GetBytes(Four);
Array.Copy(b, 0, retVal, 12, 4);
return retVal;
}
}
public class P{
public static void Main(string[] args) {
Foo foo = new Foo();
foo.One = 1;
foo.Two = 2;
foo.Three = 3;
foo.Four = 4;
byte[] arr = foo.toByteArray();
Console.WriteLine(arr.Length);
Foo bar = new Foo(arr);
Console.WriteLine(string.Format("{0} {1} {2} {3}", bar.One, bar.Two, bar.Three, bar.Four));
}
}
Output:
16
1 2 3 4

ProtoBuf corrupts byte array during deserialization (extra 0's added)

I'm using ProtoBuf.NET to serialize/deserialize some classes. I'm finding that on deserializing, I'm getting a corrupt byte[] (extra 0's). Before you ask, yes, I need the *WithLengthPrefix() versions of the ProtoBuf API since the ProtoBuf portion is at the starts of a custom stream :)
Anyway, I see
Original object is (JSON depiction):
{"ByteArray":"M+B6q+PXNuF8P5hl","ByteArray2":"DmErxVQ2y87IypSRcfxcWA==","K":2,"V
":1.0}
Protobuf: Raw Hex (42 bytes):
29-2A-20-0A-0C-33-E0-7A-AB-E3-D7-36-E1-7C-3F-98-65-12-10-0E-61-2B-C5-54-36-CB-CE
-C8-CA-94-91-71-FC-5C-58-08-02-15-00-00-80-3F
Regenerated object is (JSON depiction):
{"ByteArray":"AAAAAAAAAAAAAAAAM+B6q+PXNuF8P5hl","ByteArray2":"DmErxVQ2y87IypSRcf
xcWA==","K":2,"V":1.0}
The extra AAA*A in the ByteArray member are basically hex 0x00's in base64.
The app logic is similar to
static void Main(string[] args)
{
var parent = new Parent();
parent.Init();
Console.WriteLine("\nOriginal object is (JSON depiction):");
Console.WriteLine(JsonConvert.SerializeObject(parent));
using (var ms = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(ms, parent, PrefixStyle.Base128);
byte[] bytes2 = ms.ToArray();
var hex2 = BitConverter.ToString(bytes2);
Console.WriteLine("\nProtobuf: Hex ({0} bytes):\n{1}", bytes2.Length, hex2);
ms.Seek(0, SeekOrigin.Begin);
var backFirst = Serializer.DeserializeWithLengthPrefix<Parent>(ms,PrefixStyle.Base128);
Console.WriteLine("\nRegenerated object is (JSON depiction):");
Console.WriteLine(JsonConvert.SerializeObject(backFirst));
}
}
The DTO classes are
[DataContract]
[ProtoContract]
internal class Parent : Child
{
[DataMember(Name = "ByteArray", Order = 10)]
[ProtoMember(1)]
public byte[] ByteArray { get; set; }
[DataMember(Name = "ByteArray2", Order = 30, EmitDefaultValue = false)]
[ProtoMember(2)]
public byte[] ByteArray2 { get; set; }
public Parent()
{
ByteArray = new byte[12];
}
internal void Init(bool bindRow = false)
{
base.Init();
var rng = new RNGCryptoServiceProvider();
rng.GetBytes(ByteArray);
ByteArray2 = new byte[16];
rng.GetBytes(ByteArray2);
}
}
[DataContract]
[ProtoContract]
[ProtoInclude(5, typeof(Parent))]
public class Child
{
[DataMember(Name = "K", Order = 100)]
[ProtoMember(1)]
public Int32 K { get; set; }
[DataMember(Name = "V", Order = 110)]
[ProtoMember(2)]
public float V { get; set; }
internal void Init()
{
K = 2;
V = 1.0f;
}
}
I do see that when I move ByteArray = new byte[12] out form the Parent constructor into it's Init() method, ProtoBuf works fine. However we have app logic that prevents that in the real version (vs the SO trim down code you see above).
Are we doing something wrong or is this a bug in ProtoBuf?

Here we go:
public Parent()
{
ByteArray = new byte[12];
}
Note: protobuf is designed (by google) to be both appendable and mergeable. Where append / merge is synonymous (for lists / arrays etc) with "append".
Two options (both possible via attributes):
disable the constructor: [ProtoContract(SkipConstructor = true)]
disable the append: [ProtoMember(1, OverwriteList = true)]
There are other options too, but those are the ones I'd lean towards.
You remark that the array initialisation is different in the real code, but: I can't comment on code I can't see.

I was facing the same issue and my Bytestring data was actually an XML data I get from the server, so in my application I was already having an XML Serialaizer, thus I decided to use that instead of introducing a new serializer for Photobuf and decorate all of my models, I found this task is very time consuming.
Here is my function which will deserialize my Bytestring.
public SceneDTO ParseSceneAsyncFromByteString(ByteString byteStringData) {
var serializer = new CustomXmlSerializer(typeof(SceneDTO), _timingManager);
serializer.UnknownNode += OnUnknownNode;
serializer.UnknownAttribute += OnUnknownAttribute;
var ms = new MemoryStream(byteStringData.ToByteArray());
var scene = (SceneDTO) serializer.Deserialize(ms);
serializer.UnknownNode -= OnUnknownNode;
serializer.UnknownAttribute -= OnUnknownAttribute;
return scene;
}
The other reason why I was following this is I was getting the below error after used the Photobuf serializer.
Unexpected end-group in source data; this usually means the source
data is corrupt photobuf deserialzer
And even after fixing this, my model data was always null, and I decided not to spend many hours in fixing that issue as I already had a working solution.