I am looking for a way to serialize Avro to a byte array in Avro C# library. There is a link to do for Avro Java library as described in following link from Avro documentation:
https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Serializingtoabytearray
Code copied from above link:
ByteArrayOutputStream out = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
DatumWriter<User> writer = new SpecificDatumWriter<User>(User.getClassSchema());
writer.write(user, encoder);
encoder.flush();
out.close();
byte[] serializedBytes = out.toByteArray();
But I have not found a way to do in Avro c# library. I am basically looking for c# equivalent of above code.
You can use these methods to convert to and from an object to a byte array or vice-versa. Code extracted from https://stackoverflow.com/a/18205093/6138713
// Convert an object to a byte array
private byte[] ObjectToByteArray(Object obj)
{
if(obj == null)
return null;
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, obj);
return ms.ToArray();
}
// Convert a byte array to an Object
private Object ByteArrayToObject(byte[] arrBytes)
{
MemoryStream memStream = new MemoryStream();
BinaryFormatter binForm = new BinaryFormatter();
memStream.Write(arrBytes, 0, arrBytes.Length);
memStream.Seek(0, SeekOrigin.Begin);
Object obj = (Object) binForm.Deserialize(memStream);
return obj;
}
Maybe something as follows, I used the following code to write to a Kineses Stream
public async Task RecordAsync(ISpecificRecord record, string partitionKey)
{
using (var ms = new MemoryStream())
{
var encoder = new BinaryEncoder(ms);
var writer = new SpecificDefaultWriter(record.Schema);
writer.Write(record, encoder);
// AWS Kineses
var putRecordRequest = new PutRecordRequest
{
StreamName = _streamName,
Data = ms,
PartitionKey = partitionKey
};
await _kinesis.PutRecordAsync(putRecordRequest);
}
}
or
public byte[] Serialize(ISpecificRecord record)
{
using (var ms = new MemoryStream())
{
var encoder = new BinaryEncoder(ms);
var writer = new SpecificDefaultWriter(record.Schema);
writer.Write(record, encoder);
return ms.ToArray();
}
}
Related
I had an ASP.net app in which we serialized a string content using BinaryFormatter like this :
BinaryFormatter bf = new BinaryFormatter();
try
{
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, Content);
var SerializedContent = ms.GetBuffer();
return SerializedContent;
}
// ...
But now that BinaryFormatter has become obsolete I am getting exception. I have been trying to use XMLSerializer but haven't been able to make it work. This is what I have been trying:
XmlSerializer xmlSerializer = new XmlSerializer(Content.GetType());
using (MemoryStream ms = new MemoryStream())
{
xmlSerializer.Serialize(ms, Content);
var SerializedContent = ms.GetBuffer();
return SerializedContent;
}
// ...
But I don't get the correct output. What am I doing wrong?
I've been trying this for a long time but it keeps giving me an error. I have an array of bytes that should represent a nbt document. I would like to convert this into a c# object with a library: fNbt.
Here is my code:
byte[] buffer = Convert.FromBase64String(value);
byte[] decompressed;
using (var inputStream = new MemoryStream(buffer))
{
using var outputStream = new MemoryStream();
using (var gzip = new GZipStream(inputStream, CompressionMode.Decompress, leaveOpen: true))
{
gzip.CopyTo(outputStream);
}
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
var output = reader.ReadValueAs<AuctionItem>(); //Error: Value already read, or no value to read.
return output;
}
When I try this, it works:
decompressed = outputStream.ToArray();
outputStream.Seek(0, SeekOrigin.Begin);
outputStream.Read(new byte[1000], 0, decompressed.Count() - 1);
But when I try this, it doesn't:
outputStream.Seek(0, SeekOrigin.Begin);
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
reader.ReadValueAs<AuctionItem>();
NbtReader, like most stream readers, begins reading from the current position of whatever stream you give it. Since you're just done writing to outputStream, then that position is the stream's end. Which means at that point there's nothing to be read.
The solution is to seek the outputStream back to the beginning before reading from it:
outputStream.Seek(0, SeekOrigin.Begin); // <-- seek to the beginning
// Do the read
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
var output = reader.ReadValueAs<AuctionItem>(); // No error anymore
return output;
The solution is as follows. NbtReader.ReadValueAs does not consider a nbtCompound or nbtList as value. I made this little reader but it is not done yet (I will update the code once it is done).
public static T ReadValueAs<T>(string value) where T: new()
{
byte[] buffer = Convert.FromBase64String(value);
using (var inputStream = new MemoryStream(buffer))
{
using var outputStream = new MemoryStream();
using (var gzip = new GZipStream(inputStream, CompressionMode.Decompress, leaveOpen: true))
{
gzip.CopyTo(outputStream);
}
outputStream.Seek(0, SeekOrigin.Begin);
return new EasyNbt.NbtReader(outputStream).ReadValueAs<T>();
}
}
This is the NbtReader:
private MemoryStream MemStream { get; set; }
public NbtReader(MemoryStream memStream)
{
MemStream = memStream;
}
public T ReadValueAs<T>() where T: new()
{
return ReadTagAs<T>(new fNbt.NbtReader(MemStream, true).ReadAsTag());
}
private T ReadTagAs<T>(fNbt.NbtTag nbtTag)
{
//Reads to the root and adds to T...
}
I'm trying to convert/cast an image source to an object. I tried to convert to a byte prior to object.
ImageSource oImageSelectedByUser = (from ii in mySource where ii.PhotoImgStream != null select ii.PhotoImgStream).FirstOrDefault();
var oImageAsByte = await ConvertStreamtoByteAsync(oImageSelectedByUser);
var oImageAsObject = FromByteArray<object>(oImageAsByte);
private static async Task<byte[]> ConvertStreamtoByteAsync(ImageSource imageSource)
{
byte[] buffer = new byte[16 * 1024];
try
{
if (imageSource is FileImageSource)
{
FileImageSource objFileImageSource = (FileImageSource)imageSource;
string strFileName = objFileImageSource.File;
var webClient = new WebClient();
buffer = await webClient.DownloadDataTaskAsync(new Uri(strFileName));
return buffer;
}
}
catch (Exception ex)
{
buffer = null;
}
return buffer;
}
public T FromByteArray<T>(byte[] data)
{
if (data == null)
return default(T);
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream(data))
{
object obj = bf.Deserialize(ms);
return (T)obj;
}
}
When trying to convert to object, the error I get is:
Binary stream '0' does not contain a valid BinaryHeader. Possible
causes are invalid stream or object version change between
serialization and deserialization.
I wanted to ask is there a better way to do this?
Any direction/pointers appreciated.
You could modify the code like following
public T FromByteArray<T>(byte[] data)
{
if (data == null)
return default(T);
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream(data))
{
ms.Seek(0, SeekOrigin.Begin); // /rewinded the stream to the begining.
object obj = bf.Deserialize(ms);
return (T)obj;
}
}
I'm writing a program that needs to send data from a client to a server over a TCP socket.
I have the following code that serialises and desirialises the object.
public static byte[] serializeObjects(Object obj)
{
using (MemoryStream ms = new MemoryStream()) {
IFormatter bf = new BinaryFormatter();
bf.Serialize(ms, obj);
byte[] bytes = ms.ToArray();
return bytes;
}
}
public static Object deserialiseObject(byte[] bytes)
{
using (MemoryStream ms = new MemoryStream()) {
IFormatter bf = new BinaryFormatter();
ms.Write(bytes, 0, bytes.Length);
ms.Seek(0, SeekOrigin.Begin);
var obj = bf.Deserialize(ms);
return obj;
}
}
The code that sends the the serialised data from the server from the client.
while ((userInput = Console.ReadLine()) != null)
{
byte[] serialisedObject;
serialisedObject = serializeObjects(userInput);
byte[] userDataBytes = BitConverter.GetBytes((Int32)serialisedObject.Length);
_writer.Write(userDataBytes);
_writer.Write(serialisedObject);
_writer.Flush();
And the code that receives the serialised data from the stream.
NetworkStream stream = new NetworkStream(socket, true);
StreamWriter writer = new StreamWriter(stream, Encoding.UTF8);
byte[] readMsgLeng = new byte[4];
stream.Read(readMsgLeng, 0, 4);
int dataLength = BitConverter.ToInt32(readMsgLeng, 0);
byte[] messageData = new byte[dataLength];
stream.Read(messageData, 0, dataLength);
returnedObject = deserialiseObject(messageData);
The error that is occuring is as follows "The input stream is not a valid binary format. The starting contents (in bytes) are:......"
Not sure what it is that is causing the error.
I want to make a binary serialize of an object and the result to save it in a database.
Person person = new Person();
person.Name = "something";
MemoryStream memorystream = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(memorystream, person);
How can I transform memorystream in a string type to be saved in database, and after this to be able to deserialize the object?
What you're really asking for is a safe way of representing arbitrary binary data as text and then converting it back again. The fact that it stores a serialized object is irrelevant.
The answer is almost to use Base 64 (e.g. Convert.ToBase64String and Convert.FromBase64String). Do not use Encoding.UTF8.GetString or anything similar - your binary data is not encoded text data, and shouldn't be treated as such.
However, does your database not have a data type for binary data? Check for BLOB, IMAGE and BINARY types...
Here's the sample. TData must be marked [Serializable] and all fields type also.
private static TData DeserializeFromString<TData>(string settings)
{
byte[] b = Convert.FromBase64String(settings);
using (var stream = new MemoryStream(b))
{
var formatter = new BinaryFormatter();
stream.Seek(0, SeekOrigin.Begin);
return (TData)formatter.Deserialize(stream);
}
}
private static string SerializeToString<TData>(TData settings)
{
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, settings);
stream.Flush();
stream.Position = 0;
return Convert.ToBase64String(stream.ToArray());
}
}
//-------write to database-------------------------
Person person = new Person();
person.name = "Firstnm Lastnm";
MemoryStream memorystream = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(memorystream, person);
byte[] yourBytesToDb = memorystream.ToArray();
//here you write yourBytesToDb to database
//----------read from database---------------------
//here you read from database binary data into yourBytesFromDb
MemoryStream memorystreamd = new MemoryStream(yourBytesFromDb);
BinaryFormatter bfd = new BinaryFormatter();
Person deserializedperson = bfd.Deserialize(memorystreamd) as Person;
I used something like this
MemoryStream memoryStream = new MemoryStream();
BinaryFormatter binaryFormatter = new BinaryFormatter();
binaryFormatter.Serialize(memoryStream, Person);
memoryStream.Flush();
memoryStream.Position = 0;
string value = Convert.ToBase64String(memoryStream.ToArray());
Basically, don't save the data as string to the database, there are blob fields available to store binary data.
If you really need to have the data as string, you'll need to convert your byte[] to a string using base64 encoding, and to grab the byte[] from a string use decoding.
Have you not looked into converting the memorystream into a base64hex string to be put into the database?
byte[] mStream = memorystream.ToArray();
string sConvertdHex = System.Convert.ToBase64String(mStream)
Then you can dump the contents sConvertdHex to the database. To deserialize it you need to do the reverse
byte[] mData = System.Convert.FromBase64String(...)
then deserialize mData back to your object.