I'm using Avro to serialize objects and then add them to Kafka messages that will be consumed and deserialized by clients. I've tried several different approaches for serialization but none of them seem to embed the schema in the data stream. Here is the latest version of my serialization code. You can see the commented out attempts to use the various writers available.
public static byte[] Serialize<T>(T recordObj) where T : ISpecificRecord
{
Log.Info("Serializing {0} object to Avro.", typeof(T));
try
{
using (var ms = new MemoryStream())
{
var encoder = new BinaryEncoder(ms);
//var writer = new SpecificDefaultWriter(recordObj.Schema);
var writer = new SpecificDatumWriter<T>(recordObj.Schema);
//writer.Write(recordObj.Schema, recordObj, encoder);
writer.Write(recordObj, encoder);
return ms.ToArray();
}
}
catch (Exception ex)
{
Log.Error("Failed to Avro serialize object. {0}", ex);
return null;
}
}
I'm not really sure what else to try.
After digging around in the actual Avro code, I found out I needed a FileWriter, but could not figure out how to instantiate one as DataFileWriter has no public constructor. Turns out there is a static method on the DataFileWriter class called OpenWriter which takes in a DatumWriter and a Stream and returns a DataFileWriter. The code below now properly includes object metadata in the result data stream.
public static byte[] Serialize<T>(T recordObj) where T : ISpecificRecord
{
Log.Info("Serializing {0} object to Avro.",typeof(T));
try
{
using(var ms = new MemoryStream())
{
var specDatumWriter = new SpecificDatumWriter<T>(recordObj.Schema);
var specDataWriter = Avro.File.DataFileWriter<T>.OpenWriter(specDatumWriter, ms);
specDataWriter.Append(recordObj);
specDataWriter.Flush();
specDataWriter.Close();
return ms.ToArray();
}
}
catch(Exception ex)
{
Log.Error("Failed to Avro serialize object. {0}",ex);
return null;
}
}
Related
Currently I creating a save\load system for my game. Its purpose is to be able to convert a Dictionary<string, object> to an encrypted string, and write it to disk file, and vice versa.
Saving functionality works just fine as follows:
public void Save()
{
PrepareSavableData();
using StreamWriter writer = new(SavegamePath);
string rawData = AuxMath.SerializeObjectToString(_storedStates);
string encodedData = AuxMath.Encode(rawData, PublicKey, PrivateKey);
writer.Write(encodedData);
writer.Close();
SavegameCompleted?.Invoke(this, EventArgs.Empty);
}
private void PrepareSavableData()
{
foreach (var entity in _registeredEntities)
{
_storedStates[entity.ID] = entity.GetState();
}
}
I fetch the data from every registered SavableEntitiy as follows:
public object GetState()
{
List<object> states = new(_savables.Count);
foreach (var savable in _savables)
{
states.Add(savable.CaptureState());
}
return states;
}
In the Save() above I use DES algorithm to encrypt data that is previously serialized to a string. No problems with that as well - I tested it thoroughly.
public static string SerializeObjectToString(object obj)
{
using MemoryStream memoryStream = new();
using StreamReader reader = new(memoryStream, Encoding.UTF8);
DataContractSerializer serializer = new(obj.GetType());
serializer.WriteObject(memoryStream, obj);
memoryStream.Position = 0;
return reader.ReadToEnd();
}
public static string Encode(string input, string publicKey, string privateKey)
{
try
{
byte[] inputAsBytes = Encoding.UTF8.GetBytes(input);
byte[] publicKeyAsBytes = Encoding.UTF8.GetBytes(publicKey);
byte[] privateKeyAsBytes = Encoding.UTF8.GetBytes(privateKey);
using DESCryptoServiceProvider cryptoServiceProvider = new();
using MemoryStream memoryStream = new();
using CryptoStream cryptoStream = new(memoryStream,
cryptoServiceProvider.CreateEncryptor(publicKeyAsBytes, privateKeyAsBytes),
CryptoStreamMode.Write);
cryptoStream.Write(inputAsBytes, 0, inputAsBytes.Length);
cryptoStream.FlushFinalBlock();
return Convert.ToBase64String(memoryStream.ToArray());
}
catch (Exception e)
{
return e.Message;
}
}
So fetching the savable data, serializing it, encoding, and writing to a disc file works without a hitch. At the game startup save file loading is performed, decoding, and deserialization, so that later any SavableEntity which registers itself can restore its previous state if one exists in the state dictionary. Reading the save file and decoding it works just fine.
private bool Load()
{
if (File.Exists(SavegamePath))
{
try
{
using StreamReader reader = new(SavegamePath);
string encodedData = reader.ReadToEnd();
string decodedData = AuxMath.Decode(encodedData, PublicKey, PrivateKey);
_storedStates = AuxMath.DeserializeStringToObject<Dictionary<string, object>>(decodedData, _knownSavableTypes);
SavegameLoadSuccesful?.Invoke(this, EventArgs.Empty);
return true;
}
catch (Exception e)
{
SavegameLoadFailed?.Invoke(this, new SavegameLoadFailedEventArgs(e.Message));
return false;
}
}
return false;
}
public static string Decode(string encodedInput, string publicKey, string privateKey)
{
try
{
byte[] encodedInputAsBytes = Convert.FromBase64String(encodedInput);
byte[] publicKeyAsBytes = Encoding.UTF8.GetBytes(publicKey);
byte[] privateKeyAsBytes = Encoding.UTF8.GetBytes(privateKey);
using DESCryptoServiceProvider cryptoServiceProvider = new();
using MemoryStream memoryStream = new();
using CryptoStream cryptoStream = new(memoryStream,
cryptoServiceProvider.CreateDecryptor(publicKeyAsBytes, privateKeyAsBytes),
CryptoStreamMode.Write);
cryptoStream.Write(encodedInputAsBytes, 0, encodedInputAsBytes.Length);
cryptoStream.FlushFinalBlock();
return Encoding.UTF8.GetString(memoryStream.ToArray());
}
catch (Exception e)
{
return e.Message;
}
}
But, when it finally comes to the data deserialization phase from a successfully loaded and decoded file, an error occurs inside exactly this deserialization method right inside the if statement. DataContractSerializer cannot properly deserialize a states dictionary from a memory stream.
public static T DeserializeStringToObject<T>(string objectAsXml, IEnumerable<Type> knownTypes)
{
using MemoryStream memoryStream = new();
using StreamWriter writer = new(memoryStream, Encoding.UTF8);
DataContractSerializer deserializer = new(typeof(T), knownTypes);
writer.Write(objectAsXml);
memoryStream.Position = 0;
if (deserializer.ReadObject(memoryStream) is T value)
{
return value;
}
else
{
throw new Exception("Passed data is invalid or corrupted and cannot be properly restored!");
}
}
Currently this error occurs:
XmlException: Unexpected end of file. Following elements are not closed: Value, KeyValueOfstringanyType, ArrayOfKeyValueOfstringanyType. Line 1, position 197.
It may be something trivial, but I wasn't able to crack it on my own so far. Please, help!
Best regards, Vyacheslav.
I'm working within my PCL library and need to serialise a class and output to a file. I'm very short on space, so don't have the space for PCLStorage.
Currently I'm using this for the serialisation. IFilePath returns a file path from the non-PCL part.
IFilePath FilePath;
public void SerializeObject<T>(T serializableObject, string fileName)
{
if (serializableObject == null) { return; }
try
{
using (var ms = new MemoryStream())
{
var xmlDocument = new XDocument();
using (var writer = xmlDocument.CreateWriter())
{
var serialize = new DataContractSerializer(typeof(T));
serialize.WriteObject(writer, serializableObject);
xmlDocument.Save(ms, SaveOptions.None);
}
}
}
catch (Exception ex)
{
//Log exception here
}
}
When I try to save, nothing is showing. I have a feeling it's because I'm not outputting the stream to a file, but I'm at a loss as how to do this.
You are trying to save to a file, an action which is specific for each platform.
PCLStorage is implementing this functionality for each platform and this is what you will have to do also if you can"t use it.
In you case what you have to do is to create the stream (in each platform) in your non pcl code and then pass it to your function which will look like this:
public void SerializeObject<T>(T serializableObject, Stream fileStream)
{
if (serializableObject == null) { return; }
try
{
var xmlDocument = new XDocument();
using (var writer = xmlDocument.CreateWriter())
{
var serialize = new DataContractSerializer(typeof(T));
serialize.WriteObject(writer, serializableObject);
xmlDocument.Save(fileStream, SaveOptions.None);
}
}
catch (Exception ex)
{
//Log exception here
}
}
more on pcl here.
Problem is that your variable ms in using (var ms = new MemoryStream()) is empty and does not point to any file location of which MemoryStream does not receive a filepath as argument. I propose you use a StreamWriter instead and pass the your FileStream to it. Example
Use your fileName to create a FileStream which inherits from the Stream class then replace the Memory stream with the newly created filestream like this.
using(FileStream stream = File.OpenWrite(fileName))
{
var xmlDocument = new XDocument();
using (var writer = xmlDocument.CreateWriter())
{
var serialize = new DataContractSerializer(typeof(T));
serialize.WriteObject(writer, serializableObject);
xmlDocument.Save(stream, SaveOptions.None);
}
}
Hope this helps.
I have a working XML Serializer which serializes a C# object to an entity in AutoCAD. I'd like to be able to do the same thing but with Binary Serialization for the cases in which XML does not work. So far my serialization method looks like this:
public static void BinarySave(Entity entityToWriteTo, Object objToSerialize, string key = "default")
{
using (MemoryStream stream = new MemoryStream())
{
BinaryFormatter serializer = new BinaryFormatter();
serializer.Serialize(stream, objToSerialize);
stream.Position = 0;
ResultBuffer data = new ResultBuffer();
/*Code to get binary serialization into result buffer*/
using (Transaction tr = db.TransactionManager.StartTransaction())
{
using (DocumentLock docLock = doc.LockDocument())
{
if (!entityToWriteTo.IsWriteEnabled)
{
entityToWriteTo = tr.GetObject(entityToWriteTo.Id, OpenMode.ForWrite) as Entity;
}
if (entityToWriteTo.ExtensionDictionary == ObjectId.Null)
{
entityToWriteTo.CreateExtensionDictionary();
}
using (DBDictionary dict = tr.GetObject(entityToWriteTo.ExtensionDictionary, OpenMode.ForWrite, false) as DBDictionary)
{
Xrecord xrec;
if (dict.Contains(key))
{
xrec = tr.GetObject(dict.GetAt(key), OpenMode.ForWrite) as Xrecord;
xrec.Data = data;
}
else
{
xrec = new Xrecord();
xrec.Data = data;
dict.SetAt(key, xrec);
tr.AddNewlyCreatedDBObject(xrec, true);
}
xrec.Dispose();
}
tr.Commit();
}
data.Dispose();
}
}
}
It's heavily based on my XML Serializer except I have no idea how to get the serialized object into a resultbuffer to be added to the Xrecord of entityToWriteTo.
If XML isn't working for you for some reason, I'd suggest trying a different textual data format such as JSON. The free and open-source JSON formatter Json.NET has some support for situations that can trip up XmlSerializer, including
Dictionaries.
Classes lacking default constructors.
polymorphic types.
Complex data conversion and remapping.
Plus, JSON is quite readable so you may be able to diagnose problems in your data by visual examination.
That being said, you can convert the output stream from BinaryFormatter to a base64 string using the following helper methods:
public static class BinaryFormatterHelper
{
public static string ToBase64String<T>(T obj)
{
using (var stream = new MemoryStream())
{
new BinaryFormatter().Serialize(stream, obj);
return Convert.ToBase64String(stream.GetBuffer(), 0, checked((int)stream.Length)); // Throw an exception on overflow.
}
}
public static T FromBase64String<T>(string data)
{
using (var stream = new MemoryStream(Convert.FromBase64String(data)))
{
var formatter = new BinaryFormatter();
var obj = formatter.Deserialize(stream);
if (obj is T)
return (T)obj;
return default(T);
}
}
}
The resulting string can then be stored in a ResultBuffer as you would store an XML string.
Disclaimer: I did went through most of the solution provided here but most of them were talking about OOM exception while Deserialization.
I am trying to serialize an object( it's a Tree) into Json using Json.Net. Everything works fine for small objects but i get OOM exception when i try it with large objects. As it works with smaller object of same datatype i am assuming there is no circular reference (I did inspect my data structure for it). Is there a way where i can convert my object into stream ( this is a Windows Store app ) and generate the Json using that stream ?
public static async Task<bool> SerializeIntoJson<T>(string fileName, StorageFolder destinationFolder, Content content)
{
ITraceWriter traceWriter = new MemoryTraceWriter();
try
{
string jsonString = JsonConvert.SerializeObject(content, Formatting.Indented, new JsonSerializerSettings
{
PreserveReferencesHandling = PreserveReferencesHandling.Objects,
TypeNameHandling = TypeNameHandling.All,
Error = ReportJsonErrors,
TraceWriter = traceWriter,
StringEscapeHandling = StringEscapeHandling.EscapeNonAscii
});
System.Diagnostics.Debug.WriteLine(traceWriter);
StorageFile file = await destinationFolder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
await Windows.Storage.FileIO.WriteTextAsync(file, jsonString);
return true;
}
catch (NullReferenceException nullException)
{
System.Diagnostics.Debug.WriteLine(traceWriter);
logger.LogError("Exception happened while serializing input object, Error: " + nullException.Message);
return false;
}
catch (Exception e)
{
System.Diagnostics.Debug.WriteLine(traceWriter);
logger.LogError("Exception happened while serializing input object, Error: " + e.Message, e.ToString());
return false;
}
}
In order to convert my object into stream, the code i found out was using a BinaryFormatter which is not available in Windows store app dll's.
It is due to the large amount of records you are trying to serialize, which occupies a large amount of memory. Solutions which I have found for this error is to directly write to the documents using StreamWriter(JsonWriter or TextWriter).
If you have Object use TextWriter:
using (TextWriter textWriter = File.CreateText("LocalJsonFile.json"))
{
var serializer = new JsonSerializer();
serializer.Serialize(textWriter, yourObject);
}
If you have a string, use StringWriter:
StringBuilder sb = new StringBuilder();
StringWriter sw = new StringWriter(sb);
using (JsonWriter textWriter = new JsonTextWriter(sw))
{
var serializer = new JsonSerializer();
serializer.Serialize(textWriter, yourObject);
}
Updated Code based on suggestions in the comments on the question, This works!
public static async Task<bool> SerializeIntoJson<T>(string fileName, StorageFolder destinationFolder, Content content)
{
try
{
StorageFile file = await destinationFolder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
using (var stream = await file.OpenStreamForWriteAsync())
{
StreamWriter writer = new StreamWriter(stream);
JsonTextWriter jsonWriter = new JsonTextWriter(writer);
JsonSerializer ser = new JsonSerializer();
ser.Formatting = Newtonsoft.Json.Formatting.Indented;
ser.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
ser.TypeNameHandling = TypeNameHandling.All;
ser.Error += ReportJsonErrors;
ser.Serialize(jsonWriter, content);
jsonWriter.Flush();
}
return true;
}
catch (NullReferenceException nullException)
{
logger.LogError("Exception happened while serializing input object, Error: " + nullException.Message);
return false;
}
catch (Exception e)
{
logger.LogError("Exception happened while serializing input object, Error: " + e.Message, e.ToString());
return false;
}
}
I am using this code as my IsolatedStorage Helper.
public class IsolatedStorageHelper
{
public const string MyObjectFile = "History.xml";
public static void WriteToXml<T>(T data, string path)
{
// Write to the Isolated Storage
var xmlWriterSettings = new XmlWriterSettings { Indent = true };
try
{
using (var myIsolatedStorage = IsolatedStorageFile.GetUserStoreForApplication())
{
using (var stream = myIsolatedStorage.OpenFile(path, FileMode.Create))
{
var serializer = new XmlSerializer(typeof(T));
using (var xmlWriter = XmlWriter.Create(stream, xmlWriterSettings))
{
serializer.Serialize(xmlWriter, data); //This line generates the exception
}
}
}
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.StackTrace);
//Dispatcher.BeginInvoke(() => MessageBox.Show(ex.StackTrace));
//MessageBox.Show(ex.StackTrace);
}
}
public static T ReadFromXml<T>(string path)
{
T data = default(T);
try
{
using (var myIsolatedStorage = IsolatedStorageFile.GetUserStoreForApplication())
{
using (var stream = myIsolatedStorage.OpenFile(path, FileMode.CreateNew))
{
var serializer = new XmlSerializer(typeof(T));
data = (T)serializer.Deserialize(stream);
}
}
}
catch
{
return default(T);
//add some code here
}
return data;
}
}
I am saving an object of my class PdfFile, which has ImageSource as one of its properties.
But an exception is generated while saving the object and it states System.InvalidOperationException: The type System.Windows.Media.Imaging.BitmapImage was not expected. Use the XmlInclude or SoapInclude attribute to specify types that are not known statically.
I would like to know what this means, and how I would solve this.
Thank you
You're trying to serialize a class that contains a variable / property of type BitmapImage. You can't serialize that type, so you get the mentioned exception. Try to work around by serializing a name or something like this and instantiating the BitmapImage on deserialization from that information.
If you want to save something in iso storage which is not serializable then you can still do so if you write your own serialize method.
Presumably the BitmapImage is not an original part of your app or you would already have the file for it, so I guess it must be a newly acquired bitmap.
Convert your BitmapImage to an array of bytes and you should have no problem.