We Keep Coding

protobuf-net v3 with surrogate and inheritance hierarchies

I'm struggling migrating from protobuf-net v2.4.6 to v3.0.100 (or any 3.0.x) in regards to an existing type hierarchy used as ProtoContracts with one of the subtypes requiring a surrogate due to one of its property being of type object.
With previous configuration in place, I get the following exception thrown on creating the runtime model:
System.InvalidOperationException: 'Types with surrogates cannot be used in inheritance hierarchies'
Hence, my question is how to properly deal with this scenario using protobuf-net 3.0.x?
Here's my (over-)simplified repro of the issue:
class Program
{
static void Main(string[] args)
{
var model = RuntimeTypeModel.Create();
_ = model[typeof(Base)]; // <-- InvalidOperationException thrown here
Base value = new Complex();
var copy = model.DeepClone(value);
}
}
[ProtoContract]
[ProtoInclude(1, typeof(Simple))]
[ProtoInclude(2, typeof(Complex))]
public abstract class Base
{
}
[ProtoContract]
public class Simple : Base
{
}
[ProtoContract(Surrogate = typeof(ComplexSurrogate))]
public class Complex : Base
{
}
[ProtoContract(Name = nameof(Complex))]
public class ComplexSurrogate
{
[ProtoConverter]
public static ComplexSurrogate Convert(Complex source) => new ComplexSurrogate();
[ProtoConverter]
public static Complex Convert(ComplexSurrogate source) => new Complex();
}
As a side note: When compiling protobuf-net from source with the above mentioned exception suppressed, I'm able to defined a surrogate for the Base class which seems to serve as a workaround.

Right now, that scenario isn't supported. While reworking the code for v3, some ambiguous outcomes/intents were found, and it needs work to go in and figure out what the correct outcomes are in each case, design how to achieve that, implement it, and test it. That time has not yet been found, so right now it is safer to prevent a configuration that could lead to big problems downstream, than to just shrug and assume that whatever happens is correct. It is on my list of things to do, but: ultimately this is a project that comes entirely out of my own spare time - it isn't sponsored or part of my paid work, so: it'll get there when it gets there.

I encountered the same error in protobuf v3, and I solved that with custom serializer.
My base class is
[ProtoContract]
[ProtoInclude(500, typeof(XXXRequest))]
[ProtoInclude(501, typeof(XXXResponse))]
// ...
public class MessageBase
{
[ProtoMember(1)]
long ID { get; internal set; }
[ProtoMember(3)]
int ExecutionMilliseconds { get; set; }
}
Its equivalent proto is
message Message {
int64 ID = 1;
int32 ExecutionMilliseconds = 3;
oneof body {
PredictBonusRequest XXXRequest = 500;
PredictBonusResponse XXXResponse = 501;
// ...
}
}
I want to replace some types (e.g. XXXResponse) to use the contract-first class instead. This would allow us to migrate from code-first to contract-first smoothly.
For sub-types should be surrogated, we create custom serializer as below.
using ProtoBuf;
using ProtoBuf.Serializers;
using UnderlyingMessage = GeneratedProto.Contract.Message;
using UnderlyingResponse = GeneratedProto.Contract.XXXResponse;
[DataContract]
[Serializable]
[ProtoContract(Serializer = typeof(XXXResponseSerializer))]
public class XXXResponse : MessageBase
{
class XXXResponseSerializer : ISerializer<XXXResponse>
{
public SerializerFeatures Features => SerializerFeatures.CategoryMessage | SerializerFeatures.WireTypeString;
public XXXResponse Read(ref ProtoReader.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
return serializer.Read(ref state, value);
}
public void Write(ref ProtoWriter.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
serializer.Write(ref state, value);
}
}
private readonly UnderlyingResponse _resp;
public XXXResponse() : this(new UnderlyingResponse() { })
{
}
private XXXResponse(UnderlyingResponse msg)
{
_resp = msg;
}
public static implicit operator XXXResponse(UnderlyingMessage value)
{
if( value != null)
{
return new XXXResponse(value.XXXResponse)
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
};
}
return null;
}
public static implicit operator UnderlyingMessage(XXXResponse value)
{
if(value != null)
{
return new UnderlyingMessage()
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
XXXResponse = value._resp,
};
}
return null;
}
public Transaction[] Transactions
{
get { return _resp.Transactions?.Select(t => (Transaction)t)?.ToArray(); }
set { _resp.Transactions = value?.Select(t => (BE.Data.Contract.Transaction)t)?.ToList(); }
}
public long DomainID { get { return _resp.DomainID; } set { _resp.DomainID = value; } }
public string UniversalID { get { return _resp.UniversalID; } set { _resp.UniversalID = value; } }
public string ExtraData { get { return _resp.ExtraData; } set { _resp.ExtraData = value; } }
// other proxied fields ...
}
The key is, when ISerializer.Read or ISerializer.Write is fired, the wire-format is from the scope of the entrie message, including all fields of base class, and current sub-type is in a field whose number is identified by ProtoInclude.
In our case this works. For other sub-types which we don't want surrogate at this moment, it still works as it did.

C# Json Interface serialization parent-child

this might have a easy/quick solution, but from my investigation I didn't manage to find it.
public interface IBaseMessage
{
}
public interface IRefreshMessage : IBaseMessage
{
}
Both are implemented by concrete classes in a Library.Message
They are stored inside a List<IBaseMessage> and when I serialize the object to be send over the network (one by one) it is marked as IBaseMessage even tough some are IRefrehMessage
Issue: on deserialization (on another PC) they are reconstructed as IBaseMessage and all the information from IRefreshMessage is lost.
Assumption: I believe there must be some JSON class attribute that allows me to specify into what to be serialized?
Thank you, and sorry for a rather silly question
Edit:
Using this class to serialize/deserialize:
using Newtonsoft.Json.Linq;
public class JMessage
{
public Type Type { get; set; }
public JToken Value { get; set; }
public static JMessage FromValue<T>(T value)
{
return new JMessage { Type = typeof(T), Value = JToken.FromObject(value) };
}
public static string Serialize(JMessage message)
{
return JToken.FromObject(message).ToString();
}
public static JMessage Deserialize(string data)
{
return JToken.Parse(data).ToObject<JMessage>();
}
Once is calls
string data = JMessage.Serialize(JMessage.FromValue(message));
I get:
{
"Type": "Universal_Tool_Manager.Messages.Message.Interfaces.IBaseMessage, Universal Tool Manager.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"Value": {
"Propagation": 2,
"State": 0
}
}
Hope this helps.

JSON.NET allows you to specify how to handle serialization and deserialization using JsonSerializerSettings:
class Program
{
static void Main(string[] args)
{
var items = new List<Test>
{
new Test() {Hello = "Hi"},
new TestDerived() {Hello = "hello", Second = "World"}
};
var settings = new JsonSerializerSettings()
{
TypeNameHandling = TypeNameHandling.Objects
};
var text = JsonConvert.SerializeObject(items, settings);
var data = JsonConvert.DeserializeObject<IEnumerable<Test>>(text, settings);
}
}
public class Test
{
public string Hello { get; set; }
}
public class TestDerived : Test
{
public string Second { get; set; }
}
In this sample the generated JSON will contain a name of the actual type which was serialized and JSON.NET will use this information to deserialize back to the correct derived type - data[1] is TestDerived in my example code.
To use JsonSerializerSettings with FromObject and Parse methods, you will need to create an instance of JsonSerializer:
var serializer = sonSerializer.Create(settings);
JToken.FromObject(o, serializer)
//and
JToken.Parse(data).ToObject<JMessage>(serializer);

I've asked the wrong question here :)
public static JMessage FromValue<T>(T value)
{
return new JMessage { Type = **typeof(T)**, Value = JToken.FromObject(value) };
}
The issue was on typeof(T) which of course it will always return the base interface IBaseInterface since i'm using a
List<IBaseInterface>,
The solution was to use value.GetType(),
sorry for the rookie mistake

Property with private field doesn't get deserialized? [duplicate]

I have the following class serialized into a file using BinaryFormatter:
[Serializable]
public class TestClass
{
public String ItemTwo { get; set; }
public String ItemOne { get; set; }
}
Using this code:
FileStream fs = new FileStream("DataFile.dat", FileMode.Create);
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, new TestClass{ItemOne = "ItemOne", ItemTwo = "ItemTwo"});
fs.Close();
When deserializing using this code:
FileStream fs = new FileStream("DataFile.dat", FileMode.Open);
BinaryFormatter formatter = new BinaryFormatter();
TestClass addresses = (TestClass)formatter.Deserialize(fs);
fs.Close();
I get everything normally.
However, now I need the class to have some backing fields like so:
[Serializable]
public class TestClass
{
private string _itemTwo;
private string _itemOne;
public String ItemTwo
{
get { return _itemTwo; }
set { _itemTwo = value; }
}
public String ItemOne
{
get { return _itemOne; }
set { _itemOne = value; }
}
}
My problem is that now, for some reason, deserialization from previous version doesn't work anymore. I get the class but the Properties are left null.
I cannot affect the serialization process, or the former class state.
How can I get the file to deserialize to the current class?

If you try to serialize the first version of TestClass the backfields will be serialized automatically by the binary formatter. Auto properties are only syntactic sugar, the compiler transform them in normal properties with backing fields. If you decompile your original console application (or class library) with ILSpy for example you'll see that your private fields are declared as:
.field private string '<ItemOne>k__BackingField'
which is way different from your second declaration with backing fields which yields to:
.field private string _itemOne
An approach would be declaring the ISerializable interface on your TestClass and get the value of the original properties using the information contained in the class SerializationInfo
[Serializable]
public class TestClass : ISerializable
{
private string _itemTwo;
private string _itemOne;
public String ItemTwo
{
get { return _itemTwo; }
set { _itemTwo = value; }
}
public String ItemOne
{
get { return _itemOne; }
set { _itemOne = value; }
}
protected TestClass(SerializationInfo info, StreamingContext context)
{
_itemTwo = info.GetString("<ItemTwo>k__BackingField");
_itemOne = info.GetString("<ItemOne>k__BackingField");
}
[SecurityPermissionAttribute(SecurityAction.Demand,
SerializationFormatter = true)]
public virtual void GetObjectData(SerializationInfo info, StreamingContext context)
{
//Add data to your serialization process here
}
}
so you tell the BinaryFormatter how you want your backing fields to be initialized during deserialization.

JSON .Net Serialization not working correctly

I have this class:
[Serializable]
public class ServerResponse
{
public string[] Response { get; set; }
public object Packet { get; set; }
public ServerResponse(string[] response, object packet){
this.Response = response;
this.Packet = packet;
}
public string[] getResponse() { return this.Response; }
public object getPacket() { return this.Packet; }
}
And use the following to convert to JSON:
static void writeToClient(TcpClient client, object message)
{
string json = JsonConvert.SerializeObject(message);
NetworkStream clientStream = client.GetStream();
byte[] buffer = NetworkHelper.GetMessageAsBytes(json);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
However, the Packet is never serialized. Can anyone see what I'm doing wrong? Sample output:
{"Response":["message","Authenticated"],"Packet":{}}
And this is the actual class I'm attempting to serialize:
[Serializable]
public class ActiveGameSession
{
private LoginServerSettings serverSettings;
private List<UserCharacter> sessionCharacters = new List<UserCharacter>();
private UserAccount sessionAccount;
public ActiveGameSession()
{
this.sessionCharacters = new List<UserCharacter>();
this.sessionAccount = null;
this.serverSettings = null;
}
public ActiveGameSession(List<UserCharacter> chars, UserAccount session)
{
this.sessionCharacters = chars;
this.sessionAccount = session;
}
#region setters
public void setCharacters(List<UserCharacter> chars)
{
this.sessionCharacters = chars;
}
public void setSession(UserAccount session)
{
this.sessionAccount = session;
}
public void setServerSettings(LoginServerSettings settings)
{
this.serverSettings = settings;
}
#endregion
#region getters
public List<UserCharacter> getCharacters()
{
return this.sessionCharacters;
}
public UserAccount getAccount()
{
return this.sessionAccount;
}
public LoginServerSettings getSettings()
{
return this.serverSettings;
}
#endregion
}

JSON.NET may not support serialization of an object of type Object. Try using a more concrete type. If you want to have multiple packet types you can try using generics or derive from a common base class.
Edit:
Also there are no public properties of ActiveGameSession to be deserialized, please note that the getXXX, setXXX methods is not recognized as properties.

You need to cast the object representing the Packet to a type that JSON.NET can serialize. Try casting to a string and try again.

XML serialize base class without knowing derived class

I know that an XmlSerializer will throw an exception if the passed in type does not match the type from when the serializer was created.
My question is: Is it possible to serialize a base type without knowing the derived type? Consider this scenario:
The following classes are in a top-level domain and have few or no dependencies:
public class Serializer
{
private static string Serialize(Base baseData)
{
var serializer = new XmlSerializer(typeof(Base));
var sb = new StringBuilder();
using (var writer = XmlWriter.Create(sb))
{
serializer.Serialize(writer, baseData);
}
return sb.ToString();
}
private static string Deserialize(...) { ... }
}
public class Base
{
public string Name { get; set; }
}
The following derived class has additional dependencies on things like UI components, composition, etc. (more on this later):
[Export(typeof(IMyComponent))]
public class Derived : Base, IMyComponent, INotifyPropertyChanged
{
public new string Name
{
get { return base.Name; }
set
{
base.Name = value;
NotifyPropertyChanged("Name");
}
}
}
If I invoke the serializer like this:
Base myData = new Derived();
myData.Name = "Foo bar";
Serializer.Serialize(myData);
It throws an exception because I passed in a Derived, even though the parameters and serializer are explicitly set to Base. All I want is to serialize Base, I don't care about whatever is in Derived.
My goal here is to create the derived class with a UI, but only serialize the base class. The reason is that I am deserializing the XML in a separate assembly that must not contain additional dependencies beyond the .NET framework. The serializer and base objects are recreated/copied into this isolated assembly so that the namespaces and types will match, so that when deserialization occurs, the types will be deserialized correctly. But I can't include a copy of Derived because it has dependencies on UI, MEF, other third party libraries, etc.
So, how can I force the serializer to serialize the base class even if I pass in a derived class? Again, I don't care if I lose the information in Derived, all I am concerned about is Base.

Since you are looking to only serialize the Base properties, you have to create an instance of Base and map the properties from the derived type. Serializers are programmed to be smart, sometimes too smart in your case.
Look into AutoMapper or just do it manually.
public class Serializer
{
private static string Serialize(Base baseData)
{
var mappedBase = new Base();
// Do mapping
mappedBase.Foo = baseData.Foo;
var serializer = new XmlSerializer(typeof(Base));
var sb = new StringBuilder();
using (var writer = XmlWriter.Create(sb))
{
serializer.Serialize(writer, mappedBase);
}
return sb.ToString();
}
private static string Deserialize(...) { ... }
}

One possible solution would be:
public class Serializer
{
public static string Serialize(Base baseData)
{
var baseType = baseData.GetType().BaseType;
var baseTypeFieldInfo = baseType.GetProperties();
var newBaseInstance = new Base();
var newBaseInstanceType = newBaseInstance.GetType();
//ONLY gets public properties but could be applied to fields
var newBaseTypeFieldInfo = newBaseInstanceType.GetProperties();
foreach (var bt in baseTypeFieldInfo)
{
foreach (var nbt in newBaseTypeFieldInfo)
{
if (nbt.Name == bt.Name)
{
nbt.SetValue(newBaseInstance, bt.GetValue(baseData, null), null);
}
}
}
var serializer = new XmlSerializer(typeof(Base));
var sb = new StringBuilder();
using (var writer = XmlWriter.Create(sb))
{
serializer.Serialize(writer, newBaseInstance);
}
return sb.ToString();
}
}
public class Base
{
public string Name { get; set; }
}
public class Derived : Base
{
public string AnotherName { get; set; }
public new string Name
{
get { return base.Name; }
set { base.Name = value; }
}
}