We Keep Coding

protobuf-net v3 with surrogate and inheritance hierarchies

I'm struggling migrating from protobuf-net v2.4.6 to v3.0.100 (or any 3.0.x) in regards to an existing type hierarchy used as ProtoContracts with one of the subtypes requiring a surrogate due to one of its property being of type object.
With previous configuration in place, I get the following exception thrown on creating the runtime model:
System.InvalidOperationException: 'Types with surrogates cannot be used in inheritance hierarchies'
Hence, my question is how to properly deal with this scenario using protobuf-net 3.0.x?
Here's my (over-)simplified repro of the issue:
class Program
{
static void Main(string[] args)
{
var model = RuntimeTypeModel.Create();
_ = model[typeof(Base)]; // <-- InvalidOperationException thrown here
Base value = new Complex();
var copy = model.DeepClone(value);
}
}
[ProtoContract]
[ProtoInclude(1, typeof(Simple))]
[ProtoInclude(2, typeof(Complex))]
public abstract class Base
{
}
[ProtoContract]
public class Simple : Base
{
}
[ProtoContract(Surrogate = typeof(ComplexSurrogate))]
public class Complex : Base
{
}
[ProtoContract(Name = nameof(Complex))]
public class ComplexSurrogate
{
[ProtoConverter]
public static ComplexSurrogate Convert(Complex source) => new ComplexSurrogate();
[ProtoConverter]
public static Complex Convert(ComplexSurrogate source) => new Complex();
}
As a side note: When compiling protobuf-net from source with the above mentioned exception suppressed, I'm able to defined a surrogate for the Base class which seems to serve as a workaround.

Right now, that scenario isn't supported. While reworking the code for v3, some ambiguous outcomes/intents were found, and it needs work to go in and figure out what the correct outcomes are in each case, design how to achieve that, implement it, and test it. That time has not yet been found, so right now it is safer to prevent a configuration that could lead to big problems downstream, than to just shrug and assume that whatever happens is correct. It is on my list of things to do, but: ultimately this is a project that comes entirely out of my own spare time - it isn't sponsored or part of my paid work, so: it'll get there when it gets there.

I encountered the same error in protobuf v3, and I solved that with custom serializer.
My base class is
[ProtoContract]
[ProtoInclude(500, typeof(XXXRequest))]
[ProtoInclude(501, typeof(XXXResponse))]
// ...
public class MessageBase
{
[ProtoMember(1)]
long ID { get; internal set; }
[ProtoMember(3)]
int ExecutionMilliseconds { get; set; }
}
Its equivalent proto is
message Message {
int64 ID = 1;
int32 ExecutionMilliseconds = 3;
oneof body {
PredictBonusRequest XXXRequest = 500;
PredictBonusResponse XXXResponse = 501;
// ...
}
}
I want to replace some types (e.g. XXXResponse) to use the contract-first class instead. This would allow us to migrate from code-first to contract-first smoothly.
For sub-types should be surrogated, we create custom serializer as below.
using ProtoBuf;
using ProtoBuf.Serializers;
using UnderlyingMessage = GeneratedProto.Contract.Message;
using UnderlyingResponse = GeneratedProto.Contract.XXXResponse;
[DataContract]
[Serializable]
[ProtoContract(Serializer = typeof(XXXResponseSerializer))]
public class XXXResponse : MessageBase
{
class XXXResponseSerializer : ISerializer<XXXResponse>
{
public SerializerFeatures Features => SerializerFeatures.CategoryMessage | SerializerFeatures.WireTypeString;
public XXXResponse Read(ref ProtoReader.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
return serializer.Read(ref state, value);
}
public void Write(ref ProtoWriter.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
serializer.Write(ref state, value);
}
}
private readonly UnderlyingResponse _resp;
public XXXResponse() : this(new UnderlyingResponse() { })
{
}
private XXXResponse(UnderlyingResponse msg)
{
_resp = msg;
}
public static implicit operator XXXResponse(UnderlyingMessage value)
{
if( value != null)
{
return new XXXResponse(value.XXXResponse)
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
};
}
return null;
}
public static implicit operator UnderlyingMessage(XXXResponse value)
{
if(value != null)
{
return new UnderlyingMessage()
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
XXXResponse = value._resp,
};
}
return null;
}
public Transaction[] Transactions
{
get { return _resp.Transactions?.Select(t => (Transaction)t)?.ToArray(); }
set { _resp.Transactions = value?.Select(t => (BE.Data.Contract.Transaction)t)?.ToList(); }
}
public long DomainID { get { return _resp.DomainID; } set { _resp.DomainID = value; } }
public string UniversalID { get { return _resp.UniversalID; } set { _resp.UniversalID = value; } }
public string ExtraData { get { return _resp.ExtraData; } set { _resp.ExtraData = value; } }
// other proxied fields ...
}
The key is, when ISerializer.Read or ISerializer.Write is fired, the wire-format is from the scope of the entrie message, including all fields of base class, and current sub-type is in a field whose number is identified by ProtoInclude.
In our case this works. For other sub-types which we don't want surrogate at this moment, it still works as it did.

JSON .Net Serialization not working correctly

I have this class:
[Serializable]
public class ServerResponse
{
public string[] Response { get; set; }
public object Packet { get; set; }
public ServerResponse(string[] response, object packet){
this.Response = response;
this.Packet = packet;
}
public string[] getResponse() { return this.Response; }
public object getPacket() { return this.Packet; }
}
And use the following to convert to JSON:
static void writeToClient(TcpClient client, object message)
{
string json = JsonConvert.SerializeObject(message);
NetworkStream clientStream = client.GetStream();
byte[] buffer = NetworkHelper.GetMessageAsBytes(json);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
However, the Packet is never serialized. Can anyone see what I'm doing wrong? Sample output:
{"Response":["message","Authenticated"],"Packet":{}}
And this is the actual class I'm attempting to serialize:
[Serializable]
public class ActiveGameSession
{
private LoginServerSettings serverSettings;
private List<UserCharacter> sessionCharacters = new List<UserCharacter>();
private UserAccount sessionAccount;
public ActiveGameSession()
{
this.sessionCharacters = new List<UserCharacter>();
this.sessionAccount = null;
this.serverSettings = null;
}
public ActiveGameSession(List<UserCharacter> chars, UserAccount session)
{
this.sessionCharacters = chars;
this.sessionAccount = session;
}
#region setters
public void setCharacters(List<UserCharacter> chars)
{
this.sessionCharacters = chars;
}
public void setSession(UserAccount session)
{
this.sessionAccount = session;
}
public void setServerSettings(LoginServerSettings settings)
{
this.serverSettings = settings;
}
#endregion
#region getters
public List<UserCharacter> getCharacters()
{
return this.sessionCharacters;
}
public UserAccount getAccount()
{
return this.sessionAccount;
}
public LoginServerSettings getSettings()
{
return this.serverSettings;
}
#endregion
}

JSON.NET may not support serialization of an object of type Object. Try using a more concrete type. If you want to have multiple packet types you can try using generics or derive from a common base class.
Edit:
Also there are no public properties of ActiveGameSession to be deserialized, please note that the getXXX, setXXX methods is not recognized as properties.

You need to cast the object representing the Packet to a type that JSON.NET can serialize. Try casting to a string and try again.

Parameter Enums in C#

I'm used to programming in Java however for this project I'm supposed to be using C#, I'm trying to convert my Packet system over from my Java project, however I'm running into some issues using the C# Compiler. Here's the code.
abstract class Packet
{
public static enum PacketTypes
{
INVALID(-1), LOGIN(00);
private int packetId;
private PacketTypes(int packetId)
{
this.packetId = packetId;
}
public int getId() { return packetId; }
}
}
This is actually exactly how it's done in my Java Code, and I have the individual packets extend the Packet class. I'm trying to figure out how to make this all come together in C#. Perhaps having a separate class for each packet isn't the way it should be done here?

I'm not sure what you're trying to achieve, but you can set values for particular enum elements in C#:
public enum PacketTypes
{
INVALID = -1;
LOGIN = 0;
}
Because enum is by default backed by int, you can cast it from/to int without additional code.
enums in C# cannot content any members, so you can't add methods/properties/fields to enum declaration.

Unlike Java where enums are classes, in C# enums are plain values. They cannot have member functions or fields.
One approach that could help is to define an extension method for your enum, like this:
public static class PacketTypesExtensions {
static readonly IDictionary<PacketTypes,int> IdForType = new Dictionary<PacketTypes,int> {
{ PacketTypes.INVALID, -1 }
, { PacketTypes.LOGIN, 0 }
};
static readonly IDictionary<PacketTypes,string> DescrForType = new Dictionary<PacketTypes,string> {
{ PacketTypes.INVALID, "<invalid packet type>" }
, { PacketTypes.LOGIN, "<user login>" }
};
public static string Description(this PacketTypes t) {
return DescrForType[t];
}
public static int Id(this PacketTypes t) {
return IdForType[t];
}
}
This lets you keep Java syntax:
PacketTypes pt = ... // <<== Assign a packet type here
int id = pt.Id(); // This calls the static extension method
string d = pt.Description();

You could try this in addition to Marcins answer.
public enum PacketTypes
{
INVALID = -1;
LOGIN = 0;
}
public class Packet
{
public PacketTypes PacketType { get; set;}
}
In your code somewhere, you would do this
public void DoSomething()
{
var packet = new Packet();
packet.PacketType = PacketTypes.INVALID; // Assign packtype
Console.WriteLine(packet.PacketType.ToString()); // Retrieve and print
}

Is it possible to use a Strategy pattern for data structure with no common ancestor?

I have two data structure classes (this is a simplified version of my code)
Animal: Has one property “int Age”
Person: Has one property “DateTime Birthday”
What I am trying to accomplish is to compile “Uploading” (persisting to database), which is common across all different data structure classes.
So mainly my goal is to have a small Upload method that looks like
foreach (TypeName typeName in Enum.GetValues(typeof(TypeName)))
{
IDataPopulator populator =
new DataFactory().CreateDataPopulator(typeName);
populator.Populate(string.Empty);
}
But the problem is that, populator returns an object instances of different types, which I am having trying to encapsulate since they have no common properties.
(IDataPopulator.TResult Populate(string data) in the code below)
Is there a way to get around this? Or does Strategy pattern not fit for this kind of scenario?
Here is the code I’ve been working with
public class Program
{
public static void Main()
{
foreach (TypeName typeName in Enum.GetValues(typeof(TypeName)))
{
IDataPopulator populator = new DataFactory().CreateDataPopulator(typeName);
populator.Populate(string.Empty);
}
}
}
public enum TypeName { Person, Animal }
public class Person { public DateTime Birthday { get; set; } }
public class Animal { public int Age { get; set; } }
public interface IDataPopulator
{
TResult Populate(string data);
}
class AnimalDataPopulator : IDataPopulator
{
public Animal Populate(string data)
{
// create an instance of Animal using data
}
}
class PersonDataPopulator : IDataPopulator
{
public Person Populate(string data)
{
// create an instance of Person using data
}
}
public class DataFactory
{
public IDataPopulator CreateDataPopulator(TypeName typeName)
{
switch (typeName)
{
case TypeName.Person:
return new PersonDataPopulator();
case TypeName.Animal:
return new AnimalDataPopulator();
default:
throw new ArgumentOutOfRangeException("typeName");
}
}
}
public class UploadContext
{
private readonly IUploader _Uploader;
public UploadContext(IUploader uploader) { _Uploader = uploader; }
public void Upload() { _Uploader.Upload(); }
}
public interface IUploader
{
void Upload();
}
class PersonUploader : IUploader
{
private Person _Person;
public PersonUploader(Person person) { _Person = person; }
public void Upload()
{
Console.WriteLine("Uploading person...");
}
}
class AnimalUploader : IUploader
{
private Animal _Animal;
public AnimalUploader(Animal animal) { _Animal = animal; }
public void Upload()
{
Console.WriteLine("Uploading animal...");
}
}

I don't see how the Strategy pattern would fit here. The pattern is about a family of algorithms, e.g. line breaking strategies. You don't have strategies here, you have a single abstract method
TResult Populate(string data);
and several implementations of it.
I also don't understand the initial problem, maybe you want to queue several serialization operations? In that case the Command pattern is your friend.

I think what you are after is either serialization, if you are just storing blobs, or an ORM framework.

protobuf and List<object> - how to serialize / deserialize?

I have a List<object> with different types of objects in it like integers, strings, and custom types. All custom types are protobuf-adjusted.
What I wanna do now is to serialize / deserialize this list with protobuf.net. Up until now I suspect that I have to declare each and every type explicitly, which is unfortunately not possible with these mixed-list constructs. Because the binary formater has no problems to do these things I hope that I missed something and that you can help me out.
So my question is how to deal with objects in protobuf.net.

(disclosure: I'm the author of protobuf-net)
BinaryFormatter is a metadata-based serializer; i.e. it sends .NET type information about every object serialized. protobuf-net is a contract-based serializer (the binary equivalent of XmlSerializer / DataContractSerializer, which will also reject this).
There is no current mechanism for transporting arbitrary objects, since the other end will have no way of knowing what you are sending; however, if you have a known set of different object types you want to send, there may be options. There is also work in the pipeline to allow runtime-extensible schemas (rather than just attributes, which are fixed at build) - but this is far from complete.
This isn't ideal, but it works... it should be easier when I've completed the work to support runtime schemas:
using System;
using System.Collections.Generic;
using ProtoBuf;
[ProtoContract]
[ProtoInclude(10, typeof(DataItem<int>))]
[ProtoInclude(11, typeof(DataItem<string>))]
[ProtoInclude(12, typeof(DataItem<DateTime>))]
[ProtoInclude(13, typeof(DataItem<Foo>))]
abstract class DataItem {
public static DataItem<T> Create<T>(T value) {
return new DataItem<T>(value);
}
public object Value {
get { return ValueImpl; }
set { ValueImpl = value; }
}
protected abstract object ValueImpl {get;set;}
protected DataItem() { }
}
[ProtoContract]
sealed class DataItem<T> : DataItem {
public DataItem() { }
public DataItem(T value) { Value = value; }
[ProtoMember(1)]
public new T Value { get; set; }
protected override object ValueImpl {
get { return Value; }
set { Value = (T)value; }
}
}
[ProtoContract]
public class Foo {
[ProtoMember(1)]
public string Bar { get; set; }
public override string ToString() {
return "Foo with Bar=" + Bar;
}
}
static class Program {
static void Main() {
var items = new List<DataItem>();
items.Add(DataItem.Create(12345));
items.Add(DataItem.Create(DateTime.Today));
items.Add(DataItem.Create("abcde"));
items.Add(DataItem.Create(new Foo { Bar = "Marc" }));
items.Add(DataItem.Create(67890));
// serialize and deserialize
var clone = Serializer.DeepClone(items);
foreach (DataItem item in clone) {
Console.WriteLine(item.Value);
}
}
}

List<YourClass> list;
ProtoBuf.Serializer.Deserialize<List<YourClass>>(filestream);

There is a way of doing this, albeit not a very clean way, by using a wrapper object that utilises another serialisation mechanism that supports arbitrary objects. I am presenting an example below using JSON but, like I said, resorting to a different serialisation tool seems like defeating the purpose of using protobuf:
[DataContract]
public class ObjectWrapper
{
[DataMember(Order = 1)]
private readonly string _serialisedContent;
[DataMember(Order = 2)]
private readonly string _serialisedType;
public object Content { get; private set; }
[UsedImplicitly]
private ObjectWrapper() { }
public ObjectWrapper(object content)
{
_serialisedContent = JsonConvert.SerializeObject(content);
_serialisedType = content.GetType().FullName;
Content = content;
}
[ProtoAfterDeserialization]
private void Initialise()
{
var type = Type.GetType(_serialisedType);
Content = type != null
? JsonConvert.DeserializeObject(_serialisedContent, type)
: JsonConvert.DeserializeObject(_serialisedContent);
}
}
EDIT: This can also be done using C#'s built-in binary serialisation
[DataContract]
public class ObjectWrapper
{
[DataMember(Order = 1)]
private readonly string _serialisedContent;
public object Content { get; private set; }
[UsedImplicitly]
private ObjectWrapper() { }
public ObjectWrapper(object content)
{
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, content);
stream.Flush();
stream.Position = 0;
_serialisedContent = Convert.ToBase64String(stream.ToArray());
}
}
[ProtoAfterDeserialization]
private void Initialise()
{
var data = Convert.FromBase64String(source);
using (var stream = new MemoryStream(data))
{
var formatter = new BinaryFormatter();
stream.Seek(0, SeekOrigin.Begin);
return formatter.Deserialize(stream);
}
}
}