I'm struggling migrating from protobuf-net v2.4.6 to v3.0.100 (or any 3.0.x) in regards to an existing type hierarchy used as ProtoContracts with one of the subtypes requiring a surrogate due to one of its property being of type object.
With previous configuration in place, I get the following exception thrown on creating the runtime model:
System.InvalidOperationException: 'Types with surrogates cannot be used in inheritance hierarchies'
Hence, my question is how to properly deal with this scenario using protobuf-net 3.0.x?
Here's my (over-)simplified repro of the issue:
class Program
{
static void Main(string[] args)
{
var model = RuntimeTypeModel.Create();
_ = model[typeof(Base)]; // <-- InvalidOperationException thrown here
Base value = new Complex();
var copy = model.DeepClone(value);
}
}
[ProtoContract]
[ProtoInclude(1, typeof(Simple))]
[ProtoInclude(2, typeof(Complex))]
public abstract class Base
{
}
[ProtoContract]
public class Simple : Base
{
}
[ProtoContract(Surrogate = typeof(ComplexSurrogate))]
public class Complex : Base
{
}
[ProtoContract(Name = nameof(Complex))]
public class ComplexSurrogate
{
[ProtoConverter]
public static ComplexSurrogate Convert(Complex source) => new ComplexSurrogate();
[ProtoConverter]
public static Complex Convert(ComplexSurrogate source) => new Complex();
}
As a side note: When compiling protobuf-net from source with the above mentioned exception suppressed, I'm able to defined a surrogate for the Base class which seems to serve as a workaround.
Right now, that scenario isn't supported. While reworking the code for v3, some ambiguous outcomes/intents were found, and it needs work to go in and figure out what the correct outcomes are in each case, design how to achieve that, implement it, and test it. That time has not yet been found, so right now it is safer to prevent a configuration that could lead to big problems downstream, than to just shrug and assume that whatever happens is correct. It is on my list of things to do, but: ultimately this is a project that comes entirely out of my own spare time - it isn't sponsored or part of my paid work, so: it'll get there when it gets there.
I encountered the same error in protobuf v3, and I solved that with custom serializer.
My base class is
[ProtoContract]
[ProtoInclude(500, typeof(XXXRequest))]
[ProtoInclude(501, typeof(XXXResponse))]
// ...
public class MessageBase
{
[ProtoMember(1)]
long ID { get; internal set; }
[ProtoMember(3)]
int ExecutionMilliseconds { get; set; }
}
Its equivalent proto is
message Message {
int64 ID = 1;
int32 ExecutionMilliseconds = 3;
oneof body {
PredictBonusRequest XXXRequest = 500;
PredictBonusResponse XXXResponse = 501;
// ...
}
}
I want to replace some types (e.g. XXXResponse) to use the contract-first class instead. This would allow us to migrate from code-first to contract-first smoothly.
For sub-types should be surrogated, we create custom serializer as below.
using ProtoBuf;
using ProtoBuf.Serializers;
using UnderlyingMessage = GeneratedProto.Contract.Message;
using UnderlyingResponse = GeneratedProto.Contract.XXXResponse;
[DataContract]
[Serializable]
[ProtoContract(Serializer = typeof(XXXResponseSerializer))]
public class XXXResponse : MessageBase
{
class XXXResponseSerializer : ISerializer<XXXResponse>
{
public SerializerFeatures Features => SerializerFeatures.CategoryMessage | SerializerFeatures.WireTypeString;
public XXXResponse Read(ref ProtoReader.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
return serializer.Read(ref state, value);
}
public void Write(ref ProtoWriter.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
serializer.Write(ref state, value);
}
}
private readonly UnderlyingResponse _resp;
public XXXResponse() : this(new UnderlyingResponse() { })
{
}
private XXXResponse(UnderlyingResponse msg)
{
_resp = msg;
}
public static implicit operator XXXResponse(UnderlyingMessage value)
{
if( value != null)
{
return new XXXResponse(value.XXXResponse)
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
};
}
return null;
}
public static implicit operator UnderlyingMessage(XXXResponse value)
{
if(value != null)
{
return new UnderlyingMessage()
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
XXXResponse = value._resp,
};
}
return null;
}
public Transaction[] Transactions
{
get { return _resp.Transactions?.Select(t => (Transaction)t)?.ToArray(); }
set { _resp.Transactions = value?.Select(t => (BE.Data.Contract.Transaction)t)?.ToList(); }
}
public long DomainID { get { return _resp.DomainID; } set { _resp.DomainID = value; } }
public string UniversalID { get { return _resp.UniversalID; } set { _resp.UniversalID = value; } }
public string ExtraData { get { return _resp.ExtraData; } set { _resp.ExtraData = value; } }
// other proxied fields ...
}
The key is, when ISerializer.Read or ISerializer.Write is fired, the wire-format is from the scope of the entrie message, including all fields of base class, and current sub-type is in a field whose number is identified by ProtoInclude.
In our case this works. For other sub-types which we don't want surrogate at this moment, it still works as it did.
Related
I have a lot of duplicate code places:
if (claimSettingHistoryDto.NewClaimTypeName == claimSettingHistoryDto.OldClaimTypeName)
{
claimSettingHistoryDto.NewClaimTypeName = null;
claimSettingHistoryDto.OldClaimTypeName = null;
}
if (claimSettingHistoryDto.NewApplicantName == claimSettingHistoryDto.OldApplicantName)
{
claimSettingHistoryDto.NewApplicantName = null;
claimSettingHistoryDto.OldApplicantName = null;
}
if (claimSettingHistoryDto.NewDamageSparePartsTotalCostInsertion == claimSettingHistoryDto.OldDamageSparePartsTotalCostInsertion)
{
claimSettingHistoryDto.NewDamageSparePartsTotalCostInsertion = null;
claimSettingHistoryDto.OldDamageSparePartsTotalCostInsertion = null;
}
and so constantly for different classes of different fields
I wish I had a feature like this:
private void SetNull(object newData, object oldData)
{
if (newData == oldData)
{
newData = null;
oldData = null;
}
}
but of course I understand that this is not true, since I only change the local value inside the function. How do I change the class field?
There are multiple ways of doing that, with varying positions on the "good idea" to "bad idea" spectrum.
Fields as ref parameters (good idea)
(...) this is not true, since I only change the local value inside the function
You're wrong, because ref and out parameters allow you to change values non-locally.
If you have access to the actual fields, you can pass them as a ref parameter:
public class Dto
{
private string? _old;
private string? _new;
public string? Old => _old;
public string? New => _new;
public void Foo() {
SetNullIfEqual(ref _new, ref _old);
}
private static void SetNullIfEqual<T>(ref T? newData, ref T? oldData) where T: class
{
if (newData == oldData)
{
newData = null;
oldData = null;
}
}
}
More info on passing as reference here.
This won't work with properties, even if they have a default setter. Properties are not fields, they're methods in disguise. If you can't access the actual fields...
Properties as delegates (meh idea)
... having access to properties only you'd need to pass them as delegates like this:
public class Dto
{
public string? Old { get; set; }
public string? New { get; set; }
}
public class Outside
{
public void Foo(Dto dto) {
SetNullIfEqual(() => dto.New, () => dto.Old, v => dto.New = v, v => dto.Old = v);
}
private static void SetNullIfEqual<T>(
Func<T?> getNew,
Func<T?> getOld,
Action<T?> setNew,
Action<T?> setOld) where T: class
{
if (getNew() == getOld())
{
setNew(null);
setOld(null);
}
}
}
This is clunky though, you have to question how much space it'd actually save. An instance method working on fields as in the first suggestion works much better.
When you have reflection everything looks like a nail (probably bad idea)
You can also do this with reflection, which will remove all safety, give much worse performance, but the absolute most flexibility.
using System.Reflection;
public class Dto
{
public string? Old { get; set; }
public string? New { get; set; }
}
public class Outside
{
public void Foo(Dto dto) {
SetNullIfEqual(nameof(dto.New), nameof(dto.Old), dto);
}
private static void SetNullIfEqual<T>(
string newPropName,
string oldPropName,
T instance)
{
PropertyInfo newProp = typeof(T).GetProperty(newPropName);
PropertyInfo oldProp = typeof(T).GetProperty(oldPropName);
if (Equals(newProp.GetValue(instance), oldProp.GetValue(instance)))
{
newProp.SetValue(instance, null);
oldProp.SetValue(instance, null);
}
}
}
I removed all error handling for brevity.
Recommendation
I'd go with the fields-as-ref-parameters way. If the method in question lives outside of the type, so it can't have access to the fields (don't ever use public fields please), I'd just move it into the type. In your case it'd be a bunch of methods called SetClaimTypeName, SetApplicantName, etc.
Often i have a method where i want to return the error if something goes wrong, and instead of returning null, I want something less prone to errors at runtime and more easy to consume. Is there anything already done in .Net or maybe a nuget package?
Maybe have a constructor with optional parameters or object initializer would be enough?
This would have been the first approach but then every new Dto has to either have these Error property or inherit from a base class.
if (condition)
{
return new MyDto(null, error);
}
return new MyDto(someVariable, null);
So I've made this class to use a return type:
public class Optional<TObject> where TObject : class
{
public Optional(TObject? value)
{
Value = value;
}
public Optional(String error)
{
Error = error;
}
public TObject? Value { get; }
public String Error { get;} = String.Empty;
public Boolean IsError => !String.IsNullOrEmpty(Error);
}
I return it in the method:
if (condition)
{
return new Optional(error);
}
return new Optional(new MyDto(someVariable));
And then consume it like this:
var result = await myService.GetSomethingAsync();
if(result.IsError)
{
await DisplayAlert("error", result.Error, "Ok");
}
else
{
await DoSomethingElse(result.Value);
}
By creating a small class hierarchy, you could ensure that the Value property is only available when no error occurred
public abstract class Result
{
public virtual string Message => null;
public static Error Error(string message) => new Error(message);
public static Okay<T> Okay<T>(T value) where T : class => new Okay<T>(value);
}
public class Error : Result
{
public Error(string errorMessage) => Message = errorMessage;
override public string Message { get; }
}
public class Okay<T> : Result
where T : class
{
public Okay(T value) => Value = value;
public T Value { get; }
}
Usage
Result result = Result.Error("Something went wrong");
// OR
Result result = Result.Okay(new MyDto(someVariable));
if (result is Okay<MyDto> dtoResult) {
Console.WriteLine(dtoResult.Value);
} else {
Console.WriteLine(result.Message);
}
Or by using a recursive pattern, we can retrieve the value into a variable directly
if (result is Okay<MyDto> { Value: var dto }) {
Console.WriteLine(dto);
} else {
Console.WriteLine(result.Message);
}
Note that I have declared the Message property in the abstract base class Result, so that you don't have to cast to the Error type to get the message.
I used null as defualt value for the error message, as it allows us to write
Console.Writeline(result.Message ?? "okay");
This OneOf recommendation you got looks promising. I will personally have a look at it later.
What I do with my services is to standardize the result they return by using a SvcResult class or an inherited class.
Example:
public class SvcResult
{
public List<Error> Errors { get; } // Error is a class of my own. Add set; if deserialization is needed.
public bool Success { get; } // Add set; if deserialization is needed.
// Then parameterless constructor for a successful result.
// Then parameterized constructor to receive errors for a failed result.
}
That is the class for side-effect service calling. If The service returns data, I derive from the above to create DataSvcResult:
public class DataSvcResult<TResult> : SvcResult
{
public TResult Data { get; }
// Add constructor that receives TResult for a successful object result.
// Expose base class constructor that takes errors.
}
Basically that's what I do. But that OneOf thing, though. Looks super intersting.
I have two functions one for write and one for read when i try to use the read function first i get this error:
An error occurred while deserializing the Message property of class DDSRecorder.MessageContainer: Instances of abstract classes cannot be created
Here is what i dont get, if i use the write first at least once then the read works fine. i dont understand what happens in the background that makes it ok to initialize abstract class if we used it once to write.
Adding the map for it didn't resolve the problem:
if (BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
Here is the class i am using for the mongo collection.
[BsonIgnoreExtraElements(true)]
public class MessageContainer
{
[BsonId]
public ObjectId Id { get; set; }
[BsonDateTimeOptions(Kind = DateTimeKind.Utc)]
public DateTime TimeStamp { get; set; }
[BsonElement]
public string MessageType { get; set; }
public MessageBase Message { get; set; }
[BsonConstructor]
public MessageContainer()
{
}
[BsonConstructor]
public MessageContainer(MessageBase message)
{
Message = message ?? throw new ArgumentNullException(nameof(message));
TimeStamp = DateTime.UtcNow;
MessageType = message.GetType().Name;
}
[BsonConstructor]
public MessageContainer(DateTime timeStamp, string messageType, MessageBase message)
{
TimeStamp = timeStamp;
MessageType = messageType ?? throw new ArgumentNullException(nameof(messageType));
Message = message ?? throw new ArgumentNullException(nameof(message));
}
}
And the abstract class inside:
public abstract class MessageBase
{
protected MessageBase();
public MessageBase CreateCopy();
}
Example of write method:
public bool Write(MessageContainer message)
{
if (message != null && _mongoCollection != null)
{
try
{
if (!BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageContainer>();
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
_mongoCollection.InsertOne(message);
return true;
}
catch (Exception Ex)
{
Console.WriteLine(Ex.Message);
}
}
return false;
}
Example of read method:
public bool GetFirstAndLastMessageTime(out DateTime firstMessageTime, out DateTime lastMessageTime)
{
if (BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
var filter = Builders<MessageContainer>.Filter.Empty;
var first = _mongoCollection.Find(filter).Sort(Builders<MessageContainer>.Sort.Ascending("TimeStamp")).Limit(5).ToList().First();
var last = _mongoCollection.Find(filter).Sort(Builders<MessageContainer>.Sort.Descending("TimeStamp")).Limit(5).ToList().First();
firstMessageTime = first.TimeStamp;
lastMessageTime = last.TimeStamp;
return true;
}
What am i missing for it to be able to initialize the abstract class without the need of writing first?
Well, kind of an anti-pattern here (I don't like adding dependencies from base classes to their implementations), but a quick fix would be to add
[BsonKnownTypes(typeof(MyImplementation))]
where MyImplementation is the type that implements your abstract class.
on your MessageBase class. For me this did the trick - I was able to read the data and deserialize just fine. I didn't have to add any class maps either.
While the answer provided amitla is correct, it was not something that worked for me for multiple reasons.
One is I didn't want to change the abstract class and write down all of the implementations as tags.
Another note is that I found out from the MongoDB forums that apparently the reason for it working only after write is an auto mapper and that is why it worked for me after write.
So to make work this is what I did:
public void RegisterClassMapping(List<string> messagesType)
{
// Map base class first
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
if (messagesType.Count > 0)
{
foreach (string message in messagesType) // Do look up for each message
{
Type type = GetType("NAMEOFTHENAMESPACE." + message);
if (type == null)
{
type = GetType("NAMEOFTHENAMESPACE." + message);
}
if (type != null && !BsonClassMap.IsClassMapRegistered(type))
{
BsonClassMap.LookupClassMap(type);
}
}
}
}
And this is the generic GetType I used:
private Type GetType(string typeName)
{
var type = Type.GetType(typeName);
if (type != null) return type;
foreach (var a in AppDomain.CurrentDomain.GetAssemblies())
{
type = a.GetType(typeName);
if (type != null)
return type;
}
return null;
}
I just run this once and its working for the rest of session.
Setup:
public class Data
{
public int A { get; set; }
public int B { get; set; }
}
public class Runner
{
public static void Run(Data data)
{
data.A = data.B;
data.A = 1;
}
}
class Program
{
static void Main(string[] args)
{
var data = new Data() { A = 1, B = 2 };
Runner.Run(data);
}
}
Problem: I need to implement change tracking here for property names not values. Inside Runner.Run on the first line data.A = data.B I need to record somehow that "A" was set to "B" (literally property names) and then on the next line data.A = 1 I need to record that "A" was set to constant and say forget about it.
Constrains:
When setting one property to another (e.g. A = B) that needs to be recorded
When setting property to anything else (e.g. A = 1 or A = B * 2) this change needs to be forgotten (e.g. remember A only)
Suppose this is the tracker contract being used:
void RecordChange(string setterName, string getterName);
void UnTrackChange(string setterName);
Question:
I would like to somehow proxy the Data class so it still can be used in the interface code (e.g. Runner - is a whole bunch of a business logic code that uses Data) INCLUDING strong-typing and it can track it's changes without modifying the code (e.g. there is lots of places like 'data.A = data.B').
Is there any way to do it without resorting to I guess some magic involving IL generation?
Already investigated/tried:
PostSharp interceptors/Castle.DynamicProxy with interceptors - these alone cannot help. The most I can get out of it is to have a value of data.B inside setter interceptor but not nameof(data.B).
Compiler services - haven't found anything suitable here - getting the name of caller doesn't really help.
Runtine code generation - smth like proxy inherited from DynamicObject or using Relfection.Emit (TypeBuilder probably) - I lose typings.
Current solution:
Use the Tracker implementation of the abovementioned contract and pass it around into every function down the road. Then instead of writing data.A = data.B use method tracker.SetFrom(x => x.A, x => x.B) - tracker holds a Data instance and so this works. BUT in a real codebase it is easy to miss something and it just makes it way less readable.
It is the closest the solution I've come up with. It isn't perfect as I still need to modify all the contracts/methods in the client code to use a new data model but at least all the logic stays the same.
So I'm open for other answers.
Here's the renewed Data model:
public readonly struct NamedProperty<TValue>
{
public NamedProperty(string name, TValue value)
{
Name = name;
Value = value;
}
public string Name { get; }
public TValue Value { get; }
public static implicit operator TValue (NamedProperty<TValue> obj)
=> obj.Value;
public static implicit operator NamedProperty<TValue>(TValue value)
=> new NamedProperty<TValue>(null, value);
}
public interface ISelfTracker<T>
where T : class, ISelfTracker<T>
{
Tracker<T> Tracker { get; set; }
}
public class NamedData : ISelfTracker<NamedData>
{
public virtual NamedProperty<int> A { get; set; }
public virtual NamedProperty<int> B { get; set; }
public Tracker<NamedData> Tracker { get; set; }
}
Basically I've copy-pasted the original Data model but changed all its properties to be aware of their names.
Then the tracker itself:
public class Tracker<T>
where T : class, ISelfTracker<T>
{
public T Instance { get; }
public T Proxy { get; }
public Tracker(T instance)
{
Instance = instance;
Proxy = new ProxyGenerator().CreateClassProxyWithTarget<T>(Instance, new TrackingNamedProxyInterceptor<T>(this));
Proxy.Tracker = this;
}
public void RecordChange(string setterName, string getterName)
{
}
public void UnTrackChange(string setterName)
{
}
}
The interceptor for Castle.DynamicProxy:
public class TrackingNamedProxyInterceptor<T> : IInterceptor
where T : class, ISelfTracker<T>
{
private const string SetterPrefix = "set_";
private const string GetterPrefix = "get_";
private readonly Tracker<T> _tracker;
public TrackingNamedProxyInterceptor(Tracker<T> proxy)
{
_tracker = proxy;
}
public void Intercept(IInvocation invocation)
{
if (IsSetMethod(invocation.Method))
{
string propertyName = GetPropertyName(invocation.Method);
dynamic value = invocation.Arguments[0];
var propertyType = value.GetType();
if (IsOfGenericType(propertyType, typeof(NamedProperty<>)))
{
if (value.Name == null)
{
_tracker.UnTrackChange(propertyName);
}
else
{
_tracker.RecordChange(propertyName, value.Name);
}
var args = new[] { propertyName, value.Value };
invocation.Arguments[0] = Activator.CreateInstance(propertyType, args);
}
}
invocation.Proceed();
}
private string GetPropertyName(MethodInfo method)
=> method.Name.Replace(SetterPrefix, string.Empty).Replace(GetterPrefix, string.Empty);
private bool IsSetMethod(MethodInfo method)
=> method.IsSpecialName && method.Name.StartsWith(SetterPrefix);
private bool IsOfGenericType(Type type, Type openGenericType)
=> type.IsGenericType && type.GetGenericTypeDefinition() == openGenericType;
}
And the modified entry point:
static void Main(string[] args)
{
var data = new Data() { A = 1, B = 2 };
NamedData namedData = Map(data);
var proxy = new Tracker<NamedData>(namedData).Proxy;
Runner.Run(proxy);
Console.ReadLine();
}
The Map() function actually maps Data to NamedData filling in property names.
in numerous other Types I have created it is possible to downCast a type
and i usually Create An Extension method too so it will be easier to manage...
BaseTypeM
BTDerV : BaseTypeM
BTDerLastDescndnt : BTDerV
now i create A LastDerived Type and assign its value To ParentType
BTDerV BTDer;
BTDerLastDescndnt BTDerLastDesc = new BTDerLastDescndnt(parA, ParB);
this.BTDer = BTDerLastDesc;
then using the downCast Extension
var LDesc = this.BTDer.AsBTDerLastDescndnt();
which is actually
public static BTDerLastDescndnt AsBTDerLastDescndnt(this BTDerV SelfBTDerV )
{
return (BTDerLastDescndnt)SelfBTDerV;
}
now when i do this as the code below, here it does compile but gives me a run-time error
//BTDerV---v v---BaseTypeM
public class SqlParDefV : SqlParameterM
{
public override SqlSpParDefMeta ParDMT
{
get {
return base.ParDMT;
}
set {
base.ParDMT = value;
}
}
public SqlParDefV(int bsprpOrdinal, string bsprpParName, MSSTypesS bdprpTypeS, bool bsprpIsDbStuctured, bool bsprpIsReq = true, ParameterDirection bsprpDirection = ParameterDirection.Input)
{
this.ParDMT = new SqlSpParDefMeta(bsprpOrdinal, bsprpParName, bdprpTypeS, bsprpIsReq, bsprpIsDbStuctured, bsprpDirection);
}
}
//BTDerLastDescndnt---v
public sealed class SqlParTvDrecDefinitionVScl : SqlParDefV
{
public override SqlSpParDefMeta ParDMT
{
get {
return base.ParDMT;
}
set {
base.ParDMT = value;
}
}
public SprocTvTargetSF.currentSDTObjType SqlObjType { get; set; }
public SqlMetaData[] Meta { get; set; }
public SqlParTvDrecDefinitionVScl(int bsprpOrdinal, string bsprpParName, SprocTvTargetSF.currentSDTObjType ctrSqlObjType, SqlMetaData[] parGeneratedSqlMetaData, MSSTypesS bdprpTypeS, bool bsprpIsDbStuctured, bool bsprpIsReq = true, ParameterDirection bsprpDirection = ParameterDirection.Input)
: base(bsprpOrdinal, bsprpParName, bdprpTypeS, bsprpIsDbStuctured, bsprpIsReq, bsprpDirection)
{
this.SqlObjType = ctrSqlObjType;
this.Meta = parGeneratedSqlMetaData;
}
}
is there something unusual here or am i confused and missed some basic rule ?
I am unsure of the precise reasons a cast from Derived to MoreDerived fails here. However, a potential workaround (note: possibly code smell) is the as operator:
public static MoreDerived AsMoreDerived (this Derived d)
{
return d as MoreDerived;
}
Note that as effectively attempts the cast and returns null, so you'll need an appropriate check there.