We Keep Coding

protobuf-net v3 with surrogate and inheritance hierarchies

I'm struggling migrating from protobuf-net v2.4.6 to v3.0.100 (or any 3.0.x) in regards to an existing type hierarchy used as ProtoContracts with one of the subtypes requiring a surrogate due to one of its property being of type object.
With previous configuration in place, I get the following exception thrown on creating the runtime model:
System.InvalidOperationException: 'Types with surrogates cannot be used in inheritance hierarchies'
Hence, my question is how to properly deal with this scenario using protobuf-net 3.0.x?
Here's my (over-)simplified repro of the issue:
class Program
{
static void Main(string[] args)
{
var model = RuntimeTypeModel.Create();
_ = model[typeof(Base)]; // <-- InvalidOperationException thrown here
Base value = new Complex();
var copy = model.DeepClone(value);
}
}
[ProtoContract]
[ProtoInclude(1, typeof(Simple))]
[ProtoInclude(2, typeof(Complex))]
public abstract class Base
{
}
[ProtoContract]
public class Simple : Base
{
}
[ProtoContract(Surrogate = typeof(ComplexSurrogate))]
public class Complex : Base
{
}
[ProtoContract(Name = nameof(Complex))]
public class ComplexSurrogate
{
[ProtoConverter]
public static ComplexSurrogate Convert(Complex source) => new ComplexSurrogate();
[ProtoConverter]
public static Complex Convert(ComplexSurrogate source) => new Complex();
}
As a side note: When compiling protobuf-net from source with the above mentioned exception suppressed, I'm able to defined a surrogate for the Base class which seems to serve as a workaround.

Right now, that scenario isn't supported. While reworking the code for v3, some ambiguous outcomes/intents were found, and it needs work to go in and figure out what the correct outcomes are in each case, design how to achieve that, implement it, and test it. That time has not yet been found, so right now it is safer to prevent a configuration that could lead to big problems downstream, than to just shrug and assume that whatever happens is correct. It is on my list of things to do, but: ultimately this is a project that comes entirely out of my own spare time - it isn't sponsored or part of my paid work, so: it'll get there when it gets there.

I encountered the same error in protobuf v3, and I solved that with custom serializer.
My base class is
[ProtoContract]
[ProtoInclude(500, typeof(XXXRequest))]
[ProtoInclude(501, typeof(XXXResponse))]
// ...
public class MessageBase
{
[ProtoMember(1)]
long ID { get; internal set; }
[ProtoMember(3)]
int ExecutionMilliseconds { get; set; }
}
Its equivalent proto is
message Message {
int64 ID = 1;
int32 ExecutionMilliseconds = 3;
oneof body {
PredictBonusRequest XXXRequest = 500;
PredictBonusResponse XXXResponse = 501;
// ...
}
}
I want to replace some types (e.g. XXXResponse) to use the contract-first class instead. This would allow us to migrate from code-first to contract-first smoothly.
For sub-types should be surrogated, we create custom serializer as below.
using ProtoBuf;
using ProtoBuf.Serializers;
using UnderlyingMessage = GeneratedProto.Contract.Message;
using UnderlyingResponse = GeneratedProto.Contract.XXXResponse;
[DataContract]
[Serializable]
[ProtoContract(Serializer = typeof(XXXResponseSerializer))]
public class XXXResponse : MessageBase
{
class XXXResponseSerializer : ISerializer<XXXResponse>
{
public SerializerFeatures Features => SerializerFeatures.CategoryMessage | SerializerFeatures.WireTypeString;
public XXXResponse Read(ref ProtoReader.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
return serializer.Read(ref state, value);
}
public void Write(ref ProtoWriter.State state, XXXResponse value)
{
ISerializer<UnderlyingMessage> serializer = state.GetSerializer<UnderlyingMessage>();
serializer.Write(ref state, value);
}
}
private readonly UnderlyingResponse _resp;
public XXXResponse() : this(new UnderlyingResponse() { })
{
}
private XXXResponse(UnderlyingResponse msg)
{
_resp = msg;
}
public static implicit operator XXXResponse(UnderlyingMessage value)
{
if( value != null)
{
return new XXXResponse(value.XXXResponse)
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
};
}
return null;
}
public static implicit operator UnderlyingMessage(XXXResponse value)
{
if(value != null)
{
return new UnderlyingMessage()
{
ID = value.ID,
ExecutionMilliseconds = value.ExecutionMilliseconds,
XXXResponse = value._resp,
};
}
return null;
}
public Transaction[] Transactions
{
get { return _resp.Transactions?.Select(t => (Transaction)t)?.ToArray(); }
set { _resp.Transactions = value?.Select(t => (BE.Data.Contract.Transaction)t)?.ToList(); }
}
public long DomainID { get { return _resp.DomainID; } set { _resp.DomainID = value; } }
public string UniversalID { get { return _resp.UniversalID; } set { _resp.UniversalID = value; } }
public string ExtraData { get { return _resp.ExtraData; } set { _resp.ExtraData = value; } }
// other proxied fields ...
}
The key is, when ISerializer.Read or ISerializer.Write is fired, the wire-format is from the scope of the entrie message, including all fields of base class, and current sub-type is in a field whose number is identified by ProtoInclude.
In our case this works. For other sub-types which we don't want surrogate at this moment, it still works as it did.

Instances of abstract class cannot be created unless i write to database first

I have two functions one for write and one for read when i try to use the read function first i get this error:
An error occurred while deserializing the Message property of class DDSRecorder.MessageContainer: Instances of abstract classes cannot be created
Here is what i dont get, if i use the write first at least once then the read works fine. i dont understand what happens in the background that makes it ok to initialize abstract class if we used it once to write.
Adding the map for it didn't resolve the problem:
if (BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
Here is the class i am using for the mongo collection.
[BsonIgnoreExtraElements(true)]
public class MessageContainer
{
[BsonId]
public ObjectId Id { get; set; }
[BsonDateTimeOptions(Kind = DateTimeKind.Utc)]
public DateTime TimeStamp { get; set; }
[BsonElement]
public string MessageType { get; set; }
public MessageBase Message { get; set; }
[BsonConstructor]
public MessageContainer()
{
}
[BsonConstructor]
public MessageContainer(MessageBase message)
{
Message = message ?? throw new ArgumentNullException(nameof(message));
TimeStamp = DateTime.UtcNow;
MessageType = message.GetType().Name;
}
[BsonConstructor]
public MessageContainer(DateTime timeStamp, string messageType, MessageBase message)
{
TimeStamp = timeStamp;
MessageType = messageType ?? throw new ArgumentNullException(nameof(messageType));
Message = message ?? throw new ArgumentNullException(nameof(message));
}
}
And the abstract class inside:
public abstract class MessageBase
{
protected MessageBase();
public MessageBase CreateCopy();
}
Example of write method:
public bool Write(MessageContainer message)
{
if (message != null && _mongoCollection != null)
{
try
{
if (!BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageContainer>();
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
_mongoCollection.InsertOne(message);
return true;
}
catch (Exception Ex)
{
Console.WriteLine(Ex.Message);
}
}
return false;
}
Example of read method:
public bool GetFirstAndLastMessageTime(out DateTime firstMessageTime, out DateTime lastMessageTime)
{
if (BsonClassMap.IsClassMapRegistered(typeof(MessageContainer)))
{
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
}
var filter = Builders<MessageContainer>.Filter.Empty;
var first = _mongoCollection.Find(filter).Sort(Builders<MessageContainer>.Sort.Ascending("TimeStamp")).Limit(5).ToList().First();
var last = _mongoCollection.Find(filter).Sort(Builders<MessageContainer>.Sort.Descending("TimeStamp")).Limit(5).ToList().First();
firstMessageTime = first.TimeStamp;
lastMessageTime = last.TimeStamp;
return true;
}
What am i missing for it to be able to initialize the abstract class without the need of writing first?

Well, kind of an anti-pattern here (I don't like adding dependencies from base classes to their implementations), but a quick fix would be to add
[BsonKnownTypes(typeof(MyImplementation))]
where MyImplementation is the type that implements your abstract class.
on your MessageBase class. For me this did the trick - I was able to read the data and deserialize just fine. I didn't have to add any class maps either.

While the answer provided amitla is correct, it was not something that worked for me for multiple reasons.
One is I didn't want to change the abstract class and write down all of the implementations as tags.
Another note is that I found out from the MongoDB forums that apparently the reason for it working only after write is an auto mapper and that is why it worked for me after write.
So to make work this is what I did:
public void RegisterClassMapping(List<string> messagesType)
{
// Map base class first
BsonClassMap.RegisterClassMap<MessageBase>(cm =>
{
cm.AutoMap();
cm.SetIsRootClass(true);
});
if (messagesType.Count > 0)
{
foreach (string message in messagesType) // Do look up for each message
{
Type type = GetType("NAMEOFTHENAMESPACE." + message);
if (type == null)
{
type = GetType("NAMEOFTHENAMESPACE." + message);
}
if (type != null && !BsonClassMap.IsClassMapRegistered(type))
{
BsonClassMap.LookupClassMap(type);
}
}
}
}
And this is the generic GetType I used:
private Type GetType(string typeName)
{
var type = Type.GetType(typeName);
if (type != null) return type;
foreach (var a in AppDomain.CurrentDomain.GetAssemblies())
{
type = a.GetType(typeName);
if (type != null)
return type;
}
return null;
}
I just run this once and its working for the rest of session.

HttpSessionStateBase losing property values of inherited type

We are using HttpSessionStateBase to store messages in a set up similar to this working example:
public class HttpSessionMessageDisplayFetch : IMessageDisplayFetch
{
protected HttpSessionStateBase _session;
private IList<ICoreMessage> messages
{
get
{
if (_session[EchoCoreConstants.MESSAGE_KEY] == null)
_session[EchoCoreConstants.MESSAGE_KEY] = new List<ICoreMessage>();
return _session[EchoCoreConstants.MESSAGE_KEY] as IList<ICoreMessage>;
}
}
public HttpSessionMessageDisplayFetch()
{
if (HttpContext.Current != null)
_session = new HttpSessionStateWrapper(HttpContext.Current.Session);
}
public void AddMessage(ICoreMessage message)
{
if (message != null)
messages.Add(message);
}
public IEnumerable<IResultPresentation> FlushMessagesAsPresentations(IResultFormatter formatter)
{
var mToReturn = messages.Select(m => m.GetPresentation(formatter)).ToList();
messages.Clear();
return mToReturn;
}
}
When we pass in a QualityExplicitlySetMessage (which inherits from ICoreMessage, see below) it is saved correctly to messages.
This is how the object looks after being inserted into the messages list, at the end of AddMessage(ICoreMessage message) above.
But when we come to access it after changing controllers the inherited member's properties are null, which causes a variety of null reference exceptions.
This is how the object now looks after we call FlushMessagesAsPresentations. I've commented out var mToReturn... as this tries to access one of these null ref properties.
I'd like to ask the following:
Why is the HttpSessionStateBase failing to capture these values taken
by the inherited type?
Is this an issue in saving to the HttpSession or in retrieving?
Is this anything to do with, as I suspect, inheritance?
Or is the fact I'm potentially calling a new controller that dependency injects the HttpSessionMessageDisplayFetch causing an issue?
I'm a first-time poster so please let me know if I'm making any kind of faux pas - Super keen to learn! Any input is very welcome.
Some potentially useful code snippets:
QualityExplicitlySetMessage
public class QualityExplicitlySetMessage : QualityChangeMessage
{
public QualityExplicitlySetMessage(IQPossession before, IQPossession after, IQEffect qEffect)
: base(before, after, qEffect)
{
IsSetToExactly = true;
}
}
QualityChangeMessage - Working example
public abstract class QualityChangeMessage : CoreMessage, IQualityChangeMessage
{
protected PossessionChange Change;
public PossessionChange GetPossessionChange()
{
return Change;
}
protected QualityChangeMessage(IQPossession before, IQPossession after, IQEffect qEffect)
{
Change = new PossessionChange(before, after, qEffect);
StoreQualityInfo(qEffect.AssociatedQuality);
}
public override IResultPresentation GetPresentation(IResultFormatter formatter)
{
return formatter.GetQualityResult(this);
}
#region IQualityChangeMessage implementation
public int LevelBefore
{
get { return Change.Before.Level; }
}
//... And so on with values dependent on the Change property.
}
CoreMessage - Working example
public abstract class CoreMessage : ICoreMessage
{
public string MessageType
{
get { return GetType().ToString(); }
}
public string ImageTooltip
{
get { return _imagetooltip; }
set { _imagetooltip = value; }
}
public string Image
{
get { return _image; }
set { _image = value; }
}
public int? RelevantQualityId { get; set; }
protected void StoreQualityInfo(Quality q)
{
PyramidNumberIncreaseLimit = q.PyramidNumberIncreaseLimit;
RelevantQualityId = q.Id;
RelevantQualityName = q.Name;
ImageTooltip = "<strong>" + q.Name + "</strong><br/>" + q.Description + "<br>" +
q.EnhancementsDescription;
Image = q.Image;
}
public virtual IResultPresentation GetPresentation(IResultFormatter formatter)
{
return formatter.GetResult(this);
}
}
UserController - Working example.
public partial class UserController : Controller
{
private readonly IMessageDisplayFetch _messageDisplayFetch;
public UserController(IMessageDisplayFetch messageDisplayFetch)
{
_messageDisplayFetch = messageDisplayFetch;
}
public virtual ActionResult MessagesForStoryletWindow()
{
var activeChar = _us.CurrentCharacter();
IEnumerable<IResultPresentation> messages;
messages = _messageDisplayFetch.FlushMessagesAsPresentations(_storyFormatter);
var vd = new MessagesViewData(messages)
{
Character = new CharacterViewData(activeChar),
};
return View(Views.Messages, vd);
}
}

After Implementing an UITestPropertyProvider, AccessibleName is still not a valid Searchproperty

I need to implement automatic UI Tests for a Delphi Application with Visual Studio Coded UI Tests. I have already implemented the IAccessible Interface to my Delphi-Contols. It works fine and i get the AccessibleName from the Control.
Then i implemented an extension for visual studio. In this extension i have my own PropertyProvider-, ExtensionPackage- and WinControl-Class.
PropertyProvider:
namespace CUITExtension
{
public class AccessibleNamePropertyProvider : UITestPropertyProvider
{
private static Dictionary<string, UITestPropertyDescriptor> accessibleNamePropertyMap = null;
private static Dictionary<string, UITestPropertyDescriptor> AccessibleNamePropertyMap
{
get
{
if (accessibleNamePropertyMap == null)
{
UITestPropertyAttributes read = UITestPropertyAttributes.Readable
| UITestPropertyAttributes.DoNotGenerateProperties;
accessibleNamePropertyMap = new Dictionary<string, UITestPropertyDescriptor>
(StringComparer.OrdinalIgnoreCase);
accessibleNamePropertyMap.Add("AccessibleName", new UITestPropertyDescriptor(typeof(string), read));
}
return accessibleNamePropertyMap;
}
}
public override UITestPropertyDescriptor GetPropertyDescriptor(UITestControl uiTestControl, string propertyName)
{
return AccessibleNamePropertyMap[propertyName];
}
public override ICollection<string> GetPropertyNames(UITestControl uiTestControl)
{
if (uiTestControl.ControlType.NameEquals("Custom"))
{
// the keys of the property map are the collection of property names
return AccessibleNamePropertyMap.Keys;
}
throw new NotSupportedException();
}
public override object GetPropertyValue(UITestControl uiTestControl, string propertyName)
{
if (String.Equals(propertyName, "AccessibleName", StringComparison.OrdinalIgnoreCase))
{
object[] native = uiTestControl.NativeElement as object[];
IAccessible acc = native[0] as IAccessible;
return acc.accName;
}
throw new NotSupportedException();
}
public override int GetControlSupportLevel(UITestControl uiTestControl)
{
if (string.Equals(uiTestControl.TechnologyName, "MSAA",
StringComparison.OrdinalIgnoreCase) &&
uiTestControl.ControlType.NameEquals("Custom"))
{
return (int)ControlSupport.ControlSpecificSupport;
}
// This is not my control, so return NoSupport
return (int)ControlSupport.NoSupport;
}
public override string[] GetPredefinedSearchProperties(Type specializedClass)
{
return null;
}
public override string GetPropertyForAction(UITestControl uiTestControl, UITestAction action)
{
return null;
}
public override string[] GetPropertyForControlState(UITestControl uiTestControl, ControlStates uiState, out bool[] stateValues)
{
stateValues = null;
return null;
}
public override Type GetPropertyNamesClassType(UITestControl uiTestControl)
{
if (uiTestControl.ControlType.NameEquals("Custom"))
return typeof(AccessibleControl.PropertyNames);
return null;
}
public override Type GetSpecializedClass(UITestControl uiTestControl)
{
if (uiTestControl.ControlType.NameEquals("Custom"))
return typeof(AccessibleControl);
return null;
}
public override void SetPropertyValue(UITestControl uiTestControl, string propertyName, object value)
{
return;
}
}
}
ExtensionPackage:
[assembly: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestExtensionPackage(
"AccessibleNameExtensionPackage",
typeof(CUITExtension.AccessibleNameExtensionPackage))]
namespace CUITExtension
{
class AccessibleNameExtensionPackage : UITestExtensionPackage
{
public override string PackageDescription
{
get { return "Supports coded UI testing by using the AccessibleName"; }
}
public override string PackageName
{
get { return "AccessibleName Extension Package"; }
}
public override string PackageVendor
{
get { return "Microsoft (sample)"; }
}
public override Version PackageVersion
{
get { return new Version(1, 0); }
}
public override Version VSVersion
{
get { return new Version(14, 0); }
}
public override void Dispose() { }
public override object GetService(Type serviceType)
{
if (serviceType == typeof(UITestPropertyProvider))
{
if (propertyProvider == null)
{
propertyProvider = new AccessibleNamePropertyProvider();
}
return propertyProvider;
}
return null;
}
private UITestPropertyProvider propertyProvider = null;
}
}
WinControl:
namespace CUITExtension
{
public class AccessibleControl : WinControl
{
public AccessibleControl(UITestControl c) : base(c)
{
TechnologyName = "MSAA";
SearchProperties.Add(UITestControl.PropertyNames.ControlType, "Custom");
}
public virtual string AccessibleName
{
get
{
return (string)GetProperty("AccessibleName");
}
}
}
}
Now the Coded UI Test Builder is showing the AccessibleName and is also generating AccessibleName as a SearchProperty.
UIMap:
public AccessibleControl UIItemCustom
{
get
{
if ((this.mUIItemCustom == null))
{
this.mUIItemCustom = new AccessibleControl(this);
#region Search Criteria
this.mUIItemCustom.SearchProperties["AccessibleName"] = "UniqueName1";
this.mUIItemCustom.SearchProperties[WinControl.PropertyNames.ClassName] = "TEdit";
this.mUIItemCustom.WindowTitles.Add("Title");
#endregion
}
return this.mUIItemCustom;
}
}
*I have changed the Searchproperties here (only for the post, i didnt changed the generated code)
Now when I start the test, I get an exception that says that AccessibleName is not an valid searchproperty. I got this exception before, when i havent implemented the extension yet. But I thougth by implementing the propertyprovider AccessibleName should be a valid searchproperty now.
I tried to debug it, but it seems like by searching the Control it doesnt use the propertyprovider and i have no idea why?
I hope you can help me and if you need more information just ask.
Paul

I got the problem with the valid searchproperty to work.
I overrode the GetValidSearchProperties method from WinControl.
protected override Dictionary<string, bool> GetValidSearchProperties()
{
Dictionary<string, bool> searchProperties = base.GetValidSearchProperties();
if (!searchProperties.ContainsKey("AccessibleName"))
searchProperties.Add("AccessibleName", true);
return searchProperties;
}

run time error when Casting (down casting) a type to another subType

in numerous other Types I have created it is possible to downCast a type
and i usually Create An Extension method too so it will be easier to manage...
BaseTypeM
BTDerV : BaseTypeM
BTDerLastDescndnt : BTDerV
now i create A LastDerived Type and assign its value To ParentType
BTDerV BTDer;
BTDerLastDescndnt BTDerLastDesc = new BTDerLastDescndnt(parA, ParB);
this.BTDer = BTDerLastDesc;
then using the downCast Extension
var LDesc = this.BTDer.AsBTDerLastDescndnt();
which is actually
public static BTDerLastDescndnt AsBTDerLastDescndnt(this BTDerV SelfBTDerV )
{
return (BTDerLastDescndnt)SelfBTDerV;
}
now when i do this as the code below, here it does compile but gives me a run-time error
//BTDerV---v v---BaseTypeM
public class SqlParDefV : SqlParameterM
{
public override SqlSpParDefMeta ParDMT
{
get {
return base.ParDMT;
}
set {
base.ParDMT = value;
}
}
public SqlParDefV(int bsprpOrdinal, string bsprpParName, MSSTypesS bdprpTypeS, bool bsprpIsDbStuctured, bool bsprpIsReq = true, ParameterDirection bsprpDirection = ParameterDirection.Input)
{
this.ParDMT = new SqlSpParDefMeta(bsprpOrdinal, bsprpParName, bdprpTypeS, bsprpIsReq, bsprpIsDbStuctured, bsprpDirection);
}
}
//BTDerLastDescndnt---v
public sealed class SqlParTvDrecDefinitionVScl : SqlParDefV
{
public override SqlSpParDefMeta ParDMT
{
get {
return base.ParDMT;
}
set {
base.ParDMT = value;
}
}
public SprocTvTargetSF.currentSDTObjType SqlObjType { get; set; }
public SqlMetaData[] Meta { get; set; }
public SqlParTvDrecDefinitionVScl(int bsprpOrdinal, string bsprpParName, SprocTvTargetSF.currentSDTObjType ctrSqlObjType, SqlMetaData[] parGeneratedSqlMetaData, MSSTypesS bdprpTypeS, bool bsprpIsDbStuctured, bool bsprpIsReq = true, ParameterDirection bsprpDirection = ParameterDirection.Input)
: base(bsprpOrdinal, bsprpParName, bdprpTypeS, bsprpIsDbStuctured, bsprpIsReq, bsprpDirection)
{
this.SqlObjType = ctrSqlObjType;
this.Meta = parGeneratedSqlMetaData;
}
}
is there something unusual here or am i confused and missed some basic rule ?

I am unsure of the precise reasons a cast from Derived to MoreDerived fails here. However, a potential workaround (note: possibly code smell) is the as operator:
public static MoreDerived AsMoreDerived (this Derived d)
{
return d as MoreDerived;
}
Note that as effectively attempts the cast and returns null, so you'll need an appropriate check there.