We Keep Coding

Serializing an IEnumerable in protobuf-net

I have a library of fairly heavy-weight DTOs that is currently being used by some WCF services. We are attempting to bring it into protobuf-net world, with as little modification as possible. One particular set of items is giving me trouble in serialization. I'm going to simply them here because it gets a little complicated, but the gist of the problem is:
public class Key
{
public string Id {get; set;}
}
public class KeyCollection : IEnumerable<Key>
{
private readonly List<Key> list;
#region IEnumerable
// etc...
#endregion
}
public class Item
{
public long Id { get; set; }
}
public abstract class ContainerBase
{ }
public abstract class ContainerBase<T> : ContainerBase
where T : Item
{ }
public abstract class ContainerType1Base : ContainerBase<Item>
{
public KeyCollection Keys { get; set; }
}
public class ContainerType1 : ContainerType1Base
{ }
I've left out the decorators because I don't they're the problem, mostly because if I add void Add(Key item) { } to KeyCollection the whole thing seems to work. Otherwise, I run into problems attempting to serialize an instance of ContainerType1.
Actually, changing the signature of KeyCollection is kind of prohibitive, so I'm attempting to follow this answer to try to do it programatically. Specifically, setting itemType and defaultType to null on the "Keys" ValueMember of ContainerType1, ContainerType1Base and ContainerBase<Item>. I also set IgnoreListHandling to true on KeyCollection... which totally doesn't work. I get a generic "failed to deserialize" exception on the client, which I can post here if it would help. On the server side, I serialize it out using Serializer.Serialize(), and I spit out Serializer.GetProto<>() as well as JSON of the object, and they all seem to be work okay.
How can I turn off the list handling? Related to that, is there a way to turn on extra debugging while serializing to try to get some more information of the problem?

Fundamentally, the code shown looks fine. Unfortunately, there's currently a "feature" in gRPC that means that it discards the original exception when a marshaller (serializer) fails for some reason, so gRPC does not currently expose the actual problem. I have submitted a fix for this - it may or may not be accepted.
In the interim, I suggest that you simply remove gRPC from the equation, and simulate just the marshaller workload; to do this, on the server: generate the data you are trying to send, and do:
var ms = new MemoryStream();
Serializer.Serialize(ms, yourDataHere);
var payload = Convert.ToBase64String(ms.ToArray());
and obtain the value of payload (which is just a string). Now at the client, reverse this:
var ms = new MemoryStream(Convert.FromBase64String(thatStringValue));
Serialize.Deserialize<YourTypeHere>(ms);
My expectation here is that this should throw an exception that will tell you what the actual problem is.
If the gRPC change gets merged, then the fault should be available via:
catch (RpcException fault)
{
var originalFault = fault.Status.DebugException;
// ^^^
}

Accessing properties with a dot in their name

I am trying to deserialize JSON. My root object has a single property "en.pickthall". I am using dynamic type for reading my JSON. I thought I could just do away with "." in the property since its a local JSON file but then there must be some way to access such a property
var result = App_Code.FileIOHelper.ReadFromDefaultFile("ms-appx:///Assets/en.pickthall.json");
dynamic stuff = JsonConvert.DeserializeObject(result);
foreach(var x in stuff.(en.pickthall)) //Tried this intellisense didn't like it
{
}

You could create a root class to deserialize into and use JsonProperty
public class Root
{
// Use the proper type instead of object
[JsonProperty(PropertyName = "en.pickthall")]
public IEnumerable<object> EnPickthall { get; set; }
public Root() { }
}
Used as follows
Root stuff = JsonConvert.DeserializeObject<Root>(result);
foreach(var x in stuff.EnPickthall)
{
}

You could serialize not to dynamic but to JObject and then access your property via
JObject stuff = JsonConvert.DeserializeObject<JObject>(Jsonstring);
var x = stuff.Value<String>("my.property")

C# doesn't have any way of quoting identifiers. If it's not a valid identifier, your only option is reflection.
However, it's possible the object returned by your JSON deserializer changed the identifiers to make them useable in C# - you might want to enumerate all the properties to check if that is the case. A dynamic object with indexers might also be a solution (allowing e.g. stuff["en.pickthall"]).
Another alternative is to change the way the serializer maps properties. For example, Newtonsoft.Jsoft allows you to customize this using a IContractResolver. It's quite easy to replace the . for something more C#-sane in this way.

I know you said you were using a dynamic type for your JSON deserialization, but I just wanted to point out that there is a .NET RESTful client out there that supports this with static model definitions too. For you or for anyone else who happens upon this response when searching for an answer to their problems with dots in property names in C# REST calls.
As of the newly released RestSharp 106.1.0 (and I do mean this version because this support was just added), it can handle renaming properties with a dot in their name via the DeserializeAs attribute. An example being when I call the ElasticSearch API for a _cat call with the following model:
public class CatResponse
{
public string index { get; set; }
...
[DeserializeAs(Name = "docs.count")]
public string docscount { get; set; }
}
And actually get back the docs.count property deserialized into docscount now:
var resource = $"_cat/indices/{indexPattern}?format=json&pretty=true";
var request = new RestRequest(resource, Method.GET);
var response = client.Execute<List<CatResponse>>(request);
This support is out of the box and doesn't need to use the Newtonsoft.Json.JsonSerializer which I have also heard is a possible solution to this problem but which I couldn't get to work.

With a dynamic object and NewtonSoft.Json:
dynamic json = JValue.Parse(result);
int Opens = Convert.ToInt32(json.opens);
int Clicks = Convert.ToInt32(json.clicks);
string State = json.state;

Produce different serialized JSON for a given class in different scenarios

Update 1: for reasons I won't go into, I want to avoid having anything other than the properties to be persisted in my entity objects. This means no extra properties or methods...
I have an entity called Entity1 with (say) 10 public properties. In
one place in my code I want to output serialized JSON with (say) 3 of
those fields, in a second place I need to output 7 fields and in a
third place I might need to output (say) all 10 fields. How do I do
this using Newtonsoft's JSON library?
I can't use [JsonIgnore] or [DataMember] as that will apply to all
cases, so I won't be able to create "custom views" of the data (my own
terminology :-).
I tried to achieve this using an interface:
public interface Entity1View1
{
string Property1;
string Property2;
string Property5;
}
had Entity1 implement Entity1View1 and I passed an
IList<Entity1View1> to the JSON serializer (the objects were
actually just Entity1 objects). Didn't work: the serializer output
all the 10 public properties of Entity1.
The only other way I could think of was to implement
Entity1Wrapper1, Entity1Wrapper2 etc. type of classes where each
object would hold a corresponding instance of Entity1 and in turn
expose only those public properties that correspond to the properties
I want to show in "View1", "View2" etc. Then I pass lists of these
wrapper objects to the serializer (should work, haven't tried it yet).
Is there a better way?
If it matters, here's my configuration:
.Net 4.5
MVC 5

Don't know it that's the best way... but that's one.
One good point is that it will work either with json serialization or xml serialization, for example (which you may don't mind at all).
You can use ShouldSerialize<yourpropertyName> to manage what is serialized or not. <yourpropertyName> must match exactly the name of the property you wanna manage.
For example
public class Entity {
//assuming you want the default behavior to be "serialize all properties"
public Entity() {
ShouldSerializeProperty1 = true;
ShouldSerializeProperty2 = true;
ShouldSerializeProperty3 = true;
}
public string Property1 {get;set;}
public bool ShouldSerializeProperty1 {get;set;}
public string Property2 {get;set;}
public bool ShouldSerializeProperty2 {get;set;}
public int Property3 {get;set;}
public bool ShouldSerializeProperty3 {get;set;}
}
Then you could do, before all your serialization (of course, this could / should be extension methods).
var list = myListOfEntity;
//serialization1
foreach (var element in list) {
element.ShouldSerializeProperty3 = false;
}
//or serialization2
foreach (var element in list) {
element.ShouldSerializeProperty2 = false;
element.ShouldSerializeProperty3 = false;
}

I just wanted to make sure that this was the final step in processing.
You can create anonymous objects to serialize based on circumstance:
var json1Source1 = new {
Property1 = entityView1.Property1,
Property3 = entityView1.Property3
};
var json1Source2 = new {
Property2 = entityView1.Property2,
Property3 = entityView1.Property3
};
You can create jsonSource1 (or 2, 3, 4 etc) as anonymous objects that capture just what you need and then serialize them. The serializer will not care that they are anonymous.

Update 1:
To conditionally serialize a property, add a method that returns boolean with the same name as the property and then prefix the method name with ShouldSerialize..
This means that the solution suggested by Raphaël Althaus doesn't work as it relies on properties, whereas the serializer's documentation mentions that it has to be a method. I have verified that only a method returning a bool works as expected.
Original:
I finally went with a mix of Wrapper classes and the methodology suggested by Raphaël Althaus (with modifications): use Wrappers where some amount of sophistication may be required and use Raphaël's suggestion when simplicity will do.
Here's how I am using wrappers (intentionally left out null checks):
public class Entity1View1
{
protected Entity1 wrapped;
public Entity1View1(Entity1 entity)
{
wrapped = entity;
}
public String Property1
{
get { return wrapped.Property1; }
}
public String Property2
{
get { return wrapped.Property2; }
}
public String Property3
{
get { return wrapped.Property3.ToUpper(); }
}
}
This allows me to modify properties as their values are returned (as done with Property3 above) and lets me leverage inheritance to create new ways of serialization. For example, I can flatten the structure/hierarchy:
public class Entity1View2 : Entity1View1
{
pulic Entity1View2(Entity1 entity) : base(entity) { }
public long? SubEntityID
{
get { return wrapped.SubEntity.ID; }
}
}
For simpler cases where complexity/transformation of this sort is not required, I can simply use the ShouldSerialize* methods.
Same entity classes, different serialization outputs.

Solution for CA2227 or better approach?

I'm only using Code Analysis for cleaning, organizing and ensuring these changes are globally performed for all instances of a particular warning. I'm down to the final, and it's CA2227.
CA2227 Collection properties should be read only Change '' to be
read-only by removing the property setter.
Note this is for mapping of EDI documents. These classes are to represent a whole or part of an EDI document.
public class PO1Loop
{
public SegmentTypes.PO1LoopSegmentTypes.PO1 PO1 { get; set; }
public Collection<SegmentTypes.PO1LoopSegmentTypes.PID1> PIDRepeat1 { get; set; }
public Collection<SegmentTypes.PO1LoopSegmentTypes.PID2> PIDRepeat2 { get; set; }
public SegmentTypes.PO1LoopSegmentTypes.PO4 PO4 { get; set; }
/* Max Use: 8 */
public Collection<SegmentTypes.PO1LoopSegmentTypes.ACK> ACKRepeat { get; set; }
}
You can see all of the Collection properties will give me this warning, and there are hundreds of them. When using the above class I instantiate it without any data. Then externally I add the data and set each individual variable through its public accessor. I do not instantiate this class with all the data prepared and passed using a constructor method (IMO for the size these can reach it can easily wreak havoc on the eyes). When complete and all properties are assigned the class as a whole is then used to generate that part of a document it represents.
My question is, for the usage described above, what would be a better approach for setting this up correctly? Do I keep the public accessors and suppress this warning entirely, or is there a entirely different solution that would work?

Here's what MSDN says about the error, and also how you can avoid it.
Here's my take on the issue.
Consider, the following class:
class BigDataClass
{
public List<string> Data { get; set; }
}
This class will throw that exact same issue. Why? Because Collections do not need a setter. Now, we can do anything with that object: assign Data to an arbitrary List<string>, add elements to Data, remove elements from Data, etc. If we remove the setter, we only lose the ability to directly assign to that property.
Consider the following code:
class BigDataClass
{
private List<string> data = new List<string>();
public List<string> Data { get { return data; } } // note, we removed the setter
}
var bigData = new BigDataClass();
bigData.Data.Add("Some String");
This code is perfectly valid and in fact the recommended way to do things. Why? Because the List<string> is a reference to a memory location, that contains the remainder of the data.
Now, the only thing you cannot now do with this, is directly set the Data property. I.e. the following is invalid:
var bigData = new BigDataClass();
bigData.Data = new List<string>();
This is not necessarily a bad thing. You'll notice that on many .NET types this model is used. It's the basics of immutability. You usually do not want direct access to the mutability of Collections, as this can cause some accidental behavior that has strange issues. This is why Microsoft recommends you omit setters.
Example:
var bigData = new BigDataClass();
bigData.Data.Add("Some String");
var l2 = new List<string>();
l2.Add("String 1");
l2.Add("String 2");
bigData.Data = l2;
Console.WriteLine(bigData.Data[0]);
We might be expecting Some String, but we'll get String 1. This also means that you cannot reliably attach events to the Collection in question, so you cannot reliably determine if new values are added or values are removed.
A writable collection property allows a user to replace the collection with a completely different collection.
Essentially, if you only ever need to run the constructor, or assignment, once, then omit the set modifier. You won't need it, direct assignment of collections is against best-practices.
Now, I'm not saying never use a setter on a Collection, sometimes you may need one, but in general you should not use them.
You can always use .AddRange, .Clone, etc. on the Collections, you only lose the ability of direct assignment.
Serialization
Lastly, what do we do if we wish to Serialize or Deserialize a class that contains our Collection without a set? Well, there is always more than one way to do it, the simplest (in my opinion) is to create a property that represents the serialized collection.
Take our BigDataClass for example. If we wished to Serialize, and then Deserialize this class with the following code, the Data property would have no elements.
JavaScriptSerializer jss = new JavaScriptSerializer();
BigDataClass bdc = new BigDataClass();
bdc.Data.Add("Test String");
string serd = jss.Serialize(bdc);
Console.WriteLine(serd);
BigDataClass bdc2 = jss.Deserialize<BigDataClass>(serd);
So, to fix this, we can simply modify our BigDataClass a bit to make it use a new string property for Serialization purposes.
public class BigDataClass
{
private List<string> data = new List<string>();
[ScriptIgnore]
public List<string> Data { get { return data; } } // note, we removed the setter
public string SerializedData { get { JavaScriptSerializer jss = new JavaScriptSerializer(); return jss.Serialize(data); } set { JavaScriptSerializer jss = new JavaScriptSerializer(); data = jss.Deserialize<List<string>>(value); } }
}
Another option is always the DataContractSerializer (which is really a better option, in general.) You can find information about it on this StackOverflow question.

With current VS2019 we can simply do this:
public List<string> Data { get; } = new List<string>();
This satisfies CA2227 and can be serialized/deserialized.
The deserialization works because List<> has an "Add" method, and the serializer knows how to handle a read-only collection property with an Add method (the property is read-only but not the elements) (I use Json.Net, other serializers may behave differently).
Edit:
As pointed out it should be "=" and not "=>" (compiler will prevent you using "=>"). If we used "public List Data => new List();" then it would create a new list every time the property was accessed which is not what we want either.
Edit:
Note that this will NOT work if the type of the property is an interface, such as IList
Edit:
I think the handling of interfaces is determined by the serializer used. The following works perfectly. I'm sure all common serializers know how to handle ICollection. And if you have some custom interface that does not implement ICollection then you should be able to configure the serializer to handle it, but in that case CA2227 probably won't be triggered making it irrelevant here. (As it is a read-only property you have to assign a concrete value within the class so it should always be serializing and de-serializing a non-null value)
public class CA2227TestClass
{
public IList Data { get; } = new List<string>();
}
[TestMethod]
public void CA2227_Serialization()
{
var test = new CA2227TestClass()
{
Data = { "One", "Two", "Three" }
};
var json = JsonConvert.SerializeObject(test);
Assert.AreEqual("{\"Data\":[\"One\",\"Two\",\"Three\"]}", json);
var jsonObject = JsonConvert.DeserializeObject(json, typeof(CA2227TestClass)) as CA2227TestClass;
Assert.IsNotNull(jsonObject);
Assert.AreEqual(3, jsonObject.Data.Count);
Assert.AreEqual("One", jsonObject.Data[0]);
Assert.AreEqual("Two", jsonObject.Data[1]);
Assert.AreEqual("Three", jsonObject.Data[2]);
Assert.AreEqual(typeof(List<string>), jsonObject.Data.GetType());
}

💡 Alternative Solution 💡
In my situation, making the property read-only was not viable because the whole list (as a reference) could change to a new list.
I was able to resolve this warning by changing the properties' setter scope to be internal.
public List<Batch> Batches
{
get { return _Batches; }
internal set { _Batches = value; OnPropertyChanged(nameof(Batches)); }
}
Note one could also use private set...
The hint's (achilleas heal) of this warning seems really pointed to libraries for the documentation says (Bolding mine):
An externally visible writable property is a type that implements
System.Collections.ICollection.
For me it was, "Ok, I won't make it viewable externally...." and internal was fine for the app.

Thanks to #Matthew, #CraigW and #EBrown for helping me understanding the solution for this warning.
public class PO1Loop
{
public SegmentTypes.PO1LoopSegmentTypes.PO1 PO1 { get; set; }
public Collection<SegmentTypes.PO1LoopSegmentTypes.PID1> PIDRepeat1 { get; private set; }
public Collection<SegmentTypes.PO1LoopSegmentTypes.PID2> PIDRepeat2 { get; private set; }
public SegmentTypes.PO1LoopSegmentTypes.PO4 PO4 { get; set; }
/* Max Use: 8 */
public Collection<SegmentTypes.PO1LoopSegmentTypes.ACK> ACKRepeat { get; private set; }
public PO1Loop()
{
PIDRepeat1 = new Collection<SegmentTypes.PO1LoopSegmentTypes.PID1>();
PIDRepeat2 = new Collection<SegmentTypes.PO1LoopSegmentTypes.PID2>();
ACKRepeat = new Collection<SegmentTypes.PO1LoopSegmentTypes.ACK>();
}
}
When wanting to assign data to the collection types use AddRange, Clear or any other variation of method for modifying a collection.

Only while binding DTO, you need to suppress warnings.
otherwise a custom ModelBinder is required custom ModelBinder to bind collections.
quoting the rule documentation:
When to suppress warnings
You can suppress the warning if the property is part of a Data Transfer Object (DTO) class.
Otherwise, do not suppress warnings from this rule.
https://learn.microsoft.com/pt-br/visualstudio/code-quality/ca2227?view=vs-2019

DTOs often require serialization and deserialization. Thus, they are required to be mutable.
Having to create an alternate backing property is a pain.
Simply change the property type from List<string> to IReadOnlyList<string> then this works as expected without CA2227.
The collection is set via the property but you can also cast to List<string> if you wish to append or delete items.
class Holder
{
public IReadOnlyList<string> Col { get; set; } = new List<string>();
}
var list = new List<string> { "One", "Two" };
var holder = new Holder() { Col = list } ;
var json = JsonConvert.SerializeObject(holder);
// output json {"Col":["One","Two"]}
var deserializedHolder = JsonConvert.DeserializeObject<Holder>(json);

I had to fix some of the CA2227 violations, so i had to add the "readonly" keyword to the collection field and then of course, had to remove the setter property. Some code that have used the setter, just created a new collection object which initially was empty. This code sure did not compile so i had to add a SetXxx() method in order to realize the missing setter's functionality. I did it like this:
public void SetXxx(List<string> list)
{
this.theList.Clear();
this.theList.AddRange(list);
}
The code of callers using the setter has been replaced with a call to the method SetXxx().
Instead of creating a complete new list, the existing list now will be cleared and filled with new items from another list, passed in as a parameter. The original list, due to the fact it is readonly and created only once, will always remain.
I believe this is also a good way to avoid that the garbagae collector has to delete old objects that got out of scope and second, to create new collection objects although there is already one.

As an addition to Der Kommissar's excellent answer.
Starting with .NET 5 (C# 9.0) there are init-only properties. These properties are only settable under specific circumstances, see here for reference.
The following example should not raise a warning CA2227, yet still allow for the collection being set during object initialization.
using System.Collections.Generic;
namespace BookStore
{
public class BookModel
{
public ICollection<string> Chapters { get; init; }
}
}
Note that the current version of the .NET SDK still raises a warning when using the built-in analyzer (not the NuGet package). This is a known bug and should be fixed in the future.

To cover all the possible scenarios to resolve CA2227 error:
This covers the Entity relationship mapping when we use Entity Framework.
class Program
{
static void Main(string[] args)
{
ParentClass obj = new ParentClass();
obj.ChildDetails.Clear();
obj.ChildDetails.AddRange();
obj.LstNames.Clear();
obj.LstNames.AddRange();
}
}
public class ChildClass
{ }
public class ParentClass
{
private readonly ICollection<ChildClass> _ChildClass;
public ParentClass()
{
_ChildClass = new HashSet<ChildClass>();
}
public virtual ICollection<ChildClass> ChildDetails => _ChildClass;
public IList<string> LstNames => new List<string>();
}

C# 10gen and mongo: deserialization for members as interfaces

All
Consider this example:
private class CollectionHolder
{
public ObjectId Id { get; set; }
public MyCollection Collection { get; set; }
}
private class MyCollection : List<int>
{
public MyCollection(List<int> a)
{
this.AddRange(a);
}
}
private static void CollectionTest()
{
var collection = database.GetCollection<MyCollection>("collectionTest");
collection.RemoveAll();
collection.Save(new CollectionHolder { Collection = new MyCollection(new List<int> { 1, 2, 3, 4, 5 }) });
var x = collection.AsQueryable().First(); //exception!
x.ForEach(Console.WriteLine);
}
The marked line throws exception
An error occurred while deserializing the Collection property of class MongoDriverTest.Program+CollectionHolder: An error occurred while deserializing the Capacity property of class MongoDriverTest.Program+MyCollection: Object reference not set to an instance of an object.
I am not sure, is this a bug in 10gen mongo driver, or is this impossible to implement?
How do You think, should this be posted as a bug?
Moreover -- what is the best way to avoid such kind of errors?

The problem with custom collections in 1.4.2 and earlier is that since there is no serializer registered for your custom collection the C# driver attempts to serialize it using the BsonClassMapSerializer. But the BsonClassMapSerializer requires the class being serialized to expose all the data to be serialized as public get/set properties (which your base class List<T> does not).
The only thing that changes in 1.5 is how the driver chooses which serializer to use when a POCO implements IEnumerable or IDictionary.
You can use custom collections already in 1.4.2 and earlier by explicitly registering a serializer for your custom collection like this:
BsonSerializer.RegisterSerializer(typeof(MyCollection), new EnumerableSerializer<int>());

Currently, custom collections are not supported. There is already implemented in master and will exist in release 1.5 for this. Until then, you can't use custom collections to get the behavior you are requesting.