We Keep Coding

Handling temporarily null reference types when deserializing objects in C# 8.0?

I am trying C# 8.0 out, and I want to enable the null reference checking for the entire project. I am hoping I can improve my code design, and without disabling the nullability context in any code scopes.
I encountered a problem when I deserialize an object graph. The objects have references with one another but, for the final user view, all references in the object graph must have a value.
In other words, during the deserialization process, references may be null, but after all objects have finished loading, a final process will link all of the objects together, thus resolving those null references.
Workarounds
I have been able to address this using a few different techniques, and they each work as expected. They also expand the code considerably, however, by introducing a lot of extra scaffolding.
Option #1: Shadow Class
For example, I tried writing a paired class for each kind of object, using these as an intermediate object during deserialization. In these paired classes, all reference are allowed to be null. After deserialization completes, I copy all fields from these classes and convert them to the real object. Of course, with this approach, I need write a lot of extra code.
Option #2: Shadow Members
Alternatively, I tried to put a nullable field and a non-nullable property. This is similar to the previous approach, but I'm using paired members instead of paired classes. Then I add an internal setter for each field. This approach has less code than the first, but it still increases my code base considerably.
Option #3: Reflection
Traditionally, without considering performance, I would have managed deserialization using reflection so that there’s almost has no extra code on a per class basis. But writing my own parsing code has some benefits—for example, I can output more useful error messages including tips on how callers can resolve issues.
But when I introduce the nullable fields, my parsing code increases considerably—and with the sole purpose of satisfying the code analysis.
Sample Code
For the sake of demonstration, I simplified the code as much as possible; my actual classes obviously do much more than this.
class Person
{
private IReadOnlyList<Person>? friends;
internal Person(string name)
{
this.Name = name;
}
public string Name { get; }
public IReadOnlyList<Person> Friends => this.friends!;
internal SetFriends(IReadOnlyList<Person> friends)
{
this.friends = friends;
}
}
class PersonForSerialize
{
public string? Name { get; set; }
public IReadOnlyList<string> Friends { get; set; }
}
IReadOnlyList<Person> LoadPeople(string path)
{
PersonForSerialize[] peopleTemp = LoadFromFile(path);
Person[] people = new Person[peopleTemp.Count];
for (int i = 0; i < peopleTemp.Count; ++i)
{
people[i] = new Person(peopleTemp[i].Name);
}
for (int i = 0; i < peopleTemp.Count; ++i)
{
Person[] friends = new Person[peopleTemp[i].Friends.Count];
for (int j = 0; j < friends.Count; ++j)
{
string friendName = peopleTemp[i].Friends[j];
friends[j] = FindPerson(people, friendName);
}
people[i].SetFriends(friends);
}
}
Question
Is there a way to satisfy the null reference checking in C# 8.0 for properties that are only temporarily null during deserialization without introducing a lot of extra code for every class?

You’re concerned that while your objects aren't intended to have null members, those members will inevitably be null during the construction of your object graph.
Ultimately, this is a really common problem. It affects, yes, deserialization, but also the creation of objects during e.g., mapping or data binding of e.g. data transfer objects or view models. Often, these members are to be null for a very brief period between constructing an object and setting its properties. Other times, they might sit in limbo during a longer period as your code e.g. fully populates a dependency data set, as required here with your interconnected object graph.
Fortunately, Microsoft has addressed this exact scenario, offering us two different approaches.
Option #1: Null-Forgiving Operator
The first approach, as #andrew-hanlon notes in his answer, is to use the null-forgiving operator. What may not be immediately obvious, however, is that you can use this directly on your non-nullable members, thus entirely eliminating your intermediary classes (e.g., PersonForSerialize in your example). In fact, depending on your exact business requirements, you might be able to reduce your Person class down to something as simple as:
class Person
{
internal Person() {}
public string Name { get; internal set; } = null!;
public IReadOnlyList<Person> Friends { get; internal set; } = null!;
}
Option #2: Nullable Attributes
Update: As of .NET 5.0.4 (SDK 5.0.201), which shipped on March 9th, 2021, the below approach will now yield a CS8616 warning. Given this, you are better off using the null-forgiving operator outlined above.
The second approach gives you the same exact results, but does so by providing hints to Roslyn's static flow analysis via nullable attributes. These require more annotations than the null-forgiving operator, but are also more explicit about what's going on. In fact, I actually prefer this approach just because it's more obvious and intuitive to developers otherwise unaccustomed to the syntax.
class Person
{
internal Person() {}
[NotNull, DisallowNull]
public string? Name { get; internal set; };
[NotNull, DisallowNull]
public IReadOnlyList<Person>? Friends { get; internal set; };
}
In this case, you're explicitly acknowledging that the members can be null by adding the nullability indicator (?) to the return types (e.g., IReadOnlyList<Person>?). But you're then using the nullable attributes to tell consumers that even though the members are marked as nullable:
[NotNull]: A nullable return value will never be null.
[DisallowNull]: An input argument should never be null.
Analysis
Regardless of which approach you use, the end results are the same. Without the null-forgiving operator on a non-nullable property, you would have received the following warning on your members:
CS8618: Non-nullable property 'Name' is uninitialized. Consider declaring the property as nullable.
Alternatively, without using the [NotNull] attribute on a nullable property, you would have received the following warning when attempting to assign its value to a non-nullable variable:
CS8600: Converting null literal or possible null value to non-nullable type.
Or, similarly, upon trying to call a member of the value:
CS8602: Dereference of a possibly null reference.
Using one of these two approaches, however, you can construct the object with default (null) values, while still giving downstream consumers confidence that the values will, in fact, not be null—and, thus, allowing them to consume the values without necessitating guard clauses or other defensive code.
Conversely, when using either of these approaches, you will still get the following warning when attempting to assign a null value to these members:
CS8625: Cannot convert null literal to non-nullable reference type.
That's right: You'll even get that when assigning to the string? property because that's what the [DisallowNull] is instructing the compiler to do.
Conclusion
It’s up to you which of these approaches you take. As they both yield the same results, it’s a purely stylistic preference. Either way, you’re able to keep the members null during construction, while still realizing the benefits of C#’s non-nullable types.

If I understand correctly, your question boils down to:
How do I avoid compiler warnings
CS8618: Non-nullable property 'Name' is uninitialized. Consider declaring the property as nullable.
for simple model classes which are used for serialization?
You can solve the problem by creating a default constructor for which you suppress the warning. Now you want to make sure that this constructor is only used by your deserialization routine (e.g. System.Text.Json or Entity Framework). To avoid unintentional use add annotation [Obsolete] with parameter error=true which would raise compiler error CS0618.
As code:
public class PersonForSerialize
{
#pragma warning disable CS8618 // Non-nullable field is uninitialized. Consider declaring as nullable.
[Obsolete("Only intended for de-serialization.", true)]
public PersonForSerialize()
#pragma warning restore CS8618 // Non-nullable field is uninitialized. Consider declaring as nullable.
{
}
// Optional constructor
public PersonForSerialize(string name, IReadOnlyList<string> friends)
{
Name = name;
Friends = friends;
}
public string Name { get; set; }
public IReadOnlyList<string> Friends { get; set; }
}
Note1: You can let Visual Studio auto-generate the optional constructor using a quick action.
Note2: If you really mean to use the constructor marked as obsolete, you need to remove the error=true parameter. Now you can suppress the warning when calling the parameter-less constructor via #pragma warning disable CA1806.

While certainly highlighted due to non-nullable defaults in C# 8, this is really a common 'circular dependency' problem that has always existed with interdependent construction.
As you found, one standard solution is to use a restricted setter. With C# 8, you may want to use a 'null-forgiving' null! during deserialization of the object graph - allowing you to differentiate an unconstructed set from a valid empty set, and minimize allocation.
Example:
class Person
{
internal Person(string name, IReadOnlyList<Person> friends)
{
Name = name; Friends = friends
}
public string Name { get; }
public IReadOnlyList<Person> Friends {get; internal set;}
}
class SerializedPerson { ... } // Class no longer specifically required.
IEnumerable<Person> LoadPeople(string path)
{
var serializedPeople = LoadFromFile(path);
// Note the use of null!
var people = serializedPeople.Select(p => new Person(p.Name, null!));
foreach(var person in people)
{
person.Friends = GetFriends(person, people, serializedPeople);
}
return people;
}

I'm not sure how you would serialise your graph structure without Name+IdsOnly class especially if a person has plenty properties, but this is out of scope.
Nullability of Friends
Nulls in Friends could be addressed by always having the list.
For example:
public List<Person> Friends { get; set; } = new List<Friends>();
Partly populated objects that look like they're null safe
If you're using Json.NET you could take advantage of JsonConstructor.
Please note with this approach you can end up with null in Person unless you protect against it in the constructor.
using Newtonsoft.Json;
class Person
{
[JsonConstructor]
internal Person(
string name,
List<Person> friends)
{
Name = name; // Can end up being null if left without '?? "No name"';
Friends = friends ?? new List<Person>();
}
public string Name { get; }
public List<Person> Friends { get; set; }
public void AddFriends(IEnumerable<Person> friends)
{
Friends.AddRange(friends);
}
}
This should handle all kinds of data:
Deserialise<Person>("{}");
Deserialise<Person>("{ 'Name': 'John'}");
Deserialise<Person>("{ 'Name': 'John', 'Friends': [] }");
Deserialise<Person>("{ 'Name': 'John', 'Friends': [ {'Name': 'Jenny'}]}");
Deserialise<Person>("{ 'Friends': [{'Name': 'Jenny'}]}");
All above examples should produce a Person object. You could then do validation and fill in the missing details.

Why doesn't Protobuf-net typemodel serialize private readonly members?

I have the following class which I would like to build a protobuf-net typemodel for:
[DataContract]
public class Currency
{
[DataMember(Order = 0)]
private readonly string code;
public Currency(string code)
{
// Initialize members
this.code = code;
}
public string Code
{
get { return this.code; }
}
}
I set up the type model as follows:
var model = TypeModel.Create();
var typeInModel = model.Add(typeof(Currency), true);
typeInModel.UseConstructor = false;
When I look at the "fields" member in the type model after adding this type, I do not see anything there for the code member. What am I missing here?
UPDATE: Can now create type model but serialization still doesn't work.

My colleague found that if you change the DataMember order from:
[DataMember(Order = 0)]
to:
[DataMember(Order = 1)]
that everything suddenly works fine for building the type model, though serialization still doesn't work.

Your code, as written (with the [DataMember(Order=1)] fix from your answer) works fine. The following:
var obj = new Currency("abc");
var clone = (Currency) model.DeepClone(obj);
Console.WriteLine(ReferenceEquals(obj, clone));
Console.WriteLine(clone.Code);
outputs
False
abc
exactly as I would expect - meaning: different object instance, but created successfully with the right values.
If you are doing something different, please let me know. Also note, as per the comment, that reflection to private members is limited on some platforms - most notably Silverlight and Phone 7.

When running WCF code in Partial Trust mode, the serialization and deserialization of readonly fields (both public and private) is not supported. This is because the generated IL is unverifiable and therefore requires elevated permissions.
Take a look at the documentation.

Something in the serialize process skips properties that have no setters.
It seems like a bit of a hack, but I usually do this:
[DataMember]
public string Code {
get { return this.code; }
set { }
}
Once the set is there, even though it does nothing, it'll be picked up in the serialization process.

protobuf-net: incorrect wire-type exception deserializing Guid properties

I'm having issues deserializing certain Guid properties of ORM-generated entities using protobuf-net.
Here's a simplified example of the code (reproduces most elements of the scenario, but doesn't reproduce the behavior; I can't expose our internal entities, so I'm looking for clues to account for the exception). Say I have a class, Account with an AccountID read-only guid, and an AccountName read-write string. I serialize & immediately deserialize a clone.
Deserializing throws an Incorrect wire-type deserializing Guid exception while deserializing.
Here's example usage...
Account acct = new Account() { AccountName = "Bob's Checking" };
Debug.WriteLine(acct.AccountID.ToString());
using (MemoryStream ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize<Account>(ms, acct);
Debug.WriteLine(Encoding.UTF8.GetString(ms.GetBuffer()));
ms.Position = 0;
Account clone = ProtoBuf.Serializer.Deserialize<Account>(ms);
Debug.WriteLine(clone.AccountID.ToString());
}
And here's an example ORM'd class (simplified, but demonstrates the relevant semantics I can think of). Uses a shell game to deserialize read-only properties by exposing the backing field ("can't write" essentially becomes "shouldn't write," but we can scan code for instances of assigning to these fields, so the hack works for our purposes).
Again, this does not reproduce the exception behavior; I'm looking for clues as to what could:
[DataContract()]
[Serializable()]
public partial class Account
{
public Account()
{
_accountID = Guid.NewGuid();
}
[XmlAttribute("AccountID")]
[DataMember(Name = "AccountID", Order = 1)]
public Guid _accountID;
/// <summary>
/// A read-only property; XML, JSON and DataContract serializers all seem
/// to correctly recognize the public backing field when deserializing:
/// </summary>
[IgnoreDataMember]
[XmlIgnore]
public Guid AccountID
{
get { return this._accountID; }
}
[IgnoreDataMember]
protected string _accountName;
[DataMember(Name = "AccountName", Order = 2)]
[XmlAttribute]
public string AccountName
{
get { return this._accountName; }
set { this._accountName = value; }
}
}
XML, JSON and DataContract serializers all seem to serialize / deserialize these object graphs just fine, so the attribute arrangement basically works. I've tried protobuf-net with lists vs. single instances, different prefix styles, etc., but still always get the 'incorrect wire-type ... Guid' exception when deserializing.
So the specific questions is, is there any known explanation / workaround for this? I'm at a loss trying to trace what circumstances (in the real code but not the example) could be causing it.
We hope not to have to create a protobuf dependency directly in the entity layer; if that's the case, we'll probably create proxy DTO entities with all public properties having protobuf attributes. (This is a subjective issue I have with all declarative serialization models; it's a ubiquitous pattern & I understand why it arose, but IMO, if we can put a man on the moon, then "normal" should be to have objects and serialization contracts decoupled. ;-) )
Thanks!

Agreed, you shouldn't need an explicit dependency - DataMember is fine. And protobuf-net uses the same logic re ignore etc. How / where are you storing the data? In my experience the most common cause of this is that people are over-writing a buffer (or file) with different data, and not truncating it (leaving garbage at the end of the stream), as discussed here. Is this related to your scenario?

DataContractSerializer doesn't call my constructor?

I just realized something crazy, which I assumed to be completely impossible : when deserializing an object, the DataContractSerializer doesn't call the constructor !
Take this class, for instance :
[DataContract]
public class Book
{
public Book()
{ // breakpoint here
}
[DataMember(Order = 0)]
public string Title { get; set; }
[DataMember(Order = 1)]
public string Author { get; set; }
[DataMember(Order = 2)]
public string Summary { get; set; }
}
When I deserialize an object of that class, the breakpoint is not hit. I have absolutely no idea how it is possible, since it is the only constructor for this object !
I assumed that perhaps an additional constructor was generated by the compiler because of the DataContract attribute, but I couldn't find it through reflection...
So, what I'd like to know is this : how could an instance of my class be created without the constructor being called ??
NOTE: I know that I can use the OnDeserializing attribute to initialize my object when deserialization begins, this is not the subject of my question.

DataContractSerializer (like BinaryFormatter) doesn't use any constructor. It creates the object as empty memory.
For example:
Type type = typeof(Customer);
object obj = System.Runtime.Serialization.
FormatterServices.GetUninitializedObject(type);
The assumption is that the deserialization process (or callbacks if necessary) will fully initialize it.

There are some scenario's that wouldn’t be possible without this behavior. Think of the following:
1) You have an object that has one constructor that sets the new instance to an "initialized" state. Then some methods are called on that instance, that bring it in a "processed" state. You don’t want to create new objects having the "processed" state, but you still want de serialize / deserialize the instance.
2) You created a class with a private constructor and some static properties to control a small set of allowed constructor parameters. Now you can still serialize / deserialize them.
XmlSerializer has the behavior you expected. I have had a some problems with the XmlSerializer because it DOES need a default constructor. Related to that, sometimes it makes sense to have private property setters. But the XmlSerializer also needs public getter and setter on properties in order to serialize / deserialize.
I think of the DataContractSerializer / BinaryFormatter behavior like suspending the state of an instance during serialization and resuming during deserialization. In other words, the instances are not “constructed” but “restored” to an earlier state.
As you already mentioned, the [OnDeserializing] attribute makes it possible to keep non serialized data in sync.

FWIW, you can call the constructor explicitly from a [OnDeserializing] method:
[OnDeserializing]
public void OnDeserializing(StreamingContext context)
{
this.GetType().GetConstructor(System.Array.Empty<Type>()).Invoke(this, null);
}

Use [OnDeserialized] attribute to initialise your properties.
// This method is called after the object
// is completely deserialized. Use it instead of the
// constructror.
[OnDeserialized]
void OnDeserialized(StreamingContext context)
{
fullName = firstName + " " + lastName;
}
Please refer to microsoft guid-lines:
https://learn.microsoft.com/en-us/dotnet/standard/serialization/serialization-guidelines

In my case, i wanted to create an object to use in a lock-clause. I tried implementing IDeserializationCallback (didn't work because callback only runs after properties have been assigned), [OnDeserialized] (didn't work, same previous reason), and ISerializable (didn't work because the class is decorated with the [DataContractAttribute]).
My workaround was to initialize the field before it's used using Interlocked.CompareExchange. A bit of unnecessary work gets done, but at least now my field gets initialized when a DataContractSerializer creates it.
Interlocked.CompareExchange(ref _sync, new object(), null);

Why isn't my public property serialized by the XmlSerializer?

This is one i struggled with for ages so thought I'd document somewhere. (Apologies for asking and answering a question.)
(C# .net 2.0)
I had a class that was being serialized by XmlSerializer, I added a new public property however it wasn't being included in the output XML.
It's not mentioned in the docs anywhere I could find, but public properties must have a set as well as a get to be serialized! I guess this is because it assumes that if you're going to serialize then you'll want to deserialize from the same file, so only serializes properties that have both a set and a get.

As mentioned, most properties must have both a getter and setter; the main exception to this is lists - for example:
private readonly List<Foo> bar = new List<Foo>();
public List<Foo> Bar {get { return bar; } } // works fine
which will work fine; however, if XmlSerializer finds a setter - it demands that it is public; the following will not work:
public List<Foo> Bar {get; private set;} // FAIL
Other reasons it might not serialize:
it isn't public with get and set (or is readonly for a field)
it has a [DefaultValue] attribute, and is with that value
it has a public bool ShouldSerializeFoo() method that returned false
it has a public bool FooSpecified {get;set;} property or field that returned false
it is marked [XmlIgnore]
it is marked [Obsolete]
Any of these will cause it not to serialize

The point about getter+setter is made in the 3rd paragraph on the "Intro to Xml Serialization" page. It's actually in a call-out box. Can't miss it!
Intro-to-XML Serialization http://www.freeimagehosting.net/uploads/2f04fea2db.png
(having a little too much fun with Freeimagehosting.net)

Also properties that return null are not serialized!

if you don't want to implement proper Setters (because maybe you are neither wanting to deserialize or change an objects value) you can just use dummy setters like this set { }, so that the XMLSerializer works, but nothing happens if you use the Setter...
i.E.
public string ID { get { return _item.ID.ToString(); } set { } }

And if your class inherits a list and also has its own members, only the elements of the list get serialized. The data present in your class members is not captured.
Took some time figuring out this!

One more thing to add about serialization of collections:
The XmlSerializer ignores collections of interfaces!
And by that I mean ignore. While you will get an exception for a line like:
public IFoo Foo { get; set; }
you will not get an exception for:
public ICollection<IFoo> LotsOfFoos { get { return this.fooBackingField; } }

You can implement the IXmlSerializer and do the serialization manually, and benefit from serializing properties, and vice versa, deserializing them using constructors / private field assignment.