We Keep Coding

JSON.NET overwrites values of dictionary by populating [duplicate]

I would like to populate the objects contained within a Dictionary from a JSON file while preserving the object references themselves.
Json.net documentation on PreserveReferencesHandling clearly state that it will not work in case a type implements System.Runtime.Serialization.ISerializable:
Specifies reference handling options for the
Newtonsoft.Json.JsonSerializer. Note that references cannot be
preserved when a value is set via a non-default constructor such as
types that implement System.Runtime.Serialization.ISerializable.
Here is my failing code:
class Model
{
public int Val { get; set; } = 123;
}
...
var model = new Model();
var to_serialize = new Dictionary<int, Model> { { 0, model } }; // works ok with list<Model>
// serialize
var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);
var jsonSerializerSettings = new JsonSerializerSettings();
jsonSerializerSettings.MissingMemberHandling = MissingMemberHandling.Ignore;
jsonSerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.All; // does not work for ISerializable
Assert.AreSame(to_serialize[0], model); // ok!
JsonConvert.PopulateObject(
value: jsonString,
target: to_serialize,
settings: jsonSerializerSettings
);
Assert.AreSame(to_serialize[0], model); // not ok... works ok with list<Model>
My main requirement is that when calling PopulateObject(), the constructor of the Model class will not be invoked. Instead, only its internal field will be updated with the value from the JSON.
In my real case, the Model class contains other values which are not in the JSON and which I don't want to lose:
[JsonObject(MemberSerialization.OptIn)]
class Model
{
[JsonProperty(PropertyName = "val_prop")]
public int Val { get; set; } = 123;
// not in the json file, would like this field to maintain the value
// it had prior to PopulateObject()
public int OtherVal { get; set; } = 456;
}
Is there a way to make this work?

Your problem is similar to the one from JsonSerializer.CreateDefault().Populate(..) resets my values: you would like to populate a preexisting collection, specifically a Dictionary<int, T> for some T, and populate the preexisting values. Unfortunately, in the case of a dictionary, Json.NET will replace the values rather than populate them, as can be seen in JsonSerializerInternalReader.PopulateDictionary() which simply deserializes the value to the appropriate type, and sets it the dictionary.
To work around this limitation, you can create a custom JsonConverter for Dictionary<TKey, TValue> when TKey is a primitive type and TValue is a complex type which merges the incoming JSON key/value pairs onto the preexisting dictionary. The following converter does the trick:
public class DictionaryMergeConverter : JsonConverter
{
static readonly IContractResolver defaultResolver = JsonSerializer.CreateDefault().ContractResolver;
readonly IContractResolver resolver = defaultResolver;
public override bool CanConvert(Type objectType)
{
var keyValueTypes = objectType.GetDictionaryKeyValueType();
if (keyValueTypes == null)
return false;
var keyContract = resolver.ResolveContract(keyValueTypes[0]);
if (!(keyContract is JsonPrimitiveContract))
return false;
var contract = resolver.ResolveContract(keyValueTypes[1]);
return contract is JsonContainerContract;
// Also possibly check whether keyValueTypes[1] is a read-only collection or dictionary.
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
return null;
if (reader.TokenType != JsonToken.StartObject)
throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
IDictionary dictionary = existingValue as IDictionary ?? (IDictionary)serializer.ContractResolver.ResolveContract(objectType).DefaultCreator();
var keyValueTypes = objectType.GetDictionaryKeyValueType();
while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndObject)
{
switch (reader.TokenType)
{
case JsonToken.PropertyName:
var name = (string)reader.Value;
reader.ReadToContentAndAssert();
// TODO: DateTime keys and enums with overridden names.
var key = (keyValueTypes[0] == typeof(string) ? (object)name : Convert.ChangeType(name, keyValueTypes[0], serializer.Culture));
var value = dictionary.Contains(key) ? dictionary[key] : null;
// TODO:
// - JsonConverter active for valueType, either in contract or in serializer.Converters
// - NullValueHandling, ObjectCreationHandling, PreserveReferencesHandling,
if (value == null)
{
value = serializer.Deserialize(reader, keyValueTypes[1]);
}
else
{
serializer.Populate(reader, value);
}
dictionary[key] = value;
break;
default:
throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
}
}
return dictionary;
}
public override bool CanWrite { get { return false; } }
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) { throw new NotImplementedException(); }
}
public static partial class JsonExtensions
{
public static JsonReader ReadToContentAndAssert(this JsonReader reader)
{
return reader.ReadAndAssert().MoveToContentAndAssert();
}
public static JsonReader MoveToContentAndAssert(this JsonReader reader)
{
if (reader == null)
throw new ArgumentNullException();
if (reader.TokenType == JsonToken.None) // Skip past beginning of stream.
reader.ReadAndAssert();
while (reader.TokenType == JsonToken.Comment) // Skip past comments.
reader.ReadAndAssert();
return reader;
}
public static JsonReader ReadAndAssert(this JsonReader reader)
{
if (reader == null)
throw new ArgumentNullException();
if (!reader.Read())
throw new JsonReaderException("Unexpected end of JSON stream.");
return reader;
}
}
public static class TypeExtensions
{
public static IEnumerable<Type> BaseTypesAndSelf(this Type type)
{
while (type != null)
{
yield return type;
type = type.BaseType;
}
}
public static Type[] GetDictionaryKeyValueType(this Type type)
{
return type.BaseTypesAndSelf().Where(t => t.IsGenericType && t.GetGenericTypeDefinition() == typeof(Dictionary<,>)).Select(t => t.GetGenericArguments()).FirstOrDefault();
}
}
Having done so, you will encounter a secondary issue: Json.NET will never use a custom converter to populate the root object. To work around this you will need to call JsonConverter.ReadJson() directly, from some utility method:
public static partial class JsonExtensions
{
public static void PopulateObjectWithConverter(string value, object target, JsonSerializerSettings settings)
{
if (target == null || value == null)
throw new ArgumentNullException();
var serializer = JsonSerializer.CreateDefault(settings);
var converter = serializer.Converters.Where(c => c.CanConvert(target.GetType()) && c.CanRead).FirstOrDefault() ?? serializer.ContractResolver.ResolveContract(target.GetType()).Converter;
using (var jsonReader = new JsonTextReader(new StringReader(value)))
{
if (converter == null)
serializer.Populate(jsonReader, target);
else
{
jsonReader.MoveToContentAndAssert();
var newtarget = converter.ReadJson(jsonReader, target.GetType(), target, serializer);
if (newtarget != target)
throw new JsonException(string.Format("Converter {0} allocated a new object rather than populating the existing object {1}.", converter, value));
}
}
}
}
You will now be able to populate your dictionary as follows:
var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);
var settings = new JsonSerializerSettings
{
Converters = { new DictionaryMergeConverter() },
};
JsonExtensions.PopulateObjectWithConverter(jsonString, to_serialize, settings);
Notes:
PreserveReferencesHandling has no impact on whether dictionary values are populated or replaced. Instead this setting controls whether a serialization graph with multiple references to the same object will maintain its reference topology when round-tripped.
In your question you wrote // works ok with list<Model> but in fact this is not correct. When a List<T> is populated the new values are appended to the list, so Assert.AreSame(to_serialize[0], model); passes purely by luck. If you had additionally asserted Assert.AreSame(1, to_serialize.Count) it would have failed.
While the converter will work for primitive keys such as string and int it may not work for key types that require JSON-specific conversion such as enum or DateTime.
The converter is currently only implemented for Dictionary<TKey, TValue> and takes advantage of the fact that this type implements the non-generic IDictionary interface. It could be extended to other dictionary types such as SortedDictionary<TKey,TValue> if required.
Demo fiddle here.

Newtonsoft.Json DefaultContractResolver remove key from dictionary

I have the following class (I don't want to change the class to solve the issue..):
public class Test
{
public Dictionary<string, string> Data;
[PrivateField]
[JsonIgnore]
public string Name { get { return Data["Name"]; } set { Data.Add("Name", value); } }
public string NotPrivate { get { return Data["NotPrivate"]; } set { Data.Add("NotPrivate", value); } }
}
I want to remove specific key from Data property during serialize, in my case the key in the dictionary is 'Name' because it is marked as private.
var test = new Test();
var settings = new JsonSerializerSettings();
settings.ContractResolver = new IgnorePrivatePropertiesContractResolver();
var places = JsonConvert.SerializeObject(test, settings);
public class IgnorePrivatePropertiesContractResolver : DefaultContractResolver
in IgnorePrivatePropertiesContractResolver I have tried :
override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
but I can't get the Dictionary out of JsonProperty.
I also tried :
JsonProperty CreateProperty(MemberInfo member, MemberSerialization memberSerialization)
but I can't get the Dictionary out of MemberInfo .

Json.NET has no built-in functionality to filter keys from dictionaries while serializing. Therefore you will need to create a custom JsonConverter that does the necessary filtering. Then, since you can't change your class, you will need to apply the converter using a custom contract resolver.
First, the custom JsonConverter:
public class KeyFilteringDictionaryConverter<TKey, TValue> : JsonConverter<IDictionary<TKey, TValue>>
{
readonly HashSet<TKey> toSkip;
public KeyFilteringDictionaryConverter(IEnumerable<TKey> toSkip) => this.toSkip = toSkip?.ToHashSet() ?? throw new ArgumentNullException(nameof(toSkip));
public override void WriteJson(JsonWriter writer, IDictionary<TKey, TValue> value, JsonSerializer serializer) => serializer.Serialize(writer, new KeyFilteringDictionarySurrogate<TKey, TValue>(value, toSkip));
public override bool CanRead => false;
public override IDictionary<TKey, TValue> ReadJson(JsonReader reader, Type objectType, IDictionary<TKey, TValue> existingValue, bool hasExistingValue, JsonSerializer serializer) => throw new NotImplementedException();
}
public class KeyFilteringDictionarySurrogate<TKey, TValue> : IReadOnlyDictionary<TKey, TValue>
{
readonly IDictionary<TKey, TValue> dictionary;
readonly HashSet<TKey> toSkip;
public KeyFilteringDictionarySurrogate(IDictionary<TKey, TValue> dictionary, IEnumerable<TKey> toSkip) : this(dictionary, toSkip ?.ToHashSet()) { }
public KeyFilteringDictionarySurrogate(IDictionary<TKey, TValue> dictionary, HashSet<TKey> toSkip)
{
this.dictionary = dictionary ?? throw new ArgumentNullException(nameof(dictionary));
this.toSkip = toSkip ?? throw new ArgumentNullException(nameof(toSkip));
}
public bool ContainsKey(TKey key) => !toSkip.Contains(key) && dictionary.ContainsKey(key);
public bool TryGetValue(TKey key, out TValue value)
{
if (toSkip.Contains(key))
{
value = default(TValue);
return false;
}
return dictionary.TryGetValue(key, out value);
}
public TValue this[TKey key] => toSkip.Contains(key) ? throw new KeyNotFoundException() : dictionary[key];
public IEnumerable<TKey> Keys => this.Select(p => p.Key);
public IEnumerable<TValue> Values => this.Select(p => p.Value);
public int Count => this.Count(); // Could be made faster?
public IEnumerator<KeyValuePair<TKey, TValue>> GetEnumerator() => dictionary.Where(p => !toSkip.Contains(p.Key)).GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
Next, the custom ContractResolver. The easiest place to apply the custom converter would be in CreateProperties() after all the property information has been created. Overriding CreateObjectContract() would also work.
public class IgnorePrivatePropertiesContractResolver : DefaultContractResolver
{
protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
{
var jsonProperties = base.CreateProperties(type, memberSerialization);
// Apply to all string-keyed dictionary properties named "Data" that do not already have a converter
foreach (var dataProperty in jsonProperties.Where(p => p.PropertyName == "Data" && p.Converter == null))
{
var keyValuePairTypes = dataProperty.PropertyType.GetDictionaryKeyValueTypes().ToList();
if (keyValuePairTypes.Count == 1 && keyValuePairTypes[0][0] == typeof(string))
{
// Filter all properties with PrivateFieldAttribute applied
var ignoreProperties = jsonProperties.Where(p => p.AttributeProvider.GetAttributes(typeof(PrivateFieldAttribute), true).Any()).Select(p => p.PropertyName).ToHashSet();
if (ignoreProperties.Count > 0)
{
dataProperty.Converter = (JsonConverter)Activator.CreateInstance(typeof(KeyFilteringDictionaryConverter<,>).MakeGenericType(keyValuePairTypes[0]), new object [] { ignoreProperties });
}
}
}
return jsonProperties;
}
}
public static class TypeExtensions
{
public static IEnumerable<Type> GetInterfacesAndSelf(this Type type)
=> (type ?? throw new ArgumentNullException()).IsInterface ? new[] { type }.Concat(type.GetInterfaces()) : type.GetInterfaces();
public static IEnumerable<Type []> GetDictionaryKeyValueTypes(this Type type)
=> type.GetInterfacesAndSelf().Where(t => t.IsGenericType && t.GetGenericTypeDefinition() == typeof(IDictionary<,>)).Select(t => t.GetGenericArguments());
}
[AttributeUsage(AttributeTargets.Field | AttributeTargets.Property, AllowMultiple=false)]
public class PrivateFieldAttribute : System.Attribute { }
Finally, use e.g. as follows:
var test = new Test
{
Data = new Dictionary<string, string>
{
{"SomeAdditionalKey", "Some additional value"},
},
Name = "private value",
NotPrivate = "public value",
};
var resolver = new IgnorePrivatePropertiesContractResolver();
var settings = new JsonSerializerSettings
{
ContractResolver = resolver,
};
var json = JsonConvert.SerializeObject(test, Formatting.Indented, settings);
Notes:
This solution doesn't actually remove the private key from the dictionary, it simply skips serializing it. Generally serialization doesn't actually modify the object being serialized. If you really need to actually modify the dictionary during serialization your contract resolver could apply a custom OnSerializing() method to do that, instead of applying a converter.
Your question doesn't specify exactly how to determine the properties to which KeyFilteringDictionaryConverter should be applied. I applied it to all Dictionary<string, TValue> properties named "Data" when there is also a member with PrivateFieldAttribute in the same class. You could restrict it to just Test, or use any other logic that suits your needs, as long as the converter is applied only to a property of type IDictionary<string, TValue> for any type TValue.
You may want to statically cache the contract resolver for best performance.
The converter does not attempt to remove private properties while deserializing as this was not required by your question.
Your Test class, when serialized, includes the value of NotPrivate twice: once as a property, and once as a member of the dictionary property:
{
"Data": {
"SomeAdditionalKey": "Some additional value",
"NotPrivate": "public value"
},
"NotPrivate": "public value"
}
You may want to exclude it from one location or the other.
You wrote, I can't get the Dictionary out of JsonProperty.
That is correct. A contract resolver defines how to map .Net types to JSON. As such, specific instances are not available during contract creation. That's why it is necessary to apply a custom converter, as specific instances are passed into the converter during serialization and deserialization.
Demo fiddle here.

public class IgnorePrivatePropertiesContractResolver : DefaultContractResolver
{
protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
{
IList<JsonProperty> properties = base.CreateProperties(type, memberSerialization);
if (!(type.Name.Equals(typeof(Test).Name)))
{
return properties;
}
var dataProperty = properties.FirstOrDefault(prop => prop.PropertyName.Equals("Data"));
if (dataProperty == null)
return properties;
var privatePropertiesNames = properties.Where(prop => prop.AttributeProvider.GetAttributes(false).OfType<PrivateFieldAttribute>().Any())
.Select(privateProperty => privateProperty.PropertyName);
dataProperty.ValueProvider = new JsonPropertyExtraDataValueProvider(privatePropertiesNames);
return properties;
}
}
public class JsonPropertyExtraDataValueProvider : IValueProvider
{
private IEnumerable<string> _privateKeys;
public JsonPropertyExtraDataValueProvider(IEnumerable<string> privateKeys)
{
_privateKeys = privateKeys;
}
public void SetValue(object target, object value)
{
throw new NotImplementedException();
}
public object GetValue(object target)
{
Dictionary<string, string> value = ((Test)target).ExtraData.ToDictionary(pr => pr.Key, pr => pr.Value);
foreach (var privateKey in _privateKeys)
value.Remove(privateKey);
return value;
}
}

Custom JSON serializer for optional property with System.Text.Json

I'm trying to implement a JSON serialization mechanism which handles both null and missing JSON values, to be able to perform partial updates when needed (so that it does not touch the field in the database when the value is missing, but it clears it when the value is explicitly set to null).
I created a custom struct copied from Roslyn's Optional<T> type:
public readonly struct Optional<T>
{
public Optional(T value)
{
this.HasValue = true;
this.Value = value;
}
public bool HasValue { get; }
public T Value { get; }
public static implicit operator Optional<T>(T value) => new Optional<T>(value);
public override string ToString() => this.HasValue ? (this.Value?.ToString() ?? "null") : "unspecified";
}
Now I want to be able to serialize/deserialize to/from JSON so that any missing field in JSON is preserved when roundtripping it through the Optional<T> object:
public class CustomType
{
[JsonPropertyName("foo")]
public Optional<int?> Foo { get; set; }
[JsonPropertyName("bar")]
public Optional<int?> Bar { get; set; }
[JsonPropertyName("baz")]
public Optional<int?> Baz { get; set; }
}
Then:
var options = new JsonSerializerOptions();
options.Converters.Add(new OptionalConverter());
string json = #"{""foo"":0,""bar"":null}";
CustomType parsed = JsonSerializer.Deserialize<CustomType>(json, options);
string roundtrippedJson = JsonSerializer.Serialize(parsed, options);
// json and roundtrippedJson should be equivalent
Console.WriteLine("json: " + json);
Console.WriteLine("roundtrippedJson: " + roundtrippedJson);
I started an implementation based on JsonConverterFactory, but I can't seem to find a proper way to omit the property during serialization if the optional's HasValue is false:
public class OptionalConverter : JsonConverterFactory
{
public override bool CanConvert(Type typeToConvert)
{
if (!typeToConvert.IsGenericType) { return false; }
if (typeToConvert.GetGenericTypeDefinition() != typeof(Optional<>)) { return false; }
return true;
}
public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
{
Type valueType = typeToConvert.GetGenericArguments()[0];
return (JsonConverter)Activator.CreateInstance(
type: typeof(OptionalConverterInner<>).MakeGenericType(new Type[] { valueType }),
bindingAttr: BindingFlags.Instance | BindingFlags.Public,
binder: null,
args: null,
culture: null
);
}
private class OptionalConverterInner<T> : JsonConverter<Optional<T>>
{
public override Optional<T> Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
T value = JsonSerializer.Deserialize<T>(ref reader, options);
return new Optional<T>(value);
}
public override void Write(Utf8JsonWriter writer, Optional<T> value, JsonSerializerOptions options)
{
// Does not work (produces invalid JSON).
// Problem: the object's key has already been written in the JSON writer at this point.
if (value.HasValue)
{
JsonSerializer.Serialize(writer, value.Value, options);
}
}
}
}
Problem: this produces the following output, which is invalid:
json: {"foo":0,"bar":null}
roundtrippedJson: {"foo":0,"bar":null,"baz":}
How can I solve this?

A custom JsonConverter<T> cannot prevent the serialization of a value to which the converter applies, see [System.Text.Json] Converter-level conditional serialization #36275 for confirmation.
In .Net 5 there is an option to ignore default values which should do what you need, see How to ignore properties with System.Text.Json. This version introduces JsonIgnoreCondition.WhenWritingDefault:
public enum JsonIgnoreCondition
{
// Property is never ignored during serialization or deserialization.
Never = 0,
// Property is always ignored during serialization and deserialization.
Always = 1,
// If the value is the default, the property is ignored during serialization.
// This is applied to both reference and value-type properties and fields.
WhenWritingDefault = 2,
// If the value is null, the property is ignored during serialization.
// This is applied only to reference-type properties and fields.
WhenWritingNull = 3,
}
You will be able to apply the condition to specific properties via [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)] or globally by setting JsonSerializerOptions.DefaultIgnoreCondition.
Thus in .Net 5 your class would look like:
public class CustomType
{
[JsonPropertyName("foo")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public Optional<int?> Foo { get; set; }
[JsonPropertyName("bar")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public Optional<int?> Bar { get; set; }
[JsonPropertyName("baz")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public Optional<int?> Baz { get; set; }
}
And the HasValue check should be removed from OptionalConverterInner<T>.Write():
public override void Write(Utf8JsonWriter writer, Optional<T> value, JsonSerializerOptions options) =>
JsonSerializer.Serialize(writer, value.Value, options);
Demo fiddle #1 here.
In .Net 3, as there is no conditional serialization mechanism in System.Text.Json, your only option to conditionally omit optional properties without a value is to write a custom JsonConverter<T> for all classes that contain optional properties. This is not made easy by the fact that JsonSerializer does not provide any access to its internal contract information so we need to either handcraft a converter for each and every such type, or write our own generic code via reflection.
Here is one attempt to create such generic code:
public interface IHasValue
{
bool HasValue { get; }
object GetValue();
}
public readonly struct Optional<T> : IHasValue
{
public Optional(T value)
{
this.HasValue = true;
this.Value = value;
}
public bool HasValue { get; }
public T Value { get; }
public object GetValue() => Value;
public static implicit operator Optional<T>(T value) => new Optional<T>(value);
public override string ToString() => this.HasValue ? (this.Value?.ToString() ?? "null") : "unspecified";
}
public class TypeWithOptionalsConverter<T> : JsonConverter<T> where T : class, new()
{
class TypeWithOptionalsConverterContractFactory : JsonObjectContractFactory<T>
{
protected override Expression CreateSetterCastExpression(Expression e, Type t)
{
// (Optional<Nullable<T>>)(object)default(T) does not work, even though (Optional<Nullable<T>>)default(T) does work.
// To avoid the problem we need to first cast to Nullable<T>, then to Optional<Nullable<T>>
if (t.IsGenericType && t.GetGenericTypeDefinition() == typeof(Optional<>))
return Expression.Convert(Expression.Convert(e, t.GetGenericArguments()[0]), t);
return base.CreateSetterCastExpression(e, t);
}
}
static readonly TypeWithOptionalsConverterContractFactory contractFactory = new TypeWithOptionalsConverterContractFactory();
public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
var properties = contractFactory.GetProperties(typeToConvert);
if (reader.TokenType == JsonTokenType.Null)
return null;
if (reader.TokenType != JsonTokenType.StartObject)
throw new JsonException();
var value = new T();
while (reader.Read())
{
if (reader.TokenType == JsonTokenType.EndObject)
return value;
if (reader.TokenType != JsonTokenType.PropertyName)
throw new JsonException();
string propertyName = reader.GetString();
if (!properties.TryGetValue(propertyName, out var property) || property.SetValue == null)
{
reader.Skip();
}
else
{
var type = property.PropertyType.IsGenericType && property.PropertyType.GetGenericTypeDefinition() == typeof(Optional<>)
? property.PropertyType.GetGenericArguments()[0] : property.PropertyType;
var item = JsonSerializer.Deserialize(ref reader, type, options);
property.SetValue(value, item);
}
}
throw new JsonException();
}
public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
{
writer.WriteStartObject();
foreach (var property in contractFactory.GetProperties(value.GetType()))
{
if (options.IgnoreReadOnlyProperties && property.Value.SetValue == null)
continue;
var item = property.Value.GetValue(value);
if (item is IHasValue hasValue)
{
if (!hasValue.HasValue)
continue;
writer.WritePropertyName(property.Key);
JsonSerializer.Serialize(writer, hasValue.GetValue(), options);
}
else
{
if (options.IgnoreNullValues && item == null)
continue;
writer.WritePropertyName(property.Key);
JsonSerializer.Serialize(writer, item, property.Value.PropertyType, options);
}
}
writer.WriteEndObject();
}
}
public class JsonPropertyContract<TBase>
{
internal JsonPropertyContract(PropertyInfo property, Func<Expression, Type, Expression> setterCastExpression)
{
this.GetValue = ExpressionExtensions.GetPropertyFunc<TBase>(property).Compile();
if (property.GetSetMethod() != null)
this.SetValue = ExpressionExtensions.SetPropertyFunc<TBase>(property, setterCastExpression).Compile();
this.PropertyType = property.PropertyType;
}
public Func<TBase, object> GetValue { get; }
public Action<TBase, object> SetValue { get; }
public Type PropertyType { get; }
}
public class JsonObjectContractFactory<TBase>
{
protected virtual Expression CreateSetterCastExpression(Expression e, Type t) => Expression.Convert(e, t);
ConcurrentDictionary<Type, ReadOnlyDictionary<string, JsonPropertyContract<TBase>>> Properties { get; } =
new ConcurrentDictionary<Type, ReadOnlyDictionary<string, JsonPropertyContract<TBase>>>();
ReadOnlyDictionary<string, JsonPropertyContract<TBase>> CreateProperties(Type type)
{
if (!typeof(TBase).IsAssignableFrom(type))
throw new ArgumentException();
var dictionary = type
.GetProperties(BindingFlags.Public | BindingFlags.Instance | BindingFlags.FlattenHierarchy)
.Where(p => p.GetIndexParameters().Length == 0 && p.GetGetMethod() != null
&& !Attribute.IsDefined(p, typeof(System.Text.Json.Serialization.JsonIgnoreAttribute)))
.ToDictionary(p => p.GetCustomAttribute<System.Text.Json.Serialization.JsonPropertyNameAttribute>()?.Name ?? p.Name,
p => new JsonPropertyContract<TBase>(p, (e, t) => CreateSetterCastExpression(e, t)),
StringComparer.OrdinalIgnoreCase);
return dictionary.ToReadOnly();
}
public IReadOnlyDictionary<string, JsonPropertyContract<TBase>> GetProperties(Type type) => Properties.GetOrAdd(type, t => CreateProperties(t));
}
public static class DictionaryExtensions
{
public static ReadOnlyDictionary<TKey, TValue> ToReadOnly<TKey, TValue>(this IDictionary<TKey, TValue> dictionary) =>
new ReadOnlyDictionary<TKey, TValue>(dictionary ?? throw new ArgumentNullException());
}
public static class ExpressionExtensions
{
public static Expression<Func<T, object>> GetPropertyFunc<T>(PropertyInfo property)
{
// (x) => (object)x.Property;
var arg = Expression.Parameter(typeof(T), "x");
var getter = Expression.Property(arg, property);
var cast = Expression.Convert(getter, typeof(object));
return Expression.Lambda<Func<T, object>>(cast, arg);
}
public static Expression<Action<T, object>> SetPropertyFunc<T>(PropertyInfo property, Func<Expression, Type, Expression> setterCastExpression)
{
//(x, y) => x.Property = (TProperty)y
var arg1 = Expression.Parameter(typeof(T), "x");
var arg2 = Expression.Parameter(typeof(object), "y");
var cast = setterCastExpression(arg2, property.PropertyType);
var setter = Expression.Call(arg1, property.GetSetMethod(), cast);
return Expression.Lambda<Action<T, object>>(setter, arg1, arg2);
}
}
Notes:
CustomType remains as shown in your question.
No attempt was made to handle the presence of a naming policy in JsonSerializerOptions.PropertyNamingPolicy. You could implement this in TypeWithOptionalsConverter<T> if necessary.
I added a non-generic interface IHasValue to enable easier access to a boxed Optional<T> during serialization.
Demo fiddle #2 here.
Alternatively, you could stick with Json.NET which supports this at the property and contact level. See:
Optionally serialize a property based on its runtime value (essentially a duplicate of your question).
how to dynamic jsonignore according to user authorize?

Json.net - How to preserve dictionary value references when populating a dictionary?

I would like to populate the objects contained within a Dictionary from a JSON file while preserving the object references themselves.
Json.net documentation on PreserveReferencesHandling clearly state that it will not work in case a type implements System.Runtime.Serialization.ISerializable:
Specifies reference handling options for the
Newtonsoft.Json.JsonSerializer. Note that references cannot be
preserved when a value is set via a non-default constructor such as
types that implement System.Runtime.Serialization.ISerializable.
Here is my failing code:
class Model
{
public int Val { get; set; } = 123;
}
...
var model = new Model();
var to_serialize = new Dictionary<int, Model> { { 0, model } }; // works ok with list<Model>
// serialize
var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);
var jsonSerializerSettings = new JsonSerializerSettings();
jsonSerializerSettings.MissingMemberHandling = MissingMemberHandling.Ignore;
jsonSerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.All; // does not work for ISerializable
Assert.AreSame(to_serialize[0], model); // ok!
JsonConvert.PopulateObject(
value: jsonString,
target: to_serialize,
settings: jsonSerializerSettings
);
Assert.AreSame(to_serialize[0], model); // not ok... works ok with list<Model>
My main requirement is that when calling PopulateObject(), the constructor of the Model class will not be invoked. Instead, only its internal field will be updated with the value from the JSON.
In my real case, the Model class contains other values which are not in the JSON and which I don't want to lose:
[JsonObject(MemberSerialization.OptIn)]
class Model
{
[JsonProperty(PropertyName = "val_prop")]
public int Val { get; set; } = 123;
// not in the json file, would like this field to maintain the value
// it had prior to PopulateObject()
public int OtherVal { get; set; } = 456;
}
Is there a way to make this work?

Your problem is similar to the one from JsonSerializer.CreateDefault().Populate(..) resets my values: you would like to populate a preexisting collection, specifically a Dictionary<int, T> for some T, and populate the preexisting values. Unfortunately, in the case of a dictionary, Json.NET will replace the values rather than populate them, as can be seen in JsonSerializerInternalReader.PopulateDictionary() which simply deserializes the value to the appropriate type, and sets it the dictionary.
To work around this limitation, you can create a custom JsonConverter for Dictionary<TKey, TValue> when TKey is a primitive type and TValue is a complex type which merges the incoming JSON key/value pairs onto the preexisting dictionary. The following converter does the trick:
public class DictionaryMergeConverter : JsonConverter
{
static readonly IContractResolver defaultResolver = JsonSerializer.CreateDefault().ContractResolver;
readonly IContractResolver resolver = defaultResolver;
public override bool CanConvert(Type objectType)
{
var keyValueTypes = objectType.GetDictionaryKeyValueType();
if (keyValueTypes == null)
return false;
var keyContract = resolver.ResolveContract(keyValueTypes[0]);
if (!(keyContract is JsonPrimitiveContract))
return false;
var contract = resolver.ResolveContract(keyValueTypes[1]);
return contract is JsonContainerContract;
// Also possibly check whether keyValueTypes[1] is a read-only collection or dictionary.
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
return null;
if (reader.TokenType != JsonToken.StartObject)
throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
IDictionary dictionary = existingValue as IDictionary ?? (IDictionary)serializer.ContractResolver.ResolveContract(objectType).DefaultCreator();
var keyValueTypes = objectType.GetDictionaryKeyValueType();
while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndObject)
{
switch (reader.TokenType)
{
case JsonToken.PropertyName:
var name = (string)reader.Value;
reader.ReadToContentAndAssert();
// TODO: DateTime keys and enums with overridden names.
var key = (keyValueTypes[0] == typeof(string) ? (object)name : Convert.ChangeType(name, keyValueTypes[0], serializer.Culture));
var value = dictionary.Contains(key) ? dictionary[key] : null;
// TODO:
// - JsonConverter active for valueType, either in contract or in serializer.Converters
// - NullValueHandling, ObjectCreationHandling, PreserveReferencesHandling,
if (value == null)
{
value = serializer.Deserialize(reader, keyValueTypes[1]);
}
else
{
serializer.Populate(reader, value);
}
dictionary[key] = value;
break;
default:
throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
}
}
return dictionary;
}
public override bool CanWrite { get { return false; } }
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) { throw new NotImplementedException(); }
}
public static partial class JsonExtensions
{
public static JsonReader ReadToContentAndAssert(this JsonReader reader)
{
return reader.ReadAndAssert().MoveToContentAndAssert();
}
public static JsonReader MoveToContentAndAssert(this JsonReader reader)
{
if (reader == null)
throw new ArgumentNullException();
if (reader.TokenType == JsonToken.None) // Skip past beginning of stream.
reader.ReadAndAssert();
while (reader.TokenType == JsonToken.Comment) // Skip past comments.
reader.ReadAndAssert();
return reader;
}
public static JsonReader ReadAndAssert(this JsonReader reader)
{
if (reader == null)
throw new ArgumentNullException();
if (!reader.Read())
throw new JsonReaderException("Unexpected end of JSON stream.");
return reader;
}
}
public static class TypeExtensions
{
public static IEnumerable<Type> BaseTypesAndSelf(this Type type)
{
while (type != null)
{
yield return type;
type = type.BaseType;
}
}
public static Type[] GetDictionaryKeyValueType(this Type type)
{
return type.BaseTypesAndSelf().Where(t => t.IsGenericType && t.GetGenericTypeDefinition() == typeof(Dictionary<,>)).Select(t => t.GetGenericArguments()).FirstOrDefault();
}
}
Having done so, you will encounter a secondary issue: Json.NET will never use a custom converter to populate the root object. To work around this you will need to call JsonConverter.ReadJson() directly, from some utility method:
public static partial class JsonExtensions
{
public static void PopulateObjectWithConverter(string value, object target, JsonSerializerSettings settings)
{
if (target == null || value == null)
throw new ArgumentNullException();
var serializer = JsonSerializer.CreateDefault(settings);
var converter = serializer.Converters.Where(c => c.CanConvert(target.GetType()) && c.CanRead).FirstOrDefault() ?? serializer.ContractResolver.ResolveContract(target.GetType()).Converter;
using (var jsonReader = new JsonTextReader(new StringReader(value)))
{
if (converter == null)
serializer.Populate(jsonReader, target);
else
{
jsonReader.MoveToContentAndAssert();
var newtarget = converter.ReadJson(jsonReader, target.GetType(), target, serializer);
if (newtarget != target)
throw new JsonException(string.Format("Converter {0} allocated a new object rather than populating the existing object {1}.", converter, value));
}
}
}
}
You will now be able to populate your dictionary as follows:
var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);
var settings = new JsonSerializerSettings
{
Converters = { new DictionaryMergeConverter() },
};
JsonExtensions.PopulateObjectWithConverter(jsonString, to_serialize, settings);
Notes:
PreserveReferencesHandling has no impact on whether dictionary values are populated or replaced. Instead this setting controls whether a serialization graph with multiple references to the same object will maintain its reference topology when round-tripped.
In your question you wrote // works ok with list<Model> but in fact this is not correct. When a List<T> is populated the new values are appended to the list, so Assert.AreSame(to_serialize[0], model); passes purely by luck. If you had additionally asserted Assert.AreSame(1, to_serialize.Count) it would have failed.
While the converter will work for primitive keys such as string and int it may not work for key types that require JSON-specific conversion such as enum or DateTime.
The converter is currently only implemented for Dictionary<TKey, TValue> and takes advantage of the fact that this type implements the non-generic IDictionary interface. It could be extended to other dictionary types such as SortedDictionary<TKey,TValue> if required.
Demo fiddle here.

Replace Json.Nets Standard Serialization Dictionary with Custom Dictionary globally

Json.Net uses the default .Net Dictionary for Serialization of JSON Dictionaries when i just use the IDictionary interface. I want to replace it against my Custom Version of a Dictionary globally.
Do I need to write a JsonConverter in order to do this or do i just miss a Setting like:
config.DefaultDictType = typeof(MyDict);
somewhere

There isn't a simple setting for this. Properties declared as IDictionary<TKey,TValue> are allocated as Dictionary<TKey, TValue> if the current value is null.
However, if you pre-allocate the custom dictionaries in the default constructors for your classes, Json.NET will use the pre-existing allocated dictionary rather than allocating a new one. (This is why JsonConverter.ReadJson has an existingValue argument.)
For instance:
public class MyDictionary<TKey, TValue> : Dictionary<TKey, TValue>
{
}
public class DictionaryContainer
{
public DictionaryContainer()
{
this.Values = new MyDictionary<string, string>();
}
public IDictionary<string, string> Values { get; set; }
}
public static class TestClass
{
public static void Test()
{
var container = new DictionaryContainer();
container.Values["one"] = "first";
container.Values["two"] = "second";
container.Values["three"] = "third";
Debug.Assert(container.Values.GetType() == typeof(MyDictionary<string, string>)); // No assert
var json = JsonConvert.SerializeObject(container, Formatting.Indented);
var container2 = JsonConvert.DeserializeObject<DictionaryContainer>(json);
Debug.Assert(container.Values.GetType() == container2.Values.GetType());// No assert
}
}
If this is impractical (because of, e.g., issues with legacy code), the easiest way to achieve what you want is probably with a single global converter that, when reading, allocates dictionaries for all properties defined to return type IDictionary<TKey,TValue> for any key and value:
public abstract class CustomDictionaryTypeReaderBase : JsonConverter
{
protected abstract Type CreateDictionaryType(Type tKey, Type tValue);
bool GetIDictionaryGenericParameters(Type objectType, out Type tKey, out Type tValue)
{
tKey = tValue = null;
if (!objectType.IsGenericType)
return false;
var genericType = objectType.GetGenericTypeDefinition();
if (genericType != typeof(IDictionary<,>))
return false;
var args = objectType.GetGenericArguments();
if (args.Length != 2)
tKey = tValue = null;
tKey = args[0];
tValue = args[1];
return true;
}
public override bool CanConvert(Type objectType)
{
Type tKey, tValue;
return GetIDictionaryGenericParameters(objectType, out tKey, out tValue);
}
public override bool CanWrite { get { return false; } }
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
if (existingValue != null)
{
serializer.Populate(reader, existingValue);
return existingValue;
}
else
{
Type tKey, tValue;
bool ok = GetIDictionaryGenericParameters(objectType, out tKey, out tValue);
if (!ok)
{
return serializer.Deserialize(reader, objectType);
}
else
{
var realType = CreateDictionaryType(tKey, tValue);
Debug.Assert(objectType.IsAssignableFrom(realType));
return serializer.Deserialize(reader, realType);
}
}
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotImplementedException();
}
}
public sealed class MyCustomDictionaryTypeReader : CustomDictionaryTypeReaderBase
{
protected override Type CreateDictionaryType(Type tKey, Type tValue)
{
var dictType = typeof(MyDictionary<,>).MakeGenericType(new[] { tKey, tValue });
return dictType;
}
}
And then use it like:
var settings = new JsonSerializerSettings();
settings.Converters = new[] { new MyCustomDictionaryTypeReader() };
var multiContainer2 = JsonConvert.DeserializeObject<MultipleDictionaryContainer>(json2, settings);

Another way to achieve what you wanted is by modifying the DefaultContractResolver, override its CreateDictionaryContract() method, and then set it as the serializer's default ContractResolver.
Assuming your custom dictionary is defined as:
public class CustomDictionary<TKey, TValue> : Dictionary<TKey, TValue>
{
public string Custom { get { return "I'm Custom"; } }
}
You could implement the custom contract resolver as:
public class CustomDictionaryContractResolver : DefaultContractResolver
{
public CustomDictionaryContractResolver() : base(true) { }
protected override JsonDictionaryContract CreateDictionaryContract(Type objectType)
{
// let the base class do the heavy lifting
var contract = base.CreateDictionaryContract(objectType);
// nothing todo if the created type is already our custom dictionary type
if (IsGenericDefinition(contract.CreatedType, typeof (CustomDictionary<,>)))
return contract;
if (IsGenericDefinition(contract.UnderlyingType, typeof(IDictionary<,>)) || (typeof(IDictionary).IsAssignableFrom(contract.UnderlyingType)))
{
contract.CreatedType = typeof(CustomDictionary<,>)
.MakeGenericType(contract.DictionaryKeyType ?? typeof(object), contract.DictionaryValueType ?? typeof(object));
// Set our object instantiation using the default constructor;
// You need to modify this, if your custom dictionary does not
// have a default constructor.
contract.DefaultCreator = () => Activator.CreateInstance(contract.CreatedType);
}
return contract;
}
static bool IsGenericDefinition(Type type, Type genericDefinition)
{
return type.IsGenericType && type.GetGenericTypeDefinition() == genericDefinition;
}
}
And then use it like:
JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
ContractResolver = new CustomDictionaryContractResolver()
};
For example:
public class MyObject
{
public string FirstName { get; set; }
public string LastName { get; set; }
public IDictionary<string, object> Properties { get; set; }
}
public static class TestClass
{
public static void Test()
{
const string json = #"{""FirstName"": ""John"",""LastName"": ""Smith"",""Properties"": {""Email"": ""john.smith#example.com"",""Level"": 42,""Admin"": true}}";
JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
ContractResolver = new CustomDictionaryContractResolver()
};
var myObject = JsonConvert.DeserializeObject<MyObject>(json);
Debug.WriteLineIf(myObject.FirstName == "John", "FirstName is John");
Debug.WriteLineIf(myObject.LastName == "Smith", "LastName is Smith");
Debug.WriteLineIf(myObject.Properties != null, "Properties is not NULL");
Debug.WriteLineIf(myObject.Properties.Count == 3, "Properties has 3 items");
Debug.WriteLineIf(myObject.Properties.GetType() == typeof(CustomDictionary<string, object>), "Properties is CustomDictionary<,>");
var customDictionary = myObject.Properties as CustomDictionary<string, object>;
Debug.WriteLineIf(customDictionary != null, "Properties say " + customDictionary.Custom);
}
}
If you run the TestClass.Test() method, you should see the following in the Output:
FirstName is John
LastName is Smith
Properties is not NULL
Properties has 3 items
Properties is CustomDictionary
Properties say I'm Custom