We Keep Coding

In C#, can I edit/change the types in a json string?

I have slightly annoying use case.
So, I am calling an API . The APi returns some json, which contains a json object with x amount of fields:
{
"status": "ok",
"result": {
"firstprovider": [ .... ],
"secondprovider": [ ...],
"thirdprovider": [ ... ]
}
}
In this example, only three provider are returned, but I could get more or less than that, and their names may vary. It's quite important that I save those names.
The "providers" inside of "result" are of the same type, so I can easily deserialize those to a certain model.
Now, I would normally expext "result" to be a json array of elements, so I could easily deserialize "result" into a List<LiveShopperEventModel>.
Currently, I made a hacky solution, creating a Dictionary<string,Provider> where the key is the name of the provider, and then I later use selectmany to flatten it into a list.
But, I was wondering if there exists some way in c#, that would allow me to convert the "result", into an array, which would make deserialization a lot simpler for me.
So, does anyone know of a way, or a resource that in c# can help in changing the types of json elements, and in this case, making a json object into a json array, with the fields becoming elements in the list?
Reproducing concept with minimal example
So, let's say my json looks like this:
{
"status": "ok",
"result": {
"firstprovider": [ {"name":"John"}, {"car":"BMW"}, {"surname":"Johnson"} ],
"secondprovider": [ {"name":"Zoe"}, {"car":"Ford"}, {"surname":"johnsøn"}],
"thirdprovider": [{"name":"Elliot"}, {"car":"Volkswagen"}, {"surname":"Jackson"} ]
}
}
Then I can deserialize it as in the following code snippet:
string json = "{\r\n\"status\": \"ok\", \r\n\"result\": { \r\n \"firstprovider\": [ {\"name\":\"John\"}, {\"car\":\"BMW\"}, {\"surname\":\"Johnson\", \"age\":30, \"car\":\"fast\"} ],\r\n \"secondprovider\": [ {\"name\":\"Zoe\"}, {\"car\":\"Ford\"}, {\"surname\":\"johnsøn\", \"age\":31, \"car\":null}], \r\n \"thirdprovider\": [{\"name\":\"Elliot\"}, {\"car\":\"Volkswagen\"}, {\"surname\":\"Jackson\", \"age\":32, \"car\":null} ] \r\n }\r\n }\r\n";
// deserializing to a dictionary
var resultDict = JsonConvert.DeserializeObject<ResultModel>(json);
// and now flattening the structure, so that it is a list of "ProviderModel"
// this is the part that feels hacky to me
var providerModelList = resultDict.result.SelectMany(listOfEvents => {
listOfEvents.Value.Select(Provider =>
{
Provider.provider = listOfEvents.Key;
return Provider;
}).ToList();
return listOfEvents.Value;
}).ToList();
public class ResultModel
{
[JsonProperty("result")]
public Dictionary<string, List<ProviderModel>> result { get; set; }
}
public class ProviderModel
{
public string provider { get; set; }
public string name { get; set; }
public string surname { get; set; }
}
The `selectmany" part feels hacky to me, since, i'm combining linq queries in a way that feels overly complicated
Instead, I think it would be much nicer, is the result class could just look like:
public class ResultModel
{
[JsonProperty("result")]
public List<ProviderModel> result { get; set; }
}

you can try something like this
List<ProviderModel> providerModelList = ((JObject)JObject.Parse(json)["result"])
.Properties()
.Select(x => GetValues(x))
.ToList();
public ProviderModel GetValues(JProperty jProp)
{
var providerModel = new JObject(((JArray)jProp.Value)
.Select(jo => ((JObject)jo).Properties().First()))
.ToObject<ProviderModel>();
providerModel.provider = jProp.Name;
return providerModel;
}

Create RavenDB index based on dynamic properties

In RavenDB 4.2, I want to create an Index/Map based on a dynamic object. Better put, on dynamic properties which are not known at compile-time.
Here is an example of the raw JSON I'm ingesting:
{
"id": "A",
"detections":
[
{
"steps": [
{
"object": {
"id": "A1",
"target": {
"domain_name": "foobar.com"
}
},
"object": {
"id": "A2",
"target": {
"ipv4": "127.0.0.1"
}
}
}
]
}
]
}
The above sample is ingested from a 3rd party and stored in a RavenDB collection. Roughly translated, the following model has the challenge:
public class Step
{
public string Id { get; set; }
public DateTime When {get; set;}
public dynamic Object { get; set; } // aware that it's not handy naming
}
The pickle in this is that the object.target.X property name is dynamic. They cannot be strong-typed and can be a lot of things, like: domain_name, ipv4, ipv6, dns, shoe_size, hair_colour etc. This is why the entire steps.object is ingested and stored as either System.Object or dynamic.
My objective is to basically do a SelectMany() on each object.target and extract the property name (key) and value. This would make my RavenDB Index something like this:
public class StepsIndex : AbstractIndexCreationTask<Models.Step, StepsIndex.Result>
{
public class Result
{
public DateTime When { get; set; }
public string TargetKey { get; set; }
public string TargetValue { get; set; }
// ... removed other properties for brevity
}
public StepsIndex()
{
Map = steps =>
from block in blocks
from detection in blocks.Detections
from step in detection.Steps
select new Result
{
// extract property name (key), like 'domain_name'
TargetKey = step.Object.target.GetType().GetProperties()[0].Name,
// extract property value, like 'foobar.com'
TargetValue = step.Object.target.GetType().GetProperty(s.Object.target.GetType().GetProperties()[0].Name).GetValue(s.Object.target, null)
};
}
}
Unfortunately this doesn't work due to step.Object being dynamic and resulting in the following error during compile-time:
Error [CS1963] An expression tree may not contain a dynamic operation
Second option I've tried is to cast it to JSON in the expression, which also fails because Raven's projection is not aware of Newtonsoft.Json during runtime:
// Error CS0103: The name 'JObject' does not exist in the current context
// Error CS0103: The name 'JsonConvert' does not exist in the current context
TargetKey = JObject.Parse(JsonConvert.SerializeObject(ass.Object))["target"][0].Value<string>(),
A third option I thought of was perhaps changing the dynamic Object to System.Object Object, but haven't found a neat way to extract the property key/values without knowning the property.
The question: how can I extract these dynamic property keys and values and Map them to a RavenDB index?

RavenDB allows to index dynamic fields:
See:
https://ravendb.net/docs/article-page/4.2/Csharp/indexes/using-dynamic-fields
https://github.com/ravendb/book/blob/v4.0/Ch10/Ch10.md#dynamic-data

Web API 2 - Implementing a PATCH

I currently have a Web API that implements a RESTFul API. The model for my API looks like this:
public class Member
{
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime Created { get; set; }
public DateTime BirthDate { get; set; }
public bool IsDeleted { get; set; }
}
I've implemented a PUT method for updating a row similar to this (for brevity, I've omitted some non-relevant stuff):
[Route("{id}")]
[HttpPut]
public async System.Threading.Tasks.Task<HttpResponseMessage> UpdateRow(int id,
[FromBody]Models.Member model)
{
// Do some error checking
// ...
// ...
var myDatabaseEntity = new BusinessLayer.Member(id);
myDatabaseEntity.FirstName = model.FirstName;
myDatabaseEntity.LastName = model.LastName;
myDatabaseEntity.Created = model.Created;
myDatabaseEntity.BirthDate = model.BirthDate;
myDatabaseEntity.IsDeleted = model.IsDeleted;
await myDatabaseEntity.SaveAsync();
}
Using PostMan, I can send the following JSON and everything works fine:
{
firstName: "Sara",
lastName: "Smith",
created: "2018/05/10",
birthDate: "1977/09/12",
isDeleted: false
}
If I send this as my body to http://localhost:8311/api/v1/Member/12 as a PUT request, the record in my data with ID of 12 gets updated to what you see in the JSON.
What I would like to do though is implement a PATCH verb where I can do partial updates. If Sara gets married, I would like to be able to send this JSON:
{
lastName: "Jones"
}
I would like to be able to send just that JSON and update JUST the LastName field and leave all the other fields alone.
I tried this:
[Route("{id}")]
[HttpPatch]
public async System.Threading.Tasks.Task<HttpResponseMessage> UpdateRow(int id,
[FromBody]Models.Member model)
{
}
My problem is that this returns all the fields in the model object (all of them are nulls except the LastName field), which makes sense since I am saying I want a Models.Member object. What I would like to know is if there is a way to detect which properties have actually been sent in the JSON request so I can update just those fields?

I hope this helps using Microsoft JsonPatchDocument:
.Net Core 2.1 Patch Action into a Controller:
[HttpPatch("{id}")]
public IActionResult Patch(int id, [FromBody]JsonPatchDocument<Node> value)
{
try
{
//nodes collection is an in memory list of nodes for this example
var result = nodes.FirstOrDefault(n => n.Id == id);
if (result == null)
{
return BadRequest();
}
value.ApplyTo(result, ModelState);//result gets the values from the patch request
return NoContent();
}
catch (Exception ex)
{
return StatusCode(StatusCodes.Status500InternalServerError, ex);
}
}
Node Model class:
[DataContract(Name ="Node")]
public class Node
{
[DataMember(Name = "id")]
public int Id { get; set; }
[DataMember(Name = "node_id")]
public int Node_id { get; set; }
[DataMember(Name = "name")]
public string Name { get; set; }
[DataMember(Name = "full_name")]
public string Full_name { get; set; }
}
A valid Patch JSon to update just the "full_name" and the "node_id" properties will be an array of operations like:
[
{ "op": "replace", "path": "full_name", "value": "NewNameWithPatch"},
{ "op": "replace", "path": "node_id", "value": 10}
]
As you can see "op" is the operation you would like to perform, the most common one is "replace" which will just set the existing value of that property for the new one, but there are others:
[
{ "op": "test", "path": "property_name", "value": "value" },
{ "op": "remove", "path": "property_name" },
{ "op": "add", "path": "property_name", "value": [ "value1", "value2" ] },
{ "op": "replace", "path": "property_name", "value": 12 },
{ "op": "move", "from": "property_name", "path": "other_property_name" },
{ "op": "copy", "from": "property_name", "path": "other_property_name" }
]
Here is an extensions method I built based on the Patch ("replace") specification in C# using reflection that you can use to serialize any object to perform a Patch ("replace") operation, you can also pass the desired Encoding and it will return the HttpContent (StringContent) ready to be sent to httpClient.PatchAsync(endPoint, httpContent):
public static StringContent ToPatchJsonContent(this object node, Encoding enc = null)
{
List<PatchObject> patchObjectsCollection = new List<PatchObject>();
foreach (var prop in node.GetType().GetProperties())
{
var patch = new PatchObject{ Op = "replace", Path = prop.Name , Value = prop.GetValue(node) };
patchObjectsCollection.Add(patch);
}
MemoryStream payloadStream = new MemoryStream();
DataContractJsonSerializer serializer = new DataContractJsonSerializer(patchObjectsCollection.GetType());
serializer.WriteObject(payloadStream, patchObjectsCollection);
Encoding encoding = enc ?? Encoding.UTF8;
var content = new StringContent(Encoding.UTF8.GetString(payloadStream.ToArray()), encoding, "application/json");
return content;
}
}
Noticed that tt also uses this class I created to serialize the PatchObject using DataContractJsonSerializer:
[DataContract(Name = "PatchObject")]
class PatchObject
{
[DataMember(Name = "op")]
public string Op { get; set; }
[DataMember(Name = "path")]
public string Path { get; set; }
[DataMember(Name = "value")]
public object Value { get; set; }
}
A C# example of how to use the extension method and invoking the Patch request using HttpClient:
var nodeToPatch = new { Name = "TestPatch", Private = true };//You can use anonymous type
HttpContent content = nodeToPatch.ToPatchJsonContent();//Invoke the extension method to serialize the object
HttpClient httpClient = new HttpClient();
string endPoint = "https://localhost:44320/api/nodes/1";
var response = httpClient.PatchAsync(endPoint, content).Result;
Thanks

PATCH operations aren't usually defined using the same model as the POST or PUT operations exactly for that reason: How do you differentiate between a null, and a don't change. From the IETF:
With PATCH, however, the enclosed entity contains a set of
instructions describing how a resource currently residing on the
origin server should be modified to produce a new version.
You can look here for their PATCH suggestion, but sumarilly is:
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]

#Tipx's answer re using PATCH is spot on, but as you've probably already found, actually achieving that in a statically typed language like C# is a non-trivial exercise.
In the case where you're using a PATCH to represent a set of partial updates for a single domain entity (e.g. to update the first name and last name only for a contact with many more properties) you need to do something along the lines of looping each instruction in the 'PATCH' request and then applying that instruction to an instance of your class.
Applying an individual instruction will then comprise of
Finding the property of the instance that matches the name in the
instruction, or handling property names you weren't expecting
For an update: Trying to parse the value submitted in the patch into the instance property and handling the error if e.g. the instance property is a bool but the patch instruction contains a date
Deciding what to do with Add instructions as you can't add new properties to a statically typed C# class. One approach is to say that Add means "set the value of the instance's property only if property's existing value is null"
For Web API 2 on the full .NET Framework the JSONPatch github project looks to make a stab at providing this code, although it doesn't look like there's been a lot of development on that repo recently and the readme does state:
This is still very much an early project, don't use it in production
yet unless you understand the source and don't mind fixing a few bugs
;)
Things are simpler on .NET Core as that has a set of functionality to support this in the Microsoft.AspNetCore.JsonPatch namespace.
The rather useful jsonpatch.com site also lists out a few more options for Patch in .NET:
Asp.Net Core JsonPatch (Microsoft official implementation)
Ramone (a framework for consuming REST services, includes a JSON Patch implementation)
JsonPatch (Adds JSON Patch support to ASP.NET Web API)
Starcounter (In-memory Application Engine, uses JSON Patch with OT for client-server sync)
Nancy.JsonPatch (Adds JSON Patch support to NancyFX)
Manatee.Json (JSON-everything, including JSON Patch)
I need to add this functionality to an existing Web API 2 project of ours, so I'll update this answer if I find anything else that's useful while doing that.

I wanted to achieve exactly the same thing, but used a different method to others described here. I've created a working repo using this if you want to check it out:
https://github.com/emab/patch-example
If you have the following two models:
Database model
public class WeatherDBModel
{
[Key]
public int Id { get; set; }
public string City { get; set; }
public string Country { get; set; }
public double Temperature { get; set; }
public double WindSpeed { get; set; }
public double Rain { get; set; }
public Weather(int id, string city, string country, double temperature, double windSpeed, double rain)
{
Id = id;
City = city;
Country = country;
Temperature = temperature;
WindSpeed = windSpeed;
Rain = rain;
}
}
Update model
Containing exact names of database model properties. Includes properties which can be updated
public class WeatherUpdateModel
{
public string? City { get; set; }
public string? Country { get; set; }
public double Temperature { get; set; }
public double WindSpeed { get; set; }
public double Rain { get; set; }
}
This update model is sent to the service layer along with the id of the object you'd like to update.
You can then implement the following method in your repository layer which maps any non-null values from the updateModel into an existing entity if it has been found:
public Weather Update(int id, WeatherUpdate updateObject)
{
// find existing entity
var existingEntity = _context.Weather.Find(id);
// handle not found
if (existingEntity == null)
{
throw new EntityNotFoundException(id);
}
// iterate through all of the properties of the update object
// in this example it includes all properties apart from `id`
foreach (PropertyInfo prop in updateObject.GetType().GetProperties())
{
// check if the property has been set in the updateObject
// if it is null we ignore it. If you want to allow null values to be set, you could add a flag to the update object to allow specific nulls
if (prop.GetValue(updateObject) != null)
{
// if it has been set update the existing entity value
existingEntity.GetType().GetProperty(prop.Name)?.SetValue(existingEntity, prop.GetValue(updateObject));
}
}
_context.SaveChanges();
return existingEntity;
}
Using this method you can change your models without worrying about the update logic, as long as you ensure that the UpdateModel is kept up-to-date with the database model.

If a property of your object was omitted in your JSON, ASP.NET won't "set" that property on the object, the property will have its default value. In order to know which properties were sent with the JSON object you need to have a way to detect which properties of the object were set.
In order to detect which properties have "actually been sent" with the JSON object, you can modify your Member class to contain a collection of property names that were "set". Then, for all properties that you want to be able to know if they were sent in the JSON object make that when the property is set the name of the property should be added to the collection of set properties.
public class Member
{
private string _firstName;
private string _lastName;
...
private bool _isDeleted;
public string FirstName
{
get => _firstName;
set
{
_firstName = value;
_setProperties.Add(nameof(FirstName));
}
}
public string LastName
{
get => _lastName;
set
{
_lastName = value;
_setProperties.Add(nameof(LastName));
}
}
...
public bool IsDeleted
{
get => _isDeleted;
set
{
_isDeleted= value;
_setProperties.Add(nameof(IsDeleted));
}
}
private readonly HashSet<string> _setProperties = new HashSet<string>();
public HashSet<string> GetTheSetProperties()
{
return new HashSet<string>(_setProperties);
}
}
In the UpdateRow method you can now check whether a property was sent in the JSON by checking if it is in the _setProperties collection. So if you want to see if the LastName was sent in the JSON just do
bool lastNameWasInJson = model.Contains(nameof(model.LastName));

Following up to Avid Learners approach. I found this easy to add to an existing PUT method.
Alternatively to avoid loading twice you could apply update operations and then before saving apply the patch, but I'd rather load twice and have simple code.
public ResultModel Patch(UpdateModel model)
{
var record = LoadAsUpdateModel(model.Id);
if (record == null) return null;
foreach(var propertyName in model.SetProperties())
{
var property = model.GetType().GetProperty(propertyName);
property.SetValue(record, property.GetValue(model));
}
return Update(record);
}

configuration.GetValue list returns null

I am trying to read a list from appsettings.json file using the GetValue<T> method:
var builder = new ConfigurationBuilder().SetBasePath(System.AppContext.BaseDirectory)
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
IConfigurationRoot configuration = builder.Build();
var rr = configuration.GetValue<IList<ConnectionSettings>>("Connections");
public class ConnectionSettings
{
public string Name { get; set; }
public string Host { get; set; }
public string Account { get; set; }
public string Password { get; set; }
}
and my appsettings.json
{
"Connections": [
{
"Name": "",
"Host": "192.168.1.5",
"Account": "74687",
"Password": "asdsdadsq"
},
{
"Name": "",
"Host": "127.0.0.1",
"Account": "45654",
"Password": "asdasads"
}
]
}
Problem is that I always get null and I dont understand why.

According to the documentation for GetValue<>, it gets the value of a (single) key and converts it to the specified type. Unfortunately, it doesn't throw an error if the value can't be converted, which is the situation you're running into.
I believe that Get<> would be preferable in your situation.
var rr = configuration.GetSection("Connections").Get<IList<ConnectionSettings>>();
According to Get<>'s documentation, it:
Attempts to bind the configuration instance to a new instance of type T. If this configuration section has a value, that will be used. Otherwise binding by matching property names against configuration keys recursively.
This allows you to get the value directly or, if it can't find the property, it looks for nested objects that contain a matching property.
An alternative would be as #AthanasiosKataras says; use Bind<>. This is helpful when you might have a sparse configuration in which you want to overlay some values with either default or calculated values.

I have spotted the following issue on GitHub: GetValue<T> not working with lists
Long story short: It is by design.
So you can try this:
var result = new List<ConnectionSettings>();
var rr = configuration.GetSection("Connections").Bind(result);

Configuration.Get<T> is a better option when you have nested configuration using non-primitive structure like list or array.
{
"Email": {
"ToEmails": [
"test1#test.com",
"test2#test.com",
"test3#test.com"
]
}
List<string> emailTo = _config.GetSection("Email:ToEmails").Get<List<string>>()
foreach (string email in emailTo)
{
sendGridMessage.AddTo(new EmailAddress(email));
}
OR use Bind()
public static class ConfigurationRootExtentions
{
public static List<T> GetListValue<T>(this IConfigurationRoot configurationRoot, string section)
{
var result = new List<T>();
configurationRoot.GetSection(section).Bind(result);
return result;
}
}
Ref[1]: https://blog.bitscry.com/2017/11/14/reading-lists-from-appsettings-json/
Ref[2]: https://github.com/aspnet/Configuration/issues/451

If you develop a project from scratch i.e using the VS template then set the following properties for appsettings.json (Right-click on appsettings.json--> click on Properties ). After that update following property value.
Copy to output Directory: Copy always
Now, you can run the application and get value using GetValue method.
After that, You can reverse the same setting as pervious
Copy to output Directory: Copy if newer

It might be useful for you.
GetValue<type>("key") does not work for complex types.
Let's say you have the following appsettings.json structure:
{
"ArrayOfObjectsWithinObjects": [
{
"foo0": "bar",
"foo1": 1,
"foo2": false,
"anotherObject": {
"foo3.1": "3.1",
"foo3.2": "3.2"
}
},
{
"foo0": "bar",
"foo1": 1,
"foo2": false,
"anotherObject": {
"foo3.1": "3.1",
"foo3.2": "3.2"
}
}
...
]
}
and C# the appropriate classes
public class Foo
{
public string foo0 { get; set;}
public int foo1 { get; set;}
public bool foo2 { get; set;}
public AnotherObject anotherObject { get; set;}
}
public class AnotherObject
{
public string foo3.1 { get; set; }
public string foo3.2 { get; set; }
}
You should be able to:
//using static System.Console;
//using System.Linq;
//using Microsoft.Extensions.Configuration;
var _listOfFoos = _config.GetSection("ArrayOfObjects")
.GetChildren()
.ToList()
.Select( x => new Foo
{
x.GetValue<string>("")
x.GetValue<int>("")
x.GetValue<bool>("")
x.GetSection("anotherObject").Get<AnotherObject>()
});
WriteLine(_listOfFoos.anotherObject.foo3.1);
Note that foo3.1 is an invalid c# syntax and was used for didactic purposes.

NEST 2.0 doesn't persist some fields into ElasticSearch 2.0

This is my document:
[ElasticsearchType(Name = "MyDoc")]
public class MyDoc: Dictionary<string, object>
{
[String(Store = false, Index = FieldIndexOption.NotAnalyzed)]
public string text { get; set; }
}
As you can see, it inherits from Dictionary<string, object> so I can dinamically add fields to it (this is a requirement to make aggregation work)
Here I store the mapping:
client.Map<MyDoc>(m => m.Index("myindexname").AutoMap());
Now I create a new record and store it:
var rec= new MyDoc();
rec.Add("id", "mystuff");
rec.text = "mytext";
client.Index(rec, i => i.Index("myindexname"));
client.Refresh("myindexname");
The record get actually stored but the text field is not persisted. Here the JSON back from ElasticSearch 2.0
{
"_index": "myindexname",
"_type": "MyDoc",
"_id": "AVM3B2dlrjN2fcJKmw_z",
"_version": 1,
"_score": 1,
"_source": {
"id": "mystuff"
}
}
If I remove the base class from MyDoc the text field is stored correctly but obviously the dictionary content is not (I also need to remove the .Add() bit as the document doesn't inherit from a Dictionary).
How to store both the text field and the dictionary content?

Sorry i wrote wrong suggestion in my previous post so i deleted it.
I think issues is actually in serialization since your base class is Dictionary
I would do two things first try to serialize your object to see output string i am pretty sure that text is ignored.
Second i would change class to following
public class MyDoc : Dictionary<string, object>
{
public string text
{
get
{
object mytext;
return TryGetValue("text", out mytext) ? mytext.ToString() : null;
}
set { this.Add("text", value);}
}
}
PS. As i thought issue is on c# side since you inherit from dictionary,
var rec = new MyDoc();
rec.Add("id", "mystuff");
rec.text = "mytext";
//Text2 is property public string text2 { get; set; }
rec.text2 = "mytext2";
var test = JsonConvert.SerializeObject(rec); //{"id":"mystuff","text":"mytext"}