We Keep Coding

Modifying a JSON file using System.Text.Json

I know you can do this easily with Newtonsoft. As I am working with .NET Core 3.0, however, I am trying to use the new methods for interacting with JSON files —i.e., System.Text.Json—and I refuse to believe that what I am trying to do is all that difficult!
My application needs to list users that have not already been added to my database. In order to get the full list of all users, the app retrieves a JSON string from a web API. I now need to cycle through each of these users and check if they have already been added to my application before returning a new JSON list to my view so that it can display the new potential users to the end user.
As I am ultimately returning another JSON at the end of the process, I don't especially want to bother deserializing it to a model. Note that the structure of data from the API could change, but it will always have a key from which I can compare to my database records.
My code currently looks like this:
using (WebClient wc = new WebClient())
{
var rawJsonDownload = wc.DownloadString("WEB API CALL");
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
foreach (var user in users.ToList())
{
//Check if User is new
if (CHECKS)
{
users.Remove(user);
}
}
return Json(users);
}
This seems like a lot of hoops to jump through in order to achieve something that would be fairly trivial with Newtonsoft.
Can anyone advise me on a better way of doing this—and, ideally, without the need for the UserObject?

Your problem is that you would like to retrieve, filter, and pass along some JSON without needing to define a complete data model for that JSON. With Json.NET, you could use LINQ to JSON for this purpose. Your question is, can this currently be solved as easily with System.Text.Json?
As of .NET 6, this cannot be done quite as easily with System.Text.Json because it has no support for JSONPath which is often quite convenient in such applications. There is currently an open issue Add JsonPath support to JsonDocument/JsonElement #41537 tracking this.
That being said, imagine you have the following JSON:
[
{
"id": 1,
"name": "name 1",
"address": {
"Line1": "line 1",
"Line2": "line 2"
},
// More properties omitted
}
//, Other array entries omitted
]
And some Predicate<long> shouldSkip filter method indicating whether an entry with a specific id should not be returned, corresponding to CHECKS in your question. What are your options?
In .NET 6 and later you could parse your JSON to a JsonNode, edit its contents, and return the modified JSON. A JsonNode represents an editable JSON Document Object Model and thus most closely corresponds to Newtonsoft's JToken hierarchy.
The following code shows an example of this:
var root = JsonNode.Parse(rawJsonDownload).AsArray(); // AsArray() throws if the root node is not an array.
for (int i = root.Count - 1; i >= 0; i--)
{
if (shouldSkip(root[i].AsObject()["id"].GetValue<long>()))
root.RemoveAt(i);
}
return Json(root);
Mockup fiddle #1 here
In .NET Core 3.x and later, you could parse to a JsonDocument and return some filtered set of JsonElement nodes. This works well if the filtering logic is very simple and you don't need to modify the JSON in any other way. But do note the following limitations of JsonDocument:
JsonDocument and JsonElement are read-only. They can be used only to examine JSON values, not to modify or create JSON values.
JsonDocument is disposable, and in fact must needs be disposed to minimize the impact of the garbage collector (GC) in high-usage scenarios, according to the docs. In order to return a JsonElement you must clone it.
The filtering scenario in the question is simple enough that the following code can be used:
using var usersDocument = JsonDocument.Parse(rawJsonDownload);
var users = usersDocument.RootElement.EnumerateArray()
.Where(e => !shouldSkip(e.GetProperty("id").GetInt64()))
.Select(e => e.Clone())
.ToList();
return Json(users);
Mockup fiddle #2 here.
In any version, you could create a partial data model that deserializes only the properties you need for filtering, with the remaining JSON bound to a [JsonExtensionDataAttribute] property. This should allow you to implement the necessary filtering without needing to hardcode an entire data model.
To do this, define the following model:
public class UserObject
{
[JsonPropertyName("id")]
public long Id { get; set; }
[System.Text.Json.Serialization.JsonExtensionDataAttribute]
public IDictionary<string, object> ExtensionData { get; set; }
}
And deserialize and filter as follows:
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
users.RemoveAll(u => shouldSkip(u.Id));
return Json(users);
This approach ensures that properties relevant to filtering can be deserialized appropriately without needing to make any assumptions about the remainder of the JSON. While this isn't as quite easy as using LINQ to JSON, the total code complexity is bounded by the complexity of the filtering checks, not the complexity of the JSON. And in fact my opinion is that this approach is, in practice, a little easier to work with than the JsonDocument approach because it makes it somewhat easier to inject modifications to the JSON if required later.
Mockup fiddle #3 here.
No matter which you choose, you might consider ditching WebClient for HttpClient and using async deserialization. E.g.:
var httpClient = new HttpClient(); // Cache statically and reuse in production
var root = await httpClient.GetFromJsonAsync<JsonArray>("WEB API CALL");
Or
using var usersDocument = await JsonDocument.ParseAsync(await httpClient.GetStreamAsync("WEB API CALL"));
Or
var users = await JsonSerializer.DeserializeAsync<List<UserObject>>(await httpClient.GetStreamAsync("WEB API CALL"));
You would need to convert your API method to be async as well.

ServiceStack's RedisTypedClient - Can you use strings to define the type?

I'm trying to figure out if there is any way to create a RedisClient that has the functionality of a RedisTypedClient but able to define the URN key with a simple string instead of passing in a type. For example, at the moment we're creating the IRedisTypedClient like so:
var redisProjectClient = _redisClientsManager.GetClient().As<Project>()
We can then store/retrieve related entities based on types that we know about:
var files = redisProjectClient.GetRelatedEntities<File>(projectId);
However, we're wanting to manage simple JSON payloads of objects from external services (fetched by HTTP API endpoints). These payload objects will be types that we won't know about in c# but will provide schema's/type names that we want to be able to use to manage the relationships like we do in the redis typed client. I can't see if this is currently possible without having to manually manage all of the extra stuff that makes the typed clients so good:
Id listing for entities of that type
The ability to retrieve/update entities related to an object
GetNextSequence for next Id
These aren't available in a flat IRedisClient so I want to do something like this:
var file = "{...}" // JSON object provided by external service
// We will know it's a "Project" type with a "projectID" from the JSON payload data:
var redisProjectClient = _redisClientsManager.GetClient().As("Project");
redisProjectClient.StoreRelatedEntities("File", projectId, file);
// ...
var files = redisProjectClient.GetRelatedEntities("File", projectId);
Any idea if this is possible or how to create this type of client?

RedisTypedClient - Can you use strings to define the type?
No the Typed clients only works with Types, not unknown data structures, you'll need to store any arbitrary JSON payloads using the string IRedisClient.
To maintain the relationships you can store them in Redis sets as described in this previous answer which is what the StoreRelatedEntities() API does behind the scenes where it stores the entities using StoreAll() as normal but also maintains the related entities in an index Set using the urn (e.g. urn:File:x) as the identifier that each entity is stored against.
GetRelatedEntities() then reads all ids maintained in the "relationship set index" and uses GetValues() to fetch all entities by urns.

How to ensure that the class library represents the Json Schema perfectly

I have a JSON Schema, and a class library.
I am able to serialize this class, then convert back successfully to object.
To test it, I create a random object, serialize it.
Then convert to object and check its validity.
And deserialize it just to be sure about the values.
The code below works perfectly - but
I want to be absolutely sure that the class library represents the Json Schema.
Is there a way to achieve this? I found some online tools tries to create the class library from given schema, but none of them were so useful.
// Create random object.
MyObject myObject = new MyObject().CreateRandomMyObject();
// Serialize it.
string JSONObjectText = JsonConvert.SerializeObject(myObject);
// Check if schema is valid.
JSchema schema = JSchema.Parse(txtSchema.Value);
// Check if the serialized object is valid for schema.
JObject jsonObject = JObject.Parse(JSONObjectText);
IList<string> errorMessages;
bool valid = jsonObject.IsValid(schema, out errorMessages);
// Check if the serialized object can be deserialized.
MyObject myObjectReDeserialized = (MyObject)JsonConvert.DeserializeObject(JSONObjectText, typeof(MyObject), new JsonSerializerSettings() { MissingMemberHandling = MissingMemberHandling.Error });

A way to do test-oriented assertion of your mapping is to use FsCheck to generate plenty of random objects and then assert what you want to hold from them: in this case, that
their serialization is valid given the schema,
they can be deserialized back to the same object. (You should make sure you are using structural equality there.)
To be precise, such approach only checks that everything described by your objects is representable by the schema. You might want to do the other way also -- that every JSON that conforms to the schema is representable by your objects. Again, you can generate many possible JSONs conforming to the schema and check that
they can be deserialized to your objects,
reserialization of those objects gives you the same JSON you started with.
Beware though, this might not be practical: FsCheck probably don't have some nice, first-class support for JSON schema based generation out-of-the-box.
If the schema you have is going to change in the future, it would be really great to have a way to generate corresponding objects to have strong types even at the boundary of your application. Have you tried Swagger Codegen? Swagger describes it's endpoints using subset of JSON Schema. The corresponding tooling might help you.

EF TPH inheritance lost in Web Api JSON

I've successfully set up some classes that use TPH EF inheritance, MyBaseClass, MySubClass1, MySubClass2 etc.
When querying using Linq context.MyBaseClasses.Where(...), the objects returned all correctly use the subclass specified by the Discriminator field in the database. (So I might end up with a list containing a mix of objects of MySubClass1, or MySubClass2.)
However, when I pass these objects to a WPF application, via a JSON Web Api call, the objects received are all of MyBaseClass, rather than the correct sub class they started off at.
The object property they are returned via is of type public virtual List<MyBaseClass> MyThings, so I guess it makes sense that they all end up as this type, but I want to retain the correct sub class type for each object.
How do I achieve this? Do I need to force the EF Discriminator field to be sent along with all the other data somehow?
Edit
1.
At the client end, I'm now attempting to deserialize the JSON (including $type data) like so (with no luck, the items are still converted back to their base class)
HttpResponseMessage response = GetClient().GetAsync(url).Result;
if (response.IsSuccessStatusCode)
{
string jsonMessage;
using (Stream responseStream = response.Content.ReadAsStreamAsync().Result)
{
jsonMessage = new StreamReader(responseStream).ReadToEnd();
}
List<My.Domain.Models.MyBaseClass> thingsToReturn;
//new method
thingsToReturn = JsonConvert.DeserializeObject<List<My.Domain.Models.MyBaseClass>>(jsonMessage);
//previous method
//thingsToReturn = response.Content.ReadAsAsync<List<My.Domain.Models.MyBaseClass>>().Result;
return thingsToReturn;
}
2.
Following Anish's advice re SerializerSettings.TypeNameHandling, I've now got $type information appearing in my JSON, however this is of type System.Data.Entity.DynamicProxies.SubClass1_5E07A4CE2F037430DC7BFA00593.... is this OK for the client end to deserialize back into SubClass1 successfully?

This is a serialization concern.
You can customize Json.Net's serializer settings to include type information with your json objects.
Add this to your HttpConfiguration:
config.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = TypeNameHandling.Auto;
Where config is the HttpConfiguration instance that you use to configure and initialize Asp.Net WebApi.
This will tell Json.Net to add some type information to each json object that has type ambiguity. Such an object would look like this:
{
"$type":"MyProjectContainingMyTypes.MySubClass1, MyProjectContainingMyTypes",
"Name": "Tyrion Lannister",
"DisplayName": "The Imp",
"Traits": ["funny", "awesome", "clever"]
}
Json.Net will know how to deal with this when you deserialize this on the WPF side.
This should work, in your WPF app:
var things = JsonConvert.DeserializeObject<List<MyBaseClass>>(jsonString);
Then you can cast the objects in the things list to their respective derived types.
Of course, your WPF application will need to have a reference to the project where you define MyBaseClass and MySubClass1.
Edit
Thanks Anish, that's almost sorted it. I can see the correct $type data in the JSON, I'm just being a dunce, and I'm not sure how to get the WebApi response as a jsonString? At the minute I'm doing response.Content.ReadAsAsync>().Result; to auto deserialize the data.
In response to your comment, you can read the content as a string like this:
var jsonString = response.Content.ReadAsStringAsync().Result;
Edit 2
Anish, thanks so much for your input. As you can see, I've managed to get the JSON as a string now, but even using JsonConvert.DeserializeObject I'm still getting the same issue. Do you think it is (as I mention in Edit 2) to do with the $type being returned from the service incorrectly (as an EF proxy type)?
In response to your comment, yes you will not be able to deserialize the EF proxy type into the type you want. This issue needs to be resolved on the WebApi side.
The proxy classes are created to allow you to lazy load entities. Proxies are generally used represent referenced/nested entities. This allows the fetching of referenced entities from the database to be deferred until required, if required at all.
Here is a link to some documentation around lazy and eager loading entities with EF.
Solution
You want to hydrate the list of objects in your WebApi controller action and return it, this will tell EF to load the entities from the database and new up instances of your classes.
You have a few options here:
Option 1
Call ToList() on the query to hydrate the collection:
var result = (from t in dbContext.Things select t).ToList();
or
var result = dbContext.Things.ToList();
Naturally, you don't want to return an unbounded result set so add a range:
var result = (from t in dbContext.Things select t).Skip(0).Take(10).ToList();
or
var result = dbContext.Things.Skip(0).Take(10).ToList();
Bear in mind that with method you will have to explicitly hydrate nested objects like this:
var result = dbContext
.Things
.Include(t => t.SomePropertyThatRepresentsSomeNestedObject)
.Skip(0)
.Take(10)
.ToList();
Option 2
Turn off lazy loading for your DbContext.
Personally, I'd go with Option 1, I think it is better to know your entities and have control over when and what you hydrate.

Many, many thanks to Anish, who's answer set me off on the right track, along with this excellent article.
The steps I had to take to get the types of the inherited objects to pass through the web api service are as follows:
Server Side
The key things were to set the JsonFormatter Serializer TypeNameHandling setting to TypeNameHandling.Auto. This can be done in WebApiConfig.Register(), but it will then obviously add a $type property to all JSON objects returned by your web service calls, or, you can simply decorate the property of the object you need the $type for.
WebApiConfig Method
GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = Newtonsoft.Json.TypeNameHandling.Auto;
Property Decoration Method
[Newtonsoft.Json.JsonProperty(ItemTypeNameHandling = Newtonsoft.Json.TypeNameHandling.Auto)]
public virtual List<MyBaseClass> Things { get; set; }
To get the correct value in the $type property in the JSON, and not the EF proxy class name, I turned off the ProxyCreationEnabled property before performing the Linq query that returned the objects based on MyBaseClass.
dbContext.Configuration.ProxyCreationEnabled = false;
List<MyBaseClass> things = dbContext.MyBaseClasses.Include("This").Include("That").ToList();
Client Side
I had to add a JsonMediaTypeFormatter with SerializerSettings = { TypeNameHandling = TypeNameHandling.Auto } to the ReadAsAsync() call and then the (correctly typed) objects mapped to their sub class happily.
HttpResponseMessage response = GetClient().GetAsync(url).Result;
if (response.IsSuccessStatusCode)
{
//this formatter responds to the $type parameter passed in the JSON to allow us to correctly map object types
//https://kirmir.wordpress.com/2014/05/16/polymorphic-serialization-using-newton-json-net-in-httpcontent/
var formatter = new JsonMediaTypeFormatter
{
SerializerSettings = { TypeNameHandling = TypeNameHandling.Auto }
};
List<MyBaseClass> thingsToReturn;
thingsToReturn = response.Content.ReadAsAsync<List<MyBaseClass>>(new List<MediaTypeFormatter> { formatter }).Result;
return productTestsToReturn;
}

Automatically map asmx-provided xml data to POCO object

In my .net code I am consuming a third-party asmx service that provides me data in Xml format. So basically, I am recieving a structure in a form of XmlNode:
<PostcodeEntry>
<Postcode>13542</Postcode>
<Postcodename>Odessa</Postcodename>
</PostcodeEntry>
Currently, to map it to my POCO object I have to manually iterate through a corresponding ChildNode's and retrieve their InnerText value to get the actual data:
var PostCodeNode = entryNode.SelectSingleNode("Postcode");
if (PostCodeNode != null)
{
result.PostCode = PostCodeNode.InnerText;
}
In case I need to map a large info structure, the code grows to a messy code-scroll.
Is there a way I can improve this so I don't have to write the parsing manually? What is the best practice for this?

I believe that you have different options depending on how you get your data and how you like to design your code etc. From your brief description I can think of at least these two:
Create an XML Serializer - for example by marking up your class with Xml Attributes and de-serialize the XML directly as your desired object via the serializer. The disadvantage of this approach is that you will create a strong coupling between your serializer and your business object. Please take a look at something like this: http://www.switchonthecode.com/tutorials/csharp-tutorial-xml-serialization.
Create a proxy object and map your proxy object to your business object. You can create the proxy object either by using a WSDL exposed by the asmx service, or by using the XSD.exe tool or similar (you may need to first generate an XSD based on the XML if the XML is not already described by an XSD). Then you can map the properties of your proxy object to the properties of your business object. This will provide you a more clean separation between the objects, but at the same time it requires more work.
Br. Morten

You can create SoapClient object for WebService, then you can return the Response as List<>. You need to change the Ouput response to List<>.
example Consilder this the webservice to consume, http://xxx.xx.xxx.xxx/CosmosService/Cm_Service.asmx
then add Service Reference in your application, Click On Advanced Button, change the Collection Type System.Collections.GenericList.
then you can Consume WebService Methods as List<> directly like this
CosmosRef.CM_ServiceSoapClient client = new CosmosRef.CM_ServiceSoapClient();
List<CosmosRef.Product> listProduct = client.GetAllProducts("Computers", 1);
dataGrid1.DataContext = listProduct;