We Keep Coding

Modifying a JSON file using System.Text.Json

I know you can do this easily with Newtonsoft. As I am working with .NET Core 3.0, however, I am trying to use the new methods for interacting with JSON files —i.e., System.Text.Json—and I refuse to believe that what I am trying to do is all that difficult!
My application needs to list users that have not already been added to my database. In order to get the full list of all users, the app retrieves a JSON string from a web API. I now need to cycle through each of these users and check if they have already been added to my application before returning a new JSON list to my view so that it can display the new potential users to the end user.
As I am ultimately returning another JSON at the end of the process, I don't especially want to bother deserializing it to a model. Note that the structure of data from the API could change, but it will always have a key from which I can compare to my database records.
My code currently looks like this:
using (WebClient wc = new WebClient())
{
var rawJsonDownload = wc.DownloadString("WEB API CALL");
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
foreach (var user in users.ToList())
{
//Check if User is new
if (CHECKS)
{
users.Remove(user);
}
}
return Json(users);
}
This seems like a lot of hoops to jump through in order to achieve something that would be fairly trivial with Newtonsoft.
Can anyone advise me on a better way of doing this—and, ideally, without the need for the UserObject?

Your problem is that you would like to retrieve, filter, and pass along some JSON without needing to define a complete data model for that JSON. With Json.NET, you could use LINQ to JSON for this purpose. Your question is, can this currently be solved as easily with System.Text.Json?
As of .NET 6, this cannot be done quite as easily with System.Text.Json because it has no support for JSONPath which is often quite convenient in such applications. There is currently an open issue Add JsonPath support to JsonDocument/JsonElement #41537 tracking this.
That being said, imagine you have the following JSON:
[
{
"id": 1,
"name": "name 1",
"address": {
"Line1": "line 1",
"Line2": "line 2"
},
// More properties omitted
}
//, Other array entries omitted
]
And some Predicate<long> shouldSkip filter method indicating whether an entry with a specific id should not be returned, corresponding to CHECKS in your question. What are your options?
In .NET 6 and later you could parse your JSON to a JsonNode, edit its contents, and return the modified JSON. A JsonNode represents an editable JSON Document Object Model and thus most closely corresponds to Newtonsoft's JToken hierarchy.
The following code shows an example of this:
var root = JsonNode.Parse(rawJsonDownload).AsArray(); // AsArray() throws if the root node is not an array.
for (int i = root.Count - 1; i >= 0; i--)
{
if (shouldSkip(root[i].AsObject()["id"].GetValue<long>()))
root.RemoveAt(i);
}
return Json(root);
Mockup fiddle #1 here
In .NET Core 3.x and later, you could parse to a JsonDocument and return some filtered set of JsonElement nodes. This works well if the filtering logic is very simple and you don't need to modify the JSON in any other way. But do note the following limitations of JsonDocument:
JsonDocument and JsonElement are read-only. They can be used only to examine JSON values, not to modify or create JSON values.
JsonDocument is disposable, and in fact must needs be disposed to minimize the impact of the garbage collector (GC) in high-usage scenarios, according to the docs. In order to return a JsonElement you must clone it.
The filtering scenario in the question is simple enough that the following code can be used:
using var usersDocument = JsonDocument.Parse(rawJsonDownload);
var users = usersDocument.RootElement.EnumerateArray()
.Where(e => !shouldSkip(e.GetProperty("id").GetInt64()))
.Select(e => e.Clone())
.ToList();
return Json(users);
Mockup fiddle #2 here.
In any version, you could create a partial data model that deserializes only the properties you need for filtering, with the remaining JSON bound to a [JsonExtensionDataAttribute] property. This should allow you to implement the necessary filtering without needing to hardcode an entire data model.
To do this, define the following model:
public class UserObject
{
[JsonPropertyName("id")]
public long Id { get; set; }
[System.Text.Json.Serialization.JsonExtensionDataAttribute]
public IDictionary<string, object> ExtensionData { get; set; }
}
And deserialize and filter as follows:
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
users.RemoveAll(u => shouldSkip(u.Id));
return Json(users);
This approach ensures that properties relevant to filtering can be deserialized appropriately without needing to make any assumptions about the remainder of the JSON. While this isn't as quite easy as using LINQ to JSON, the total code complexity is bounded by the complexity of the filtering checks, not the complexity of the JSON. And in fact my opinion is that this approach is, in practice, a little easier to work with than the JsonDocument approach because it makes it somewhat easier to inject modifications to the JSON if required later.
Mockup fiddle #3 here.
No matter which you choose, you might consider ditching WebClient for HttpClient and using async deserialization. E.g.:
var httpClient = new HttpClient(); // Cache statically and reuse in production
var root = await httpClient.GetFromJsonAsync<JsonArray>("WEB API CALL");
Or
using var usersDocument = await JsonDocument.ParseAsync(await httpClient.GetStreamAsync("WEB API CALL"));
Or
var users = await JsonSerializer.DeserializeAsync<List<UserObject>>(await httpClient.GetStreamAsync("WEB API CALL"));
You would need to convert your API method to be async as well.

How To Serve Dynamic Objects From Odata In MVC

I am writing a generic rest service against a json database. I would like to serve my data as IQuerable OData. Do to the nature of the data, my objects are untyped dynamic objects. For instance :
[EnableQuery]
public IQueryable<dynamic> GetObjects()
{
return db.BigListOfJsonStrings
// returns dynamic object
.Select(o => System.Web.Helpers.Json.Decode(o))
.AsQueryable();
}
Now this does not work, but you get an Idea of what I am going for. The exception received is
The query specified in the URI is not valid. Could not find a property named 'propertyName' on type 'System.Object
I inspected the objects being returned, and they are in-fact type-less. They are JsonDynamicObjects a type of string dictionary (like the ExpandoObjects).
This said, what is the next step I need to take to resolve this issue ? Do I need to write a custom query provider or a custom OData provider or something ? A link to the correct documentation would be helpful.
Edit
For the time being I have this solution using dynamics on hold. I have achieved partial OData support by deserializing the objects as JObjects and manually applying the ODataQuery. It is not an ideal solution, but it works.

MongoDB CSUUID Fields

A bit of a strange scenario, but I basically have a strongly typed model, lets call it Person. This model is saved into MongoDB using the C# driver. Then another application pulls out the raw BSON document (via QueryDocument) then calls ToJson() and spits it out somewhere else for something else to consume.
However the JSON spat out has custom CSUUID fields in the JSON and the serialization framework doesn't know how to deal with them, so is there any way to just remove them and have it just have the GUID without the CSUUID wrapper?

Yes, when you do .ToJson() with an overload that takes JsonWriterSettings. JsonWriterSettings has a property call OutputMode which corresponds to the JsonOutputMode. With it, you can choose the level of "strictness" you want.
var settings = new JsonWriterSettings
{
OutputMode = JsonOutputMode.Strict
};
return doc.ToJson(settings);

Serialize an object when posting data with RestSharp

I've recently started using RestSharp to consume a REST service which uses XML.
It makes deserializing objects from XML to a collection of custom objects trivial. But my question is what is the best way to reserialize when posting back to the service?
Should I use LINQ-to-XML to reserialize? I tried using the Serializeable attribute and a SerializeToXml utility function, but when I do so it seems to break the deserializing performed by RestSharp.

I have been able to use attributes to get all of what I need, although my situation is relatively simple. For example, to get it to deserialize nodes with dashes in them, and then to be able to serialize to the same node name I used this:
[XmlElement(ElementName = "short-name")]
[SerializeAs(Name = "short-name")]
public string shortName { get; set; }
So, in your example, serialization doesn't respect [XmlElement("elementName")]. Instead, you will need to use [SerializeAs(Name = "elementName")].
I found this by trolling through the test code in the RestSharp project.

After looking at the source code for RestSharp I found that they actually have a built in wrapper for System.Xml.Serialization.XmlSerializer named DotNetXmlSerializer, it's just not used by default. To use it, just add the following line:
var request = new RestRequest();
request.RequestFormat = RequestFormat.Xml;
request.XmlSerializer = new DotNetXmlSerializer();
request.AddBody(someObject);

On a recent project I used XElement (from the System.Xml.Linq assembly) to manually build up my requests. I only had a handful of properties to deal with though. RestSharp solved the real problem which was deserializing the large XML graph responses from the server.
If your object model is dissimilar to XML schema you will have to create another object model, and map to that, just so it can be serialized automagically, using some library. In that situation you may be better off manually mapping to the schema.

RestSharp supports some basic XML serialization, which you can override if needed:
var request = new RestRequest();
request.RequestFormat = RequestFormat.Xml;
request.XmlSerializer = new SuperXmlSerializer(); // optional override, implements ISerializer
request.AddBody(person); // object serialized to XML using current XML serializer

ASP.NET C# serialize HttpValueCollection

I'm trying to serialize a Request object for logging purposes. The code
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(obj.GetType());
// obj is a Request object
gives me the following exception:
To be XML serializable, types which inherit from ICollection must have an implementation of Add(System.String) at all levels of their inheritance hierarchy. System.Web.HttpValueCollection does not implement Add(System.String).
How to solve the problem? Thanks.

In short, trying to serialize a http request object may not end well; even if you get past the current issue, I would expect it to fail in a few more places.
You should construct your own object model that includes those parts of the request you care about, in a simple form. In the case of the HttpValueCollection, you may need to add a basic collection of some type that is a name/value pair.
Then: populate your new model from the actual request, and serialize your model.

If you're interested in the request as a whole (ie: bytes), you can use the HttpRequest.Filter Property. It allows to install a filter (an object that derives from Stream) that can read and write from the raw input HTTP request.
Here is an article on the subject: Filtering HTTP Requests with .NET

I didn't try it but this would likely work and would be available to serialize.
HttpContext.Current.Request.GetType()
.GetProperties()
.Select(
a =>
new KeyValuePair<object, object>(a, HttpContext.Current.Request.GetType().GetProperty(a.GetType().Name)))
.ToList();