I know you can do this easily with Newtonsoft. As I am working with .NET Core 3.0, however, I am trying to use the new methods for interacting with JSON files —i.e., System.Text.Json—and I refuse to believe that what I am trying to do is all that difficult!
My application needs to list users that have not already been added to my database. In order to get the full list of all users, the app retrieves a JSON string from a web API. I now need to cycle through each of these users and check if they have already been added to my application before returning a new JSON list to my view so that it can display the new potential users to the end user.
As I am ultimately returning another JSON at the end of the process, I don't especially want to bother deserializing it to a model. Note that the structure of data from the API could change, but it will always have a key from which I can compare to my database records.
My code currently looks like this:
using (WebClient wc = new WebClient())
{
var rawJsonDownload = wc.DownloadString("WEB API CALL");
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
foreach (var user in users.ToList())
{
//Check if User is new
if (CHECKS)
{
users.Remove(user);
}
}
return Json(users);
}
This seems like a lot of hoops to jump through in order to achieve something that would be fairly trivial with Newtonsoft.
Can anyone advise me on a better way of doing this—and, ideally, without the need for the UserObject?
Your problem is that you would like to retrieve, filter, and pass along some JSON without needing to define a complete data model for that JSON. With Json.NET, you could use LINQ to JSON for this purpose. Your question is, can this currently be solved as easily with System.Text.Json?
As of .NET 6, this cannot be done quite as easily with System.Text.Json because it has no support for JSONPath which is often quite convenient in such applications. There is currently an open issue Add JsonPath support to JsonDocument/JsonElement #41537 tracking this.
That being said, imagine you have the following JSON:
[
{
"id": 1,
"name": "name 1",
"address": {
"Line1": "line 1",
"Line2": "line 2"
},
// More properties omitted
}
//, Other array entries omitted
]
And some Predicate<long> shouldSkip filter method indicating whether an entry with a specific id should not be returned, corresponding to CHECKS in your question. What are your options?
In .NET 6 and later you could parse your JSON to a JsonNode, edit its contents, and return the modified JSON. A JsonNode represents an editable JSON Document Object Model and thus most closely corresponds to Newtonsoft's JToken hierarchy.
The following code shows an example of this:
var root = JsonNode.Parse(rawJsonDownload).AsArray(); // AsArray() throws if the root node is not an array.
for (int i = root.Count - 1; i >= 0; i--)
{
if (shouldSkip(root[i].AsObject()["id"].GetValue<long>()))
root.RemoveAt(i);
}
return Json(root);
Mockup fiddle #1 here
In .NET Core 3.x and later, you could parse to a JsonDocument and return some filtered set of JsonElement nodes. This works well if the filtering logic is very simple and you don't need to modify the JSON in any other way. But do note the following limitations of JsonDocument:
JsonDocument and JsonElement are read-only. They can be used only to examine JSON values, not to modify or create JSON values.
JsonDocument is disposable, and in fact must needs be disposed to minimize the impact of the garbage collector (GC) in high-usage scenarios, according to the docs. In order to return a JsonElement you must clone it.
The filtering scenario in the question is simple enough that the following code can be used:
using var usersDocument = JsonDocument.Parse(rawJsonDownload);
var users = usersDocument.RootElement.EnumerateArray()
.Where(e => !shouldSkip(e.GetProperty("id").GetInt64()))
.Select(e => e.Clone())
.ToList();
return Json(users);
Mockup fiddle #2 here.
In any version, you could create a partial data model that deserializes only the properties you need for filtering, with the remaining JSON bound to a [JsonExtensionDataAttribute] property. This should allow you to implement the necessary filtering without needing to hardcode an entire data model.
To do this, define the following model:
public class UserObject
{
[JsonPropertyName("id")]
public long Id { get; set; }
[System.Text.Json.Serialization.JsonExtensionDataAttribute]
public IDictionary<string, object> ExtensionData { get; set; }
}
And deserialize and filter as follows:
var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
users.RemoveAll(u => shouldSkip(u.Id));
return Json(users);
This approach ensures that properties relevant to filtering can be deserialized appropriately without needing to make any assumptions about the remainder of the JSON. While this isn't as quite easy as using LINQ to JSON, the total code complexity is bounded by the complexity of the filtering checks, not the complexity of the JSON. And in fact my opinion is that this approach is, in practice, a little easier to work with than the JsonDocument approach because it makes it somewhat easier to inject modifications to the JSON if required later.
Mockup fiddle #3 here.
No matter which you choose, you might consider ditching WebClient for HttpClient and using async deserialization. E.g.:
var httpClient = new HttpClient(); // Cache statically and reuse in production
var root = await httpClient.GetFromJsonAsync<JsonArray>("WEB API CALL");
Or
using var usersDocument = await JsonDocument.ParseAsync(await httpClient.GetStreamAsync("WEB API CALL"));
Or
var users = await JsonSerializer.DeserializeAsync<List<UserObject>>(await httpClient.GetStreamAsync("WEB API CALL"));
You would need to convert your API method to be async as well.
Related
I've recently started working with JSON structures from Microsoft Dynamics 365 dataverse. A lot of their odata is structured like this:
{
"#odata.context":"https://....",
"value":[
{
"#odata.etag":"W/\"Jz....",
"dataAreaId":"foo",
"ItemNumber":"TEST",
"IsPhantom":"No"
}
]
}
I simply want to get the value of ItemNumber, which in this case is TEST. This seems like it should be very simple, but after an hour and a dozen different approaches, I'm wondering what I'm missing.
When using Newtsonsoft, it seems like all their approaches require a fully-baked class that perfectly matches the json structure, and the correct specification of complex combinations of <T>. This is tedious because I work with many different json data sets and almost all are different from each other in terms of the attributes and their types. I tried things like:
dynamic try1 = JsonConvert.DeserializeObject<dynamic>(jsonResult);
dynamic try2 = JObject.Parse(jsonResult);
dynamic try3 = JsonConvert.DeserializeObject<ExpandoObject>(jsonResult, new ExpandoObjectConverter());
JObject try4 = (JObject)JsonConvert.DeserializeObject<dynamic>(jsonResult);
Then I gave up on Newtonsoft and tried JavaScriptSerializer, but ran into similar dead ends.
var try5 = new JavaScriptSerializer().Deserialize<Dictionary<string, object>>(jsonResult);
string try6 = Utilities.SafeTrim(jsonObj["value"]);
In every case I end up with a valid object, but I can never figure out how to traverse down into the object and grab the value I want. I frequently end up with completely useless constructs like this from the VS watch window, which it delivers just to mock me:
new System.Linq.SystemCore_EnumerableDebugView<System.Collections.Generic.KeyValuePair<string, Newtonsoft.Json.Linq.JToken>>(new System.Linq.SystemCore_EnumerableDebugView<Newtonsoft.Json.Linq.JToken>((new System.Linq.SystemCore_EnumerableDebugView<System.Collections.Generic.KeyValuePair<string, Newtonsoft.Json.Linq.JToken>>(try1).Items[1]).Value).Items[0]).Items[2]
I'm sure someone has the correct and elegant way to solve this, which would be fine. But what I would really love is a way to take any JSON string and convert to an object, array, or collection that I can easily tease out any value with nested array syntax (or something equally simple). So for my example, maybe something like this:
jsonObject["value"]["ItemNumber"]
Is something like that so difficult?
This code:
const string jsonInput = #"{
'odata.context':'https://....',
'value':[{
'#odata.etag':'W/z....',
'dataAreaId':'foo',
'ItemNumber':'TEST',
'IsPhantom':'No'
}]}";
dynamic try1 = JsonConvert.DeserializeObject(jsonInput)!;
Console.WriteLine(try1["value"][0]["ItemNumber"]);
returns
TEST
as expected
I have this JSON:
{
"response":
{
"data":
[
{
"start":1,
"subjects":["A"]
},
{
"start":3,
"subjects":["B"]
},
{
"start":2,
"subjects":["C"]
}
]
}
}
And I want to get only the "subject" data from the object with it's "start" value to be the smallest one that is greater than 1.3, which in this case would be C. Would anybody happen to know how such a thing can be achieved using C#?
I want to extend a bit on the other answers and shed more light into the subject.
A JSON -- JavaScript Object Notation - is just a way to move data "on a wire". Inside .NET, you shouldn't really consider your object to be a JSON, although you may colloquially refer to a data structure as such.
Having said that, what is a JSON "inside" .NET? It's your call. You can, for instance treat it as a string, but you will have a hard time doing this operation of finding a specific node based on certain parameters/rules.
Since a JSON is a tree-like structure, you could build your on data structure or use the many available on the web. This is great if you are learning the workings of the language and programming in general, bad if you are doing this professionally because you will probably be reinventing the wheel. And parsing the JSON is not a easy thing to do (again, good exercise).
So, the most time-effective way of doing? You have two options:
Use a dynamic object to represent your JSON data. A dynamic is a "extension" to .NET (actually, an extension to the CLR, that is called DLR) which lets you create objects that doesn't have classes (they can be considered to be "untyped", or, better, to use duck typing).
Use a typed structure that you defined to hold your data. This is the canonical, object-oriented, .NET way of doing it, but there's a trade-off in declaring classes and typing everything, which is costly in terms of time. The payoff is that you get better intellisense, performance (DLR objects are slower than traditional objects) and more safe code.
To go with the first approach, you can refer to #YouneS answer. You need to add a dependency to your project, Newtonsoft.Json (a nuget), and call deserialize to convert the JSON string to a dynamic object. As you can see from his answer, you can access properties in this object as you would access then on a JavaScript language. But you'll also realize that you have no intellisense and things such as myObj.unexistentField = "asd" will be allowed. That is the nature of dynamic typed objects.
The second approach is to declare all types. Again, this is time consuming and on many cases you'll prefer not to do it. Refer to Microsoft Docs to get more insight.
You should first create your data contracts, as below (forgive me for any typos, I'm not compiling the code).
[DataContract]
class DataItem
{
[DataMember]
public string Start { get; set; }
[DataMember]
public string[] Subjects { get; set; }
}
[DataContract]
class ResponseItem
{
[DataMember]
public DateItem[] Data { get; set; }
}
[DataContract]
class ResponseContract
{
[DataMember]
public ResponseItem Response { get; set; }
}
Once you have all those data structures declared, deserialize your json to it:
using (var ms = new MemoryStream(Encoding.Unicode.GetBytes(json)))
{
var deserializer = new DataContractJsonSerializer(typeof(ResponseContract));
return (T)deserializer.ReadObject(ms);
}
The code above may seem a bit complicated, but follow a bit of .NET / BCL standards. The DataContractJsonSerializer work only with streams, so you need to open a stream that contains your string. So you create a memory stream with all the bytes from the json string.
You can also use Newtonsoft to do that, which is much simpler but, of course, still requires that extra dependency:
DataContract contract = JsonConvert.DeserializeObject<DataContract>(output);
If you use this approach you don't need the annotations (all those DataMember and DataContract) on your classes, making code a bit more clean. I very much prefer using this approach than DataContractJsonSerializer, but it's your call.
I've talked a lot about serializing and deserializing objects, but your question was, "How do I find a certain node?". All the discussion above was just a prerequisite.
There are, again and as usual, a few ways of achieving what you want:
#YouneS answer. It's very straightforward and achieves what you are looking for.
Use the second approach above, and then use your typed object to get what you want. For instance:
var contract = JsonConvert.DeserializeObject<DataContract>(output);
var query = from dataItem in contract.Response.Data
where dataItem.Start > 1.3
order by dataItem.Start;
var item = query.FirstOrNull();
Which will return the first item which, since it's ordered, should be the smallest. Remember to test the result for null.
You can use a feature from Newtonsoft that enables to directly find the node you want. Refer to the documentation. A warning, it's a bit advanced and probably overkill for simple cases.
You can make it work with something like the following code :
// Dynamic object that will hold your Deserialized json string
dynamic myObj = JsonConvert.DeserializeObject<dynamic>(YOUR-JSON-STRING);
// Will hold the value you are looking for
string[] mySubjectValue = "";
// Looking for your subject value
foreach(var o in myObj.response.data) {
if(o.start > 1.3)
mySubjectValue = o.subjects;
}
I'm using Auth0 for a new app I'm writing using ASP.NET Core 1.0. It's working great so far, but I have bumped into something that is stumping me for some reason.
When the user logs in, Auth0 will pass claims back to my app. Auth0 has the ability to add custom data to the user, it is stored in JSON format and comes over as a list of claims.
One of the claims will look something like this:
Type "app_metadata"
Value "\"IsPublisherFor\":[\"p56\",\"p124\",\"p258\"]"
ValueType "JSON"
My question is, how would I convert that claim value into something I can work with? For example, the ability to do something like IsPublisherFor.Contains("p56");
I tried to pass the value to NewtonSoft.Json.JsonConvert.DeserializeObject(value) but it throws an exception. JsonReaderException: Additional text encountered after finished reading JSON content
Any way to convert this cleanly?
You probably need to wrap your JSON string in curly braces. You generally need a single top-level array or object to have valid JSON. Don't know about NewtonSoft, but using the JavascriptSerializer, this works:
var json = "{\"IsPublisherFor\":[\"p56\",\"p124\",\"p258\"]}";
var serializer = new JavaScriptSerializer();
stuff obj = serializer.Deserialize<stuff>(json);
With the following class defined to receive the data:
public class stuff
{
public string[] IsPublisherFor { get; set; }
}
I've successfully set up some classes that use TPH EF inheritance, MyBaseClass, MySubClass1, MySubClass2 etc.
When querying using Linq context.MyBaseClasses.Where(...), the objects returned all correctly use the subclass specified by the Discriminator field in the database. (So I might end up with a list containing a mix of objects of MySubClass1, or MySubClass2.)
However, when I pass these objects to a WPF application, via a JSON Web Api call, the objects received are all of MyBaseClass, rather than the correct sub class they started off at.
The object property they are returned via is of type public virtual List<MyBaseClass> MyThings, so I guess it makes sense that they all end up as this type, but I want to retain the correct sub class type for each object.
How do I achieve this? Do I need to force the EF Discriminator field to be sent along with all the other data somehow?
Edit
1.
At the client end, I'm now attempting to deserialize the JSON (including $type data) like so (with no luck, the items are still converted back to their base class)
HttpResponseMessage response = GetClient().GetAsync(url).Result;
if (response.IsSuccessStatusCode)
{
string jsonMessage;
using (Stream responseStream = response.Content.ReadAsStreamAsync().Result)
{
jsonMessage = new StreamReader(responseStream).ReadToEnd();
}
List<My.Domain.Models.MyBaseClass> thingsToReturn;
//new method
thingsToReturn = JsonConvert.DeserializeObject<List<My.Domain.Models.MyBaseClass>>(jsonMessage);
//previous method
//thingsToReturn = response.Content.ReadAsAsync<List<My.Domain.Models.MyBaseClass>>().Result;
return thingsToReturn;
}
2.
Following Anish's advice re SerializerSettings.TypeNameHandling, I've now got $type information appearing in my JSON, however this is of type System.Data.Entity.DynamicProxies.SubClass1_5E07A4CE2F037430DC7BFA00593.... is this OK for the client end to deserialize back into SubClass1 successfully?
This is a serialization concern.
You can customize Json.Net's serializer settings to include type information with your json objects.
Add this to your HttpConfiguration:
config.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = TypeNameHandling.Auto;
Where config is the HttpConfiguration instance that you use to configure and initialize Asp.Net WebApi.
This will tell Json.Net to add some type information to each json object that has type ambiguity. Such an object would look like this:
{
"$type":"MyProjectContainingMyTypes.MySubClass1, MyProjectContainingMyTypes",
"Name": "Tyrion Lannister",
"DisplayName": "The Imp",
"Traits": ["funny", "awesome", "clever"]
}
Json.Net will know how to deal with this when you deserialize this on the WPF side.
This should work, in your WPF app:
var things = JsonConvert.DeserializeObject<List<MyBaseClass>>(jsonString);
Then you can cast the objects in the things list to their respective derived types.
Of course, your WPF application will need to have a reference to the project where you define MyBaseClass and MySubClass1.
Edit
Thanks Anish, that's almost sorted it. I can see the correct $type data in the JSON, I'm just being a dunce, and I'm not sure how to get the WebApi response as a jsonString? At the minute I'm doing response.Content.ReadAsAsync>().Result; to auto deserialize the data.
In response to your comment, you can read the content as a string like this:
var jsonString = response.Content.ReadAsStringAsync().Result;
Edit 2
Anish, thanks so much for your input. As you can see, I've managed to get the JSON as a string now, but even using JsonConvert.DeserializeObject I'm still getting the same issue. Do you think it is (as I mention in Edit 2) to do with the $type being returned from the service incorrectly (as an EF proxy type)?
In response to your comment, yes you will not be able to deserialize the EF proxy type into the type you want. This issue needs to be resolved on the WebApi side.
The proxy classes are created to allow you to lazy load entities. Proxies are generally used represent referenced/nested entities. This allows the fetching of referenced entities from the database to be deferred until required, if required at all.
Here is a link to some documentation around lazy and eager loading entities with EF.
Solution
You want to hydrate the list of objects in your WebApi controller action and return it, this will tell EF to load the entities from the database and new up instances of your classes.
You have a few options here:
Option 1
Call ToList() on the query to hydrate the collection:
var result = (from t in dbContext.Things select t).ToList();
or
var result = dbContext.Things.ToList();
Naturally, you don't want to return an unbounded result set so add a range:
var result = (from t in dbContext.Things select t).Skip(0).Take(10).ToList();
or
var result = dbContext.Things.Skip(0).Take(10).ToList();
Bear in mind that with method you will have to explicitly hydrate nested objects like this:
var result = dbContext
.Things
.Include(t => t.SomePropertyThatRepresentsSomeNestedObject)
.Skip(0)
.Take(10)
.ToList();
Option 2
Turn off lazy loading for your DbContext.
Personally, I'd go with Option 1, I think it is better to know your entities and have control over when and what you hydrate.
Many, many thanks to Anish, who's answer set me off on the right track, along with this excellent article.
The steps I had to take to get the types of the inherited objects to pass through the web api service are as follows:
Server Side
The key things were to set the JsonFormatter Serializer TypeNameHandling setting to TypeNameHandling.Auto. This can be done in WebApiConfig.Register(), but it will then obviously add a $type property to all JSON objects returned by your web service calls, or, you can simply decorate the property of the object you need the $type for.
WebApiConfig Method
GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = Newtonsoft.Json.TypeNameHandling.Auto;
Property Decoration Method
[Newtonsoft.Json.JsonProperty(ItemTypeNameHandling = Newtonsoft.Json.TypeNameHandling.Auto)]
public virtual List<MyBaseClass> Things { get; set; }
To get the correct value in the $type property in the JSON, and not the EF proxy class name, I turned off the ProxyCreationEnabled property before performing the Linq query that returned the objects based on MyBaseClass.
dbContext.Configuration.ProxyCreationEnabled = false;
List<MyBaseClass> things = dbContext.MyBaseClasses.Include("This").Include("That").ToList();
Client Side
I had to add a JsonMediaTypeFormatter with SerializerSettings = { TypeNameHandling = TypeNameHandling.Auto } to the ReadAsAsync() call and then the (correctly typed) objects mapped to their sub class happily.
HttpResponseMessage response = GetClient().GetAsync(url).Result;
if (response.IsSuccessStatusCode)
{
//this formatter responds to the $type parameter passed in the JSON to allow us to correctly map object types
//https://kirmir.wordpress.com/2014/05/16/polymorphic-serialization-using-newton-json-net-in-httpcontent/
var formatter = new JsonMediaTypeFormatter
{
SerializerSettings = { TypeNameHandling = TypeNameHandling.Auto }
};
List<MyBaseClass> thingsToReturn;
thingsToReturn = response.Content.ReadAsAsync<List<MyBaseClass>>(new List<MediaTypeFormatter> { formatter }).Result;
return productTestsToReturn;
}
A bit of a strange scenario, but I basically have a strongly typed model, lets call it Person. This model is saved into MongoDB using the C# driver. Then another application pulls out the raw BSON document (via QueryDocument) then calls ToJson() and spits it out somewhere else for something else to consume.
However the JSON spat out has custom CSUUID fields in the JSON and the serialization framework doesn't know how to deal with them, so is there any way to just remove them and have it just have the GUID without the CSUUID wrapper?
Yes, when you do .ToJson() with an overload that takes JsonWriterSettings. JsonWriterSettings has a property call OutputMode which corresponds to the JsonOutputMode. With it, you can choose the level of "strictness" you want.
var settings = new JsonWriterSettings
{
OutputMode = JsonOutputMode.Strict
};
return doc.ToJson(settings);