We Keep Coding

How To Create a JSON Object within an Existing Object in JSON using JObject

I am trying to edit an existing Json File that contains an object called "forge" ("forge" itself is contained within an object called "profiles"), and within that object, I am trying to add another object called resolution that contains height and width. The native application it belongs to can modify and add this element, but I can only seem to be able to read, and write/create all other objects within the forge object except the resolution object and its children.
But no matter how much I read in newtonsoft documentation, nothing appears to be working. This is what I have written so far. Note that this method is just for overwriting all the contents in the "forge" object, and I tired to add the resolution object, but it fails. Anyone know how to do this? (Connection path is the json file)

I think the "resolution" is not array, but an object.
Try this
dynamic forge = new JObject();
forge.Name = "forge";
forge.Type = "Brasil";
dynamic resolutionObj = new JObject();
resolutionObj.Width = "10";
resolutionObj.Height = "100";
forge.resolution = resolutionObj;
You will see anything like this:
{
"Name": "forge",
"Type": "Brasil",
"resolution": {
"Width": "10",
"Height": "100"
}
}

Deserialize Avro Spark

I'm pushing a stream of data to Azure EventHub with the following code leveraging Microsoft.Hadoop.Avro.. this code runs every 5 seconds, and simply plops the same two Avro serialised items 👍🏼:
var strSchema = File.ReadAllText("schema.json");
var avroSerializer = AvroSerializer.CreateGeneric(strSchema);
var rootSchema = avroSerializer.WriterSchema as RecordSchema;
var itemList = new List<AvroRecord>();
dynamic record_one = new AvroRecord(rootSchema);
record_one.FirstName = "Some";
record_one.LastName = "Guy";
itemList.Add(record_one);
dynamic record_two = new AvroRecord(rootSchema);
record_two.FirstName = "A.";
record_two.LastName = "Person";
itemList.Add(record_two);
using (var buffer = new MemoryStream())
{
using (var writer = AvroContainer.CreateGenericWriter(strSchema, buffer, Codec.Null))
{
using (var streamWriter = new SequentialWriter<object>(writer, itemList.Count))
{
foreach (var item in itemList)
{
streamWriter.Write(item);
}
}
}
eventHubClient.SendAsync(new EventData(buffer.ToArray()));
}
The schema used here is, again, v. simple:
{
"type": "record",
"name": "User",
"namespace": "SerDes",
"fields": [
{
"name": "FirstName",
"type": "string"
},
{
"name": "LastName",
"type": "string"
}
]
}
I have validated this is all good, with a simple view in Azure Stream Analytics on the portal:
So far so good, but i cannot, for the life of me correctly deserialize this in Databricks leverage the from_avro() command under Scala..
Load (the exact same) schema as a string:
val sampleJsonSchema = dbutils.fs.head("/mnt/schemas/schema.json")
Configure EventHub
val connectionString = ConnectionStringBuilder("<CONNECTION_STRING>")
.setEventHubName("<NAME_OF_EVENT_HUB>")
.build
val eventHubsConf = EventHubsConf(connectionString).setStartingPosition(EventPosition.fromEndOfStream)
val eventhubs = spark.readStream.format("eventhubs").options(eventHubsConf.toMap).load()
Read the data..
// this works, and i can see the serialised data
display(eventhubs.select($"body"))
// this fails, and with an exception: org.apache.spark.SparkException: Malformed records are detected in record parsing. Current parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
display(eventhubs.select(from_avro($"body", sampleJsonSchema)))
So essentially, what is going on here.. i am serialising the data with the same schema as deserializing, but something is malformed.. the documentation is incredibly sparse on this front (very very minimal on the Microsoft website).

The issue
After additional investigation, (and mainly with the help of this article) I found what my problem was: from_avro(data: Column, jsonFormatSchema: String) expects spark schema format and not avro schema format. The documentation is not very clear on this.
Solution 1
Databricks provides a handy method from_avro(column: Column, subject: String, schemaRegistryUrl: String)) that fetches needed avro schema from kafka schema registry and automatically converts to correct format.
Unfortunately, it is not available for pure spark, nor is it possible to use it without a kafka schema registry.
Solution 2
Use schema conversion provided by spark:
// define avro deserializer
class AvroDeserializer() extends AbstractKafkaAvroDeserializer {
override def deserialize(payload: Array[Byte]): String = {
val genericRecord = this.deserialize(payload).asInstanceOf[GenericRecord]
genericRecord.toString
}
}
// create deserializer instance
val deserializer = new AvroDeserializer()
// register deserializer
spark.udf.register("deserialize_avro", (bytes: Array[Byte]) =>
deserializer.deserialize(bytes)
)
// get avro schema from registry (but I presume that it should also work with schema read from a local file)
val registryClient = new CachedSchemaRegistryClient(kafkaSchemaRegistryUrl, 128)
val avroSchema = registryClient.getLatestSchemaMetadata(topic + "-value").getSchema
val sparkSchema = SchemaConverters.toSqlType(new Schema.Parser().parse(avroSchema))
// consume data
df.selectExpr("deserialize_avro(value) as data")
.select(from_json(col("data"), sparkSchema.dataType).as("data"))
.select("data.*")

Newtonsoft.Json Usage in long trees

I'm trying so scrape some info from a JSON response but it fails; I think it may have to do with the way I'm making use of a library I'm new to.
Below is the JSON instance that I'm trying to get data from;
{
"profileChanges":[
{
"changeType":"fullProfileUpdate",
"profile":{
"stats":{
"attributes":{
"allowed_to_receive_gifts":true,
"allowed_to_send_gifts":true,
"ban_history":{},
//This is what I'm trying to scrape the EpicPCKorea string
"current_mtx_platform":"EpicPCKorea",
"daily_purchases":{},
"gift_history":{},
"import_friends_claimed":{},
"in_app_purchases":{
"fulfillmentCounts":{
"2E5AC9924F2247325BBB22AC9AF9965B":1
},
"receipts":[
"EPIC:543a35e70dde4e0aaf56a9e0b76c8f67"
]
},
"inventory_limit_bonus":0,
"mfa_enabled":false,
"monthly_purchases":{},
"mtx_affiliate":"",
"mtx_purchase_history":{},
"weekly_purchases":{}
}
},
"updated":"2018-12-06T14:37:27.797Z",
}
}
],
"profileChangesBaseRevision":31,
"serverTime":"2018-12-30T18:44:35.451Z"
}
Here, my code in C#;
//str4 is the json response that I just posted up.
dynamic json = JsonConvert.DeserializeObject(str4);
string platform = json.profileChanges.profile.stats.attributes.current_mtx_platform;
But it doesn't work at all.
I debugged it and found out this Exception:
'Newtonsoft.Json.Linq.JArray' does not contain a definition for 'profile'
What am I doing wrong and how can I fix it?

As mentioned in the comments under your question, profileChanges is an array so you need to specify which item in the array to access.
Are you sure the the below does not work? It works for me...
string platform = json.profileChanges[0].profile.stats.attributes.current_mtx_platform;

Parsing/Iterating over Json

I'm fairly new to parsing Json with C# and i'm having a little issue i can't work my head around.
My data looks something like this:
{
"languages": {
"ja_lang": {
"data": {
"name": "Japanese"
},
"files": [["ja",
"Japanese File",
"lang_ja.txt"]]
},
"en_lang": {
"data": {
"name": "English"
},
"files": [["en",
"English File",
"lang_en.txt"]]
}
}
}
Now i want to iterate over the items in languages and only work with the one where the object-name starts with "ja_" (in this case it would only work with "ja_lang" and ignore "en_lang"), then extract the name inside data and the "lang_ja.txt" in files.
To Parse the Json in C# i downloaded the Newtonsoft.Json library and came up with this:
dynamic json_obj = JsonConvert.DeserializeObject("json string");
// when debugging language holds { "ja_lang": { "data": { "name": "Japanese" }, "files": [["ja", "Japanese File", "lang_ja.txt"]] } }
foreach (var language in json_obj.languages)
{
// not sure how i can access the object-name
/*if(!language.StartsWith("ja_"))
continue;*/
// Exception: 'Newtonsoft.Json.Linq.JProperty' does not contain a definition for 'data' - Not sure why it is treated as a property?
var name = language.data.name;
var file = language.files[2];
}
I'm sorry for this probably dumb question, but i've been trying to cast it to different types and searched the web for solutions, but i just couldn't figure it out. So if someone could help me out with this i would be really greatful.
Thanks in advance!

Since you're stating in a comment (on an answer that has been deleted) that the data changes so a fixed model won't work, you can still fix what is known:
Here's a LINQPad program that demonstrates:
void Main()
{
var collection = JsonConvert.DeserializeObject<LanguagesCollection>(File.ReadAllText(#"c:\temp\test.json"));
foreach (var keyValuePair in collection.Languages)
if (keyValuePair.Key.StartsWith("ja_"))
keyValuePair.Value.Dump();
}
public class LanguagesCollection
{
public Dictionary<string, JObject> Languages { get; } = new Dictionary<string, JObject>();
}
This will deserialize the outer object, with the "languages" key, and inside you have a dictionary with the keys, "ja_lang", "en_lang", and you can just process the values as you see fit. These are left as JObject which means they will contain whatever json was present as a value for that key in the dictionary.

Using a site like json2sharp you can just pass your json data in and get a ready to use c# model out.
Then you can easily deserialize your json data into that c# model and use the properties for much easier handling:
string jsonData = #"{
'languages': {
'ja_lang': {
'data': {
'name': 'Japanese'
},
'files': [['ja',
'Japanese File',
'lang_ja.txt']]
},
'en_lang': {
'data': {
'name': 'English'
},
'files': [['en',
'English File',
'lang_en.txt']]
}
}
}";
RootObject data = JsonConvert.DeserializeObject<RootObject>(jsonData);
foreach(Languages lang in data.languages) //would work if Languages was a listing
{
}
Although I admit that your Json is a bit strange and that Languages most likly should be a listing and not a property for each language.

JSON.net Serialize With Namespaces

If I have json that looks something like this: (wrote this by hand, so it may have errors)
{
"http://devserver.somesite.com/someendpoint/1/2/3$metadata#Something.Start": [
{
"Title": "start",
"Endpoint": "https://devserver.somesite.com/someendpoint/1/2/3/start"
}
],
"http://devserver.somesite.com/someendpoint/1/2/3$metadata#Something.Stop": [
{
"Title": "stop",
"Endpoint": "https:// devserver.somesite.com/someendpoint/1/2/3/stop"
}
]
}
Is there any easy, built in way (JSON.net) to have it understand that there’s a namespace in play here? Or is there a way to set a variable or pattern based JsonProperty via an attribute?
I can't have the URL as part of my business object, because that will change from environment to environment.
I know I can create a custom json converter, but before going down that route I’d like to see if there’s something more out of box that handles this. Another option is to get the data via xml and handle that by hand.

Assuming you are taking this as a string that you have received from a web call you can do the following in JSON.NET.
var json = "your string here";
var obj = JObject.Parse(json);
foreach(var ns in obj.Properties) {
var arr = (JArray)ns.Value;
foreach(var obj2 in arr) {
// do you logic here to get the properties
}
}
Another option that James Newton-King provided you can do this, which seems a little cleaner:
var list = JsonConvert.DeserializeObject<Dictionary<string, List<MyClass>>>(json);