Deserialization and Exporting of Nested JSON API Data to CSV - c#

I have requirement to take some data from an API from nested JSON and export it to CSV. I can't find the right syntax for taking that data to export it to CSV without creating multiple classes (there are hundreds of properties, and new ones can be added dynamically).
The JSON response structure from the API is below:
{
"data": [
{
"id": "1",
"type": "Bus",
"attributes": {
"property-two": "2020-12-10",
"property-three": "D",
"property-four": null,
"property-five": 5
}
},
{
"id": "2",
"type": "Car",
"attributes": {
"property-two": "2020-12-10",
"property-three": "D",
"property-four": null,
"property-five": 5
}
}
]
}
We only need to export the "attributes" node from each dataset to CSV, but cannot seem to flatten the data or extract just those nodes.
The following code gets a list of JToken objects, but I'm not sure how this can be exported to CSV without re-serializing and de-serializing again. The datatype is dynamic since columns can be added and removed from it.
var jsonObject = JObject.Parse(apiResponseString);
var items = jsonObject["data"].Children()["attributes"].ToList();
//TODO: Export to CSV, DataTable etc
Is it possible to deserialize the data from the attributes nodes only, and how would that be done (whether on the initial JObject.Parse or serializing the JToken list again)? I'm not tied to JObject, so can use Newtonsoft or other as well.

Using Json.Net's LINQ-to-JSON API (JObjects), you can convert your JSON data to CSV as shown below.
First define a couple of short extension methods:
using System;
using System.Collections.Generic;
using System.Linq;
using Newtonsoft.Json.Linq;
public static class JsonHelper
{
public static string ToCsv(this IEnumerable<JObject> items, bool includeHeaders = true)
{
if (!items.Any()) return string.Empty;
var rows = new List<string>();
if (includeHeaders)
{
rows.Add(items.First().Properties().Select(p => p.Name).ToCsv());
}
rows.AddRange(items.Select(jo =>
jo.Properties().Select(p => p.Value.Type == JTokenType.Null ? null : p.Value).ToCsv()
));
return string.Join(Environment.NewLine, rows);
}
public static string ToCsv(this IEnumerable<object> values)
{
const string quote = "\"";
const string doubleQuote = "\"\"";
return string.Join(",", values.Select(v =>
v != null ? string.Concat(quote, v.ToString().Replace(quote, doubleQuote), quote) : string.Empty
));
}
}
Then you can do:
var obj = JObject.Parse(json);
var csv = obj.SelectTokens("..attributes").Cast<JObject>().ToCsv();
Here is a working demo: https://dotnetfiddle.net/NF2G2l

I ended up creating an extension method in my Json Helper class to convert the JToken enumerable to a datatable. This meets the original requirements to export to CSV simply (we use Aspose Cells that has a method for DatatTable to CSV), but also allows us to work with the datatable as an object without defining the columns. The end result looks like this:
Worker Class:
var stringResponse = await apiResponse.Content.ReadAsStringAsync();
var jsonObject = JObject.Parse(stringResponse);
var dataRows = jsonObject.SelectTokens("$..attributes").ToList();
var outputData = new DataTable("MyDataTable");
dataRows.AddToDataTable(outputData, pageNumber);
//TODO: Do something with outputData (to file, to object, to DB etc)
JsonHelper Class new extension method
public static void AddToDataTable(this List<JToken> jTokens, DataTable dt, int pageNumber)
{
foreach (var token in jTokens)
{
JObject item;
JToken jtoken;
if (pageNumber == 1 && dt.Rows.Count == 0)
{
item = (JObject)token;
jtoken = item.First;
while (jtoken != null)
{
dt.Columns.Add(new DataColumn(((JProperty)jtoken).Name));
jtoken = jtoken.Next;
}
}
item = (JObject)token;
jtoken = item.First;
var dr = dt.NewRow();
while (jtoken != null)
{
dr[((JProperty)jtoken).Name] = ((JProperty)jtoken).Value.ToString();
jtoken = jtoken.Next;
}
dt.Rows.Add(dr);
}
}

Related

Replace ID from json entity with random number without creating entity class

I have testdata.json with object like this:
"Entity": {
"ID": "1",
"name": "some name",
"City": "some city",
"address": "some address"
}
Here are my 2 methods for withdrawing and deserializing this entity:
public static string ReadSettings(string name)
{
var parts = name.Split('.', StringSplitOptions.RemoveEmptyEntries);
JObject jObj = GetObject();
foreach (var part in parts)
{
var token = jObj[part];
if (token == null)
{
return null;
}
else
{
return token.ToString();
}
}
return null;
}
private static JObject GetObject()
{
if (_jObject != null)
{
return _jObject;
}
var filename = Path.Combine(
Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)!,
"testdata.json");
var json = File.ReadAllText(filename);
_jObject = JObject.Parse(json);
return _jObject;
}
My question is next: is there any way I can replace Entity.ID with random number (I need this in tests, where I create new Entity - every time test runs - there should be new Entity with new unique ID)?
p.s. I'm just learning C# so don't judge me hardly if it's simple question =)
I suppose the given testdata.json content is not the real thing since running JObject.Parse() on it gives an exception. However, you can navigate to a property with the string key indexer of the JObject class and change it as:
// Generate a new random value.
var random = new Random();
var randomValue = random.Next(minValue, maxValue);
// Navigate to the property and assign the random value as a string.
_jObject["Entity"]["ID"] = randomValue.ToString();
The property navigation of course needs to match the structure of your Json.

How to output JSON array as a single field in CSV using ChoETL

I'm using ChoETL to convert JSON to CSV. Currently, if a property in the JSON object is an array it is output into separate fields in JSON.
Example:
{
"id", 1234,
"states": [
"PA",
"VA"
]
},
{
"id", 1235,
"states": [
"CA",
"DE",
"MD"
]
},
This results in CSV like this (using pipe as a delimeter)
"id"|"states_0"|"states_1"|"states_2"
"1234"|"PA"|"VA"
"1235"|"CA"|"DE"|"MD"
What I would like is for the array to be displayed in a single states field as a comma separated string
"id"|"states"
"1234"|"PA,VA"
"1235"|"CA,DE,MD"
Here is the code I have in place to perform the parsing and transformation.
public static class JsonCsvConverter
{
public static string ConvertJsonToCsv(string json)
{
var csvData = new StringBuilder();
using (var jsonReader = ChoJSONReader.LoadText(json))
{
using (var csvWriter = new ChoCSVWriter(csvData).WithFirstLineHeader())
{
csvWriter.WithMaxScanRows(1000);
csvWriter.Configuration.Delimiter = "|";
csvWriter.Configuration.QuoteAllFields = true;
csvWriter.Write(jsonReader);
}
}
return csvData.ToString();
}
}
Edited: Removed test code that wasn't useful
This is how you can produce the expected output using the code below
var csvData = new StringBuilder();
using (var jsonReader = ChoJSONReader.LoadText(json))
{
using (var csvWriter = new ChoCSVWriter(csvData)
.WithFirstLineHeader()
.WithDelimiter("|")
.QuoteAllFields()
.Configure(c => c.UseNestedKeyFormat = false)
.WithField("id")
.WithField("states", m => m.ValueConverter(o => String.Join(",", ((Array)o).OfType<string>())))
)
{
csvWriter.Write(jsonReader);
}
}
Console.WriteLine(csvData.ToString());
Output:
id|states
"1234"|"PA,VA"
"1235"|"CA,DE,MD"
PS: on the next release, this issue will be handled automatically without using valueconverters

JObject in C# for independent data fetching of JSON

I am using Json.Net to parse my JSON
This is my JSON:
"OptionType": {
"C": [
"C",
"Call"
],
"P": [
"P",
"Put"
]
}
Before this step, when processed, as a result, I would get a value from any of this.
For example Option Type: Call
Whatever value I get, I need it to be transcodified according to the above JSON.
Example: Option Type: C
First of all your sample data is not a valid JSON. You need to wrap it to the curvy braces.
If your sample is a part of the JSON object, you can map OptionType to the Dictionary<string, List<string>>.
To find valid option you will need to iterate this dictionary like in the sample below:
var valueToCheck = "Call";
string type = null;
foreach (var kvp in optionTypes)
{
if (kvp.Value.Contains(valueToCheck))
{
type = kvp.Key;
break;
}
}
Same using JObject with fixed JSON data:
var json = #"{
""OptionType"":{
""C"": [
""C"",
""Call""
],
""P"": [
""P"",
""Put""
]
}
}";
var valueToCheck = "Call";
string type = null;
var ot = JObject.Parse(json);
var objectType = ot["OptionType"];
foreach (var token in objectType)
{
var prop = token.ToObject<JProperty>();
var key = prop.Name;
var values = prop.Value.ToObject<List<string>>();
if (values.Contains(valueToCheck))
{
type = key;
break;
}
}
Code is not perfect but it is just an idea what to do.
You need to iterate over properties of JObject and then search your option type and then replace your search option with its parent key.
This is custom function can do above task.
public static JObject GetJObject(JObject jObject, List<string> optionType)
{
foreach (var type in jObject["OptionType"])
{
var key = type.ToObject<JProperty>().Name;
var values = type.ToObject<JProperty>().Value.ToObject<List<string>>();
foreach (var option in optionType)
{
if (values.Contains(option))
{
int index = values.IndexOf(option);
values.RemoveAt(index);
values.Insert(index, key);
}
}
JToken jToken = JToken.FromObject(values);
jObject.SelectToken("OptionType." + key).Replace(jToken);
}
return jObject;
}
And you can use above custom function this like
string json = File.ReadAllText(#"Path to your json file");
JObject jObject = JObject.Parse(json);
List<string> optionType = new List<string> { "Call", "Put" };
JObject result = GetJObject(jObject, optionType);
Output:

How to get keys and values of .json file which is generated by parser dynamically in c#

We have following scenario,
We need to parse resumes of candidates and i am using below parser to parse resume in Json format, right.
https://github.com/antonydeepak/ResumeParser
and Json files which i get are in valid format as i checked them in online jsonviewer, but it is cleared that each resume is in different format. so each time parser introduced new pair of Keys ,
Example 1.
Example 2.
Above two formats are of two different resumes and so on...
so now i need to iterate through every key and value that are dynamically generated.
as for as i did to get JObject and JArray at level 0, now i need to iterate through each JObject and JArray to get its values.
I used Json.net to get them
string text = System.IO.File.ReadAllText(#"C:\abc.json");
var d = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(text);
and it showed me data as
abc.json has JObjects and JArray, so now i need to iterate through each and every line and need to get every key and value from parsed json file and load it to datatable and tried it using google , but it missed some of keys and values.
System.Data.DataTable dt = new System.Data.DataTable();
dt.Columns.Add("Key", typeof(string));
dt.Columns.Add("Value", typeof(string));
string text = System.IO.File.ReadAllText(#"C:\antony_thomas.json");
var d = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(text);
foreach (var item in d)
{
var key = "";
var val = item.Value;
if (val is JObject)
{
dynamic dynObj = JsonConvert.DeserializeObject(Convert.ToString(val));
foreach (var ite in dynObj)
{
DataRow row = dt.NewRow();
string jsonvalue = Convert.ToString(ite).Replace("[", "").Replace("]", "").Replace("{", "").Replace("}", "").Replace("\"", "");
string jkey = jsonvalue.Split(':')[0];
string jval = jsonvalue.Split(':')[1];
row["Key"] = jkey;
row["Value"] = jval;
dt.Rows.Add(row);
}
// key = item.Key;
}
if (val is JArray)
{
//key = item.Key;
foreach (var it in val)
{
// var newkey=
DataRow row = dt.NewRow();
string jsonvalue = Convert.ToString(it).Replace("[", "").Replace("]", "").Replace("{", "").Replace("}", "").Replace("\"", "");
//Convert.ToString(test);
string jkey = jsonvalue.Split(':')[0];
string jval = jsonvalue.Split(':')[1];
row["Key"] = jkey;
row["Value"] = jval;
dt.Rows.Add(row);
}
}
}
I am using asp.net , C#, Json.net , if anyone have any idea so please guide me how can i get my desired result..
I used this code recently to traverse an arbitrary JSON string.
Each element is dumped with a row of dots at the beginning indicating its level in the hierarchy.
You could modify it to output to a DataTable as you walk through an JArray for example.
public static void JsonFileDump(string path)
{
//Parse the data
string jsonStr = File.ReadAllText(path);
JToken token = JToken.Parse(jsonStr); // get parent token
JsonTokenDump(token);
}
public static void JsonTokenDump(JToken node, int lvl = 0, string nodeName = null)
{
if (nodeName != null)
Console.WriteLine("{0}Node Name={1}, Type={2}", new string('.', lvl), nodeName, node.Type);
else
Console.WriteLine("{0}Node Type={1}", new string('.', lvl), node.Type);
if (node.Type == JTokenType.Object)
{
foreach (JProperty child in node.Children<JProperty>())
{
JsonTokenDump(child.Value, lvl + 1, child.Name);
}
}
else if (node.Type == JTokenType.Array)
{
foreach (JToken child in node.Children())
{
JsonTokenDump(child, lvl + 1);
}
}
}

Write a JObject from a JArray using the object's Key-Pair values in C#

I have a JArray which looks as follows:
[
{
"key": "S8710 Server",
"value": "M"
},
{
"key": "Java",
"value": "M"
}
]
This JArray needs to be converted into a JObject by taking the key and value such that the output object looks like this:
{
"S8710 Server": "M",
"Java": "M"
}
Is this conversion Possible? Any help would be greatly appreciated.
What I tried is extracting the keys and values from a DataTable and serializing the result of that. Then I tried JObject.Parse.
var skillList = from skill in ds.Tables[4].AsEnumerable()
where skill.Field<Int64>("ResumeID") == applicantValue.ResumeID
select new clsResume.ExtractRatingInfo
{
key = skill.Field<string>("Skill"),
value = skill.Field<string>("SkillRating")
};
string skillResult = JsonConvert.SerializeObject(skillList, Newtonsoft.Json.Formatting.None);
JObject obj = JObject.Parse(skillResult);
Yes, you can transform your JSON like this:
JObject result = new JObject(
jArray.Select(jt => new JProperty((string)jt["key"], jt["value"]))
);
Fiddle: https://dotnetfiddle.net/CcLj3D
Note: the keys must be distinct across all objects in the array for this to work properly.
Create JToken by passing the content string into JArray`s static method Parse.
get the inner childs from JToken and then look for the properties and their values.
like this:
JArray jArray = JArray.Parse(YOUR_CONTENT_GOES_HERE);
foreach (var token in jArray.Children<JToken>())
{
var firstProp = token.Children<JProperty>().First(param => param.Name == "S8710 Server");
var firstValue = firstProp.Value<string>();
var secondProp = token.Children<JProperty>().First(param => param.Name == "Java");
var secondValue = firstProp.Value<string>();
}

Categories