We Keep Coding

Convert Nested JSON to CSV in C# via ChoETL

Does anyone know how to convert the below nested JSON to CSV via CHOETL (An ETL framework for .NET)? Thank you!
I'm using this code but it will only return the first equipment record.
CODE:
{
using (var json = new ChoJSONReader("./test.json"))
{
csv.Write(json.Cast<dynamic>().Select(i => new
{
EquipmentId = i.GpsLocation.Equipment[0].EquipmentId,
InquiryValue = i.GpsLocation.Equipment[0].InquiryValue,
Timestamp = i.GpsLocation.Equipment[0].Timestamp
}));
}
}
JSON:
"GpsLocation": {
"Equipment": [
{
"EquipmentId": "EQ00001",
"InquiryValue": [
"IV00001"
],
"Timestamp": "2020-01-01 01:01:01.01",
},
{
"EquipmentId": "EQ00002",
"InquiryValue": [
"IV00002"
],
"Timestamp": "2020-01-01 01:01:01.01"
}
]
}
}````

As others suggest, the issue is you are only looking at the first element of the array.
It appears that the easiest way to control what you serialise into CSV is by correctly defining your source objects from JSON. JSON Path expressions come in pretty handy.
What I ended up doing here is query all JSON to return an array of Equipment objects regardless of where they are in the hierarchy (which means you may need to filter it a bit better depending on your full JSON).
Then it's pretty easy to define each field based on JSON path and just pass the result to CSVWriter.
Also check out some gotchas that I outlined in the respective comment lines.
void Main()
{
var jsonString = "{\"GpsLocation\":{\"Equipment\":[{\"EquipmentId\":\"EQ00001\",\"InquiryValue\":[\"IV00001\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"},{\"EquipmentId\":\"EQ00002\",\"InquiryValue\":[\"IV00002\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"}]}}";
var jsonReader = new StringReader(jsonString);
var csvWriter = new StringWriter(); // outputs to string, comment out if you want file output
//var csvWriter = new StreamWriter(".\\your_output.csv"); // writes to a file of your choice
using (var csv = new ChoCSVWriter(csvWriter))
using (var json = new ChoJSONReader(jsonReader)
.WithJSONPath("$..Equipment[*]", true) // firstly you scope the reader to all Equipment objects. take note of the second parameter. Apparently you need to pass true here as otherwise it just won't return anythig
.WithField("EquipmentId", jsonPath: "$.EquipmentId", isArray: false) // then you scope each field in the array to what you want it to be. Since you want scalar values, pass `isArray: false` for better predictability
.WithField("InquiryValue", jsonPath: "$.InquiryValue[0]", isArray: false) // since your InquiryValue is actually an array, you want to obtain first element here. if you don't do this, fields names and values would go askew
.WithField("Timestamp", jsonPath: "$.Timestamp", fieldType: typeof(DateTime), isArray: false)) // you can also supply field type, otherwise it seems to default to `string`
{
csv.WithFirstLineHeader().Write(json);
}
Console.WriteLine(csvWriter.GetStringBuilder().ToString()); // comment this out if writing to file - you won't need it
}
Update summary:
Pivoted to update the code to rely on JSON Path scoping - this seems to allow for field name manipulation with pretty low effort
Looking at your comment, you could probably simplify your file writer a little bit - use StreamWriter instead of StringWriter - see updated code for example

Here is the working sample of producing CSV from your JSON
string json = #"{
""GpsLocation"": {
""Equipment"": [
{
""EquipmentId"": ""EQ00001"",
""InquiryValue"": [
""IV00001""
],
""Timestamp"": ""2020-02-01 01:01:01.01"",
},
{
""EquipmentId"": ""EQ00002"",
""InquiryValue"": [
""IV00002""
],
""Timestamp"": ""2020-01-01 01:01:01.01""
}
]
}
}";
StringBuilder csv = new StringBuilder();
using (var r = ChoJSONReader.LoadText(json)
.WithJSONPath("$.GpsLocation.Equipment")
.WithField("EquipmentId")
.WithField("InquiryValue", jsonPath: "InquiryValue[0]", fieldType: typeof(string))
.WithField("Timestamp", fieldType: typeof(DateTime))
)
{
using (var w = new ChoCSVWriter(csv)
.WithFirstLineHeader())
w.Write(r);
}
Console.WriteLine(csv.ToString());
Output:
EquipmentId,InquiryValue,Timestamp
EQ00001,IV00001,2/1/2020 1:01:01 AM
EQ00002,IV00002,1/1/2020 1:01:01 AM
Sample fiddle: https://dotnetfiddle.net/hJWtqH

Your code is sound, but the issue is that you're only writing the first variable in the array by using i.GpsLocation.Equipment[0]. Instead, try looping over everything by putting it into a for loop, and changing the [0] to your iterating variable inside of said loop.

How to parse json value that is stored in an array and store selected value in array in c#?

I read json values from a text and store it in array using this code.
string[] allLines = System.IO.File.ReadAllLines("D:\\tweets.txt");
I need to extract certain fields from this array containing Json.
My Json is of this type:
{"Name":"John","Id":"45","Time":"11 pm"}
{"Name":"Pear","Id":"34","Time":"3 pm"}
I want to extract each "Name" in one array and each "Id" in one array, something like this.
string[] Name= null;
string[] Id= null;
for (var i = 0; i < allLines[i].length; i++)
{
Name = allLines[i].Name;
Id = allLines[i].Id;
}
I tried another way using json parsing as well. I can obtain one row at one time json deserialized this way. But then confused how to obtain the selected fields.
StreamReader streamReader = System.IO.File.OpenText("D:\\tweets.txt");
string lineContent = streamReader.ReadLine();
do
{
if (lineContent != null)
{
var a = JsonConvert.DeserializeObject(lineContent);
}
lineContent = streamReader.ReadLine();
}
while (streamReader.Peek() != -1);
streamReader.Close();
Please help.

I recommend using Json.NET to parse JSON, you can download it as a NuGet package.
It has some great documentation about querying JSON here
Querying your JSON with LINQ would look something like:
JObject json = JObject.Parse(json);
var name = from p in json
select (string)p["Name"];

This example uses the NewtonSoft Json library to deserialize your Json into an object. Then, linq is used to pull out the lists that you are interested in.
I have written this example as a Console Application in Visual Studio. You will need to create a new Console Application, then copy this code into it.
Also, to use the NewtonSoft library in your new Console Application, you will need to load it from NuGet. To do this in VisualStudio, you will need to
Right-click on the project name
Click on Manage NuGet Packages...
In the search box on the top right, enter "newtonsoft" (without the quotes)
Newtonsoft.Json should show up in the list. Click it and press the Install button. This will load the binary and set up the references in your project. After that, you can use the sample code shown in this example.
static void Main(string[] args)
{
TestParseJson();
Console.WriteLine();
Console.WriteLine("Press Any Key to End");
Console.ReadLine(); // Wait for input so we can see our output text
}
// 1 - Construct an object used for deserialization. You will need to make this class match the format of the records in your json text file.
public class User
{
public string Name { get; set; }
public int Id { get; set; }
public DateTime Time { get; set; }
}
// 2 - Simulate Json creation and then use the NewtonSoft library to deserialize. You will need to just extract from the DeserializeObject line down
public static void TestParseJson()
{
// Read list of json objects from file, one line at a time. The json in the test file users.json is in this format:
// {"Name":"John","Id":45,"Time":"2015-11-05T19:18:02.0324468Z"}
// {"Name":"Pear","Id":34,"Time":"2015-11-06T19:18:02.0329474Z"}
var jsonLines = File.ReadLines(#"c:\temp\users.json");
var deserializedUsers = jsonLines.Select(JsonConvert.DeserializeObject<User>).ToList();
// Use Linq to project the list of deserializedUsers into the lists that you want
var names = deserializedUsers.Select(user => user.Name);
var ids = deserializedUsers.Select(user => user.Id);
// Output list of names and ids for debugging purposes
Console.WriteLine("");
Console.WriteLine(" Names:");
foreach (var name in names)
{
Console.WriteLine(" " + name);
}
Console.WriteLine(" Ids:");
foreach (var id in ids)
{
Console.WriteLine(" " + id);
}
}

Your JSON does not actually represent an array, but rather a series of individual objects back-to-back. To be considered a valid JSON array, the objects would need to be enclosed in square brackets and separated by commas (see JSON.org). Regardless, it is still possible to read and parse this non-standard JSON using Json.Net. The JsonTextReader class has a special SupportMultipleContent setting that is designed to cope with this situation. You can process your file using the following code:
List<string> names = new List<string>();
List<string> ids = new List<string>();
JsonSerializer serializer = new JsonSerializer();
using (StreamReader sr = new StreamReader("D:\\tweets.txt"))
using (JsonTextReader reader = new JsonTextReader(sr))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
JObject jo = JObject.Load(reader);
names.Add((string)jo["Name"]);
ids.Add((string)jo["Id"]);
}
}
}
Console.WriteLine("Names: " + string.Join(", ", names));
Console.WriteLine("Ids: " + string.Join(", ", ids));
Fiddle: https://dotnetfiddle.net/tYVLLr

Parsing JSON to Controller in MVC

I'm parsing a json which has 3 properties that have arrays as values, to my controller. I managed to save the json to a Dictionary, and now i need those 3 arrays to make a query to my database.
The setup is the following
public JsonResult FilterMultiselects(string json)
{
Dictionary<string, string[]> filters = JsonConvert.DeserializeObject<Dictionary<string, string[]>>(json);
string[] lines = filters["lines"];
string[] countries = filters["countries"];
string[] users = filters["users"];
var query = ... //do my query which returns the results i desire
string result = JsonConvert.SerializeObject(query, Formatting.Indented,
new JsonSerializerSettings {
PreserveReferencesHandling = PreserveReferencesHandling.Objects
});
return Json(result, JsonRequestBehavior.AllowGet);
}
The problem im having is that when i get to the part in which i initialize those 3 arrays only the second one(countries) exists. The other two dont even create which makes no sense to me. Even thought i have not worked with Dictionaries(hashes) too much in C# it looks good.
Could anyone help me with what am i missing?
The json im parsing has the following look:
{
lines: ["line 1", "line 2"]
countries: ["England"]
users: []
}
The string my controller receives is:
"{\"lines\":[\"line 1\",\"line 2\"],\"countries\":[\"England\"],\"users\":[]}"
The dictionary gets created and has 3 keys that have the correct arrays as values.

Parsing JSON in ASP.NET

I have the following JSOn I need to parse:
{"items":[{"dict":"es","words":[{"word":"car","id":"3487"},{"word":"dog","id":"443"},{"word":"plane","id":"1171"}]},{"dict":"fr","words":[{"word":"house","id":"134"}]}]}
Using JavaScriptSerializer, how could I iterate first through each dict and then retrieve the id of each word?

make anonymouse type, acording your json, for example:
var result = new[] {new {action = "", value = false}}; // put your item structure here
var list = JsonConvert.DeserializeObject(myJson, result.GetType());
then you might want to itterate through. For example:
foreach (dynamic val in ((dynamic) list)) { ...

Parsing large json

I'm trying to parse a json file and then operate on it to insert data into a SQL Server 2008 Database.
Example:
var sr = new StreamReader("C:\\path to file\file.json");
var json = JsonSerializer.SerializeToString(sr);
var o = JsonObject.Parse(json);
But I always get this error at the second line - "Timeouts are not supported on this stream."
The Json file looks like this:
"main":{
"prg": [
{
"Id": 1,
"name": "A&E",
more fields
}
"prg": [
{
"Id": 2,
"name": "asda",
more fields
}
}
I need to make something like this
foreach (prg in main)
entity.id = prg.id
entity.name = prg.name
How can I do this and why I get that timeout exception?
EDIT: To better understand my question this is how I do for an XML file
XmlDocument sourceDoc = new XmlDocument();
sourceDoc.Load(SourcesElement2); // where SourcesElement2 is the path to my XML
XmlNodeList prg = sourceDoc.GetElementsByTagName("prg");
foreach (XmlNode item in prg)
{
entity.Name= item.SelectSingleNode("name").InnerText;
...
}
I have converted the XML to Json and I want to do same thing. For every "prg" node in the Json File insert a new item in the database
EDIT2:
This is what I've done.
using (
StreamReader stream =
File.OpenText(
"C:\\path\\Sources.json")
)
{
JObject sources = (JObject) JToken.ReadFrom(new JsonTextReader(stream));
var a = sources["on"];
var b = a["sources"];
var c = b["prgs"];
foreach (var item in c)
{
var d= item.SelectToken("prg");
// Here d is null
}
I have the same question as the one from above. For every "prg" node in the Json File insert a new item in the database. How can I do this ? ( path to prg is on/sources/prgs/ )

I don't think you want to serialize the stream.
JsonSerializer.SerializeToString(sr)
You want to deserialize from the stream.
JsonSerializer.Deserialize
You might want to use JsonReader for performance reasons.
Your XML example load the whole file in the memory - you don't want to do that for large documents. reader.Read() pattern is better suited for processing large files.

For everyone with the same problem here is what I've done ( NOTE: this is just an example )
By the way, thank you for everyone who tried to answer my question and I'm sorry for my mistakes.
List<string> d = new List<string>();
using (
StreamReader stream =
File.OpenText(
"C:\\path\\Sources.json")
)
{
JObject sources = (JObject) JToken.ReadFrom(new JsonTextReader(stream));
var a = sources["on"];
var b = a["sources"];
var c = b["prgs"];
foreach (JObject item in c["prg"].ToList())
{
d.Add(item.Value<string>("name"));
}
}
//part below is just for testing
foreach (var VARIABLE in d)
{
Console.WriteLine(VARIABLE);
}
Console.ReadLine();

I would approach it by converting that JSON object into a C# class and then applying logic to the C# object and/or use DataTables
[Note: There are solutions online that show you would to easily pass an Object or List<Object> into a DataTable and then pass it "easily" to SQL]
The first step is still your hiccup in either solution, how do I pull in a large JSON string from filesystem?
if you have the JSON, use json2csharp.com and/or jsonutils.com to retrieve the classes in order to Deserialize it to your object.
StreamReader re = new StreamReader(#"C:\path to file\file.json");
JsonTextReader reader = new JsonTextReader(re);
YourClass DeserializedObject = se.Deserialize<YourClass>(reader);
Console.WriteLine(DeserializeObject.SomeProperty);