Does anyone know how to convert the below nested JSON to CSV via CHOETL (An ETL framework for .NET)? Thank you!
I'm using this code but it will only return the first equipment record.
CODE:
{
using (var json = new ChoJSONReader("./test.json"))
{
csv.Write(json.Cast<dynamic>().Select(i => new
{
EquipmentId = i.GpsLocation.Equipment[0].EquipmentId,
InquiryValue = i.GpsLocation.Equipment[0].InquiryValue,
Timestamp = i.GpsLocation.Equipment[0].Timestamp
}));
}
}
JSON:
"GpsLocation": {
"Equipment": [
{
"EquipmentId": "EQ00001",
"InquiryValue": [
"IV00001"
],
"Timestamp": "2020-01-01 01:01:01.01",
},
{
"EquipmentId": "EQ00002",
"InquiryValue": [
"IV00002"
],
"Timestamp": "2020-01-01 01:01:01.01"
}
]
}
}````
As others suggest, the issue is you are only looking at the first element of the array.
It appears that the easiest way to control what you serialise into CSV is by correctly defining your source objects from JSON. JSON Path expressions come in pretty handy.
What I ended up doing here is query all JSON to return an array of Equipment objects regardless of where they are in the hierarchy (which means you may need to filter it a bit better depending on your full JSON).
Then it's pretty easy to define each field based on JSON path and just pass the result to CSVWriter.
Also check out some gotchas that I outlined in the respective comment lines.
void Main()
{
var jsonString = "{\"GpsLocation\":{\"Equipment\":[{\"EquipmentId\":\"EQ00001\",\"InquiryValue\":[\"IV00001\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"},{\"EquipmentId\":\"EQ00002\",\"InquiryValue\":[\"IV00002\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"}]}}";
var jsonReader = new StringReader(jsonString);
var csvWriter = new StringWriter(); // outputs to string, comment out if you want file output
//var csvWriter = new StreamWriter(".\\your_output.csv"); // writes to a file of your choice
using (var csv = new ChoCSVWriter(csvWriter))
using (var json = new ChoJSONReader(jsonReader)
.WithJSONPath("$..Equipment[*]", true) // firstly you scope the reader to all Equipment objects. take note of the second parameter. Apparently you need to pass true here as otherwise it just won't return anythig
.WithField("EquipmentId", jsonPath: "$.EquipmentId", isArray: false) // then you scope each field in the array to what you want it to be. Since you want scalar values, pass `isArray: false` for better predictability
.WithField("InquiryValue", jsonPath: "$.InquiryValue[0]", isArray: false) // since your InquiryValue is actually an array, you want to obtain first element here. if you don't do this, fields names and values would go askew
.WithField("Timestamp", jsonPath: "$.Timestamp", fieldType: typeof(DateTime), isArray: false)) // you can also supply field type, otherwise it seems to default to `string`
{
csv.WithFirstLineHeader().Write(json);
}
Console.WriteLine(csvWriter.GetStringBuilder().ToString()); // comment this out if writing to file - you won't need it
}
Update summary:
Pivoted to update the code to rely on JSON Path scoping - this seems to allow for field name manipulation with pretty low effort
Looking at your comment, you could probably simplify your file writer a little bit - use StreamWriter instead of StringWriter - see updated code for example
Here is the working sample of producing CSV from your JSON
string json = #"{
""GpsLocation"": {
""Equipment"": [
{
""EquipmentId"": ""EQ00001"",
""InquiryValue"": [
""IV00001""
],
""Timestamp"": ""2020-02-01 01:01:01.01"",
},
{
""EquipmentId"": ""EQ00002"",
""InquiryValue"": [
""IV00002""
],
""Timestamp"": ""2020-01-01 01:01:01.01""
}
]
}
}";
StringBuilder csv = new StringBuilder();
using (var r = ChoJSONReader.LoadText(json)
.WithJSONPath("$.GpsLocation.Equipment")
.WithField("EquipmentId")
.WithField("InquiryValue", jsonPath: "InquiryValue[0]", fieldType: typeof(string))
.WithField("Timestamp", fieldType: typeof(DateTime))
)
{
using (var w = new ChoCSVWriter(csv)
.WithFirstLineHeader())
w.Write(r);
}
Console.WriteLine(csv.ToString());
Output:
EquipmentId,InquiryValue,Timestamp
EQ00001,IV00001,2/1/2020 1:01:01 AM
EQ00002,IV00002,1/1/2020 1:01:01 AM
Sample fiddle: https://dotnetfiddle.net/hJWtqH
Your code is sound, but the issue is that you're only writing the first variable in the array by using i.GpsLocation.Equipment[0]. Instead, try looping over everything by putting it into a for loop, and changing the [0] to your iterating variable inside of said loop.
Related
I have a Discord slash command that works like a Dokkan simulator, the user creates their own passive and they can generate some stats based on their input.
My main issue is trying to read the JSON file where I store all the passives and I want to search for the "PassiveName" and see if it matches what the user entered.
Once it's confirmed that a correct match has been found I will then execute my function where I generate the Stats
So far this is what I have when im trying to read the JSON file, my approach is that i read the JSON and store all its members in a list and then search the list for the correct passive name that the user entered
private void ReadJSONFile(string nameSearch)
{
List<object> Passives = new List<object>(); //List to temporarily store everything from the JSON
try
{
using (StreamReader sr = new StreamReader("UserPassivesStorage.json"))
{
string jsonFile = sr.ReadToEnd();
JSONObject100 jsonObject = JsonConvert.DeserializeObject<JSONObject100>(jsonFile);
Passives.Add(jsonObject);
for (int i = 0; i < jsonObject.members.Length; i++)
{
if (i < jsonObject.members.Length)
{
var data = jsonObject.members[i];
Passives.Add(data);
}
else
{
Console.Write("Error");
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}
This current code reads the JSON and successfully puts it into the list, but I can't even read from the list to compare the user input to whatever is in storage
This is my JSON file, I have 2 entries in it as an example
{
"userPassiveStorage": "Storage for user created passives using /passivecreator",
"members": [
{
"UserName": "𝕤𝕒𝕞.𝕛𝕖𝕤𝕦𝕤𝟠",
"PassiveName": "TEQ Gogeta",
"UnitHP": 19199,
"UnitATK": 20744,
"UnitDEF": 10909,
"UnitLeaderName": "Fusion",
"UnitLeaderSkill": 170,
"UnitPassiveATK": 200,
"UnitPassiveDEF": 200,
"Support": 0,
"Links": "No Links Provided by the user"
},
{
"UserName": "𝕤𝕒𝕞.𝕛𝕖𝕤𝕦𝕤𝟠",
"PassiveName": "AGL Gogeta",
"UnitHP": 15745,
"UnitATK": 16492,
"UnitDEF": 11088,
"UnitLeaderName": "Movie Heroes",
"UnitLeaderSkill": 170,
"UnitPassiveATK": 100,
"UnitPassiveDEF": 100,
"Support": 0,
"Links": "No Links Provided by the user"
}
]
}
So what I want to figure out is how can i read this JSON file and search it for the "PassiveName" property so I can access what is stored in it and if it's correct, I load the whole member in to do the stat calculations later on. (For example user enters "TEQ Gogeta", i want it to match to the TEQ gogeta in the JSON)
I'd recommend using LINQ to avoid all the trouble with for loops and checking array lengths,etc. To find all members, whose PassiveName matches the search name you can use Where method:
var matchingMembers = jsonObject.members.Where(member => member.PassiveName == nameSearch);
and LINQ does the job returning an IEnumerable. If you want a list out of it:
var membersList = matchingMembers.ToList();
You can replace member.PassiveName == nameSearch condition to anything like StartsWith or Contains or add further processing.
Of course your JSONObject100 class must have all the needed properties to map to the Json object.
Note, the above requires importing Linq namespace. See docs for more details. You should also add null checks to cover cases where json is missing some parts.
EDIT: If for some reasons you'd prefer to stick to your original approach, what you're missing is mostly the comparison of the pattern with the values youre iterating:
for (int i = 0; i < jsonObject.members.Length; i++)
{
var member = jsonObject.members[i];
if(member.PassiveName == nameSearch)
return member;
}
You don't need to check condition i < jsonObject.members.Length again as for loop does it for you. Even better than for would be foreach here:
foreach(var member in jsonObject.members) {...}
If you can have more than 1 result, you should add them to a list and then return a list, or print, or whatever is the purpose.
for some legacy code, I need to return a list of rows than come from a list of the element, but, the client needs it without the property name.
This is what I need. (I know that inside this JSON return, it's an array list then it's not a valid JSON, but is the way that needs to be received by the front end I don't know why
{ "success":true,"msg":"","draw":"1","recordsTotal":14,"recordsFiltered":14,
{ "data":
[
["Xiaomi","Redmi 8","10","6.0.1","","sadsadasd"],
["Xiaomi","Redmi 8","10","6.0.1","","sadsadasd"],
]
}
This is what I got now
{"success":true,"msg":"","draw":"1","recordsTotal":14,"recordsFiltered":14,
"data" :[
{"Brand":"Xiaomi","Model":"Redmi 8","Version":10,"App":"4.3.3","Phone":"","AsignedUSer":"ceererer"},
{"Brand":"Xiaomi","Model":"Redmi 8","Version":10,"App":"4.3.3","Phone":"","AsignedUSer":"ceererer"},
]
}
And this is the code how I am retrieving the data
return new JsonResult(new { success = true, msg = "", draw ="1", recordsTotal = listToReturn.Count(), recordsFiltered = listToReturn.Count() ,
data = listToReturn ,
});
You could project your list into arrays to extract the property values:
var data = listToReturn
.Select(d => new string[]{d.Brand, d.Model, d.Version, etc..})
.ToList();
Note that a "JSON" content without the property names will be a not valid JSON document.
What you need seems to be more like a CSV-like response.
Have you tried to format the result as CSV?
I have been searching and searching for a way to convert a json file to a csv and the vice versa using C#. I have searched google and have not come up with anything. Everything I've tried so far from the answers on stack overflow just do not work from me. Does anyone know of any tooling or tutorials I could have look at how to accomplish this with the .NET Framework? Usually I post what I've tried however I'm clearly far off here so it is pointless.
Compromises and Problems
You can accomplish this with the .NET Framework but there's not a clear and obvious way to just do this straight-up because of hierarchies and collections. What I mean by that is that CSV data is very flat and unstructured whereas JSON data is very organized and iterative. Let's take a simple chunk of JSON data that could look like this:
{
"Data": [
{
"Name":"Mickey Mouse",
"Friends":[ "Pluto", "Minnie", "Donald" ]
},
{
"Name":"Pluto",
"Friends":[ "Mickey" ]
}
]
}
The most obvious CSV file for that could be:
Name,Friend
Mickey Mouse,Pluto
Mickey Mouse,Minnie
Mickey Mouse,Donald
Pluto,Mickey
That's the easier conversion but let's say you just have that CSV file. It's not so obvious what the JSON should look like. One could argue that the JSON should look like this:
{
"Data": [
{ "Name":"Mickey Mouse", "Friend":"Pluto" },
{ "Name":"Mickey Mouse", "Friend":"Minnie" },
{ "Name":"Mickey Mouse", "Friend":"Donald" },
{ "Name":"Pluto", "Friend":"Mickey" },
]
}
That resulting JSON file is very different than the input JSON file. My point is that this isn't a simple/obvious conversion so any off-the-shelf or copy/paste solution will be imperfect. Whatever your solution is, you're going to have to make compromises or intelligent decisions.
.NET Framework Options
Now that we've gotten that out of the way, .NET gives you some capabilities to accomplish this out of the box and there are some good Nuget-supplied options as well. If you want to utilize pure .NET capabilities, you could use a combination of these two SO Answers:
Not perfect but this answer has some great code to get you started in the logic to generate a CSV file
This question and the resulting answers have some good info about generating JSON using just the .NET Framework and without any third-party utilities.
You should be able to apply the concepts in those two links PLUS the compromises and intelligent decisions you need to make from my first "Compromises and Problems" section of this post to accomplish what you need.
Something I've Done Before
I've done something similar where I actually used some functionality in the Microsoft.VisualBasic.FileIO namespace (works great in a C# app) in addition to Web API's serialization functionality to accomplish a CSV->JSON conversion using Dynamic objects (using the dynamic keyword) as an intermediary. The code is provided below. It's not terribly robust and makes some significant compromises but it has worked well for me. If you want to try this, you'll have to create your own version that goes in reverse, but as I mentioned in my first section, that's really the easy part.
using System.Collections.Generic;
using System.Dynamic;
using System.IO;
using System.Linq;
using System.Web.Http;
// NOTE: This is not purely my code. This was put together
// with the help of other SO questions that I wish I had the
// links to so I could credit them. You probably will find
// some chunk(s) of this code elsewhere on SO.
namespace Application1.Controllers
{
public class Foo
{
public string Csv { get; set; }
}
public class JsonController : ApiController
{
[HttpPost]
[Route("~/Csv/ToJson")]
public dynamic[] ConvertCsv([FromBody] Foo input)
{
var data = CsvToDynamicData(input.Csv);
return data.ToArray();
}
internal static List<dynamic> CsvToDynamicData(string csv)
{
var headers = new List<string>();
var dataRows = new List<dynamic>();
using (TextReader reader = new StringReader(csv))
{
using (var parser = new Microsoft.VisualBasic.FileIO.TextFieldParser(reader))
{
parser.Delimiters = new[] {","};
parser.HasFieldsEnclosedInQuotes = true;
parser.TrimWhiteSpace = true;
var rowIdx = 0;
while (!parser.EndOfData)
{
var colIdx = 0;
dynamic rowData = new ExpandoObject();
var rowDataAsDictionary = (IDictionary<string, object>) rowData;
foreach (var field in parser.ReadFields().AsEnumerable())
{
if (rowIdx == 0)
{
// header
headers.Add(field.Replace("\\", "_").Replace("/", "_").Replace(",", "_"));
}
else
{
if (field == "null" || field == "NULL")
{
rowDataAsDictionary.Add(headers[colIdx], null);
}
else
{
rowDataAsDictionary.Add(headers[colIdx], field);
}
}
colIdx++;
}
if (rowDataAsDictionary.Keys.Any())
{
dataRows.Add(rowData);
}
rowIdx++;
}
}
}
return dataRows;
}
}
}
If you want something more robust, then you can always leverage these great projects:
JSON.NET (This works VERY WELL with creating JSON from dynamic objects. Given that you're not using Web API, this would be the first place I would look to take the dynamic[] return value and convert it to JSON.)
CsvHelper
Besides using combination of multiple libraries to do the conversion of JSON to CSV and vice versa, Cinchoo ETL gives you unified interface to do the conversion between those 2 formats.
For a sample JSON file:
[
{
"Name" : "Xytrex Co.",
"Description" : "Industrial Cleaning Supply Company",
"AccountNumber" : "ABC15797531"
},
{
"Name" : "Watson and Powell, Inc.",
"Description" : "Law firm. New York Headquarters",
"AccountNumber" : "ABC24689753"
}
]
To produce CSV file:
Name,Description,AccountNumber
Xytrex Co.,Industrial Cleaning Supply Company,ABC15797531
Watson and Powell Inc.,Law firm. New York Headquarters,ABC24689753
JSON to CSV:
using (var p = ChoJSONReader.LoadText(json))
{
using (var w = new ChoCSVWriter(Console.Out)
.WithFirstLineHeader()
)
{
w.Write(p);
}
}
Sample fiddle: https://dotnetfiddle.net/T3u4W2
CSV to JSON:
using (var p = ChoCSVReader.LoadText(csv)
.WithFirstLineHeader()
)
{
using (var w = new ChoJSONWriter(Console.Out))
{
w.Write(p);
}
}
Sample fiddle: https://dotnetfiddle.net/gVlJVX
like Jaxidian mentioned, the problem is, that json can have a hierarchy, csv not.
So, there are two solutions I could suggest you:
create a hierarchical csv, shouldn't be much effort:
"Id";"Name";"Age";"Type"
"FriendId"
1;"Mickey Mouse";20;"mouse"
2
3
4
2;"Pluto";7;"dog"
1
3;"Minnie";20;"mouse"
4;"Donald";22;"duck"
create multiple files, could be more effort, but is more beautiful and more dynamic, when you eg. export from/import into database. Maybe this link could help you: http://www.snellman.net/blog/archive/2016-01-12-json-to-multicsv
all.csv (store all characters)
"Id";"Name";"Age";"Type"
1;"Mickey Mouse";20;"mouse"
2;"Pluto";7;"dog"
3;"Minnie";20;"mouse"
4;"Donald";22;"duck"
friends.csv (store all relations)
"FriendKey1";"FriendKey2"
1;2
2;1
1;3
1;4
I am trying to read child elements in the Json object below. Below is the Json example. I want to read RecommendedCount and TotalReviewCount in testdata1, testdata2 and testdata3.
{
"HasErrors": false,
"Includes": {
"test ": {
"testdata1": {
"ReviewStatistics": {
"RecommendedCount": 0,
"TotalReviewCount": 2
}
},
"testdata2": {
"ReviewStatistics": {
"RecommendedCount": 0,
"TotalReviewCount": 2
}
},
"testdata3": {
"ReviewStatistics": {
"RecommendedCount": 0,
"TotalReviewCount": 2
}
}
}
}
}
I tried the code below.
RecommendedCount = apiResponse.Includes.Products[key].ReviewStatistics.RecommendedCount,
TotalReviewCount = apiResponse.Includes.Products[key].ReviewStatistics.TotalReviewCount
But this is very slow as the Json response has more than 1000 lines so it is taking time. I want to know is there any linq i can use to find the relevant data or any other methods i can use?
Thanks in advance.
var jObj = (JObject)JsonConvert.DeserializeObject(rawJson);
foreach (var child in jObj["test"].Children())
{
}
The above is the deserialize code i am trying to use but getting Object reference not set to an instance of an object. error
My solution:
JObject obj = JObject.Parse(jsonString);
var recList= obj.SelectTokens("$..ReviewStatistics.RecommendedCount").ToList();
var totalList= obj.SelectTokens("$..ReviewStatistics.TotalReviewCount").ToList();
Then you can get the data that you want. For example, if you want RecommendedCount from testdata2, you do like this
var dataYouWant = (int)recList[1];
References:
http://www.newtonsoft.com/json/help/html/LINQtoJSON.htm
Deserializing JSON to .NET object using Newtonsoft (or LINQ to JSON maybe?)
https://msdn.microsoft.com/en-us/library/cc197957(v=vs.95).aspx
If there is anything wrong, please feel free to correct my answer. Thanks!
I suspect the problem is in your Json you have a space in the field "test " which is causing a null reference exception.
I'm not entirely sure whether is it copy paste mistake or your getting Json string in this format.
Check this working code on removing the space.
I'm having trouble converting a string of json facebook graph api, I used the facebook C# and json.Net.
But at conversion time it returns this error: Name can not begin with the '0 'character, hexadecimal value 0x30.
This is the code:
dynamic result = await _fb.GetTaskAsync ("me / feed");
FBxml JsonConvert.DeserializeXNode string = (result.ToString ()). ToString ();
It looks like there is a problem with portion of the json string as mentioned below (taken from your link http://jsfiddle.net/btripoloni/PaLC2/)
"story_tags": {
"0": [{
"id": "100000866891334",
"name": "Bruno Tripoloni",
"offset": 0,
"length": 15,
"type": "user"}]
},
Json cannot create class that begins with a numeric value such as '0'. Try creating the classes using the link http://json2csharp.com/ you will get an idea.
To solve this problem you can create a dynamic object and go through each properties OR create a JsonConverter and write your code in the ReadJson to convert the "0" to a meaningful name. May be this can help you http://blog.maskalik.com/asp-net/json-net-implement-custom-serialization
If this is not your problem then update the question with more information like class structure of FBxml, call stack of the exception (from which line of the json code is throwing the exception), Json version etc.
As keyr says, the problem is with those JSON properties that have numeric names. In XML names can contain numeric characters but cannot begin with one: XML (see the Well-formedness and error-handling section).
My idea was to recursively parse the JSON with JSON.Net, replacing properties that had numeric names:
var jsonObject = JObject.Parse(json);
foreach (var obj in jsonObject)
{
Process(obj.Value);
}
XDocument document = JsonConvert.DeserializeXNode(jsonObject.ToString());
....
private static void Process(JToken jToken)
{
if (jToken.Type == JTokenType.Property)
{
JProperty property = jToken as JProperty;
int value;
if (int.TryParse(property.Name, out value))
{
JToken token = new JProperty("_" + property.Name, property.Value);
jToken.Replace(token);
}
}
if (jToken.HasValues)
{
//foreach won't work here as the call to jToken.Replace(token) above
//results in the collection modifed error.
for(int i = 0; i < jToken.Values().Count(); i++)
{
JToken obj = jToken.Values().ElementAt(i);
Process(obj);
}
}
}
This seemed to work well, prefixing numeric names with _. At this line:
XDocument document = JsonConvert.DeserializeXNode(jsonObject.ToString());
it crashed with an error saying that invalid/not well formed XML had been created. I don't have the actual error with me, but you can run the above code to replicate it.
I think from here you may need to revisit converting the JSON to XML in the first place. Is this a specific requirement?