I have a formatted log file I'm trying to parse; the file is divided in sections with an header and the data inside each section is formatted with JSON like follows. Link to an extract of the log file here
[UnityCrossThreadLogger]1/8/2019 7:49:19 PM
==> Deck.GetDeckLists(112):
{
"jsonrpc": "2.0",
"method": "Deck.GetDeckLists",
"params": {},
"id": "112"
}
My issue here is manipulating the whole string in a way I get to the section I want and there strip the meaningless data and parse the remaining through Newtonsoft JSON. For now I'm cutting everything I don't need using this function, since the log file is in chronological order and only the latest occurrence of the entry is needed:
//Cut the whole log to the last entry
private static string CutLog(string fromWhereToCut)
{
string log = GetLog();
//In this case fromWhereToCut would be "Deck.GetDeckLists"
string s = log.Substring(log.LastIndexOf(fromWhereToCut));
return s;
}
The problem is the fact it leaves the header in place I need to remove before deserializing the JSON and it's prone to breaking because the name of the sections aren't that unique and they could be repeated further down as non-header titles (as can be seen in my example). Furthermore I don't know how to stop at the end of the section I need before another one begins.
I thought RegEx could be used but this seems way to big even for a RegEx and maybe there's a better solution.
If the Log is the same as the one found in PasteBin, this deserializes fine.
I'm using a support class (JSON_Logs) to contain the extracted data.
The JSON is read from a file in this simulation.
Reading the structure of the data, the most probable candidate to identify the start of the actual data, is the recurring string "Deck.GetDeckLists". In the parsing method it's assigned to a variable called excludedSection.
The data starts right after the last one of those string. I'm using logFile.LastIndexOf(excludedSection) to find the index of the last of these entries, then use this index to identify the first data structure.
JsonConvert.DeserializeObject is then used to deserialize the data into a List of class objects.
I didn't find any problem during the deserialization process.
string searchString = "Deck.GetDeckLists";
List<JSON_Logs.Header> jsonLogs = ParseJsonLog(searchString, "JSON_Logs.txt");
private List<JSON_Logs.Header> ParseJsonLog(string excludedSection, string fileName)
{
string logFile = File.ReadAllText(fileName);
int refIndex = logFile.LastIndexOf(excludedSection);
logFile = logFile.Substring(logFile.IndexOf("[", refIndex));
return JsonConvert.DeserializeObject<List<JSON_Logs.Header>>(logFile);
}
Support class:
public class JSON_Logs
{
public class Header
{
public string id { get; set; }
public string name { get; set; }
public string description { get; set; }
public string format { get; set; }
public string resourceId { get; set; }
public int deckTileId { get; set; }
public MainDeck[] mainDeck { get; set; }
public object[] sideboard { get; set; }
public DateTime lastUpdated { get; set; }
public bool lockedForUse { get; set; }
public bool lockedForEdit { get; set; }
public bool isValid { get; set; }
}
public class MainDeck
{
public string id { get; set; }
public int quantity { get; set; }
}
}
I hope this is what you need. :) Actually, regex finds json in all sections, but I included getting only last section (matches[matches.Count - 1]). Since JToken doesn't have TryParse method, you have to use try/catch:
static void ParseLog()
{
var s = File.ReadAllText(#"C:\log.json");
var pattern =
#"(?s)(?'header'\[\w+\]\d{1,2}/\d{1,2}/\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}\s(A|P)M\r\n" +
#"<?==>?.+?\r\n)" +
#"(?'body'.+?)(?=$|\[\w+\]\d{1,2}/\d{1,2}/\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}\s(A|P)M)";
var matches = Regex.Matches(s, pattern);
if (matches.Count > 0)
{
JToken last_json = null;
try
{
var text = matches[matches.Count - 1].Groups["body"].Value;
last_json = JToken.Parse(text);
WriteLine(last_json.ToString());
}
catch (Exception ex) { WriteLine(ex.ToString()); }
}
else
{
WriteLine("No matches found");
}
}
Related
I want to get the price of any crypto coin from BitZ api.
I have the code like this:
string coinName;
string jsonURL = "https://apiv2.bitz.com/Market/coinRate?coins=" + coinName;
I will give the variable coinName the value I want for example coinName = "btc" and I want the price in USDT
The problem here is the Json structure it contains the coin name I will end up with tons of code lines if do this for every coin,
public class Btc
{
public string usdt { get; set; }
}
public class Data
{
public Btc btc { get; set; }
}
public class Root
{
public int status { get; set; }
public string msg { get; set; }
public Data data { get; set; }
public int time { get; set; }
public string microtime { get; set; }
public string source { get; set; }
}
Unlike Bittrex api for example which is easier to read using JsonDotNet asset from unity store and :
BittrexJsonUrl = "https://api.bittrex.com/api/v1.1/public/getticker?market=USDT-" + coinName;
and then I use this code to get the data:
private IEnumerator GetData()
{
/////bittrex
UnityWebRequest request = UnityWebRequest.Get(BittrexJsonUrl);
yield return request.SendWebRequest();
if (request.error == null)
{
Bittrex_proccessJsonData(request.downloadHandler.text);
}
else
{
Debug.Log("Something went wrong!!");
}
}
private void Bittrex_proccessJsonData (string _url) {
var _bittrexJsonData = JsonConvert.DeserializeObject<BittrexJsonData>(_url);
bittrexPrice = _bittrexJsonData.result.Last;
}
this works perfectly with with bittrex's Json structure, since it doesnt contain the coin name all I do is change the Json URL.
Now I want to do like the same thing for BitZ's if you have any idea how to please help :) thank you in advance.
For such thing you could use good old SimpleJson.
Here you don't need to implement the entire c# structure but rather access the data field by field via it's ID. You can imagine it like a nested Dictionary like thing.
Simply create that file with given content from the link somewhere in your project and do e.g.
var json = JSON.Parse(the_JSON_string);
var usdt = json["Data"]["bst"]["usdt"].AsFloat;
Currently I am working with the Shopify GraphQL Bulk Query.
This Query returns a JSON Lines file. Such a file may look like this:
{"id":"gid:\/\/shopify\/Product\/5860091625632","title":"Levis Jeans","description":"Cool Jeans","vendor":"Levis","status":"ACTIVE"}
{"id":"gid:\/\/shopify\/ProductImage\/20289865679008","__parentId":"gid:\/\/shopify\/Product\/5860091625632"}
{"id":"gid:\/\/shopify\/ProductVariant\/37178118963360","title":"32","position":1,"image":null,"selectedOptions":[{"name":"Size","value":"32"}],"inventoryItem":{},"__parentId":"gid:\/\/shopify\/Product\/5860091625632"}
{"available":10,"location":{"id":"gid:\/\/shopify\/Location\/57510625440"},"__parentId":"gid:\/\/shopify\/ProductVariant\/37178118963360"}
{"id":"gid:\/\/shopify\/ProductVariant\/37178118996128","title":"31","position":2,"image":null,"selectedOptions":[{"name":"Size","value":"31"}],"inventoryItem":{},"__parentId":"gid:\/\/shopify\/Product\/5860091625632"}
{"available":5,"location":{"id":"gid:\/\/shopify\/Location\/57510625440"},"__parentId":"gid:\/\/shopify\/ProductVariant\/37178118996128"}
{"available":3,"location":{"id":"gid:\/\/shopify\/Location\/57951518880"},"__parentId":"gid:\/\/shopify\/ProductVariant\/37178118996128"}
{"id":"gid:\/\/shopify\/ProductVariant\/37178119028896","title":"34","position":3,"image":null,"selectedOptions":[{"name":"Size","value":"34"}],"inventoryItem":{},"__parentId":"gid:\/\/shopify\/Product\/5860091625632"}
{"available":5,"location":{"id":"gid:\/\/shopify\/Location\/57510625440"},"__parentId":"gid:\/\/shopify\/ProductVariant\/37178119028896"}
{"available":15,"location":{"id":"gid:\/\/shopify\/Location\/57951518880"},"__parentId":"gid:\/\/shopify\/ProductVariant\/37178119028896"}
Each line of this file is a valid JSON-object and the lines are connected via __parentId with each other.
My Goal is to Deserialize this into C# Classes like this:
class Product
{
public string Id { get; set; }
public string Title { get; set; }
public string Description { get; set; }
public IEnumerable<ProductImage> Images { get; set; }
public IEnumerable<ProductVariant> Variants { get; set; }
}
class ProductImage
{
public string Id { get; set; }
}
class ProductVariant
{
public string Id { get; set; }
public IEnumerable<IDictionary<string, string>> SelectedOptions { get; set; }
public IEnumerable<InventoryLevel> Levels { get; set; }
}
class InventoryLevel
{
public int Available { get; set; }
}
And the output of a potential function performing the deserialization:
var file = new System.IO.StreamReader(#"c:\test.jsonl");
var products = DeserializeJsonL<IEnumerable<Product>>(file);
Shopify suggests to read the file in reverse. I get the Idea.
But I cannot imagine how to deserialize this file in a type safe way. How could I determine if the current line is a ProductVariant, a ProductImage or something else? I cannot influence the JSONL Output to include type information.
I am pretty sure without type information I cannot deserialize it safely. But how should I handle this data then to insert into a database for example?
EDIT the classname in {"id":"gid:\/\/shopify\/Product\/5860091625632"} cannot be used to determine the Type!
I ended up adding some sort of type information to my graphql-query by defining a unique fieldname for each type which may be on a new line in the resulting JSON Lines file.
For that i used GraphQL field aliases:
someQuery {
uniqueFieldAlias : fieldName
}
When i read the file i search on each line for the unique fieldname. Then i deserialize the line into the corresponding class.
using (var file = new StreamReader(await res.Content.ReadAsStreamAsync()))
{
string line;
while ((line = await file.ReadLineAsync()) != null)
{
if (line.Contains("\"uniqueFieldAlias\""))
{
var product = JsonSerializer.Deserialize<Product>(line);
products.Add(product);
continue;
}
if (line.Contains("\"otherUniqueAlias\""))
{
var somethingElse = JsonSerializer.Deserialize<SomeClass>(line);
products[productIndex].Something.Add(somethingElse);
continue;
}
}
}
The idea is inspired by #Caius Jard comments
Firstly thank you for taking the time to look at this. It's quite alot.
Question:
I'm basically trying to download a json as a string and then deserialize it to a list. The reason why is so i can then call a specific property of that list (in my case 'ips' because it's all i actually need) and insert it into a table if requirements are met.
The problem is that it moves all null values into the array. 114 columns of null, or empty array and i can't figure out why?
I think i'll attach a link to the JSON because its a massive file its here https://endpoints.office.com/endpoints/Worldwide?clientRequestId=b10c5ed1-bad1-445f-b386-b919946339a7
Here is my code:
Getters and setters for JSON
public class GetSetJsonIP {
[JsonProperty("id")]
public int id { get; set; }
[JsonProperty("serviceArea")]
public string ServiceArea { get; set; }
[JsonProperty("serviceAreaDisplayName")]
public string ServiceAreaDisplayName { get; set; }
[JsonProperty("urls")]
public IList<string> urls { get; set; }
[JsonProperty("ips")]
public IList<string> ips { get; set; }
[JsonProperty("tcpPorts")]
public string tcpPorts { get; set; }
[JsonProperty("expressRoute")]
public bool expressRoute { get; set; }
[JsonProperty("category")]
public string category { get; set; }
[JsonProperty("required")]
public bool required { get; set; }
[JsonProperty("notes")]
public string notes { get; set; }
[JsonProperty("udpPorts")]
public string udpPorts { get; set; }
}
List class
public class ConvertJsonIP{
public List<GetSetJsonIP> jsonIpConvert { get; set; }
public List<GetSetJsonIP> jsonIPConvert = new List<GetSetJsonIP>();
}
3.I download the JSON using an empty string called o365IP
o365IP = wc.DownloadString(wc.BaseAddress + "/endpoints/Worldwide?clientRequestId=b10c5ed1-bad1-445f-b386-b919946339a7");
I deserialize using my List to a seperate var
var o365IpVerion = JsonConvert.DeserializeObject<List<ConvertJsonIP>>(o365IP);
This code shows no errors. so i can only assume its a logical one on my part. It should be noted that i had to put the <List< in to stop an error stating that it couldnt convert an object to an array.
Seriously, i've been stuck on this for 3 days so any help on this would be greatly appreciated! Thanks in advance!
the json you have is a list of objects and each of these objects conform to GetSetJsonIp. You should deserialize using List<GetSetJsonIP>
var o365IpVerion = JsonConvert.DeserializeObject<List<GetSetJsonIP>>(o365IP);
public class GetJsonIP works fine.
The reason you must Deserialize into a List<> is because the json object starts with a bracket making the entire object a List or array.
var O365IpVersion = JsonConvert.DeserializeObject<List<GetJsonIP>(O365IP);
There are different ways to fetch the value of a certain property. If you just need ips and want to check the value then update it, then you could loop:
JArray arr = JArray.Parse(O365IP);
foreach (JObject obj in arr.Children<JObject>())
{
foreach (JPRoperty prop in obj.Properties().Where(x => x.Name == "ips"))
{
//use prop.Value and perform tasks
}
}
Or just simply loop like this:
for (int i = 0; i < O365IpVersion.Count; i++)
{
//use O365IpVersion.ElementAt(i).ips
I wrote a parse class trying to handle parsing the data from a string array into it's appropriate value. I am trying to test this program to see if it will print out the value parse.open, and it is not. It is printing up 0's for the moment (which isn't accurate), until i could figure out why it's not showing what I need.
while (!r.EndOfStream)
{
ParseFileRead parse = new ParseFileRead();
string line = r.ReadLine();
//Send this to Parse class
string [] values = line.Split(',');
//parse records
Console.WriteLine(values[6]); //This is printing the accurate value for parse.open
ParseFileRead.Parse(values);
Console.WriteLine(parse.open); //This is not printing the accurate value
}
Console.Read();
vWriteFile.Close();
And here is my ParseFileRead class:
public class ParseFileRead
{
public int open { get; set; }
public int buy { get; set; }
public int sell { get; set; }
public double settleMM { get; set; }
public string account { get; set; }
public string underlying { get; set; }
public string symbol { get; set; }
public static void Parse(string[] arr)
{
ParseFileRead parse = new ParseFileRead();
parse.account = arr[0];
parse.underlying = arr[12];
parse.symbol = arr[1];
parse.open = Convert.ToInt32(arr[6]);
parse.buy = Convert.ToInt32(arr[7]);
parse.sell = Convert.ToInt32(arr[8]);
parse.settleMM = Convert.ToDouble(arr[10]);
}
}
This is actually correct.
The default value for an uninitialized int is 0.
You are creating a new instance of your ParseFileRead class which will have a value of 0 for open. You then check your parsed value to make sure it's reading in correctly using Console.WriteLine(values[6]);.
Next, you try to parse your values using the Parse function of your ParseFileRead class; which is a void function so it has no return value.
Inside your Parse function you have: ParseFileRead parse = new ParseFileRead(); which creates yet another new instance of your class with a value of 0 for open. This particular instance is never used anywhere and is not the same as the values of the properties created with your initial instance of ParseFileRead
If you put a Console.Write in your Parse function, I'm sure that you will see it being parsed correctly.
So you have 2 options:
Set the properties of your ParseFileRead inside the Parse class without creating a new instance of ParseFileRead
Return the newly created ParseFileRead instance out of your Parse function.
Or a 3rd Option, which is probably best as suggested by Plutonix:
/*Parse class*/
public class ParseFileRead
{
public int open { get; set; }
public int buy { get; set; }
public int sell { get; set; }
public double settleMM { get; set; }
public string account { get; set; }
public string underlying { get; set; }
public string symbol { get; set; }
public ParseFileRead(string[] arr)
{
this.account = arr[0];
this.underlying = arr[12];
this.symbol = arr[1];
this.open = Convert.ToInt32(arr[6]);
this.buy = Convert.ToInt32(arr[7]);
this.sell = Convert.ToInt32(arr[8]);
this.settleMM = Convert.ToDouble(arr[10]);
}
}
/*Parsing code*/
while (!r.EndOfStream)
{
string line = r.ReadLine();
//Send this to Parse class
string [] values = line.Split(',');
//parse records
Console.WriteLine(values[6]); //This is printing the accurate value for parse.open
ParseFileRead parse = new ParseFileRead(values);
Console.WriteLine(parse.open); //This is not printing the accurate value
}
I'm trying to parse the following XML string that is being returned from a service.
DataReference.USZipSoapClient blah = new DataReference.USZipSoapClient("USZipSoap");
var results = blah.GetInfoByCity(tbCityName.Text).OuterXml;
returns the following
<NewDataSet xmlns=""><Table><CITY>Marana</CITY><STATE>AZ</STATE><ZIP>85653</ZIP><AREA_CODE>520</AREA_CODE><TIME_ZONE>M</TIME_ZONE></Table></NewDataSet>
I'm having no luck parsing the data:
I just want to display the results like City = Marana, State = AZ etc.
Why not just use XPath?
XmlDocument doc = new XmlDocument()
doc.LoadXml(results); // probably some try-catch here
var city = doc.SelectSingleNode("//CITY").InnerXml; //Handle null as well
You can create a serializing object and then serialize the data you get against that object, i used http://xmltocsharp.azurewebsites.net/ to generate the following xml object:
[XmlRoot(ElementName="Table")]
public class Table {
[XmlElement(ElementName="CITY")]
public string CITY { get; set; }
[XmlElement(ElementName="STATE")]
public string STATE { get; set; }
[XmlElement(ElementName="ZIP")]
public string ZIP { get; set; }
[XmlElement(ElementName="AREA_CODE")]
public string AREA_CODE { get; set; }
[XmlElement(ElementName="TIME_ZONE")]
public string TIME_ZONE { get; set; }
}
[XmlRoot(ElementName="NewDataSet")]
public class NewDataSet {
[XmlElement(ElementName="Table")]
public Table Table { get; set; }
[XmlAttribute(AttributeName="xmlns")]
public string Xmlns { get; set; }
}
Then just use the .net XML serializer to cast it to that object and use it in your code.
If you just need a string for visual inspection, a first step could be to convert the XMl to JSON with this code:
http://techhasnoboundary.blogspot.no/2011/08/convert-xml-to-json-using-c.html
Then you could go on by removing braces, replace comma with line break and colon with an equal sign.