We Keep Coding

C# Create Deedle DataFrame from JSON response

I'm having a bit of trouble loading a JSON response from this request into a Deedle DataFrame:https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?where=OBJECTID%3C10&returnGeometry=false&f=json
In the JSON, what I'm interested are the features. More specifically, for each feature there are attributes - I want essentially just a collection of those attributes to load into a DataFrame. In this particular case, there is only one attribute "name" so my expectation is that the resulting DataFrame would have a column "name" with the values shown.
I've tried using json2csharp and creating my own class, but the result either doesn't have the column header/values or the values are missing. I'm not really sure what I'm doing wrong or if I'm even approaching this the right way. My understanding from the Deedle documentation is that it should be possible to create a DataFrame from a collection of objects: https://bluemountaincapital.github.io/Deedle/csharpframe.html#Creating-and-loading-data-frames. Certainly, using the Enumerable example listed on the page works as expected.
Here is the pertinent section of my code:
string url = "https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?";
WebClient wc = new WebClient();
wc.QueryString.Add("where", "OBJECTID<10");
wc.QueryString.Add("returnGeometry", "false");
wc.QueryString.Add("f", "json");
var data = wc.UploadValues(url, "POST", wc.QueryString);
var responseString = UnicodeEncoding.UTF8.GetString(data);
JObject o = JObject.Parse(responseString);
dynamic x = JsonConvert.DeserializeObject(responseString);
var testObjList = new List<dynamic>();
foreach (dynamic element in x.features)
{
testObjList.Add(new myClass { name = element.attributes.name});
Console.WriteLine($"{element.attributes.name}");
}
var dfObjects = Frame.FromRecords(testObjList);
dfObjects.Print();
var df = Frame.FromRecords(test);
df.Print(); // No headers or values shown
where myClass is just this:
public class myClass{
public string name { get; set; }
}
Any help/pointers would be much appreciated!

The Frame.FromRecords operation relies on static type information to figure out what properties the class has. In your case, you define the list of objects as List<dynamic> - this is compiled as Object and so Deedle does not see any members.
To fix this, all you need to do is to define the type as a list of myClass objects:
var testObjList = new List<myClass>();
A more compact approach using an anonymous type would work too:
var testObjList =
((IEnumerable<dynamic>)x.features).Select(element =>
new { name = element.attributes.name });
var dfObjects = Frame.FromRecords(testObjList);

.NET API Can't Find Data I Want

I'm trying to save two variables out of a json request but i'm just trying to get the first one working this is my request:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://api.majestic.com/api/json?app_api_key=KEY&cmd=GetIndexItemInfo&items=1&item0=http://www.majestic.com&datasource=fresh");
{
WebResponse response = request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
JObject jObject = JObject.Parse(reader.ReadToEnd());
JToken Trusty = jObject["DataTables"]["Results"]["Data"][2];
var newdomain = new Identifier { domain = model.domain, contact = model.contact, contactname = model.contactname, price = model.price, type = model.type, TrustFlow = Int32.Parse(Trusty.ToString()), CitationFlow = 65, RI = model.RI, MJTopicsID = model.MJTopicsID, UserTableID = model.UserTableID };
ViewBag.newdomain = newdomain;
db.Identifiers.Add(newdomain);
This returns this error:
System.ArgumentOutOfRangeException: 'Index was out of range. Must be non-negative and less than the size of the collection.'
I've also tried
Token Trusty = jObject["DataTables"]["Results"]["Data"]["TrustFlow"][0];
this returns:
'Accessed JArray values with invalid key value: "TrustFlow". Int32 array index expected.'
This is the json I tried separating it myself as on the url it just came as one long line:
{
"Code":"OK","ErrorMessage":"","FullError":"","FirstBackLinkDate":"2017-08-17","IndexBuildDate":"2017-11-20 10:51:56","IndexType":1,"MostRecentBackLinkDate":"2017-11-18","QueriedRootDomains":0,"QueriedSubDomains":0,"QueriedURLs":1,"QueriedURLsMayExist":0,"ServerBuild":"2017-10-25 14:33:44","ServerName":"QUACKYO","ServerVersion":"1.0.6507.24412","UniqueIndexID":"20171120105156-FRESH",
"DataTables":{
"Results":{
"Headers":{
"MaxTopicsRootDomain":30,"MaxTopicsSubDomain":20,"MaxTopicsURL":10,"TopicsCount":3
},
"Data":[{
"RefDomainTypeProtocolHTTPS":"228","CitationFlow":42,"TrustFlow":29,"TrustMetric":29,"TopicalTrustFlow_Topic_0":"Health/Animal","TopicalTrustFlow_Value_0":26,"TopicalTrustFlow_Topic_1":"Business","TopicalTrustFlow_Value_1":25,"TopicalTrustFlow_Topic_2":"Computers/Internet/Domain Names","TopicalTrustFlow_Value_2":24
}
]}}}
What am I doing wrong? Thanks.

Your Data property is an array of size 1. Arrays are 0 index based. So you will access the first item as someArray[0] and second item as someArray[1] and so on
To read the int value stored inside the TrustFlow property of the first item in the Data array, you can do this
int trustFlow = jObject["DataTables"]["Results"]["Data"][0]["TrustFlow"].Value<int>();
This should work for the JSON Data you provided in the question. Keep in mind that this code expects the data to be in that structure . For example, if your Data array does not have any item, or your Results does not have a Data property, the code will crash ( probably with a null reference exception). You can add the null check yourself before trying to access the value as needed.

Newtonsoft Object serialized to String. JObject instance expected

Hi so am trying to parse this JSON line but i got some others that are like this in files thats why i want to automate this so i can remove the invalid lines to make the file a valid JSON for reading, The problem is that the JSON contains multiple JSON in 1 line
Example:
{"item":"value"}{"anotheritem":"value"}
Is there anyway to remove
{"anotheritem":"value"}
So it turns in to a valid JSON that is readable to start parsing the files
I tried doing using StreamReader cause there in a file i have multiple files that contain these invalid JSON
So i got it to be able to detect the Invalid JSON but for some reason i can't get it to read the JSON so i can use .remove to remove the invalid line
using (StreamReader r = new StreamReader(itemDir))
{
string json = r.ReadToEnd();
if (json.Contains("anotheritem"))
{
JObject NoGood = JObject.FromObject(json);
MessageBox.Show(NoGood.ToString());
}
}
The Error:
Object serialized to String. JObject instance expected.
Thank you all for your time and help.

If each object are side by side without space or any other character, you can convert your string to an json array.
string value = "{\"item\":\"value\"}{\"anotheritem\":\"value\"}";
string arrayValue = "[" + value.Replace("}{", "},{") + "]";
var array = JArray.Parse(arrayValue);
var goopArray = array.OfType<JObject>().Where(o => o.Property("anotheritem") == null);
Edit : see my second answer. More robust solution. More modern. And support dotnet core builtin json serializer.

Json.Net
Even better solution, Json.NET have a builtin feature for this exact scenario. See Read Multiple Fragments With JsonReader
The JsonTextReader have a property SupportMultipleContent that allow to read consecutive items when set to true
string value = "{\"item\":\"value\"}{\"anotheritem\":\"value\"}";
var reader = new JsonTextReader(new System.IO.StringReader(value));
reader.SupportMultipleContent = true;
var list = new List<JObject>();
while (reader.Read())
{
var item = JObject.Load(reader);
list.Add(item);
}
System.Text.Json
If you want to use System.Text.Json, it's also acheivable. They are no SupportMultipleContent property but Utf8JsonReader will do the job for you.
string value = "{\"item\":\"value\"}{\"anotheritem\":\"value\"}";
var bytes = Encoding.UTF8.GetBytes(value).AsSpan();
var list = new List<JsonDocument>();
while (bytes.Length != 0)
{
var reader = new Utf8JsonReader(bytes);
var item = JsonDocument.ParseValue(ref reader);
list.Add(item);
bytes = bytes.Slice((int) reader.BytesConsumed);
}

How to get Object values which have multiple rows in it

I am converting a JSON string into a object and this object contains multiple records of a table like a row structure. each row contains some set values. It's like an object within an object. I am trying to read the value of these objects but I am unsuccessful.
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(responseStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
//converting json data into array
JavaScriptSerializer ss = new JavaScriptSerializer();
object itm = ss.DeserializeObject(responseFromServer);
In this response stream object gives the json string with multiple values and I am converting this into object using javascriptserializer. At debugging mode I have seen that all the data is there in this object in multiple rows where each rows in itself is also a object .
These rows store information like offer_id, name as keys and there values. I want to write a loop to call this keys and its value to perform further action. But I am unable to call these data in the loop.

Usually the way I would use a JavaScriptSerializer would be to
//Serialise
JavaScriptSerializer js = new JavaScriptSerializer();
string str = js.Serialize(new Thing() { ID = "111" });
//Deserialise
Thing thing = js.Deserialize<Thing>(str);
however if you do not know the Type to convert back to you can
Dictionary<string, object> dic = (Dictionary<string, object>)js.DeserializeObject(str);
foreach (KeyValuePair<string, object> keyValue in dic)
{
keyValue.Key; <-- this is the property name (ID)
keyValue.Value; <-- this is the property value ("111")
}
and then just handle the key value pairs you receive.

How do I get properties in the array in this JSON string?

In the JSON string below, how do I access the values of the "at" and "current_value" properties within the "datastreams" array ?
In this example, there is only one datastream but in reality there could be many. I need to access the datastream by "id" property. Once I figure out this issue, I plan to use a where clause with the id == to the id of the desired datastream.
I tried using the approach discussed here, under "JSON in Windows 8 – A Simpler Approach" but it's not working.
In this code, json contains the JSON returned from the service I'm calling. prop is populated with a JsonArray. current results in an exception with an inner message of "JSON value not found"
var json = JsonObject.Parse(responseBodyAsText);
var prop = json.GetNamedArray("datastreams");
var current = from p in prop
select new
{
datastream = p.GetObject().GetNamedString("datastreams"),
datetime = p.GetObject().GetNamedString("at"),
value = p.GetObject().GetNamedString("current_value")
};
Here is the JSON string:
{
"title":"X",
"status":"X",
"creator":"X",
"datastreams":
[
{
"at":"x",
"max_value":"X",
"current_value":"X",
"id":"X",
"min_value":"X"
}
],
"location":{"exposure":"x","domain":"x","disposition":"x","lat":X,"lon":-X},
"created":"X",
"tags":["X"],
"feed":"X",
"private":"X",
"id":X,
"description":"X",
"version":"X",
"updated":"X"
}

datastream = p.GetObject().GetNamedString("datastreams")
The code above should return an array of objects. You'll need to loop through the array of objects to check the value of each object's "at" and "current_value" properties.

datastream return an array as you can notice by [ ] so:
datetime = datastream[0].at
value = datastream[0].current_value