I'm having a bit of trouble loading a JSON response from this request into a Deedle DataFrame:https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?where=OBJECTID%3C10&returnGeometry=false&f=json
In the JSON, what I'm interested are the features. More specifically, for each feature there are attributes - I want essentially just a collection of those attributes to load into a DataFrame. In this particular case, there is only one attribute "name" so my expectation is that the resulting DataFrame would have a column "name" with the values shown.
I've tried using json2csharp and creating my own class, but the result either doesn't have the column header/values or the values are missing. I'm not really sure what I'm doing wrong or if I'm even approaching this the right way. My understanding from the Deedle documentation is that it should be possible to create a DataFrame from a collection of objects: https://bluemountaincapital.github.io/Deedle/csharpframe.html#Creating-and-loading-data-frames. Certainly, using the Enumerable example listed on the page works as expected.
Here is the pertinent section of my code:
string url = "https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?";
WebClient wc = new WebClient();
wc.QueryString.Add("where", "OBJECTID<10");
wc.QueryString.Add("returnGeometry", "false");
wc.QueryString.Add("f", "json");
var data = wc.UploadValues(url, "POST", wc.QueryString);
var responseString = UnicodeEncoding.UTF8.GetString(data);
JObject o = JObject.Parse(responseString);
dynamic x = JsonConvert.DeserializeObject(responseString);
var testObjList = new List<dynamic>();
foreach (dynamic element in x.features)
{
testObjList.Add(new myClass { name = element.attributes.name});
Console.WriteLine($"{element.attributes.name}");
}
var dfObjects = Frame.FromRecords(testObjList);
dfObjects.Print();
var df = Frame.FromRecords(test);
df.Print(); // No headers or values shown
where myClass is just this:
public class myClass{
public string name { get; set; }
}
Any help/pointers would be much appreciated!
The Frame.FromRecords operation relies on static type information to figure out what properties the class has. In your case, you define the list of objects as List<dynamic> - this is compiled as Object and so Deedle does not see any members.
To fix this, all you need to do is to define the type as a list of myClass objects:
var testObjList = new List<myClass>();
A more compact approach using an anonymous type would work too:
var testObjList =
((IEnumerable<dynamic>)x.features).Select(element =>
new { name = element.attributes.name });
var dfObjects = Frame.FromRecords(testObjList);
Related
Is there a more efficient way of converting dynamo db data into concrete types? For example, when I query the data everything is in:
List<Dictionary<string, AttributeValue>>
Is it possible to easily convert the type without having to loop through each item and doing this all manually?
For example I am doing:
return items.Select(item => new Connection
{
ConnectionId = Guid.Parse(item["connectionId"].S),
ClientId = item["clientId"].S,
ProviderId = item["providerId"].S,
Scopes = item["scopes"].SS.ToArray(),
CredentialsId = item["credentialsId"].S,
Evidences = ToEvidences(item["consentEvidences"].L)
})
.ToList();
This then returns a list of my type Connection however I am explicitly mapping each field. Is there an easier way or a helper library that can do the mapping?
I think you'll have luck with the higher-level .NET Document model. It presents more natural data types.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DotNetSDKMidLevel.html
The easiest way I have found is to use the Document.FromAttributeMap function to convert it to a Document object and then convert it again to the .NET type using the DynamoDBContext.FromDocument method as shown below.
public async Task<IEnumerable<WeatherForecast>> GetAll(string cityName)
{
var queryRequest = new QueryRequest()
{
TableName = nameof(WeatherForecast),
KeyConditionExpression = "CityName = :cityName",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":cityName", new AttributeValue(cityName)},
}
};
var response = await _dynamoDbClient.QueryAsync(queryRequest);
return response.Items.Select(a =>
{
var doc = Document.FromAttributeMap(a);
return _dynamoDbContext.FromDocument<WeatherForecast>(doc);
});
}
I'm having trouble converting JSON data to a dynamic type definition. I've looked at JObject, JsonConvert serialize/deserialize and nothing works. The closest thing to making this work is JsonConvert.DeserializeAnonymousType but this requires a definition that matches the JSON. In my case, the JSON is quite complicated so I really need to convert an instance of the JSON to a fully anonymous, dynamic type.
dynamic rawjson = #"{ 'tags': { 'abcd' : '12345' },'properties': { 'desired': { 'PropOne' : '2345', 'PropTwo' : '6789' } } }";
#region reference object
dynamic reference = new
{
SomeName = $"xxxx",
initialTwin = new
{
tags = new { abcd = 12345 },
properties = new
{
desired = new
{
PropOne = "2345",
PropTwo = "6789"
}
}
}
};
#endregion
dynamic anonobject = JsonConvert.DeserializeAnonymousType(rawjson, reference.initialTwin);
dynamic testobject = new
{
SomeName = $"xxxx",
initialTwin = $"{anonobject}"
};
I need the "testobject" in the code above to look exactly like the "reference" object. Using the DesializeAnonymousType gets me very close but the definition would be very hard to create and maintain.
How can I get the same results working from an instance of JSON data without typing the definition for DeserializeAnonyousType? Is that possible?
I have tested the way below and it worked for me:
var anonobject = JsonConvert.DeserializeObject<ExpandoObject>(rawjson);
dynamic testobject = new
{
SomeName = $"xxxx",
initialTwin = anonobject
};
But since this is a dynamic object, you need to know the properties you may want to use.
Example:
Console.WriteLine(testobject.initialTwin.tags.abcd);
The following code sample writes a simple object to a couchbase lite (version 2) database and reads all objects afterwards. This is what you can find in the official documentation here
This is quite a lot of manual typing since every property of every object must be transferred to the MutableObject.
class Program
{
static void Main(string[] args)
{
Couchbase.Lite.Support.NetDesktop.Activate();
const string DbName = "MyDb";
var db = new Database(DbName);
var item = new Item { Name = "test", Value = 5 };
// Serialization HERE
var doc = new MutableDocument();
doc.SetString("Name", item.Name);
doc.SetInt("Value", item.Value);
db.Save(doc);
using (var qry = QueryBuilder.Select(SelectResult.All())
.From(DataSource.Database(db)))
{
foreach (var result in qry.Execute())
{
var resultItem = new Item
{
// Deserialization HERE
Name = result[DbName].Dictionary.GetString("Name"),
Value = result[DbName].Dictionary.GetInt("Value")
};
Console.WriteLine(resultItem.Name);
}
}
Console.ReadKey();
}
class Item
{
public string Name { get; set; }
public int Value { get; set; }
}
}
From my research Couchbase lite uses JsonConvert internally, so there might be a way to simplify all that with the help of JsonConvert.
Anything like:
var json = JsonConvert.SerializeObject(item);
var doc = new MutableDocument(json); // No overload to provide raw JSON
or maybe
var data = JsonConvert.SerializeToDict(item); // JsonConvert does not provide this
var doc = new MutableDocument(data);
Is there or is this some kind of optimization and the preferred approach is by intend?
People ask about this quite often, but Couchbase Lite does not actually store JSON strings in the database. They are stored in a different format so this would not give the benefit that you think (the JSON would need to be reparsed and then broken down into the other format). I'd been pushing for a way to serialize classes directly instead of going through dictionary objects (which seems like the ultimate goal here) but our priority is on things that enterprise clients want and this doesn't seem to be one of them. Note that for it to make it in, it needs to be implemented in C# Java and Objective-C / Swift.
I don't know about JsonConvert but there seems to be a constructor that takes IDictionary<string, object> as argument. So I would try something like this (brain-compiled):
MutableDocument CreateDocument(object data)
{
if (data == null) return null;
var propertyValues = new Dictionary<string, object>();
foreach (var property in data.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
propertyValues[property.Name] = property.GetValue(data);
}
return new MutableDocument(propertyValues);
}
See if this works.
how can I access in this JSON (http://www.pegelonline.wsv.de/webservices/rest-api/v2/stations.json?includeTimeseries=true&includeCurrentMeasurement=true) the nested array like
timeseries.shortname? I tried like this but it doesn't work.
string url = "http://www.pegelonline.wsv.de/webservices/rest-api/v2/stations.json?includeTimeseries=true&includeCurrentMeasurement=true";
HttpWebRequest request = HttpWebRequest.CreateHttp(url);
WebResponse response = await request.GetResponseAsync();
using (Stream stream = response.GetResponseStream())
{
JsonReader reader = new JsonTextReader(new StreamReader(stream));
dynamic info = JArray.Load(reader);
foreach (var item in info)
{
myModel.Add(new ItemModel()
{
uuid = item.uuid,
number = item.number,
city_longname = item.longname,
timeseries = item.timeseries.shortname
});
}
}
The 3 items works, but the last (timeseries) gives the following error: Cannot perform runtime binding on a null reference
The dynamic properties give you JToken objects. Using the Value property on those gives you the string representation. In order to get it type-safe you need to parse/convert. Since you did not provide your ItemModel class details I cannot help you here.
myModel.Add(new ItemModel()
{
uuid = item.uuid.Value,
number = item.number.Value,
city_longname = item.longname.Value
});
The timeseries property is a JArray object. You cannot get to the shortname property directly. You have to choose an index first (item.timeseries[5], for instance, give you the JObject instance you are after). The details regarding getting the actual values in a type-safe manner from above apply here as well.
I am trying to deserialize an instance of the following class from a JSON string using JavaScriptSerializer:
public class Filter
{
public HashSet<int> DataSources { get; set; }
}
Here is the code I am trying out:
Filter f = new Filter();
f.DataSources = new HashSet<int>(){1,2};
string json = (new JavaScriptSerializer()).Serialize(f);
var g= (new JavaScriptSerializer()).Deserialize<Filter>(json);
It errors out with the following message:
Object of type 'System.Collections.Generic.List1[System.Int32]'
cannot be converted to type
'System.Collections.Generic.HashSet1[System.Int32]'.
Apparently, the serializer is unable to distinguish between a list and set from JSON representation. What is the solution to this?
Note : I would prefer avoiding the use of external libraries due to constraints at work.
What is the solution to this?
Use Json.Net. This code works...
Filter f = new Filter();
f.DataSources = new HashSet<int>() { 1, 2 };
string json = JsonConvert.SerializeObject(f);
var g = JsonConvert.DeserializeObject<Filter>(json);
EDIT
DataContractJsonSerializer seems to work too...
DataContractJsonSerializer dcjs = new DataContractJsonSerializer(typeof(Filter));
var g2 = dcjs.ReadObject(new MemoryStream(Encoding.UTF8.GetBytes(json))) as Filter;
Here's my simple NOT so great solution, but it works.
var dataList = new JavaScriptSerializer().Deserialize<List<int>>(returnData);
var data = new HashSet<int>(dataList);