We Keep Coding

C# Create Deedle DataFrame from JSON response

I'm having a bit of trouble loading a JSON response from this request into a Deedle DataFrame:https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?where=OBJECTID%3C10&returnGeometry=false&f=json
In the JSON, what I'm interested are the features. More specifically, for each feature there are attributes - I want essentially just a collection of those attributes to load into a DataFrame. In this particular case, there is only one attribute "name" so my expectation is that the resulting DataFrame would have a column "name" with the values shown.
I've tried using json2csharp and creating my own class, but the result either doesn't have the column header/values or the values are missing. I'm not really sure what I'm doing wrong or if I'm even approaching this the right way. My understanding from the Deedle documentation is that it should be possible to create a DataFrame from a collection of objects: https://bluemountaincapital.github.io/Deedle/csharpframe.html#Creating-and-loading-data-frames. Certainly, using the Enumerable example listed on the page works as expected.
Here is the pertinent section of my code:
string url = "https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?";
WebClient wc = new WebClient();
wc.QueryString.Add("where", "OBJECTID<10");
wc.QueryString.Add("returnGeometry", "false");
wc.QueryString.Add("f", "json");
var data = wc.UploadValues(url, "POST", wc.QueryString);
var responseString = UnicodeEncoding.UTF8.GetString(data);
JObject o = JObject.Parse(responseString);
dynamic x = JsonConvert.DeserializeObject(responseString);
var testObjList = new List<dynamic>();
foreach (dynamic element in x.features)
{
testObjList.Add(new myClass { name = element.attributes.name});
Console.WriteLine($"{element.attributes.name}");
}
var dfObjects = Frame.FromRecords(testObjList);
dfObjects.Print();
var df = Frame.FromRecords(test);
df.Print(); // No headers or values shown
where myClass is just this:
public class myClass{
public string name { get; set; }
}
Any help/pointers would be much appreciated!

The Frame.FromRecords operation relies on static type information to figure out what properties the class has. In your case, you define the list of objects as List<dynamic> - this is compiled as Object and so Deedle does not see any members.
To fix this, all you need to do is to define the type as a list of myClass objects:
var testObjList = new List<myClass>();
A more compact approach using an anonymous type would work too:
var testObjList =
((IEnumerable<dynamic>)x.features).Select(element =>
new { name = element.attributes.name });
var dfObjects = Frame.FromRecords(testObjList);

Couchbase Lite 2 + JsonConvert

The following code sample writes a simple object to a couchbase lite (version 2) database and reads all objects afterwards. This is what you can find in the official documentation here
This is quite a lot of manual typing since every property of every object must be transferred to the MutableObject.
class Program
{
static void Main(string[] args)
{
Couchbase.Lite.Support.NetDesktop.Activate();
const string DbName = "MyDb";
var db = new Database(DbName);
var item = new Item { Name = "test", Value = 5 };
// Serialization HERE
var doc = new MutableDocument();
doc.SetString("Name", item.Name);
doc.SetInt("Value", item.Value);
db.Save(doc);
using (var qry = QueryBuilder.Select(SelectResult.All())
.From(DataSource.Database(db)))
{
foreach (var result in qry.Execute())
{
var resultItem = new Item
{
// Deserialization HERE
Name = result[DbName].Dictionary.GetString("Name"),
Value = result[DbName].Dictionary.GetInt("Value")
};
Console.WriteLine(resultItem.Name);
}
}
Console.ReadKey();
}
class Item
{
public string Name { get; set; }
public int Value { get; set; }
}
}
From my research Couchbase lite uses JsonConvert internally, so there might be a way to simplify all that with the help of JsonConvert.
Anything like:
var json = JsonConvert.SerializeObject(item);
var doc = new MutableDocument(json); // No overload to provide raw JSON
or maybe
var data = JsonConvert.SerializeToDict(item); // JsonConvert does not provide this
var doc = new MutableDocument(data);
Is there or is this some kind of optimization and the preferred approach is by intend?

People ask about this quite often, but Couchbase Lite does not actually store JSON strings in the database. They are stored in a different format so this would not give the benefit that you think (the JSON would need to be reparsed and then broken down into the other format). I'd been pushing for a way to serialize classes directly instead of going through dictionary objects (which seems like the ultimate goal here) but our priority is on things that enterprise clients want and this doesn't seem to be one of them. Note that for it to make it in, it needs to be implemented in C# Java and Objective-C / Swift.

I don't know about JsonConvert but there seems to be a constructor that takes IDictionary<string, object> as argument. So I would try something like this (brain-compiled):
MutableDocument CreateDocument(object data)
{
if (data == null) return null;
var propertyValues = new Dictionary<string, object>();
foreach (var property in data.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
propertyValues[property.Name] = property.GetValue(data);
}
return new MutableDocument(propertyValues);
}
See if this works.

Set Index name by SearchRequest class in Nest

I use Nest client to use ElasticSearch .I want to search in ElasticSearch :
SearchRequest countRequest = new SearchRequest
{
//Somthing
};
client.Search<Post>(countRequest);
On other hand :
client.Search<Post>(s=>s.Index("IndexName").Query(...))
How i can set index name by SearchRequest class search ?

This is for those using newer versions of NEST. In 2.0.1, I am unable to find the Indices property in SearchRequest. However, you can pass them in through the constructor:
var request = new SearchRequest<Post>("IndexName", "TypeName");
I map the index and type on the ConnectionSettings like so.
ConnectionSettings settings = new ConnectionSettings("url");
settings.MapDefaultTypeIndices(t => t.Add(typeof(Post), "IndexName"));
settings.MapDefaultTypeNames(t => t.Add(typeof(Post), "TypeName"));
Other ways to tell NEST the index and type:
client.Search<Post>(s => s.Index("IndexName").Type("TypeName").From(0));
or apply the ElasticsearchTypeAttribute on the type.
[ElasticsearchType(Name = "TypeName")]
public class Post{ }

SearchRequest contains an Indices property, so that you can specify multiple indices to search across. In your case, you could just pass the single index like so:
var request = new SearchRequest
{
Indices = new IndexNameMarker[] { "IndexName" }
};
Another option would be to map your Post type to the index it belongs to, and use the typed SearchRequest<T> to let NEST infer the index name.

I was trying to solve a bit different task with ES v5 (json request was pushed from the file) but also had the same problem with setting the indexName. So, my solution was to add index querystring parameter. Using this in integration tests:
public static class ElasticSearchClientHelper
{
public static ISearchResponse<T> SearchByJson<T>(this IElasticClient client, string json, string indexName, Dictionary<string, object> queryStringParams = null) where T : class
{
var qs = new Dictionary<string, object>()
{
{"index", indexName}
};
queryStringParams?.ForEach(pair => qs.Add(pair.Key, pair.Value));
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
{
var searchRequest = client.Serializer.Deserialize<SearchRequest>(stream);
((IRequestParameters)((IRequest<SearchRequestParameters>)searchRequest).RequestParameters).QueryString = qs;
return client.Search<T>(searchRequest);
}
}
}

Parsing json files of different schemas to different tables json.net

I'm receiving json files of different schema and have to dump them in sql data base.
The json files have the schema
{'type':'abc','data':{'column1':'x','column2':'y',.........}}
Corresponding to each type of schema I have a strongly typed class named similar to the type but with word 'Table' attached..
eg. 'abcTable' which has only the schema of json.data (column1, column2, ...)
So, what I can do is do a dynamic deserializing of the main json and then based on the type value do a strongly typed json parsing of the corresponding data
dynamic jsondata = JsonConvert.DeserializeObject<dynamic>(json);
if (jsonata.type=='abc')
{
var abcobj = JsonConvert.DeserializeObject<abcTable>(jsondata.data);
}
Here I'm deserializing the object twice, so don't look like the right way of doing..
Also I have 25+ such schemas and a similar number of classes/tables
So, I will be using a lot of if / else if /else statements...
I would like to understand if there are other better ways of solving what I'm trying to do..
Any help is sincerely appreciated..
Thanks

As usually, JObject is your friend:
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
if (type == "abc")
{
var abcObject = parsed["data"].ToObject<abcTable>();
}
I order to avoid many ifs, you can use the follwing pattern:
public interface ITableType
{
bool Match(string type);
void Handle(JToken jsonTable);
}
public AbcTableHandler: ITableType
{
public bool Match(string type)
{
return type == "abc";
}
public void Handle(JToken jsonTable)
{
var abcTable = jsonTable.ToObject<abcTable>();
// other code
}
}
usage:
var handlers = new[] { new AbcTableHandler() };
// ...
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
var handler = handlers.SingleOfDefault(h => h.Match(type));
if (handler == null)
throw new InvalidOperationException("Cannot find handler for " + type);
handler.Handle(parsed["data"]);
EDIT:
Adding multiple handlers:
var handlers = new ITableType[] { new AbcTableHandler(), new OtherHandler, etc.. };
or
var handlers = new List<ITableType>();
handlers.Add(new AbcTableHandler());
handlers.Add(new OtherHandler());

LINQ and creating NON anonymous return values

I think I understand returning records of an anonymous type from But in this I want to create NEW CatalogEntries, and set them from the values selected. (context is a Devart LinqConnect database context, which lets me grab a view).
My solution works, but it seems clumsy. I want to do this in one from statement.
var query = from it in context.Viewbostons
select it;
foreach (GPLContext.Viewboston item in query)
{
CatalogEntry card = new CatalogEntry();
card.idx = item.Idx;
card.product = item.Product;
card.size = (long)item.SizeBytes;
card.date = item.Date.ToString();
card.type = item.Type;
card.classification = item.Classification;
card.distributor = item.Distributor;
card.egplDate = item.EgplDate.ToString();
card.classificationVal = (int)item.ClassificationInt;
card.handling = item.Handling;
card.creator = item.Creator;
card.datum = item.Datum;
card.elevation = (int)item.ElevationFt;
card.description = item.Description;
card.dirLocation = item.DoLocation;
card.bbox = item.Bbox;
card.uniqID = item.UniqId;
values.Add(card);
}
CatalogResults response = new CatalogResults();
I just tried this:
var query2 = from item in context.Viewbostons
select new CatalogResults
{ item.Idx,
item.Product,
(long)item.SizeBytes,
item.Date.ToString(),
item.Type,
item.Classification,
item.Distributor,
item.EgplDate.ToString(),
(int)item.ClassificationInt,
item.Handling,
item.Creator,
item.Datum,
(int)item.ElevationFt,
item.Description,
item.DoLocation,
item.Bbox,
item.UniqId
};
But I get the following error:
Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a
collection initializer because it does not implement
'System.Collections.IEnumerable' C:\Users\ysg4206\Documents\Visual
Studio
2010\Projects\CatalogService\CatalogService\CatalogService.svc.cs 91 25 CatalogService
I should tell you what the definition of the CatalogResults is that I want to return:
[DataContract]
public class CatalogResults
{
CatalogEntry[] _results;
[DataMember]
public CatalogEntry[] results
{
get { return _results; }
set { _results = value; }
}
}
My mind is dull today, apologies to all. You are being helpful. The end result is going to be serialized by WCF to a JSON structure, I need the array wrapped in a object with some information about size, etc.

Since .NET 3.0 you can use object initializer like shown below:
var catalogResults = new CatalogResults
{
results = context.Viewbostons
.Select(it => new CatalogEntry
{
idx = it.Idx,
product = it.Product,
...
})
.ToArray()
};
So if this is only one place where you are using CatalogEntry property setters - make all properties read-only so CatalogEntry will be immutable.
MSDN, Object initializer:
Object initializers let you assign values to any accessible fields or properties of an
object at creation time without having to explicitly invoke a constructor.

The trick here is to create a IQueryable, and then take the FirstOrDefault() value as your response (if you want a single response) or ToArray() (if you want an array). The error you are getting (Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a collection initializer because it does not implement 'System.Collections.IEnumerable') is because you're trying to create an IEnumerable within the CatalogEntry object (by referencing the item variable).
var response = (from item in context.Viewbostons
select new CatalogEntry()
{
idx = item.Idx,
product = item.Product,
size = (long)item.SizeBytes,
...
}).ToArray();

You don't have to create anonymous types in a Linq select. You can specify your real type.
var query = context.Viewbostons.Select( it =>
new CatalogEntry
{
idx = it.idx,
... etc
});

This should work:
var query = from it in context.Viewbostons
select new CatalogEntry()
{
// ...
};