I'm using JsonPath for C# to query some JSON data. JsonPath doesn't come with its own parser, so as per Rick Sladkey's advice, I'm using Json.NET to parse my Json string into a collection of nested IDictionary objects, IList arrays, and primitives. Then I use JsonPath to filter it (after adding the class suggested in Rick Sladkey's answer).
For reference, here's the part of my code that actually handles all this:
// string exampleJsonString defined below
string query = "$.store.book[*].title" // we should get a list of all titles
// step 1: use Json.NET to parse the json string into a JObject
JObject parsedJson = JObject.Parse(exampleJsonString);
// step 2: use JsonPath with a custom ValueSystem (JsonNetValueSystem)
JsonPathContext context = new JsonPathContext
{ ValueSystem = new JsonNetValueSystem() };
// step 3: get results using JsonPath
List<object> toRet = context.SelectNodes(parsedJson,
query).Select(node => node.Value).ToList();
The reason I was using JsonPath in the first place was because of its filter functionality. You can not only do normal queries like "$.store.book[*].title" (to get an array of all the titles in the book store), but also queries like "$.store.book[?(#.category == 'fiction')].title" (to get an array of all titles in the book store whose category matches 'fiction'). I need to be able to pass entire queries as a string, and so being able to filter while querying is extremely helpful.
Unfortunately, I'm having some trouble getting this filter functionality to work. I expect that I'll have to make adjustments to either the JsonNetValueSystem class (defined originally in the aforementioned stack overflow answer) or the JsonPath namespace (you can get JsonPath.cs from JsonPath's google code page). If there's some external tool or an alternative parsing mechanism to Json.NET that would allow me to keep JsonPath's filtering without having to write too much extra code, that would be ideal, but I'm pretty sure I'll have to alter either JsonNetValueSystem or JsonPath itself. (Both of these are fairly easy to alter since they're just .cs files, but I can't really dig into Json.NET without a lot more work.)
I can't actually seem to figure out where the original JsonPath code handles filtering, nor can I figure out why the JsonNetValueSystem class robs it of that functionality. Any advice for how to add the ability to filter in the query string would be very much appreciated. Even if it's just "don't mess with JsonPath, just change JsonNetValueSystem" or vice versa.
string exampleJsonString = "{
"store": {
"book": [ {
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
}, {
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}, {
"category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
}, {
"category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
} ],
"bicycle": [ {
"color": "red",
"price": 19.95,
"style": [ "city", "hybrid" ]
}, {
"color": "blue",
"price": 59.91,
"style": [ "downhill", "freeride" ]
} ]
}
}"
When you use the script expression in a query (the ?(...) part), you need to provide a ScriptEvaluator method to evaluate the script. Unfortunately they don't provide a default implementation for the C# version. You'll need to provide that implementation.
Out of the box, this won't be the most trivial problem to solve, you'll need to write an interpreter. You have a couple of options: use CodeDOM to compile and execute the script as C# code (or any language you would prefer), use Roslyn to parse the script and evaluate, whatever works.
A quick and dirty solution for this particular case would be to do something like this in your script evaluator method:
object EvaluateScript(string script, object value, string context)
{
if (script == "#.category == 'fiction'")
{
var obj = value as JObject;
return (string)obj["category"] == "fiction";
}
return null;
}
Here's another solution which utilizes IronPython to evaluate the script.
public class IronPythonScriptEvaluator
{
private Lazy<ScriptEngine> engine = new Lazy<ScriptEngine>(() => Python.CreateEngine());
private ScriptEngine Engine { get { return engine.Value; } }
private const string ItemName = "_IronPythonScriptEvaluator_item";
public object EvaluateScript(string script, object value, string context)
{
var cleanScript = CleanupScript(script);
var scope = Engine.CreateScope();
scope.SetVariable(ItemName, value);
return Engine.Execute<bool>(cleanScript, scope);
}
private string CleanupScript(string script)
{
return Regex.Replace(script, #"#", ItemName);
}
}
Another solution is to use Manatee.Json instead. It has a native JSONPath implementation and parser all built in (along with schema and a serializer). Moreover, you aren't required to represent the path as a string. Manatee.Json has a fluent interface that you can use to build paths, including expression support.
To represent $.store.book[*].title, you would have
var path = JsonPathWith.Name("store")
.Name("book")
.Array()
.Name("title");
For your example $.store.book[?(#.category == 'fiction')].title, you'd use
var path = JsonPathWith.Name("store")
.Name("book")
.Array(jv => jv.Name("category") == "fiction")
.Name("title");
What's more is that there is limited support for fields within those expressions as well.
If your path source is a string, Manatee.Json handles path parsing, too!
Related
I have a document like this:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF/Paris"
]
}
And I have an user that has a list of authorized groups, like for example the following:
"groups": [
"France/IDF",
"USA/NY/NYC"
]
What I'm trying to achieve is to retrieve all documents in the database that the user is authorized to retrieve, essentially I want to be able to check in the list "groupsAuthorized" if one of the group contains a subset of an element of the other list "groups" contained in my user authorizations
using the following values:
my document:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF/Paris"
]
}
my user permissions:
"groups": [
"France/IDF",
"USA/NY/NYC"
]
the user should be able to retrieve this document as the string "France/IDF" is correctly contained in the string "France/IDF/Paris", however, if the values would've been like this:
my document:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF"
]
}
my user permissions:
"groups": [
"France/IDF/Paris",
"USA/NY/NYC"
]
it should not work, because my user is only authorized to view documents from France/IDF/Paris and USA/NY/NYC and none of the string inside of the authorizedGroups of my document contains those sequences
I've tried to use a standard LINQ query to achieve this which is fairly simple:
var userAuthorizedGroups = new List<string> { "France/IDF/Paris", "USA/NY/NYC" };
var results = collection.AsQueryable()
.Where(entity => userAuthorizedGroups
.Any(userGroup => entity.authorizedGroups
.Any(entityAuthorizedGroup => entityAuthorizedGroup.Contains(userGroup))));
But i'm getting the famous unsupported filter exception that it seems lot of people is having, i've tried different options found on the internet like the following:
var userAuthorizedGroups = new List<string> { "France/IDF/Paris", "USA/NY/NYC" };
var filter = Builders<PartitionedEntity<Passport>>.Filter.AnyIn(i => i.authorizedGroups, userAuthorizedGroups);
var results = (await collection.FindAsync(filter)).ToList();
return results;
But the problem is this will only check if one of the element of the array is contained inside the other array, It will not correctly work for case like "France/IDF" that should correctly match "France/IDF/Paris" because "France/IDF" string is contained inside the "France/IDF/Paris" string inside of my document
I'm getting a bit clueless on how to achieve this using the mongodb C# driver, i'm starting to think that I should just pull all documents to client and do the filtering manually but that would be quite messy
Has anyone an Idea on this subject ?
i'm starting to think that I should just pull all documents to client and do the filtering manually but that would be quite messy
don't do it :)
One place you can start with is here. It describes all the LINQ operators that are supported by the MongoDB .NET driver. As you can see .Contains() isn't mentioned there which means you can't use it and you'll get an arror in the runtime but it does not mean that there's no way to do what you're trying to achieve.
The operator closest to contains you can use is $indexOfBytes which returns -1 if there's no match and the position of a substring otherwise. Also since you need to match an array against another array you need two pairs of $map and $anyElementTrue to do exactly what .NET's .Any does.
Your query (MongoDB client) can look like this:
db.collection.find({
$expr: {
$anyElementTrue: {
$map: {
input: "$groupsAuthorized",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: ["France/IDF/Paris", "USA/NY/NYC"],
as: "userGroup",
in: { $ne: [ -1, { $indexOfBytes: [ "$$userGroup", "$$group" ] } ] }
}
}
}
}
}
}
})
Mongo Playground,
You can run the same query from .NET using BsonDocument class which takes a string (JSON) and converts into a query:
var query = BsonDocument.Parse(#"{
$expr: {
$anyElementTrue:
{
$map:
{
input: '$groupsAuthorized',
as: 'group',
in: {
$anyElementTrue:
{
$map:
{
input: ['France/IDF/Paris', 'USA/NY/NYC'],
as: 'userGroup',
in: { $ne: [-1, { $indexOfBytes: ['$$userGroup', '$$group'] } ] }
}
}
}
}
}
}
}");
var result = col.Find(query).ToList();
There is a series of JSON objects called "issues", which each have one or more "issue links", which have the following format:
// an issue link
{
"id": "000000",
"self": "some link",
"type": {
"id": "0000",
"name": "some name",
"inward": "is met by",
"outward": "meets",
"self": "some link"
},
"outwardIssue": {
"id": "000000",
"key": "the required key",
"self": "some link",
"fields": {
// the remainder is not applicable
}
}
}
}
These "issue links" have been extracted as follows. Create a JArray for the JSON for the "issue" itself, and extract the child JObjects:
public void Deserialize(dynamic jsonObject)
{
// get the issue links
if (jsonObject["fields"]["issuelinks"]!=null)
{
JArray issueLinksArray = jsonObject["fields"]["issuelinks"];
var issueLinkObjects = issueLinksArray.Children();
foreach (var issueLink in issueLinkObjects)
{
// now need the "key" in the "outwardIssue" for this object, if the value of "inward" is "is met by".
}
}
}
How to go about extracting the value of the second property "key" of "outwardIssue"?
Not sure whether I fully understand but following excerpt gets u the value (or null if condition not met) like this.
var key = issueLink["type"]["inward"].ToString()=="is met by" ? issueLink["outwardIssue"]["key"]: null;
Hint: Try to avoid dynamic.
Nowadays loops can in certain conditions considered old-school. Think LINQ: The problem can be divided into smaller probs and distributed across multiple lines (think-steps).
The additional variables might improve readability. As the project grows, for loops are prone to span more and more lines. So if you just need requested values following might be of interest:
var inwardLinks = issueLinkObject.Where(i=>i["type"]["inward"].ToString()=="is met by");
var keys = inwardLinks.Select(iwl=>iwl["outwardIssue"]["key"]);
I'm trying to pull one specific piece of data from the json received from a geocoding request.
This is not a duplicate of the myriads of similar-sounding questions, because the json is dynamic and has extremely irregular arrays that change slightly between responses, so I can't create a model to parse to, neither can I navigate through it using key names.
This is the json of one response:
{
"Response":{
"MetaInfo":{
"Timestamp":"2019-07-28T13:23:04.898+0000"
},
"View":[
{
"_type":"SearchResultsViewType",
"ViewId":0,
"Result":[
{
"Relevance":1.0,
"MatchLevel":"houseNumber",
"MatchQuality":{
"City":1.0,
"Street":[
0.9
],
"HouseNumber":1.0
},
"MatchType":"pointAddress",
"Location":{
"LocationId":"NT_Opil2LPZVRLZjlWNLJQuWB_0ITN",
"LocationType":"point",
"DisplayPosition":{
"Latitude":41.88432,
"Longitude":-87.63877
},
"NavigationPosition":[
{
"Latitude":41.88449,
"Longitude":-87.63877
}
],
"MapView":{
"TopLeft":{
"Latitude":41.8854442,
"Longitude":-87.64028
},
"BottomRight":{
"Latitude":41.8831958,
"Longitude":-87.63726
}
},
"Address":{
"Label":"425 W Randolph St, Chicago, IL 60606, United States",
"Country":"USA",
"State":"IL",
"County":"Cook",
"City":"Chicago",
"District":"West Loop",
"Street":"W Randolph St",
"HouseNumber":"425",
"PostalCode":"60606",
"AdditionalData":[
{
"value":"United States",
"key":"CountryName"
},
{
"value":"Illinois",
"key":"StateName"
},
{
"value":"Cook",
"key":"CountyName"
},
{
"value":"N",
"key":"PostalCodeType"
}
]
}
}
}
]
}
]
}
}
I'm trying to get just the two coordinates inside NavigationPosition.
Before I start the tedious work of creating a method to manually pull out the data I need from the unparsed string, does anyone have a solution? Is there a way to iterate anonymously through the json arrays using a foreach or if statement?
If the response is dynamic then one option is to use dynamic deserialization
var dynamicObject = JsonConvert.DeserializeObject<dynamic>(jsonAsString);
You can iterate this object and write defensive code by checking null.
I am assuming that "NavigationPosition" will always lie under "location" and in turn under a list of "Result" under list of "View"
something like this.. (symbolic code)
if(dynamicObject.View != null && dynamicObject.View[0] != null && dynamicObject.View[0].Result[0] != null && dynamicObject.View[0].Result[0].Location != null )
{
var navigationPosition = dynamicObject.View[0].Result[0].Location.NavigationPosition;
// do something with it
}
It seems really cumbersome but if you have no control over json response, I think you should be able to get it right after some iterations :)
I am using MongoDB C# driver version 2.2. My collection contains "Parent" objects. Each parent object has an array of children objects. Each child has name value:
"parent": {
"children":[
{ "name": "Bob", "age": 10},
{ "name": "Alice", "age": 7},
{ "name": "Tobias", "age": 11}
]
}
I need to translate the following code into C# statements / LINQ syntax:
db.getCollection('Parents').find({'parent.children': { $elemMatch: { 'name': { $regex: '.*ob.*', $options: 'im' } }}})
I have found there are methods like
var builder = Builders<BsonDocument>.Filter;
builder.Regex("parent.children.name", new BsonRegularExpression(".*ob.*")); //does not work with array
and
builder.AnyEq("parent.children.name", "ob"); //without regex
But I cannot understand how to combine them. Please advise.
UPDATE:
I am using the following for now, please correct me if you know a reason why it should not work correctly:
builder.AnyEq("parent.children.name", new BsonRegularExpression(".*ob.*"))
I am using the following for now:
builder.AnyEq("parent.children.name", new BsonRegularExpression(".*ob.*"))
Can't test C# on this machine. Let me know if this doesn't work:
var filter = Builders<People>.Filter.ElemMatch(x => x.Parent.Children, x => Regex.IsMatch(x.Name, "regex"));
var res = await collection.Find(filter).ToListAsync();
Here's a trick you might like btw:
// Take your inputted `find` query string:
string bsonQuery = "{'parent.children': { $elemMatch: { 'name': { $regex: '.*ob.*', $options: 'im' } }}}";
// Use it as the filter!
var filter = MongoDB.Bson.Serialization.BsonSerializer.Deserialize<BsonDocument>(bsonQuery);
// Results:
var result = col.FindSync (filter).ToList();
I've tested your current expression (builder.AnyEq("parent.children.name", new BsonRegularExpression(".*ob.*")) on a database at home and I don't believe it behaves in the way you intend.
Although the c# documentation doesn't explicitly state this, mongoDB supports the Regex filter on an array field. I have tested the below expression in C# and have correct results for the Regex despite the field being an array.
builder.Regex(MONGO_FIELD_NAME, new BsonRegularExpression("SOME REGEX"));
In addition, I've tested Regex on the toy example in the [online mongo webshell for query arrays] (https://docs.mongodb.com/manual/tutorial/query-arrays/). Querying for db.inventory.find({tags : {$regex : "^bl"}} will return results with "blank" or "blue" despite the "tag" field being an array.
Before I get flagged for duplicate, I have the code from Dynamic json object with numerical keys working quite well now. The question with my numeric keys is that unfortunately, the JSON string I am getting is initially delimited by year, so would I use reflection to attempt to create a dynamic property on a dynamic object, and if so how? I know with a dynamic object I can't have obj["2010"] or obj[0]. In JavaScript this is no problem, just trying to get it working in C#. Ideas?
Example of JSON being returned:
{
"2010": [
{
"type": "vacation",
"alloc": "90.00"
},
Alternatively, sometimes the year is the second element as such:
I have no control over this json.
{
"year": [],
"2010": [
{
"type": "vacation",
"alloc": "0.00"
},
Maybe I'm misunderstanding your question, but here's how I'd do it:
static void Main(string[] args) {
var json = #"
{
'2010': [
{
'type': 'vacation',
'alloc': '90.00'
},
{
'type': 'something',
'alloc': '80.00'
}
]}";
var jss = new JavaScriptSerializer();
var obj = jss.Deserialize<dynamic>(json);
Console.WriteLine(obj["2010"][0]["type"]);
Console.Read();
}
Does this help?
I wrote a blog post on serializing/deserializing JSON with .NET: Quick JSON Serialization/Deserialization in C#
I have up-voted the question and JP's answer and am glad I dug around the internet to find this.
I have included a separate answer to simplify my use case for others to benefit from. The crux of it is:
dynamic myObj = JObject.Parse("<....json....>");
// The following sets give the same result
// Names (off the root)
string countryName = myObj.CountryName;
// Gives the same as
string countryName = myObj["CountryName"];
// Nested (Country capital cities off the root)
string capitalName = myObj.Capital.Name;
// Gives the same as
string capitalName = myObj["Capital"]["Name"];
// Gives the same as
string capitalName = myObj.Capital["Name"];
Now it all seems quite obvious but I just did not think of it.
Thanks again.