I thought this question might be ill-formed, so I created a "sister" question that's far more to-the-point, which specific output cited. Please see to:
Querying JSON Nested Arrays with Linq, JSON.NET, C#
If that question gets answered before this one, I'll try to answer this one myself using the information from the other question... :) Thanks!
In a previous question (Picking Out Simple Properties from Hierarchical JSON), I asked how to pick up simple properties from hierarchical JSON. The answer there [pasted as a Linq query at the end of this post] is extremently useful (and, since posting that, I've studied quite a bit about Linq and JSON.NET). So I'm not using this forum because I'm lazy--I use it when I'm really stuck and can't seem to find the answers in the books I have access to.
I'm stumped on how to continue with the example provided by my previous question. This question builds on the previous one, so here (as succinctly as I could express) is what I'd love to learn how to do in a single Linq query.
To recap: I'm working with dynamic JSON like this (it is more complex than the JSON presented in my earlier Part I question because it contains arrays):
{
"Branch1": {
"Prop1A": "1A",
"Prop1B": "1B",
"Prop1C": "1C",
"Branch2": {
"Prop2A": "2A",
"Prop2B": "2B",
"Prop2C": "2C",
"Branch3": [{
"Prop3A": "3A",
"Prop3B": "3B",
"Prop3C": "3C"
}, {
"Prop3D": "3D",
"Prop3E": "3E",
"Prop3F": "3F"
}, {
"Prop3G": "3G",
"Prop3H": "3H",
"Prop3I": "3I"
}]
},
"Branch4": [{
"Prop4A": "4A",
"Prop4B": "4B",
"Prop4C": "4C"
}, {
"Prop4E": "4E",
"Prop4F": "4F",
"Prop4G": "4G"
}, {
"Prop4H": "4H",
"Prop4I": "4I",
"Prop4I": "4I"
}]
}
}
As you can see, the dynamic JSON is composed of hierarchical objects, and these objects are JSON objects, JSON arrays, and JSON properties.
Ultimately, I want to convert this JSON to a List object I can work with in C#. I plan to use that List object to effectively process each JSON branch in document order from the top, down.
Each item in the List collection would be a JObject; each of these objects would have a synthetic "parent" string property that would point back to the branch under which that JObject appears in the original JSON (my examples below explain what I mean by "parent"). [The previous question correctly came up with a solution for this "parent" value, so that's not too relevant to this question... What's new/relevant here is dealing with JArray objects in the JSON...]
The key is that I want each List item object to contain only the string-value properties for the object. For example, Branch1 has string properties Prop1A, 1B, and 1C. Thus, I would want query[0] to contain:
{"Prop1A":"1A","Prop1B":"1B","Prop1C":"1C", Parent:""}
Next, I would want query[2] to contain the string-value properties for Branch2:
{"Prop2A":"2A","Prop2B":"2B","Prop2C":"2C", Parent:"Branch1"}
Next, I would want query[2] to contain the string properties for only Branch3, but because Branch3 is an array of objects, I would want that array to end up together in query[2]:
[
{"Prop3A": "3A","Prop3B": "3B","Prop3C": "3C"},
{"Prop3D": "3D","Prop3E": "3E","Prop3F": "3F"},
{"Prop3G": "3G","Prop3H": "3H","Prop3I": "3I"}
]
Notice that this Branch doesn't yet have a reference to its "Parent"... I'd be happy getting an array in query[2] that looks like the above. (I plan to use dbc's logic to add a "Parent" property to each array element or figure out a way to create a new JObject that contains the array and cites the Parent only once):
[{"Prop3A": "3A","Prop3B": "3B","Prop3C": "3C","Parent":"Branch2"},
{"Prop3D": "3D","Prop3E": "3E","Prop3F": "3F","Parent":"Branch2"},
{"Prop3G": "3G","Prop3H": "3H","Prop3I": "3I","Parent":"Branch2"}
]
So, as you can see:
I would want any JSON branch that is not an array to be inserted as a new JObject in the query result along with only its string properties and a reference to its parent branch.
I would want any JSON branch that is an array to be inserted as a new JObject array in the query result along with only its string properties and a reference to its parent branch.
The trouble I had solving this on my own has to do with figuring out how to create the "if myJsonObject is JArray" condition in Linq and assigning just the string-property properties when the branch is not an array and iterating the array's elements when it is a JArray. I suspect I need to somehow leverage the ? : ternary expression, but I don't exactly know how to do that.
The query from the previous question is here:
var query3 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Skip(1).Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list3 = query3.ToList();
That code doesn't handle arrays in the way described above.
A suitable and more general answer to this question effectively appears in my related post:
Picking Out Simple Properties from Hierarchical JSON
Related
I want to query for values in a dynamic-key nested dictionary structure in order to apply a permission filter using LINQ to SQL (preferably utilizing index queries and not causing scans).
The JSON structure I have contains a Permissions property which is basically just a C# dictionary<string, dictionary<string, dictionary<string, string>>>
JSON example:
{
"Permissions": {
"Account 1 GUID": {
"Location": "Location1 GUID"
},
"Account 2 GUID: {
"Location": "Location 2 GUID"
},
"Account 3 GUID": {
"Location": "Location 3 GUID"
}
}
}
The account key properties are dynamic, meaning they are in fact the GUIDs of the accounts owning the nested location.
When a query is performed I have a list of location and account IDs that should be used to perform the permission filtering by giving access only to those locations.
The filtering query I want to apply is SIMILAR TO SOMETHING like this (THIS IS WHAT I WANT TO ACHIEVE):
var allowedLocations = new List<string> { "Location 1 GUID", "Location 2 GUID" };
var result = AllDocsIQueryable.Where(model =>
model.Permissions.Values.Where(secondLevel =>
secondLevel.Values.Where(thirdLevel =>
allowedLocations.Contains(thirdLevel)))).ToList();
As you can see, I want to filter the docs a user can see to those the user has defined an access to, by querying for docs that contains any of the location IDs in my access list.
I have no idea how to go about construction the query for querying a structure like this. I have tried many combinations of .Select and .SelectMany queries, but nothing I have tried succeeds, all I have tried yields no results or fails.
A query like the following yields the correct results, but I would really like to avoid having to build this query dynamically with predicate builder for example:
var result = AllDocsIQueryable.Where(model =>
(model.Permissions["Account 1 GUID"]["Location"].IsDefined() &&
model.Permissions["Account 1 GUID"]["Location"] == "Location 1 GUID")
||
(model.Permissions["Account 2 GUID"]["Location"].IsDefined() &&
model.Permissions["Account 2 GUID"]["Location"] == "Location 2 GUID")
// .. etc ..
).ToList();
I hope my question is clear and there is a LINQ to SQL/Cosmos/NoSQL/C# super wiz that can help me construct this query? Maybe someone knows that I can't construct this query at all, and I need to change the JSON structure (which I know would be preferable)? I need to query using LINQ and not SQL query string!
Conclusion:
It seems, as I was afraid of, that there's just no way of building the query.
As far as I know, and have tried, and read, there's just no way to do it without building the dynamic account keys into the query.
I've changed the schema to include arrays instead.
"Permissions": [
{
"LocationId": "Loc1",
"AccountId": "Acc1"
},
{
"LocationId": "Loc2",
"AccountId": "Acc2"
}
]
I'm afraid there is no way you can achieve that just using LINQ. Here are your options:
Use a UDF to retrieve an array of locations from a permissions object. Unfortunately, you can't call a UDF from LINQ (at least not in SDK v3 - you can in SDK v2), so you need to use raw SQL. (Actually there is hacky way.) This solution is bad because it's really slow.
Leave your schema but introduce an additional property that contains a list of locations that occur nested within your permissions object. Now you can easily use LINQ, and the query is fast.
I'm very new to Elasticsearch and Want to know How to create index and index following json document to Elasticsearch using NEST C#?
{
"BookName": "Book1",
"ISBN": "978-3-16-148410-0",
"chapter" : [
{
"chapter_name": "Chapter1",
"chapter_desc": "Before getting into computer programming, let us first understand computer programs and what they..."
},
{
"chapter_name": "Chapter2",
"chapter_desc": "Today computer programs are being used in almost every field, household, agriculture, medical, entertainment, defense.."
},
{
"chapter_name": "Chapter3",
"chapter_desc": "MS Word, MS Excel, Adobe Photoshop, Internet Explorer, Chrome, etc., are..."
},
{
"chapter_name": "Chapter4",
"chapter_desc": "Computer programs are being used to develop graphics and special effects in movie..."
}
]
}
To create an index with NEST is as simple as
var client = new ElasticClient();
client.CreateIndex("index-name");
This will create an index with the default number of shards and replicas defined for the node.
To index a document represented as json into the index would be
var json = #"{
""BookName"": ""Book1"",
""ISBN"": ""978-3-16-148410-0"",
""chapter"" : [
{
""chapter_name"": ""Chapter1"",
""chapter_desc"": ""Before getting into computer programming, let us first understand computer programs and what they...""
},
{
""chapter_name"": ""Chapter2"",
""chapter_desc"": ""Today computer programs are being used in almost every field, household, agriculture, medical, entertainment, defense..""
},
{
""chapter_name"": ""Chapter3"",
""chapter_desc"": ""MS Word, MS Excel, Adobe Photoshop, Internet Explorer, Chrome, etc., are...""
},
{
""chapter_name"": ""Chapter4"",
""chapter_desc"": ""Computer programs are being used to develop graphics and special effects in movie...""
}
]
}";
var indexResponse = client.LowLevel.Index<string>("index-name", "type-name", json);
if (!indexResponse.Success)
Console.WriteLine(indexResponse.DebugInformation);
Here we use the low level client to index json, available in NEST through the .LowLevel property on ElasticClient.
To search the indexed document would be
// refresh the index so that newly indexed documents are available
// for search without waiting for the refresh interval
client.Refresh("index-name");
var searchResponse = client.Search<dynamic>(s => s
.Index("index-name")
.Type("type-name")
.Query(q => q
.Match(m => m
.Query("Photoshop")
.Field("chapter.chapter_desc")
)
)
);
This returns the document indexed. The generic type parameter dynamic used in Search<T>() here means that the resulting documents will be deserialized to Json.Net JObject types.
When we created the index, we didn't specify a mapping for our type, type-name, so Elasticsearch inferred the mapping from the structure of the json document. This is dynamic mapping and can be useful for many situations however, if you know the structure of documents that you're going to be sending and it's not going to be destructively changed, then you specify a mapping for the type. The advantage of doing this in this particular example is that the chapter array will be inferred as an object type mapping, but if you wanted to search across both chapter name and chapter description of an individual chapter, then you probably want to map chapter as a nested type.
I've been looking at Dynamic Linq today (installed into VS via NuGet)...but all the examples I have found so far assume OrderBy is to be done on a known property or column name; however I am trying to OrderBy a field which is not strongly typed; but actually a key value of a row object which is derived from a Dictionary; e.g.
class RowValues : Dictionary<string, string>
{
...
}
So the list to be ordered is specifically a list of RowValues objects, filled with Name,Value pairs. For a given list of RowValues, the OrderBy field could by any of keys of the named value pairs entries (fyi: I want the orderby field to be specified in an xml config file ultimately so the ordering can be changed without re-deployment of binaries etc).
I've got a hunch the solution lies in writing a custom ordering function passed to the OrderBy??? This function would obviously know how to get a specific value from the RowValues object given a field name from the xml config....?? The answers I have seen so far show passing a string which contains a custom order by clause into the OrderBy, which is close to where I want to be, but how in my case would the runtime know where to find the fields referred to in the OrderBy string??
Input will be very much appreciated, or have I completely misunderstand the Dynamic Linq functions?
If you're using dynamic LINQ, it would just be:
var sortColumn = GetConfigValue(...);
var sorted = RowValues.OrderBy(sortColumn);
You could of course use a concatenated string to create a multiple sort ("column1, column2 DESC"). As far as I'm aware, there's no custom sort function unless you're using regular LINQ.
Also, I would make sure you know the performance characteristics of Dynamic LINQ.
Edit:
Is this what you're looking for? This will order it based on the value of the "Key" entry in the dictionary. If you need multiple sort by-s, you can use it in a loop with .ThenBy()
void Main()
{
List<RowValues> v = new List<RowValues>();
var key = "Key"; //GetFromConfig();
var v1 = new RowValues();
v1.Add("Key", "1");
v1.Add("3", "5");
var v2 = new RowValues();
v2.Add("Key", "3");
v2.Add("2", "2");
var v3 = new RowValues();
v3.Add("Key", "2");
v3.Add("2", "2");
v.Add(v1);
v.Add(v2);
v.Add(v3);
v.OrderBy(r => r[key]).Dump();
}
class RowValues : Dictionary<string, string>
{
}
Kyle, thanks again. Apologies for late reply, I have moved on from this issue now but out of interest and courtesy I wanted to come back and agree your code is much closer to where I wanted to get to, but we have lost the dynamic linq aspect. So, where you are calling the OrderBy and ordering on the key, I would want to pass a string containing the order command e.g "r[key] desc". The reason being I would want to leave the determination as to which direction to order until runtime. I suspect ths would be accomplished using an expression tree possibly? e.g: here
I have a string of Json which looks like this:
{"Linkin Park":{"mbid_id":"f59c5520-5f46-4d2c-b2c4-822eabf53419","artistbackground":
[{"id":"34862","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4fcd2d8e4764f.jpg",
"likes":"4"},{"id":"3953","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4de522dac2c13.jpg","likes":"1"},{"id":"3954","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4de523586c57e.jpg","likes":"1"},{"id":"4251","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4dea53983f4cf.jpg","likes":"1"},{"id":"30196","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4fabb53ba80a3.jpg","likes":"1"},{"id":"52251","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-5058a071d9782.jpg","likes":"1"},{"id":"52252","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-5058a071d9f41.jpg","likes":"1"},{"id":"52254","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-5058a0ebd0350.jpg","likes":"1"},{"id":"52255","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-5058a0ebd098b.jpg","likes":"1"},{"id":"63902","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-50c2f127eeae6.jpg","likes":"1"},{"id":"3951","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4de521f409aa2.jpg","likes":"0"},{"id":"3952","url":
"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4de5225cdd595.jpg","likes":"0"},
{"id":"28038","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-4f8ecbd3c004d.jpg","likes":"0"},{"id":"88261","url":"http://assets.fanart.tv/fanart/music/f59c5520-5f46-4d2c-b2c4-822eabf53419/artistbackground/linkin-park-51d1d122f3029.jpg","likes":"0"}]}}
If you did have the time to read all of that, you might see my problem. There are Json nodes which change name depending on what query I send. This time I sent "Linkin Park", and I got nodes named Linkin park. If I then were to send a query of Madonna I would end up with nodes called Madonna.
How would you go about de-serializing this into classes?
You have a few options:
Dictionary
dynamic
JObject, and then deserialize to strongly-typed one node down (assuming the subtree is static in its structure and naming) (this is based on Newtonsoft.JSON)
What library are you using for JSON deserialization?
Example of #1 using Json.NET:
var json = JsonConvert.DeserializeObject<Dictionary<string, StronglyTyped>>(jsonStr);
var data = json.Values.FirstOrDefault();
Example of #3 using Json.NET:
var json = JsonConvert.DeserializeObject<JObject>(jsonStr);
var topProp = json.Properties().FirstOrDefault().Name;
var data = json[topProp].ToObject<StronglyTyped>();
Since you know the query, the simplest solution would be to replace {"%QUERY%":{ with {"MyCommonDeserializableClass":{ before deserializing the string. :-)
But you should make sure that the JSON contains only one object like this. Logically it seems they return the artist name because several artists may match the query.
Update: False alarm! The source of the error was elsewhere. (See at the end of the question.)
Is it possible that either Linq or foreach can mess up the order of an array?
One of my testers reported to have experienced that the order of a list of items he fed in as input didn't match the order of the final list that was saved in the database. More precisely, the first element became the last.
So I reviewed my code step by step, and I have no clue what should change the order. There is a Linq query and a foreach loop however. Is it possible that one of these can mess up the order of an array?
Code, simplified:
List<FooBar> fooBarList = new List<FooBar>();
string[][] theData = new string[][] {
new string[] { "a", "x" },
new string[] { "b", "y" },
new string[] { "c", "z" } };
FooBar[] fooBarArray = theData.Select(
row => new FooBar { Foo = row[0], Bar = row[1] }
).ToArray();
foreach (FooBar item in fooBarArray)
{
int iRank = fooBarList.Count + 1;
item.Ranking = iRank;
fooBarList.Add(item);
}
The array of arrays of strings theData is in fact given as an input. It is transformed into an array of business objects. These are then added to a list and assigned a ranking field. This field is written to the database together with "Foo" and "Bar".
After saving the list in the database, the rank of "a" was 3 in that particular case. For me, however, I cannot reproduce the misbehavior...
Update: I was wrong, the data written to the database was correct. The data I looked to was from a business object that was copied from the original one. When copying, the order was mixed up while reading it from the database, and this wrong order was then persisted in the copy of the object... => Accepted Jon's answer saying "LINQ to Objects generally has a predictable ordering - other providers often don't."
Well, your sample code is only showing LINQ to Objects. How are you inserting the data into the database? If that's using LINQ as well, I strongly suspect that it's the LINQ to SQL (or whatever) side which is causing the issue, not LINQ to Objects.
LINQ to Objects generally has a predictable ordering - other providers often don't.
EDIT: If this happens reproducibly, then you ought to be able to catch it happening in the debugger... that should give you some hints. I suspect that if you try to create a short but complete program which demonstrates the problem, you'll end up finding out what's wrong.