We Keep Coding

Searching in multiple elasticsearch indexes using NEST

I am trying to search for a text in elasticsearch using nest 7.10.1. I want to search in two different indexes, I get a response in the form of documents, but I cannot access its properties because the result has the combination of two indexes. Below is the code I tried. Both the indexes has same properties. What do I use in the foreach loop to access the key and values of the result documents.
public void searchIndices(string query) {
var response = client.Search<object>(
s => s.Index("knowledgearticles_index, index2")
.Query(q => q.Match(m => m.Field("locationName")
.Query(query))));
Console.WriteLine(response.Documents);
foreach(object r in response.Documents) {
}
}
I am using elasticsearch 7.10.2

Each raw hit coming back in the search response has the _index meta field associated with it:
"hits" : {
"total" : {
"value" : 91,
"relation" : "eq"
},
"hits" : [
{
"_index" : "knowledgearticles_index", <---
"_type" : "_doc",
"_id" : "r_oLl3cBZOT6A8Qby8Qd",
"_score" : 1.0,
"_source" : {
...
}
}
Now, in NEST,
.Documents is a convenient shorthand for retrieving the _source for each hit
-- meaning that you'll have lost access to the meta properties.
So the trick is to use a loop like this instead:
foreach (var hit in response.HitsMetadata.Hits) {
Console.WriteLine(hit);
}

If the two indices that you're searching over contain different JSON structures, then the T in Search<T> will need to be a type that the differing JSON structures can be deserialize to. object will work as per the example in your question, but then there is no typed access for any of the properties.
A simple approach to overcoming this is to hook up the JsonNetSerializer and then use JObject for T
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings =
new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default);
var client = new ElasticClient(connectionSettings);
var response = client.Search<JObject>(s => s
.Index("knowledgearticles_index, index2")
.Query(q => q
.Match(m => m
.Field("locationName")
.Query(query)
)
)
);
We now have a way of accessing properties on JObject, but will still need to handle the type of each property.

Extract value from json in a database column using linq

Suppose I have two columns in my database table as below :
Id Actions
-- -------
id1 [{"Id": "create", "Schema": {"Id": "create", "Type": "Create"}, "Workflow": "basic"}]
I want to extract the value of the Workflow and check if it equals basic, then create a list with the entries in the database that meet this condition.
What I have tried is:
var data = db.Where(wf => (string) JObject.Parse(wf.Actions)["Workflow"] == "basic").AsEnumerable();
Then I check:
if (data.Any())
{
// do something
}
But the if statement throws me the following error:
System.ArgumentNullException: Value cannot be null.

Use JArray.Parse:
var data = db.Where(wf => (string) JArray.Parse(wf.Actions)[0]["Workflow"] == "basic")
.AsEnumerable();

Your json object is actually json array, so you need to parse it to JArray.
Then, when getting the value from JToken, you need to call .Value<string>, then you can compare it to "basic" string.
var data = db.Where(wf => JArray.Parse(wf.Actions).FirstOrDefault()?["Workflow"].Value<string>() == "basic").ToList();

Need to know how to search in ES using c# searching in arrays

Hello I am a newbie on ElasticSearch and need help. I'm working with c# (thought I could use a QueryRaw in String I think...).
Below the scenario:
Data
{
"id": "1",
"title": "Small cars",
"tagsColours": ["grey",
"black",
"white"],
"tagsCars": ["Suzuki",
"Ford"],
"tagsKeywords": []
},
{
"id": "2",
"title": "Medium cars",
"tagsColours": [],
"tagsCars": ["VW",
"Audi",
"Peugeot"],
"tagsKeywords": ["Sedan"]
},
{
"id": "3",
"title": "Big cars",
"tagsColours": ["red",
"black"],
"tagsCars": ["Jeep",
"Dodge"],
"tagsKeywords": ["Van",
"Big"]
}
Objective
Id' like to apply filters on tags columns based on users' selection. the values will be populated in the tagsXXX array columns withselected values.
if parameter array value is not empty then the result should contain at least one instance. Same for every parameter array. the more the parameters have values, the more specific search should be done
if at least there's one value coming from the parameter that matches amongst all values in any document's tag column array, then get that document.
but if there's another value on another tagsXXX array then it should take it into account.
if the tag parameter array has no values, then disregard that filter
Desired responses
A) If user select only 1 tag Color (i.e= black) as formatted below:
{
id: "",
title: "",
tagsColours: ["black"],
tagsCars: [],
tagsKeywords: []
}
I'd like to get documents with Id=2 and id=3 since they have black in their tagsColours and disregard tagsCars and tagsKeywords since there are no values on the parameters
B) If user select only 2 diff tags (i.e= colour=black and cars= audi, and mercedez benz) as formatted below:
{
id: "",
title: "",
tagsColours: ["black",
"yellow"],
tagsCars: ["Audi",
"Mercedes Benz"],
tagsKeywords: []
}
I'd like to get documents with id=2 since it found black on tagsColours and it found Audi in tagsCars, AND it should not pull document id=1 because
even when black is on tagsColours, none of paramters values (audi, mercedez benz) is on its tagsCars values
Hello everyone, I'm having issues when trying to search on ElasticSearch and look for in arrays with values, and when parameters have no values.
If anyone could helpe me on this I'd appreciatte.
I did this:
termsQuery = Query<StructuredData>.Terms(t => t.Field(f =>f.TagsColours).Terms(dataToSearch.TagsColours));
termsQuery = termsQuery && Query<StructuredData>.Terms(t => t.Field(f =>f.TagsCars).Terms(dataToSearch.TagsCars));
and I stopped here (did not add third filter) because I could not mix two filters together
dataToSearch has the values from parameters (same structure object, cause .Search makes me do that here .Search()
var settings = new ConnectionSettings(node);
var response = new ElasticClient(settings)
.Search<StructuredData>(
s => s.AllIndices()
.AllTypes()
.From(0)
.Size(50)
.Query(_ => termsQuery)
);
But I'm having problems when using more than 1 filter..
any ideas? is ".Terms" the correct property?

If you are using regular mappings on ES 5 > This will get you results you want. If not you will need to change the mapping.
QueryContainer query = null;
if(dataToSearch.TagsColours != null && dataToSearch.TagsCars.Length > 0)
{
query = Query<StructuredData>.Terms(t=>t.Field("tagsColours.keyword").Terms(dataToSearch.TagsColours));
}
if(dataToSearch.TagsColours != null && dataToSearch.TagsCars.Length > 0)
{
var q = Query<StructuredData>.Terms(t=>t.Field("tagsCars.keyword").Terms(dataToSearch.TagsCars));
query = query == null ? q : query && q;
}
if(dataToSearch.TagsKeywords != null && dataToSearch.TagsKeywords.Length > 0)
{
var q = Query<StructuredData>.Terms(t=>t.Field("tagsKeywords.keyword").Terms(dataToSearch.TagsKeywords));
query = query == null ? q : query && q;
}
The problem you are having is that the term query is done on a non-analyzed value and default text fields use standard analyzer. As of 5 they added keyword sub field that uses the keyword analyzer it essentially just places the terms as is and you can do a search by raw values. The standard analyzer dose tokenization for words and lowercases all the terms so it was unable to find Audi because the term was audi. If you want to just lowercase the input string this will not solve the Mercedes Benz problem since in the standard terms this will became mercedes a benz terms two terms instead of one in other words terms will return results if you put mercedes or benz but not mercedes benz. If you want to da a case insensitive search with the match query you will need to add a custom analyzer.

Getting the values from this specific JSON

I'm trying to get all of the "last" values from this JSON here:
{"btc":{
"usd": {
"bitfinex": {
"last": "1191.60",
"volume": "1.99324e+7"
},
"bitstamp": {
"last": "1193.06",
"volume": "8.73693e+6"
},
"btce": {
"last": "1174.27",
"volume": "6.03521e+6"
}
}
}
But for some reason I can only access "btc" and "usd". I can't get anything out of it including the "last" values. Here is the code i'm using:
private string GetPrice()
{
WebClient wc = new WebClient();
var data = wc.DownloadString("http://preev.com/pulse/units:btc+usd/sources:bitfinex+bitstamp+btce");
JObject o = JObject.Parse(data);
string response = o["btc"].ToString();
return response;
}
If I change it to:
o["last"].ToString();
It just doesn't return anything. Can someone please provide me with a solution? I also tried making a key/value dict out of it and looping over each pair. Did not work.

The JObject structure is similar to a class with properties, so the first-level indexer ["btc"] returns another object that you have to query for its own properties ["usd"]
You can also opt for using JObject.SelectToken, generally not a bad idea. Other answers have shown how to chain the indexers but that's hard to read and maintain. Instead you can do:
jObj.SelectToken("btc[0].usd[0].bitstamp[0].last").ToString();
Further you can use the power of this syntax for other queries:
// a list o all the 'last' values
jObj.SelectTokens("btc.usd.*.last").Select(t=>t.ToString()).ToList();
Another advantage, if you're building a more complex system, is that you could put the queries in a config file or attributes etc to make them more manageable or deploy logic changes without rebuilding.
Yet another approach would be to build your own class structure and deserialize your json into it, so you have strongly typed values (double instead of string for the values for example)
public class btc {
public usd usd {get;set;}
}
public class usd....
var btcLoaded = JsonConvert.DeserializeObject<btc>(jsonString);
var lastBitstamp = btc.usd.bitstamp.last;

Use: o["btc"]["usd"]["bitfinex"]["last"].ToString() to get the 'last' value of 'bitfinex'.

After parsing the JSON whenever you index the o variable you are indexing from the root of the JSON. In order to access nested properties like 'last' you will need to index into the next level of the JSON as such:
var bitfinex = o["btc"]["usd"]["bitfinex"]["last"].ToString();
var bitstamp = o["btc"]["usd"]["bitstamp"]["last"].ToString();
var btce = o["btc"]["usd"]["btce"]["last"].ToString();
To reduce the repetition you could iterate over the properties under the btc.usd field.

if u want to all last values use this..
decimal[] lastValues = obj.SelectTokens("$..last").ToArray()
.Select(a => a.Parent.ToObject<decimal>()).ToArray();
if u want to dictionary, use this..
var dictionary = obj["btc"]["usd"].Select(a =>
new
{
Key = ((JProperty)a).Name,
Value = a.First["last"].ToObject<decimal>()
})
.ToDictionary(a => a.Key, a => a.Value);

Picking Out Simple Properties from Hierarchical JSON

* Despite the edit to my title by another user, I am seeking a solution that uses JSON.NET's library from C# *
A reply containing psuedocode is fine! :)
I'm trying to work with hierarchical data provided by a JSON dataset. I'm using C# and JSON.NET. I'm open to using Linq in general and Linq for JSON.NET in particular if it would help; otherwise, using non-Linq C#/JSON.NET is fine.
Ideally, I am trying to accomplish two things elegantly:
I want to extract JSON that represents each branch and that branch's own properties--not its child (nested) branch objects (I will explain more in a moment).
I want to track the parent node as I create my branch objects.
For further consideration, please refer to the following JSON excerpt:
{
"Branch1": {
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C",
"Branch2" : {
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C",
"Branch3" : {
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C"
}
}
}
}
Related to Goal 1 (from above):
Given JSON that is composed of nested JSON objects, I want to pick out only the simple (string) properties for each branch. For instance, I would like to extract the JSON for Branch1 that would contain only Prop1A, Prop1B, and Prop1C properties. I would then like to extract the JSON for Branch2 that would contain only Prop2A, Prop2B, and Prop2C properties, etc. I realize that I can represent the entire JSON as a JSON.NET JToken object then iterate through its Children() and look only for JTokenType.Property types, but perhaps there is a more elegant way to quickly pick out just the property types using Linq...? In the end, I would have three separate JSON objects that would look like this:
JSON Object 1:
{
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C"
}
JSON Object 2:
{
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C"
}
JSON Object 3:
{
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C"
}
Related to Goal 2 (from above):
Ideally, each extracted JSON above would also have a property indicating its parent. Thus, the final JSON objects would look something like this:
{
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C",
"Parent" : ""
}
And:
{
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C",
"Parent" : "Branch1"
}
And:
{
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C",
"Parent" : "Branch2"
}
Any thoughts?

You can use JContainer.DescendantsAndSelf() to find all objects in the JSON hierarchy, then for each object, loop through its properties and filter out those whose value is a JValue primitive. Thus the following query creates a List<JObject> containing the property names and values you require:
var root = (JContainer)JToken.Parse(jsonString);
var query1 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
select new JObject(l); // And return a JObject
var list1 = query1.ToList();
To always skip the root object even if it has primitive properties, use JContainer.Descendants(). And if you really only want string-valued properties (rather than primitive properties), you can check the JToken.Type property:
let l = o.Properties().Where(p => p.Value.Type == JTokenType.String) // Select their string-valued properties
The query can be enhanced to include a synthetic "Parent" property giving the name of the immediate parent property containing the object, using JToken.Ancestors:
var query2 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list2 = query2.ToList();
However, in your desired output you seem to want the property name of the parent of the object, rather than the property name of the object. If so, you can do:
var query3 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Skip(1).Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list3 = query3.ToList();
For the final query, if I do:
Console.WriteLine(JsonConvert.SerializeObject(list3, Formatting.Indented));
The following output is generated, showing the JObject list has the contents you require:
[
{
"Prop1A": "1A",
"Prop1B": "1B",
"Prop1C": "1C",
"Parent": ""
},
{
"Prop2A": "2A",
"Prop2B": "2B",
"Prop2C": "2C",
"Parent": "Branch1"
},
{
"Prop3A": "3A",
"Prop3B": "3B",
"Prop3C": "3C",
"Parent": "Branch2"
}
]
Note that if the JSON objects themselves have a property named "Parent", the JObject constructor may throw a duplicated key exception.