Searching in multiple elasticsearch indexes using NEST - c#

I am trying to search for a text in elasticsearch using nest 7.10.1. I want to search in two different indexes, I get a response in the form of documents, but I cannot access its properties because the result has the combination of two indexes. Below is the code I tried. Both the indexes has same properties. What do I use in the foreach loop to access the key and values of the result documents.
public void searchIndices(string query) {
var response = client.Search<object>(
s => s.Index("knowledgearticles_index, index2")
.Query(q => q.Match(m => m.Field("locationName")
.Query(query))));
Console.WriteLine(response.Documents);
foreach(object r in response.Documents) {
}
}
I am using elasticsearch 7.10.2

Each raw hit coming back in the search response has the _index meta field associated with it:
"hits" : {
"total" : {
"value" : 91,
"relation" : "eq"
},
"hits" : [
{
"_index" : "knowledgearticles_index", <---
"_type" : "_doc",
"_id" : "r_oLl3cBZOT6A8Qby8Qd",
"_score" : 1.0,
"_source" : {
...
}
}
Now, in NEST,
.Documents is a convenient shorthand for retrieving the _source for each hit
-- meaning that you'll have lost access to the meta properties.
So the trick is to use a loop like this instead:
foreach (var hit in response.HitsMetadata.Hits) {
Console.WriteLine(hit);
}

If the two indices that you're searching over contain different JSON structures, then the T in Search<T> will need to be a type that the differing JSON structures can be deserialize to. object will work as per the example in your question, but then there is no typed access for any of the properties.
A simple approach to overcoming this is to hook up the JsonNetSerializer and then use JObject for T
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings =
new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default);
var client = new ElasticClient(connectionSettings);
var response = client.Search<JObject>(s => s
.Index("knowledgearticles_index, index2")
.Query(q => q
.Match(m => m
.Field("locationName")
.Query(query)
)
)
);
We now have a way of accessing properties on JObject, but will still need to handle the type of each property.

Related

How get List of JSON Objects from within a JSON object (non-array) (NOT Deserializing)

I'm using C# with Json.NET NuGet.
I have JSON that looks like this:
{
"pre" : "",
"options": {
"0001" : {
"id" : "0001",
"desc" : "first"
},
"0002" : {
"id" : "0002",
"desc" : "second"
},
"0003" : {
"id" : "0003",
"desc" : "third"
}
},
"post" : ""
}
How can I query the above Json to get a List<JObject> with the 3 option items in it?
Or/Also How can I get just the second item where item 2 should be:
{
"id" : "0002",
"desc" : "second"
}
I've tried stuff like
var items = json.SelectTokens("options[*]").ToList();
and
var item = json.SelectTokens("options[1]");
but those clearly don't work.
EDIT:
In case I wasn't clear I DO NOT want to deserialize. I want a List<JObject>.
Your json is valid but your thinking of this json is not quite accurate.
You are thinking that options have a list of objects that you want to iterate over, but, thats not the case. options is not a list but an object that has more objects within it.. Not an array.
You can access each of the element within the JObject by first looking up its properties. Properties are 0001, 0002 etc. Once you have those, you can iterate over the properties of options and get the values you need.
JObject options = (JObject)JObject.Parse(json)["options"];
// Get a list of all tokens within this object.
List<JObject> allObjects = new List<JObject>();
foreach (var node in options.Properties())
allObjects.Add((JObject)options[node.Name]);
// Access the IDs
allObjects.ForEach(x => Console.WriteLine(x["id"].ToString()));
// Access the 2nd ID only
Console.WriteLine(); // Just to space it out.
Console.WriteLine(allObjects[1]["id"].ToString());
Output
0001
0002
0003
0002
You could create the required List using Linq. For example,
var list = ((JObject)JObject.Parse(str)["options"])
.Properties()
.Select(x=>x.Value)
.Cast<JObject>()
.ToList();
For accessing the second element, you could use
var secondId = (string)list[1]["id"];
var secondDesc = (string)list[1]["desc"];
I was actually close but trying too hard:
var items = json.SelectToken("options").ToList();
I was unable to figure out how to get a single option from a query but since I got the whole list I did it like this:
var item = json.SelectToken("options").ToList()[1];

Getting the values from this specific JSON

I'm trying to get all of the "last" values from this JSON here:
{"btc":{
"usd": {
"bitfinex": {
"last": "1191.60",
"volume": "1.99324e+7"
},
"bitstamp": {
"last": "1193.06",
"volume": "8.73693e+6"
},
"btce": {
"last": "1174.27",
"volume": "6.03521e+6"
}
}
}
But for some reason I can only access "btc" and "usd". I can't get anything out of it including the "last" values. Here is the code i'm using:
private string GetPrice()
{
WebClient wc = new WebClient();
var data = wc.DownloadString("http://preev.com/pulse/units:btc+usd/sources:bitfinex+bitstamp+btce");
JObject o = JObject.Parse(data);
string response = o["btc"].ToString();
return response;
}
If I change it to:
o["last"].ToString();
It just doesn't return anything. Can someone please provide me with a solution? I also tried making a key/value dict out of it and looping over each pair. Did not work.
The JObject structure is similar to a class with properties, so the first-level indexer ["btc"] returns another object that you have to query for its own properties ["usd"]
You can also opt for using JObject.SelectToken, generally not a bad idea. Other answers have shown how to chain the indexers but that's hard to read and maintain. Instead you can do:
jObj.SelectToken("btc[0].usd[0].bitstamp[0].last").ToString();
Further you can use the power of this syntax for other queries:
// a list o all the 'last' values
jObj.SelectTokens("btc.usd.*.last").Select(t=>t.ToString()).ToList();
Another advantage, if you're building a more complex system, is that you could put the queries in a config file or attributes etc to make them more manageable or deploy logic changes without rebuilding.
Yet another approach would be to build your own class structure and deserialize your json into it, so you have strongly typed values (double instead of string for the values for example)
public class btc {
public usd usd {get;set;}
}
public class usd....
var btcLoaded = JsonConvert.DeserializeObject<btc>(jsonString);
var lastBitstamp = btc.usd.bitstamp.last;
Use: o["btc"]["usd"]["bitfinex"]["last"].ToString() to get the 'last' value of 'bitfinex'.
After parsing the JSON whenever you index the o variable you are indexing from the root of the JSON. In order to access nested properties like 'last' you will need to index into the next level of the JSON as such:
var bitfinex = o["btc"]["usd"]["bitfinex"]["last"].ToString();
var bitstamp = o["btc"]["usd"]["bitstamp"]["last"].ToString();
var btce = o["btc"]["usd"]["btce"]["last"].ToString();
To reduce the repetition you could iterate over the properties under the btc.usd field.
if u want to all last values use this..
decimal[] lastValues = obj.SelectTokens("$..last").ToArray()
.Select(a => a.Parent.ToObject<decimal>()).ToArray();
if u want to dictionary, use this..
var dictionary = obj["btc"]["usd"].Select(a =>
new
{
Key = ((JProperty)a).Name,
Value = a.First["last"].ToObject<decimal>()
})
.ToDictionary(a => a.Key, a => a.Value);

Picking Out Simple Properties from Hierarchical JSON

* Despite the edit to my title by another user, I am seeking a solution that uses JSON.NET's library from C# *
A reply containing psuedocode is fine! :)
I'm trying to work with hierarchical data provided by a JSON dataset. I'm using C# and JSON.NET. I'm open to using Linq in general and Linq for JSON.NET in particular if it would help; otherwise, using non-Linq C#/JSON.NET is fine.
Ideally, I am trying to accomplish two things elegantly:
I want to extract JSON that represents each branch and that branch's own properties--not its child (nested) branch objects (I will explain more in a moment).
I want to track the parent node as I create my branch objects.
For further consideration, please refer to the following JSON excerpt:
{
"Branch1": {
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C",
"Branch2" : {
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C",
"Branch3" : {
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C"
}
}
}
}
Related to Goal 1 (from above):
Given JSON that is composed of nested JSON objects, I want to pick out only the simple (string) properties for each branch. For instance, I would like to extract the JSON for Branch1 that would contain only Prop1A, Prop1B, and Prop1C properties. I would then like to extract the JSON for Branch2 that would contain only Prop2A, Prop2B, and Prop2C properties, etc. I realize that I can represent the entire JSON as a JSON.NET JToken object then iterate through its Children() and look only for JTokenType.Property types, but perhaps there is a more elegant way to quickly pick out just the property types using Linq...? In the end, I would have three separate JSON objects that would look like this:
JSON Object 1:
{
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C"
}
JSON Object 2:
{
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C"
}
JSON Object 3:
{
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C"
}
Related to Goal 2 (from above):
Ideally, each extracted JSON above would also have a property indicating its parent. Thus, the final JSON objects would look something like this:
{
"Prop1A" : "1A",
"Prop1B" : "1B",
"Prop1C" : "1C",
"Parent" : ""
}
And:
{
"Prop2A" : "2A",
"Prop2B" : "2B",
"Prop2C" : "2C",
"Parent" : "Branch1"
}
And:
{
"Prop3A" : "3A",
"Prop3B" : "3B",
"Prop3C" : "3C",
"Parent" : "Branch2"
}
Any thoughts?
You can use JContainer.DescendantsAndSelf() to find all objects in the JSON hierarchy, then for each object, loop through its properties and filter out those whose value is a JValue primitive. Thus the following query creates a List<JObject> containing the property names and values you require:
var root = (JContainer)JToken.Parse(jsonString);
var query1 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
select new JObject(l); // And return a JObject
var list1 = query1.ToList();
To always skip the root object even if it has primitive properties, use JContainer.Descendants(). And if you really only want string-valued properties (rather than primitive properties), you can check the JToken.Type property:
let l = o.Properties().Where(p => p.Value.Type == JTokenType.String) // Select their string-valued properties
The query can be enhanced to include a synthetic "Parent" property giving the name of the immediate parent property containing the object, using JToken.Ancestors:
var query2 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list2 = query2.ToList();
However, in your desired output you seem to want the property name of the parent of the object, rather than the property name of the object. If so, you can do:
var query3 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Skip(1).Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list3 = query3.ToList();
For the final query, if I do:
Console.WriteLine(JsonConvert.SerializeObject(list3, Formatting.Indented));
The following output is generated, showing the JObject list has the contents you require:
[
{
"Prop1A": "1A",
"Prop1B": "1B",
"Prop1C": "1C",
"Parent": ""
},
{
"Prop2A": "2A",
"Prop2B": "2B",
"Prop2C": "2C",
"Parent": "Branch1"
},
{
"Prop3A": "3A",
"Prop3B": "3B",
"Prop3C": "3C",
"Parent": "Branch2"
}
]
Note that if the JSON objects themselves have a property named "Parent", the JObject constructor may throw a duplicated key exception.

elasticsearch autocompletion mapping using nest

I am using nest to implement elasticsearch in .net and am new to it. I am trying to map suggesters please help me with it. how to do that in c# using nest
curl -X PUT localhost:9200/songs/song/_mapping -d '{
"song" : {
"properties" : {
"name" : { "type" : "string" },
"suggest" : { "type" : "completion",
"index_analyzer" : "simple",
"search_analyzer" : "simple",
"payloads" : true
}
}
}
}'
Find the complete code below. It creates a new ElasticClient object and then adds the mapping song to the index songs. Make sure that index songs already exists before you execute this code. You can also create the index songs before creating the mapping through code anyways. I'll leave that upto you to figure out. Find an exhaustive example of how mappings can be created in Nest here.
var client = new ElasticClient(new ConnectionSettings(new Uri("http://localhost:9200")));
var response = client.Map<object>(d => d
.Index("songs")
.Type("song")
.Properties(props => props
.String(s => s
.Name("name"))
.Completion(c => c
.Name("suggest")
.IndexAnalyzer("simple")
.SearchAnalyzer("simple")
.Payloads())));
Debug.Assert(response.IsValid);

MongoDB Get names of all keys in collection using c#

How can I get names of all the keys in a MongoDB collection using c#.
I am using mongocsharpdriver.
I am able to get all the records using
var collection1 = database1.GetCollection("Games").FindAll();
Now I need the key names to display/use it. I need key names of collection that I have fetched.
e.g. If I have collection which
{ "_id" : ObjectId("c3"), "GameID" : 20, "GameName" : "COD5", "Cost" : 100}
{ "_id" : ObjectId("c4"), "GameID" : 21, "GameName" : "NFS", "Publisher" : "EA"}
{ "_id" : ObjectId("c5"), "GameID" : 22, "GameName" : "CS", "Cost" : 200}
So I should get list of keys like GameID, GameName, Cost, Publisher.
I also went through MongoDB Get names of all keys in collection but was not able to implement it, didnot understood it & got problem with mapreduce.
Inspired from the link in your question:
string map = #"function() {
for (var key in this) { emit(key, null); }
}";
string reduce = #"function(key, stuff) { return null; }";
string finalize = #"function(key, value){
return key;
}";
MapReduceArgs args = new MapReduceArgs();
args.FinalizeFunction = new BsonJavaScript(finalize);
args.MapFunction = new BsonJavaScript(map);
args.ReduceFunction = new BsonJavaScript(reduce);
var results = collection1.MapReduce(args);
foreach (BsonValue result in results.GetResults().Select(item => item["_id"]))
{
Console.WriteLine(result.AsString);
}
A very inefficient, but simple way to do this is
HashSet<string> keys = new HashSet<string>();
foreach (var rover in collection1.FindAll())
{
rover.Names.ToList().ForEach(p => keys.Add(p));
}
Keep in mind that finding the set of keys, no matter how it's implemented, will always have to iterate the entire collection, so it will be terribly slow on larger collections.
It makes sense to use Map/Reduce for this problem on larger collections, because that avoids all the data transfer and deserialization overhead that is incurred by the solution I posted above, but you should generally try to avoid doing something like this at all. At least, don't do it where it's needed synchronously.
If you somehow need to know the set of all fields quickly, you're better off keeping track of the fields during writes and store the list of fields in a separate collection somewhere.

Categories