We Keep Coding

Elastic - updating multiple documents in a single request

I need to update several thousand items every several minutes in Elastic and unfortunately reindexing is not an option for me. From my research the best way to update an item is using _update_by_query - I have had success updating single documents like so -
{
"query": {
"match": {
"itemId": {
"query": "1"
}
}
},
"script": {
"source": "ctx._source.field = params.updateValue",
"lang": "painless",
"params": {
"updateValue": "test",
}
}
}
var response = await Client.UpdateByQueryAsync<dynamic>(q => q
.Index("masterproducts")
.Query(q => x.MatchQuery)
.Script(s => s.Source(x.Script).Lang("painless").Params(x.Params))
.Conflicts(Elasticsearch.Net.Conflicts.Proceed)
);
Although this works it is extremely inefficient as it generates thousands of requests - is there a way in which I can update multiple documents with a matching ID in a single request? I have already tried Multiple search API which it would seem cannot be used for this purpose. Any help would be appreciated!

If possible, try to generalize your query.
Instead of targeting a single itemId, perhaps try using a terms query:
{
"query": {
"terms": {
"itemId": [
"1", "2", ...
]
}
},
"script": {
...
}
}
From the looks of it, your (seemingly simplified) script sets the same value, irregardless of the document ID / itemId. So that's that.
If the script does indeed set different values based on the doc IDs / itemIds, you could make the params multi-value:
"params": {
"updateValue1": "test1",
"updateValue2": "test2",
...
}
and then dynamically access them:
...
def value_to_set = params['updateValue' + ctx._source['itemId']];
...
so the target doc is updated with the corresponding value.

JSON.NET - getting particular properties inside objects from Jarray

There is a series of JSON objects called "issues", which each have one or more "issue links", which have the following format:
// an issue link
{
"id": "000000",
"self": "some link",
"type": {
"id": "0000",
"name": "some name",
"inward": "is met by",
"outward": "meets",
"self": "some link"
},
"outwardIssue": {
"id": "000000",
"key": "the required key",
"self": "some link",
"fields": {
// the remainder is not applicable
}
}
}
}
These "issue links" have been extracted as follows. Create a JArray for the JSON for the "issue" itself, and extract the child JObjects:
public void Deserialize(dynamic jsonObject)
{
// get the issue links
if (jsonObject["fields"]["issuelinks"]!=null)
{
JArray issueLinksArray = jsonObject["fields"]["issuelinks"];
var issueLinkObjects = issueLinksArray.Children();
foreach (var issueLink in issueLinkObjects)
{
// now need the "key" in the "outwardIssue" for this object, if the value of "inward" is "is met by".
}
}
}
How to go about extracting the value of the second property "key" of "outwardIssue"?

Not sure whether I fully understand but following excerpt gets u the value (or null if condition not met) like this.
var key = issueLink["type"]["inward"].ToString()=="is met by" ? issueLink["outwardIssue"]["key"]: null;
Hint: Try to avoid dynamic.
Nowadays loops can in certain conditions considered old-school. Think LINQ: The problem can be divided into smaller probs and distributed across multiple lines (think-steps).
The additional variables might improve readability. As the project grows, for loops are prone to span more and more lines. So if you just need requested values following might be of interest:
var inwardLinks = issueLinkObject.Where(i=>i["type"]["inward"].ToString()=="is met by");
var keys = inwardLinks.Select(iwl=>iwl["outwardIssue"]["key"]);

How to pull a specific value from dynamic JSON dictionary in C#

I'm trying to pull one specific piece of data from the json received from a geocoding request.
This is not a duplicate of the myriads of similar-sounding questions, because the json is dynamic and has extremely irregular arrays that change slightly between responses, so I can't create a model to parse to, neither can I navigate through it using key names.
This is the json of one response:
{
"Response":{
"MetaInfo":{
"Timestamp":"2019-07-28T13:23:04.898+0000"
},
"View":[
{
"_type":"SearchResultsViewType",
"ViewId":0,
"Result":[
{
"Relevance":1.0,
"MatchLevel":"houseNumber",
"MatchQuality":{
"City":1.0,
"Street":[
0.9
],
"HouseNumber":1.0
},
"MatchType":"pointAddress",
"Location":{
"LocationId":"NT_Opil2LPZVRLZjlWNLJQuWB_0ITN",
"LocationType":"point",
"DisplayPosition":{
"Latitude":41.88432,
"Longitude":-87.63877
},
"NavigationPosition":[
{
"Latitude":41.88449,
"Longitude":-87.63877
}
],
"MapView":{
"TopLeft":{
"Latitude":41.8854442,
"Longitude":-87.64028
},
"BottomRight":{
"Latitude":41.8831958,
"Longitude":-87.63726
}
},
"Address":{
"Label":"425 W Randolph St, Chicago, IL 60606, United States",
"Country":"USA",
"State":"IL",
"County":"Cook",
"City":"Chicago",
"District":"West Loop",
"Street":"W Randolph St",
"HouseNumber":"425",
"PostalCode":"60606",
"AdditionalData":[
{
"value":"United States",
"key":"CountryName"
},
{
"value":"Illinois",
"key":"StateName"
},
{
"value":"Cook",
"key":"CountyName"
},
{
"value":"N",
"key":"PostalCodeType"
}
]
}
}
}
]
}
]
}
}
I'm trying to get just the two coordinates inside NavigationPosition.
Before I start the tedious work of creating a method to manually pull out the data I need from the unparsed string, does anyone have a solution? Is there a way to iterate anonymously through the json arrays using a foreach or if statement?

If the response is dynamic then one option is to use dynamic deserialization
var dynamicObject = JsonConvert.DeserializeObject<dynamic>(jsonAsString);
You can iterate this object and write defensive code by checking null.
I am assuming that "NavigationPosition" will always lie under "location" and in turn under a list of "Result" under list of "View"
something like this.. (symbolic code)
if(dynamicObject.View != null && dynamicObject.View[0] != null && dynamicObject.View[0].Result[0] != null && dynamicObject.View[0].Result[0].Location != null )
{
var navigationPosition = dynamicObject.View[0].Result[0].Location.NavigationPosition;
// do something with it
}
It seems really cumbersome but if you have no control over json response, I think you should be able to get it right after some iterations :)

Multiple Filters and Updates in One Mongo Query [duplicate]

I have a collection which elements can be simplified to this:
{tags : [1, 5, 8]}
where there would be at least one element in array and all of them should be different. I want to substitute one tag for another and I thought that there would not be a problem. So I came up with the following query:
db.colll.update({
tags : 1
},{
$pull: { tags: 1 },
$addToSet: { tags: 2 }
}, {
multi: true
})
Cool, so it will find all elements which has a tag that I do not need (1), remove it and add another (2) if it is not there already. The problem is that I get an error:
"Cannot update 'tags' and 'tags' at the same time"
Which basically means that I can not do pull and addtoset at the same time. Is there any other way I can do this?
Of course I can memorize all the IDs of the elements and then remove tag and add in separate queries, but this does not sound nice.

The error is pretty much what it means as you cannot act on two things of the same "path" in the same update operation. The two operators you are using do not process sequentially as you might think they do.
You can do this with as "sequential" as you can possibly get with the "bulk" operations API or other form of "bulk" update though. Within reason of course, and also in reverse:
var bulk = db.coll.initializeOrderedBulkOp();
bulk.find({ "tags": 1 }).updateOne({ "$addToSet": { "tags": 2 } });
bulk.find({ "tags": 1 }).updateOne({ "$pull": { "tags": 1 } });
bulk.execute();
Not a guarantee that nothing else will try to modify,but it is as close as you will currently get.
Also see the raw "update" command with multiple documents.

If you're removing and adding at the same time, you may be modeling a 'map', instead of a 'set'. If so, an object may be less work than an array.
Instead of data as an array:
{ _id: 'myobjectwithdata',
data: [{ id: 'data1', important: 'stuff'},
{ id: 'data2', important: 'more'}]
}
Use data as an object:
{ _id: 'myobjectwithdata',
data: { data1: { important: 'stuff'},
data2: { important: 'more'} }
}
The one-command update is then:
db.coll.update(
'myobjectwithdata',
{ $set: { 'data.data1': { important: 'treasure' } }
);
Hard brain working for this answer done here and here.

Starting in Mongo 4.4, the $function aggregation operator allows applying a custom javascript function to implement behaviour not supported by the MongoDB Query Language.
And coupled with improvements made to db.collection.update() in Mongo 4.2 that can accept an aggregation pipeline, allowing the update of a field based on its own value,
We can manipulate and update an array in ways the language doesn't easily permit:
// { "tags" : [ 1, 5, 8 ] }
db.collection.updateMany(
{ tags: 1 },
[{ $set:
{ "tags":
{ $function: {
body: function(tags) { tags.push(2); return tags.filter(x => x != 1); },
args: ["$tags"],
lang: "js"
}}
}
}]
)
// { "tags" : [ 5, 8, 2 ] }
$function takes 3 parameters:
body, which is the function to apply, whose parameter is the array to modify. The function here simply consists in pushing 2 to the array and filtering out 1.
args, which contains the fields from the record that the body function takes as parameter. In our case, "$tag".
lang, which is the language in which the body function is written. Only js is currently available.

In case you need replace one value in an array to another check this answer:
Replace array value using arrayFilters

Using filters with JsonPath in C#

I'm using JsonPath for C# to query some JSON data. JsonPath doesn't come with its own parser, so as per Rick Sladkey's advice, I'm using Json.NET to parse my Json string into a collection of nested IDictionary objects, IList arrays, and primitives. Then I use JsonPath to filter it (after adding the class suggested in Rick Sladkey's answer).
For reference, here's the part of my code that actually handles all this:
// string exampleJsonString defined below
string query = "$.store.book[*].title" // we should get a list of all titles
// step 1: use Json.NET to parse the json string into a JObject
JObject parsedJson = JObject.Parse(exampleJsonString);
// step 2: use JsonPath with a custom ValueSystem (JsonNetValueSystem)
JsonPathContext context = new JsonPathContext
{ ValueSystem = new JsonNetValueSystem() };
// step 3: get results using JsonPath
List<object> toRet = context.SelectNodes(parsedJson,
query).Select(node => node.Value).ToList();
The reason I was using JsonPath in the first place was because of its filter functionality. You can not only do normal queries like "$.store.book[*].title" (to get an array of all the titles in the book store), but also queries like "$.store.book[?(#.category == 'fiction')].title" (to get an array of all titles in the book store whose category matches 'fiction'). I need to be able to pass entire queries as a string, and so being able to filter while querying is extremely helpful.
Unfortunately, I'm having some trouble getting this filter functionality to work. I expect that I'll have to make adjustments to either the JsonNetValueSystem class (defined originally in the aforementioned stack overflow answer) or the JsonPath namespace (you can get JsonPath.cs from JsonPath's google code page). If there's some external tool or an alternative parsing mechanism to Json.NET that would allow me to keep JsonPath's filtering without having to write too much extra code, that would be ideal, but I'm pretty sure I'll have to alter either JsonNetValueSystem or JsonPath itself. (Both of these are fairly easy to alter since they're just .cs files, but I can't really dig into Json.NET without a lot more work.)
I can't actually seem to figure out where the original JsonPath code handles filtering, nor can I figure out why the JsonNetValueSystem class robs it of that functionality. Any advice for how to add the ability to filter in the query string would be very much appreciated. Even if it's just "don't mess with JsonPath, just change JsonNetValueSystem" or vice versa.
string exampleJsonString = "{
"store": {
"book": [ {
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
}, {
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}, {
"category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
}, {
"category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
} ],
"bicycle": [ {
"color": "red",
"price": 19.95,
"style": [ "city", "hybrid" ]
}, {
"color": "blue",
"price": 59.91,
"style": [ "downhill", "freeride" ]
} ]
}
}"

When you use the script expression in a query (the ?(...) part), you need to provide a ScriptEvaluator method to evaluate the script. Unfortunately they don't provide a default implementation for the C# version. You'll need to provide that implementation.
Out of the box, this won't be the most trivial problem to solve, you'll need to write an interpreter. You have a couple of options: use CodeDOM to compile and execute the script as C# code (or any language you would prefer), use Roslyn to parse the script and evaluate, whatever works.
A quick and dirty solution for this particular case would be to do something like this in your script evaluator method:
object EvaluateScript(string script, object value, string context)
{
if (script == "#.category == 'fiction'")
{
var obj = value as JObject;
return (string)obj["category"] == "fiction";
}
return null;
}
Here's another solution which utilizes IronPython to evaluate the script.
public class IronPythonScriptEvaluator
{
private Lazy<ScriptEngine> engine = new Lazy<ScriptEngine>(() => Python.CreateEngine());
private ScriptEngine Engine { get { return engine.Value; } }
private const string ItemName = "_IronPythonScriptEvaluator_item";
public object EvaluateScript(string script, object value, string context)
{
var cleanScript = CleanupScript(script);
var scope = Engine.CreateScope();
scope.SetVariable(ItemName, value);
return Engine.Execute<bool>(cleanScript, scope);
}
private string CleanupScript(string script)
{
return Regex.Replace(script, #"#", ItemName);
}
}

Another solution is to use Manatee.Json instead. It has a native JSONPath implementation and parser all built in (along with schema and a serializer). Moreover, you aren't required to represent the path as a string. Manatee.Json has a fluent interface that you can use to build paths, including expression support.
To represent $.store.book[*].title, you would have
var path = JsonPathWith.Name("store")
.Name("book")
.Array()
.Name("title");
For your example $.store.book[?(#.category == 'fiction')].title, you'd use
var path = JsonPathWith.Name("store")
.Name("book")
.Array(jv => jv.Name("category") == "fiction")
.Name("title");
What's more is that there is limited support for fields within those expressions as well.
If your path source is a string, Manatee.Json handles path parsing, too!