We Keep Coding

How to update over 100K items in MongoDB efficiently when new value is a calculation result from another collection using

I have a large list (100K+) where have to update all products by getting values from another collection. Basically I have to get the lowest offer and the number of offers for each product.
I am using C# driver and using a for loop it is easily achievable to do this, but when you have over 100K products and within each loop you need to do two calls to the productOffers collection (1: get lowest offer, 2: get count of offers), it becomes unacceptably slow.
There must be a better way of doing this. I read about JS procedures, but seems like everyone advice against it and not sure if you can run something like this within a JS procedures. I have also thought about running the loop parallel, but that sounds unstable and probably will run into network or other type of errors.
Product example:
{
"_id": "..",
"minPrice: "40",
"numberOfOffers": "2"
}
Product offer example:
{
"_id": "..",
"productId": ".."
"price: "40"
}

Starting from mongo v4.2 you can merge results of aggregation pipeline into documents in other collection https://docs.mongodb.com/manual/reference/operator/aggregation/merge/
The aggregation query on productOffers to calculate minPrice and count per product id and update products with results:
db.productOffers.aggregate([
{ $group: {
_id: "$productId",
minPrice: {$min: "$price"},
numberOfOffers: {$sum: 1}
} },
{ $merge : { into : "products" } }
])
Think about what you gonna do with product offers that has no matching product id and add whenNotMatched option to the $merge stage to insert or ignore the document or fail the query.
The whole thing will run on mongodb side so you will not transfer data to and from the app.

Query a dynamic-key nested dictionary structure in Cosmos DB with LINQ to SQL in C#

I want to query for values in a dynamic-key nested dictionary structure in order to apply a permission filter using LINQ to SQL (preferably utilizing index queries and not causing scans).
The JSON structure I have contains a Permissions property which is basically just a C# dictionary<string, dictionary<string, dictionary<string, string>>>
JSON example:
{
"Permissions": {
"Account 1 GUID": {
"Location": "Location1 GUID"
},
"Account 2 GUID: {
"Location": "Location 2 GUID"
},
"Account 3 GUID": {
"Location": "Location 3 GUID"
}
}
}
The account key properties are dynamic, meaning they are in fact the GUIDs of the accounts owning the nested location.
When a query is performed I have a list of location and account IDs that should be used to perform the permission filtering by giving access only to those locations.
The filtering query I want to apply is SIMILAR TO SOMETHING like this (THIS IS WHAT I WANT TO ACHIEVE):
var allowedLocations = new List<string> { "Location 1 GUID", "Location 2 GUID" };
var result = AllDocsIQueryable.Where(model =>
model.Permissions.Values.Where(secondLevel =>
secondLevel.Values.Where(thirdLevel =>
allowedLocations.Contains(thirdLevel)))).ToList();
As you can see, I want to filter the docs a user can see to those the user has defined an access to, by querying for docs that contains any of the location IDs in my access list.
I have no idea how to go about construction the query for querying a structure like this. I have tried many combinations of .Select and .SelectMany queries, but nothing I have tried succeeds, all I have tried yields no results or fails.
A query like the following yields the correct results, but I would really like to avoid having to build this query dynamically with predicate builder for example:
var result = AllDocsIQueryable.Where(model =>
(model.Permissions["Account 1 GUID"]["Location"].IsDefined() &&
model.Permissions["Account 1 GUID"]["Location"] == "Location 1 GUID")
||
(model.Permissions["Account 2 GUID"]["Location"].IsDefined() &&
model.Permissions["Account 2 GUID"]["Location"] == "Location 2 GUID")
// .. etc ..
).ToList();
I hope my question is clear and there is a LINQ to SQL/Cosmos/NoSQL/C# super wiz that can help me construct this query? Maybe someone knows that I can't construct this query at all, and I need to change the JSON structure (which I know would be preferable)? I need to query using LINQ and not SQL query string!
Conclusion:
It seems, as I was afraid of, that there's just no way of building the query.
As far as I know, and have tried, and read, there's just no way to do it without building the dynamic account keys into the query.
I've changed the schema to include arrays instead.
"Permissions": [
{
"LocationId": "Loc1",
"AccountId": "Acc1"
},
{
"LocationId": "Loc2",
"AccountId": "Acc2"
}
]

I'm afraid there is no way you can achieve that just using LINQ. Here are your options:
Use a UDF to retrieve an array of locations from a permissions object. Unfortunately, you can't call a UDF from LINQ (at least not in SDK v3 - you can in SDK v2), so you need to use raw SQL. (Actually there is hacky way.) This solution is bad because it's really slow.
Leave your schema but introduce an additional property that contains a list of locations that occur nested within your permissions object. Now you can easily use LINQ, and the query is fast.

OData query filter in Get request URL

I have an OData end point using OData V4. When I do a GET
https://localhost:323/Staff?$Filter=Company+eq+Microsoft
then I would get the multiple people back each with this basic JSON data:
"Company": "Microsoft",
"LastName": "Bob",
"FirstName: "Saget",
"Details": [
{
"Age": "63",
"Sex": "Male"
}]
How would I write a URL to query using OData convention if I want to find all staff who work for Microsoft and has age bigger than 30?
My attempt failed with couple of the version of URL below:
https://localhost:323/Staff?$Filter=Company+eq+Microsoft+and+Details/Age+gt+30
https://localhost:323/Staff?$Filter=Company+eq+Microsoft+and+$expand=Details($Filter=Age+gt+30)

I tried your examples and had trouble with using $Filter as opposed to $filter. When I used that on OData's own official website endpoints, it did not like $Filter but would work with $filter.
Your example might work if you set $Filter to $filter ? Have you tried it?
https://localhost:323/Staff?$filter=Company+eq+Micrsofoft+and+Details/Age+gt+30
My answer above is based off of trying it on OData's website (copy and paste this and try it yourself, this works and uses nested property in the filter):
https://services.odata.org/V4/OData/OData.svc/PersonDetails?$filter=(Address/City eq 'Boise' and PersonID gt 0)
So I think this will work:
https://localhost:323/Staff?$filter=(Company eq 'Microsoft' and Details/Age gt 30)
Btw, "Microsoft" was misspelled in your example query. That too can effect your results and expected output, since your JSON example has it spelled correctly.

MongoDB Delete records with date field less than days from another field

I have a collection which stores application logs in documents:
{
"_id" : ObjectId("5d92d5d01518a620ccaf015c"),
"MessageType" : "ApplocationLog",
"FireAndForget" : true,
"CreatedTimestamp" : ISODate("2019-10-01T06:28:00.198+01:00"),
"OriginReference" : "OriginReference",
"OriginName" : "OriginName",
"LogMessage" : "The log message",
"RetentionDays" : 1,
"LogSeverity" : "Error",
"ApplicationUserContextId" : "User1",
"ApplicationUserContextName" : "User1Name",
"Exception" : null,
"ErrorRelatedObjects" : null
}
I need to run a query to delete all records from the collection where the Current System Date minus 'RetentionDays' is greater than the 'CreatedTimestamp' of the document, i.e. remove expired log documents.
I am using MongoDB C# driver in my application and trying to figure the best way to do this. I have considered adding an expiry date when inserting the document which in hindsight might make things easier? But I still need to deal with the existing records using a query.
I have not got very far but started trying to work out the criteria for finding all records that have expired:
db.ApplicationLog.find({
"CreatedTimestamp": {
$gte:
new Date(new Date().setDate(new Date().getDate()-1))
}
})
If I could subtract the 'RetentionDays' instead of the hardcoded 1 in the above maybe I could then add this to the criteria of a delete query? I'm new to MongoDB so struggling a bit with this.

The Bad News is you cant really do that in one query.
You will have to first "calculate" the new field and then use the documents that matched and call a remove operator, this can be done fairly easily using an aggregation by subtracting from the date and matching documents with a negative result.
However if i may suggest a solution i personally find better:
Use TTL index's this way each document will expire when its time for it to do so instead of you having to maintain a cron'd deletion query.
Here is an example for a "dynamic" TTL index that you can use as reference.

One solution is to aggregate on ApplicationLog and filter the documents which need to be retained. Finally, replace the existing data with the output of aggregation.
For example:
db.ApplicationLog.aggregate([
{
$match:{
$expr:{
$gt:[
"$CreatedTimestamp",
{
$toDate:{
$subtract:[
new Date(),
{
$multiply:[
"$RetentionDays",
86400000
]
}
]
}
}
]
}
}
},
{
$out:"ApplicationLog"
}
])

Picking Out Simple Properties from Hierarchical JSON Part II

I thought this question might be ill-formed, so I created a "sister" question that's far more to-the-point, which specific output cited. Please see to:
Querying JSON Nested Arrays with Linq, JSON.NET, C#
If that question gets answered before this one, I'll try to answer this one myself using the information from the other question... :) Thanks!
In a previous question (Picking Out Simple Properties from Hierarchical JSON), I asked how to pick up simple properties from hierarchical JSON. The answer there [pasted as a Linq query at the end of this post] is extremently useful (and, since posting that, I've studied quite a bit about Linq and JSON.NET). So I'm not using this forum because I'm lazy--I use it when I'm really stuck and can't seem to find the answers in the books I have access to.
I'm stumped on how to continue with the example provided by my previous question. This question builds on the previous one, so here (as succinctly as I could express) is what I'd love to learn how to do in a single Linq query.
To recap: I'm working with dynamic JSON like this (it is more complex than the JSON presented in my earlier Part I question because it contains arrays):
{
"Branch1": {
"Prop1A": "1A",
"Prop1B": "1B",
"Prop1C": "1C",
"Branch2": {
"Prop2A": "2A",
"Prop2B": "2B",
"Prop2C": "2C",
"Branch3": [{
"Prop3A": "3A",
"Prop3B": "3B",
"Prop3C": "3C"
}, {
"Prop3D": "3D",
"Prop3E": "3E",
"Prop3F": "3F"
}, {
"Prop3G": "3G",
"Prop3H": "3H",
"Prop3I": "3I"
}]
},
"Branch4": [{
"Prop4A": "4A",
"Prop4B": "4B",
"Prop4C": "4C"
}, {
"Prop4E": "4E",
"Prop4F": "4F",
"Prop4G": "4G"
}, {
"Prop4H": "4H",
"Prop4I": "4I",
"Prop4I": "4I"
}]
}
}
As you can see, the dynamic JSON is composed of hierarchical objects, and these objects are JSON objects, JSON arrays, and JSON properties.
Ultimately, I want to convert this JSON to a List object I can work with in C#. I plan to use that List object to effectively process each JSON branch in document order from the top, down.
Each item in the List collection would be a JObject; each of these objects would have a synthetic "parent" string property that would point back to the branch under which that JObject appears in the original JSON (my examples below explain what I mean by "parent"). [The previous question correctly came up with a solution for this "parent" value, so that's not too relevant to this question... What's new/relevant here is dealing with JArray objects in the JSON...]
The key is that I want each List item object to contain only the string-value properties for the object. For example, Branch1 has string properties Prop1A, 1B, and 1C. Thus, I would want query[0] to contain:
{"Prop1A":"1A","Prop1B":"1B","Prop1C":"1C", Parent:""}
Next, I would want query[2] to contain the string-value properties for Branch2:
{"Prop2A":"2A","Prop2B":"2B","Prop2C":"2C", Parent:"Branch1"}
Next, I would want query[2] to contain the string properties for only Branch3, but because Branch3 is an array of objects, I would want that array to end up together in query[2]:
[
{"Prop3A": "3A","Prop3B": "3B","Prop3C": "3C"},
{"Prop3D": "3D","Prop3E": "3E","Prop3F": "3F"},
{"Prop3G": "3G","Prop3H": "3H","Prop3I": "3I"}
]
Notice that this Branch doesn't yet have a reference to its "Parent"... I'd be happy getting an array in query[2] that looks like the above. (I plan to use dbc's logic to add a "Parent" property to each array element or figure out a way to create a new JObject that contains the array and cites the Parent only once):
[{"Prop3A": "3A","Prop3B": "3B","Prop3C": "3C","Parent":"Branch2"},
{"Prop3D": "3D","Prop3E": "3E","Prop3F": "3F","Parent":"Branch2"},
{"Prop3G": "3G","Prop3H": "3H","Prop3I": "3I","Parent":"Branch2"}
]
So, as you can see:
I would want any JSON branch that is not an array to be inserted as a new JObject in the query result along with only its string properties and a reference to its parent branch.
I would want any JSON branch that is an array to be inserted as a new JObject array in the query result along with only its string properties and a reference to its parent branch.
The trouble I had solving this on my own has to do with figuring out how to create the "if myJsonObject is JArray" condition in Linq and assigning just the string-property properties when the branch is not an array and iterating the array's elements when it is a JArray. I suspect I need to somehow leverage the ? : ternary expression, but I don't exactly know how to do that.
The query from the previous question is here:
var query3 = from o in root.DescendantsAndSelf().OfType<JObject>() // Find objects
let l = o.Properties().Where(p => p.Value is JValue) // Select their primitive properties
where l.Any() // Skip objects with no properties
// Add synthetic "Parent" property
let l2 = l.Concat(new[] { new JProperty("Parent", o.Ancestors().OfType<JProperty>().Skip(1).Select(a => a.Name).FirstOrDefault() ?? "") })
select new JObject(l2); // And return a JObject.
var list3 = query3.ToList();
That code doesn't handle arrays in the way described above.

A suitable and more general answer to this question effectively appears in my related post:
Picking Out Simple Properties from Hierarchical JSON