We Keep Coding

Find MongoDB document and only matching array elements w/ C# driver

I'm trying to return a document, and that document should have it's array filtered such that it only contains one item. I've seen many similar questions, but none of them deal with dynamic queries. There may be several constraints so I have to be able to keep adding to the filter.
{
"_id" : ObjectId("6058f722e9e41a3d243258dc"),
"fooName" : "foo1",
"fooCode" : 1,
"bar" : [
{
"barCode" : "123",
"barName" : "Rick's Cafe",
"baz" : [
{
"bazId" : "00",
"bazDescription" : "Ilsa"
},
{
"bazId" : "21",
"bazDescription" : "Victor"
}
]
},
{
"barCode" : "456",
"barName" : "Draco Tavern",
"baz" : [
{
"bazId" : "00",
"bazDescription" : "Rick Shumann"
}
]
}
]
}
This is my attempt, it returns a document who's array contains the barCode, and the array's entire contents are included.
Expression<Func<Foo, bool>> filter = x => x.FooCode == 1;
string barCode = "456"
if (!String.IsNullOrEmpty(barCode))
{
Expression<Func<Foo, bool>> newPred =
x => x.Bar.Any(s => s.BarCode == barCode);
filter = filter.CombineAnd(newPred);
}
var fooQuery =
_fooCollection
.Find(filter);
How do I remove non-matching array elements, but only if an array element was specified?

Unwind to convert the single document into a document per nested-array element in the shape of:
{
"_id" : ObjectId("6058f722e9e41a3d243258dc"),
"fooName" : "foo1",
"fooCode" : 1,
"bar": {
"barCode" : "123",
"barName" : "Rick's Cafe",
...
}
}
Match to find the element you want
Group to recombine the nested-array
So the resulting C# might look like:
var fooQuery = _fooCollection.Aggregate()
.Unwind("bar")
.Match(BsonDocument.Parse("{ 'bar.barcode': '"+ barCode + "'}"))
.Group(BsonDocument.Parse("{​​​​​ '_id':'$fooCode' }​​​​​"))

You need to use aggregate in MongoDB.
You can split the array elements with unwind, filter with match, select the keys that you want with project and group with common column like id or something.
MongoDB Aggregation docs: https://docs.mongodb.com/manual/aggregation/

in mongodb, what is the most efficient way to get the first and last document

I have documents like this:
class A
{
DateTime T;
...
}
and I would like to find the earliest and the newest document.
Is it better to do this:
var First = db.Collection.AsQueryable().OrderBy(_ => _.t).FirstOrDefault();
var Last = db.Collection.AsQueryable().OrderByDescending(_ => _.t).FirstOrDefault();
or,
var First = db.Collection.AsQueryable().OrderBy(_ => _.t).FirstOrDefault();
var Last = db.Collection.AsQueryable().OrderBy(_ => _.t).LastOrDefault();
or
var C = db.Collection.AsQueryable().OrderBy(_ => _.t);
var First = C.FirstOrDefault();
var Last = C.LastOrDefault();
I am wondering if there is anything with the underlying implementation that would change the speed between these options?
I am wondering if the sort has been done once already, is it possible that the result is cached and getting the first and the last elements would be faster?

Profiler becomes your friend when you're not sure about driver syntax. To enable logging for all queries you need to run on your database below statement:
db.setProfilingLevel(2)
Then to check last query executed on the database you need to run:
db.system.profile.find().limit(1).sort( { ts : -1 } ).pretty()
So for the first code snippet you will get:
"pipeline" : [ { "$sort" : { "t" : 1 } },
{ "$limit" : 1 }
]
"pipeline" : [ { "$sort" : { "t" : -1 } },
{ "$limit" : 1 }
]
For the second pair it prints
"pipeline" : [ { "$sort" : { "t" : 1 } },
{ "$limit" : 1 }
]
and throws NotSupportedException for LastOrDefault on my machine, if it works on your MongoDB driver's version you can check generated MongoDB statement using profiler
For the last one when you hover on c in your Visual Studio it prints
{aggregate([{ "$sort" : { "t" : 1 } }])}
but since it's of type IOrderedQueryable<T> it is only not materialized query so it will get executed on the database when you run FirstOrDefault generating the same aggregation body as previous statements. I'm getting NotSupportedException here as well. Here you can find a list of supported LINQ operators and both Last and LastOrDefault are not implemented so you need to sort descending.

How to fix freeze in update documents mongoDb through C#?

I got some freeze in update documents in mongoDb through C#. What i need to do to fix it?
I have MongoDb collection with ~230000 documents. Signature is
{
"_id" : ObjectId("5c5d4c3a086dad5e5ab8d397"),
"Uid" : "SomeiDString",
"SomeTextField" : "SomeText"
}
Indexes - two fields:
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "db.mycollection"
},
{
"v" : 2,
"unique" : true,
"key" : {
"Uid" : 1.0
},
"name" : "Uid_1",
"ns" : "db.mycollection",
"background" : true
}
In C# I make async operation UpdateOneAsync, with filter on Uid and update SomeTextField.
private async Task <bool> PutSomeTextMongoDb(string Uid, string SomeText) {
var collection = database.GetCollection<User>(MONGO_DB_COLLEСTION);
var filter = Builders<User>.Filter.Eq("Uid", Uid);
var update = Builders<User>.Update.Set("SomeText", SomeText);
var option = new UpdateOptions {IsUpsert = true};
var result = await collection.UpdateOneAsync(filter, update, option);
return result.ModifiedCount != 0;
}
Problem that I have: 5000 request got good avr time. Is about 2-3 ms. But some of them grow up to ~800-1000 ms... I know, it's only 5-6 long operation from 5k, but what is it? May be some locks for update operation, or when Mongo flush data i got some queue? mongostat and mongotop didn't answered me on this question...

Find if an element in array has value equal to parent element value

Let's say we have a collection of documents like this one:
{
"_id" : ObjectId("591c54faf1c1f419a830b9cf"),
"fingerprint" : "3121733676",
"screewidth" : "1920",
"carts" : [
{
"cartid" : 391796,
"status" : "New",
"cart_created" : ISODate("2017-05-17T13:50:37.388Z"),
"closed" : false,
"items" : [
{
"brandid" : "PIR",
"cai" : "2259700"
}
],
"updatedon" : ISODate("2017-05-17T13:51:24.252Z")
},
{
"cartid" : 422907,
"status" : "New",
"cart_created" : ISODate("2017-10-23T08:57:06.846Z"),
"closed" : false,
"items" : [
{
"brandid" : "PIR",
"cai" : "IrHlNdGtLfBoTlKsJaRySnM195U"
}
],
"updatedon" : ISODate("2017-10-23T09:46:08.579Z")
}
],
"createdon" : ISODate("2016-11-08T10:29:55.120Z"),
"updatedon" : ISODate("2017-10-23T09:46:29.486Z")
}
How do you extract only the documents where no item in the array $.carts have $.carts.closed set to true and $.carts.updatedon greater than $.updatedon minus 3 days ?
I know how to do find all the documents where no item in the array satisfy the condition $and: [closed: {$eq: true}, {updatedon: {$gt : new ISODate("2017-10-20T20:15:31Z")}}]
But how can you reference the parent element $.updatedon for the comparison?
In plain mongodb shell query language it would aleady be of help.
But I am actually accessing it using c# driver, so my query filter is like this:
FilterDefinition<_visitorData> filter;
filter = Builders<_visitorData>.Filter
.Gte(f => f.updatedon, DateTime.Now.AddDays(-15));
filter = filter & (
Builders<_visitorData>.Filter
.Exists(f => f.carts, false)
| !Builders<_visitorData>.Filter.ElemMatch(f =>
f.carts, c => c.closed && c.updatedon > DateTime.Now.AddDays(-15)
)
);
How can I replace DateTime.Now.AddDays(-15) with a reference to the document root element updatedon?

You can project the difference of carts.updatedon and updatedon and then filter out the results from this aggregation pipeline.
coll.aggregate([{'$unwind':'$carts'},
{'$match':{'closed':{'$ne':true}}},
{'$project':{'carts.cartid':1,'carts.status':1,'carts.cart_created':1,'carts.closed':1,'carts.items':1,'carts.updatedon':1,'updatedon':1,'diff':{'$subtract':['$carts.updatedon','$createdon']}}},
{'$match': {'diff': {'$gte': 1000 * 60 * 60 * 24 * days}}}])
days = 3 will filter out results more than 3 days difference documents.
I have just given the example of how you can use $subtract to find date difference and filter documents based on that.

well I was in a similar situation few days back.
I tackled it by using Jobject of Newtonsoft.Json.
Create a function to return bool which actually process each document take it as input.
Jobject jOb=Jobject.parse(<Your document string>);
JArray jAr=JArray.Parse(jOb["carts"]);
If(jOb["updateon"]=<Your Business Validation>)
{
foreach(var item in jAr)
if(item["closed"]==<Your validation>){ return true}
}
return false;
I hope this helps :)
If you handling with any null values in those properties then please use Tryparse and out variable.

Aggregation with MongoDB C# Driver [duplicate]

Is there a query for calculating how many distinct values a field contains in DB.
f.e I have a field for country and there are 8 types of country values (spain, england, france, etc...)
If someone adds more documents with a new country I would like the query to return 9.
Is there easier way then group and count?

MongoDB has a distinct command which returns an array of distinct values for a field; you can check the length of the array for a count.
There is a shell db.collection.distinct() helper as well:
> db.countries.distinct('country');
[ "Spain", "England", "France", "Australia" ]
> db.countries.distinct('country').length
4
As noted in the MongoDB documentation:
Results must not be larger than the maximum BSON size (16MB). If your results exceed the maximum BSON size, use the aggregation pipeline to retrieve distinct values using the $group operator, as described in Retrieve Distinct Values with the Aggregation Pipeline.

Here is example of using aggregation API. To complicate the case we're grouping by case-insensitive words from array property of the document.
db.articles.aggregate([
{
$match: {
keywords: { $not: {$size: 0} }
}
},
{ $unwind: "$keywords" },
{
$group: {
_id: {$toLower: '$keywords'},
count: { $sum: 1 }
}
},
{
$match: {
count: { $gte: 2 }
}
},
{ $sort : { count : -1} },
{ $limit : 100 }
]);
that give result such as
{ "_id" : "inflammation", "count" : 765 }
{ "_id" : "obesity", "count" : 641 }
{ "_id" : "epidemiology", "count" : 617 }
{ "_id" : "cancer", "count" : 604 }
{ "_id" : "breast cancer", "count" : 596 }
{ "_id" : "apoptosis", "count" : 570 }
{ "_id" : "children", "count" : 487 }
{ "_id" : "depression", "count" : 474 }
{ "_id" : "hiv", "count" : 468 }
{ "_id" : "prognosis", "count" : 428 }

With MongoDb 3.4.4 and newer, you can leverage the use of $arrayToObject operator and a $replaceRoot pipeline to get the counts.
For example, suppose you have a collection of users with different roles and you would like to calculate the distinct counts of the roles. You would need to run the following aggregate pipeline:
db.users.aggregate([
{ "$group": {
"_id": { "$toLower": "$role" },
"count": { "$sum": 1 }
} },
{ "$group": {
"_id": null,
"counts": {
"$push": { "k": "$_id", "v": "$count" }
}
} },
{ "$replaceRoot": {
"newRoot": { "$arrayToObject": "$counts" }
} }
])
Example Output
{
"user" : 67,
"superuser" : 5,
"admin" : 4,
"moderator" : 12
}

I wanted a more concise answer and I came up with the following using the documentation at aggregates and group
db.countries.aggregate([{"$group": {"_id": "$country", "count":{"$sum": 1}}}])

You can leverage on Mongo Shell Extensions. It's a single .js import that you can append to your $HOME/.mongorc.js, or programmatically, if you're coding in Node.js/io.js too.
Sample
For each distinct value of field counts the occurrences in documents optionally filtered by query
> db.users.distinctAndCount('name', {name: /^a/i})
{
"Abagail": 1,
"Abbey": 3,
"Abbie": 1,
...
}
The field parameter could be an array of fields
> db.users.distinctAndCount(['name','job'], {name: /^a/i})
{
"Austin,Educator" : 1,
"Aurelia,Educator" : 1,
"Augustine,Carpenter" : 1,
...
}

To find distinct in field_1 in collection but we want some WHERE condition too than we can do like following :
db.your_collection_name.distinct('field_1', {WHERE condition here and it should return a document})
So, find number distinct names from a collection where age > 25 will be like :
db.your_collection_name.distinct('names', {'age': {"$gt": 25}})
Hope it helps!

I use this query:
var collection = "countries"; var field = "country";
db[collection].distinct(field).forEach(function(value){print(field + ", " + value + ": " + db[collection].count({[field]: value}))})
Output:
countries, England: 3536
countries, France: 238
countries, Australia: 1044
countries, Spain: 16
This query first distinct all the values, and then count for each one of them the number of occurrences.

If you're on MongoDB 3.4+, you can use $count in an aggregation pipeline:
db.users.aggregate([
{ $group: { _id: '$country' } },
{ $count: 'countOfUniqueCountries' }
]);