I got some freeze in update documents in mongoDb through C#. What i need to do to fix it?
I have MongoDb collection with ~230000 documents. Signature is
{
"_id" : ObjectId("5c5d4c3a086dad5e5ab8d397"),
"Uid" : "SomeiDString",
"SomeTextField" : "SomeText"
}
Indexes - two fields:
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "db.mycollection"
},
{
"v" : 2,
"unique" : true,
"key" : {
"Uid" : 1.0
},
"name" : "Uid_1",
"ns" : "db.mycollection",
"background" : true
}
In C# I make async operation UpdateOneAsync, with filter on Uid and update SomeTextField.
private async Task <bool> PutSomeTextMongoDb(string Uid, string SomeText) {
var collection = database.GetCollection<User>(MONGO_DB_COLLEŠ”TION);
var filter = Builders<User>.Filter.Eq("Uid", Uid);
var update = Builders<User>.Update.Set("SomeText", SomeText);
var option = new UpdateOptions {IsUpsert = true};
var result = await collection.UpdateOneAsync(filter, update, option);
return result.ModifiedCount != 0;
}
Problem that I have: 5000 request got good avr time. Is about 2-3 ms. But some of them grow up to ~800-1000 ms... I know, it's only 5-6 long operation from 5k, but what is it? May be some locks for update operation, or when Mongo flush data i got some queue? mongostat and mongotop didn't answered me on this question...
Related
Let's say we have a collection of documents like this one:
{
"_id" : ObjectId("591c54faf1c1f419a830b9cf"),
"fingerprint" : "3121733676",
"screewidth" : "1920",
"carts" : [
{
"cartid" : 391796,
"status" : "New",
"cart_created" : ISODate("2017-05-17T13:50:37.388Z"),
"closed" : false,
"items" : [
{
"brandid" : "PIR",
"cai" : "2259700"
}
],
"updatedon" : ISODate("2017-05-17T13:51:24.252Z")
},
{
"cartid" : 422907,
"status" : "New",
"cart_created" : ISODate("2017-10-23T08:57:06.846Z"),
"closed" : false,
"items" : [
{
"brandid" : "PIR",
"cai" : "IrHlNdGtLfBoTlKsJaRySnM195U"
}
],
"updatedon" : ISODate("2017-10-23T09:46:08.579Z")
}
],
"createdon" : ISODate("2016-11-08T10:29:55.120Z"),
"updatedon" : ISODate("2017-10-23T09:46:29.486Z")
}
How do you extract only the documents where no item in the array $.carts have $.carts.closed set to true and $.carts.updatedon greater than $.updatedon minus 3 days ?
I know how to do find all the documents where no item in the array satisfy the condition $and: [closed: {$eq: true}, {updatedon: {$gt : new ISODate("2017-10-20T20:15:31Z")}}]
But how can you reference the parent element $.updatedon for the comparison?
In plain mongodb shell query language it would aleady be of help.
But I am actually accessing it using c# driver, so my query filter is like this:
FilterDefinition<_visitorData> filter;
filter = Builders<_visitorData>.Filter
.Gte(f => f.updatedon, DateTime.Now.AddDays(-15));
filter = filter & (
Builders<_visitorData>.Filter
.Exists(f => f.carts, false)
| !Builders<_visitorData>.Filter.ElemMatch(f =>
f.carts, c => c.closed && c.updatedon > DateTime.Now.AddDays(-15)
)
);
How can I replace DateTime.Now.AddDays(-15) with a reference to the document root element updatedon?
You can project the difference of carts.updatedon and updatedon and then filter out the results from this aggregation pipeline.
coll.aggregate([{'$unwind':'$carts'},
{'$match':{'closed':{'$ne':true}}},
{'$project':{'carts.cartid':1,'carts.status':1,'carts.cart_created':1,'carts.closed':1,'carts.items':1,'carts.updatedon':1,'updatedon':1,'diff':{'$subtract':['$carts.updatedon','$createdon']}}},
{'$match': {'diff': {'$gte': 1000 * 60 * 60 * 24 * days}}}])
days = 3 will filter out results more than 3 days difference documents.
I have just given the example of how you can use $subtract to find date difference and filter documents based on that.
well I was in a similar situation few days back.
I tackled it by using Jobject of Newtonsoft.Json.
Create a function to return bool which actually process each document take it as input.
Jobject jOb=Jobject.parse(<Your document string>);
JArray jAr=JArray.Parse(jOb["carts"]);
If(jOb["updateon"]=<Your Business Validation>)
{
foreach(var item in jAr)
if(item["closed"]==<Your validation>){ return true}
}
return false;
I hope this helps :)
If you handling with any null values in those properties then please use Tryparse and out variable.
Below is an example of my document. I am trying to update the CostReports part based on the id of the CostReportingPeriods element.
{
"_id" : "240106",
"CostReportingPeriods" : [
{
"FyBgnDt" : ISODate("2000-01-01T05:00:00.000Z"),
"FyEndDt" : ISODate("2000-12-31T05:00:00.000Z"),
"_id" : "240106-20000101-20001231",
"CostReports" : []
},
{
"FyBgnDt" : ISODate("2001-01-01T05:00:00.000Z"),
"FyEndDt" : ISODate("2001-12-31T05:00:00.000Z"),
"_id" : "240106-20010101-20011231",
"CostReports" : []
},
{
"FyBgnDt" : ISODate("2002-01-01T05:00:00.000Z"),
"FyEndDt" : ISODate("2002-12-31T05:00:00.000Z"),
"_id" : "240106-20020101-20021231",
"CostReports" : []
},
{
"FyBgnDt" : ISODate("2003-01-01T05:00:00.000Z"),
"FyEndDt" : ISODate("2003-12-31T05:00:00.000Z"),
"_id" : "240106-20030101-20031231",
"CostReports" : []
}
]
I am using the following code to try and update that element but am getting an error that says cannot use the element (CostReportingPeriods of CostReportingPeriods.CostReports) to traverse the element. If I add CostReportingPeriods.0.CostReports it will add it to the first element of the array, regardless of the filter.
var builder = Builders<MongoModels.Provider>.Filter;
var filter = builder.Eq("_id",costReport.PRVDR_NUM) & builder.Eq("CostReportingPeriods._id", costReport.UNIQUE_ID);
var update = Builders<MongoModels.Provider>.Update.AddToSet("CostReportingPeriods.CostReports", Mapper.Map<CostReport, MongoModels.CostReport>(costReport, opt =>
{
opt.AfterMap((src, dest) => dest.Worksheets = CreateWorksheets(dest.RptRecNum).ToList());
}));
How do I get it to update the element I want it to update based on the id of the subdocument?
After trying a bunch of different things, by channging my update filter from "CostReportingPeriods.CostReports" to "CostReportingPeriods.$.CostReports" it works flawlessly.
we have a problem with our indexes. We have an index on our emails but it throws errors like such:
> db.User.insert({email: "hell33o#gmail.com", "_id" : BinData(3,"iKyq6FvBCdd54TdxxX0JhA==")})
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: placetobe.User.$email_text dup key: { : \"com\", : 0.6666666666666666 }"
}
})
when we have the index created with our C# driver like this
Created by C# with:
CreateIndexOptions options = new CreateIndexOptions {Unique = true};
_collection.Indexes.CreateOneAsync(Builders<User>.IndexKeys.Text(_ => _.email), options);
resulted in
{
"v" : 1,
"unique" : true,
"key" : {
"_fts" : "text",
"_ftsx" : 1
},
"name" : "email_text",
"ns" : "placetobe.User",
"weights" : {
"email" : 1
},
"default_language" : "english",
"language_override" : "language",
"textIndexVersion" : 2
}
but if we create it with the MongoDB console like this it works:
{
"v" : 1,
"unique" : true,
"key" : {
"email" : 1
},
"name" : "email_1",
"ns" : "placetobe.User"
}
I don't understand the difference between the two indexes, but they have an effect on our DB. We also have problems with a Collectin that saves names. we get duplicate exceptions on "Molly" if we try to insert "Molli". With the emails is seems to give us errors whenever we have two "gmail" emails in the collection or two ".com" emails etc.
This is a University project and we have to turn it in tomorrow. we're really in trouble, help would be much appreciated
You don't want your email to be a text Index. Text indices allow you to search large amounts of text in MongoDB like if you were parsing through comments or something. All you want is to make sure your emails aren't duplicated so you should use an ascending or descending index.
CreateIndexOptions options = new CreateIndexOptions {Unique = true};
_collection.Indexes.CreateOneAsync(Builders<User>.IndexKeys.Ascending(_ => _.email), options)
Mongo isn't expiring old collections. I checked to make sure my index is type date.
var keys = IndexKeys.Ascending("expiry");
var options = IndexOptions.SetTimeToLive(TimeSpan.FromMinutes(1));
collection.EnsureIndex(keys, options);
this.ExpireDate = new BsonDateTime(DateTime.UtcNow.AddMinutes(5));
var insertResult = collection.Insert(this);
Any tips would be gladly appreciated.
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "Showsv1.ShowInfo",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"expiry" : 1
},
"ns" : "Showsv1.ShowInfo",
"name" : "expiry_1",
"expireAfterSeconds" : 60
}
]
"expiry" : ISODate("2013-02-15T02:40:45.876Z")
The code was missing [BsonElement("expiry")] on top of the ExpireTime property.
Thanks #WiredPrairie for the tip.
I have a "Payee" BsonDocument like this:
{
"Token" : "0b21ae960f25c6357286ce6c206bdef2",
"LastAccessed" : ISODate("2012-07-11T02:14:59.94Z"),
"Firstname" : "John",
"Lastname" : "Smith",
"PayrollInfo" : [{
"Tag" : "EARNINGS",
"Value" : "744.11",
}, {
"Tag" : "DEDUCTIONS",
"Value" : "70.01",
}],
},
"Status" : "1",
"_id" : ObjectId("4fc263158db2b88f762f1aa5")
}
I retrieve this document based on the Payee _id.
var collection = database.GetCollection("Payee");
var query = Query.EQ("_id", _id);
var bDoc = collection.FindOne(query);
Then, at various times I need to update a specific object inside the PayrollInfo array. So I search for the object with appropriate "Tag" inside the array and update the "Value" into the database. I use the following logic to do this:
var bsonPayrollInfo = bDoc["PayrollInfo", null];
if (bsonPayrollInfo != null)
{
var ArrayOfPayrollInfoObjects = bsonPayrollInfo.AsBsonArray;
for (int i = 0; i < ArrayOfPayrollInfoObjects.Count; i++)
{
var bInnerDoc = ArrayOfPayrollInfoObjects[i].AsBsonDocument;
if (bInnerDoc != null)
{
if (bInnerDoc["Tag"] == "EARNINGS")
{
//update here
var update = Update
.Set("PayrollInfo."+ i.ToString() + ".Value", 744.11)
collection.FindAndModify(query, null, update);
bUpdateData = true;
break;
}
}
}
}
if (!bUpdateData)
{
//Use Update.Push. This works fine and is not relevant to the question.
}
All this code works fine, but I think I am being cumbersome in achieving the result. Is there a more concise way of doing this? Essentially, I am trying to find a better way of updating an object inside of an array in a BsonDocument.
Mongo has a positional operator that will let you operate on the matched value in an array. The syntax is: field1.$.field2
Here's an example of how you'd use it from the Mongo shell:
db.dots.insert({tags: [{name: "beer", count: 2}, {name: "nuts", count: 3}]})
db.dots.update({"tags.name": "beer"}, {$inc: {"tags.$.count" : 1}})
result = db.dots.findOne()
{ "_id" : ObjectId("50078284ea80325278ff0c63"), "tags" : [ { "name" : "beer", "count" : 3 }, { "name" : "nuts", "count" : 3 } ] }
Putting my answer here in case it helps you. Based on #MrKurt's answer (thank you!), here is what I did to rework the code.
var collection = database.GetCollection("Payee");
var query = Query.EQ("_id", _id);
if (collection.Count(query) > 0)
{
//Found the Payee. Let's save his/her Tag for EARNINGS
UpdateBuilder update = null;
//Check if this Payee already has any EARNINGS Info saved.
//If so, we need to update that.
query = Query.And(query,
Query.EQ("PayrollInfo.Tag", "EARNINGS"));
//Update will be written based on whether we find the Tag:EARNINGS element in the PayrollInfo array
if (collection.Count(query) > 0)
{
//There is already an element in the PayrollInfo for EARNINGS
//Just update that element
update = Update
.Set("PayrollInfo.$.Value", "744.11");
}
else
{
//This user does not have any prior EARNINGS data. Add it to the user record
query = Query.EQ("_id", _id);
//Add a new element in the Array for PayrollInfo to store the EARNINGS data
update = Update.Push("PayrollInfo",
new BsonDocument {{"Tag", "EARNINGS"}, {"Value", "744.11"}}
);
}
//Run the update
collection.FindAndModify(query, null, update);
}
It doesn't look any lesser than my original code, but it is much more intuitive, and I got to learn a lot about positional operators!