We Keep Coding

Why are getmore ops so fast from mongo shell while they are slow using the c# driver?

If I run a query using mongo shell that loads 10.000 documents, it starts multiple getmore operations to fetch all 10.000 documents from the cursor. 6 getmore operations that seem like they run in parallel, so the query executes quite fast
These operations are very fast (about 200 ms time between them):
db.coll
.find({
"start": {
"$gte": ISODate("2018-06-01T00:00:00+02:00"),
"$lte": ISODate("2018-07-01T00:00:00+02:00"),
},
"state": {
$in: ["OK"]
}
})
.limit(10000)
.toArray()
But if I run the same query using the C# driver, that only uses 1 getmore at a time until all documents are retrieved
the time between getmore operations significantly higher (1-2 seconds):
This of course slows down the query when executed from C#. Is there something I am doing wrong in C#?
List<Tb> tbs = new List<Tb>();
List<ObjectId> tbIds = new List<ObjectId>();
if (keys != null && keys.Count > 0)
{
tbFilter = tbBuilder.In("categories.values.key", keys);
var tbProjection = Builders<Tb>.Projection.Include(t => t.Id);
List<TbIdOnly> tbOnlyIds = await context._tbs
.Find(tbFilter)
.Project<TbIdOnly>(tbProjection)
.ToListAsync();
tbIds = tbOnlyIds.Select(t => t.Id).ToList();
filter = filter & builder.In(t => t.IdTb, tbIds);
}
if (startTime != null)
filter = filter & builder.Gte(t => t.TsStart, startTime);
if (endTime != null)
filter = filter & builder.Lte(t => t.TsStart, endTime);
if (status != null && status.Count > 0)
filter = filter & builder.In(t => t.State, status);
var query = context._trs
.Find(filter, new FindOptions() {BatchSize = maxNumValues})
.Limit(maxNumValues);
Console.WriteLine($"Running query:\n {query.ToString()}");
Stopwatch sw = Stopwatch.StartNew();
List<Tr> trs = new List<Tr>();
using (var cursor = await query.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var cursorRuns = cursor.Current;
trs.AddRange(cursorRuns);
Console.WriteLine($"Fetched batch of trs.. {cursor.Current.Count()}");
}
}
sw.Stop();

getMore operations do not provide the starting position - they return the next batch of the cursor. As such it is not possible to have more than one getMore running on the same cursor at a time.
You made a mistake somewhere in your analysis.

Use command monitoring (e.g. see here, apply to your driver) to verify the time to execute the queries is the same in your driver and in the shell. If not, use command monitoring to retrieve exact commands sent and issue them in the shell.
On the sending side, drivers may have more things they are doing than the shell, for example:
Managing and using implicit sessions
Connection pools
Note that the drivers connect asynchronously (creating a Client object doesn't connect in foreground) while the shell connects synchronously. When you are benchmarking a driver vs the shell you need to ensure you already have the driver connected to and monitoring all servers, otherwise you are including server connection and monitoring time in the driver time but not in the shell's time.
On both sending and receiving side but especially on the receiving side, the driver generally deserializes the server response into language objects. Some languages are more efficient at this, others less so. Benchmark retrieving documents with simple fields like integers vs complex fields like arrays and times.
If you have application-level overrides for serialization or deserialization, that would increase the time to "execute a query" substantially.

Deadlock on timer controlled db operation

I have 3 identical programs (made using topshelf) that run as 3 separate services for 3 different departments. When they run, they keep accessing database using timers - mainly to keep sending Alive signal or to purge the old logs. Recently I've been asked to add additional timer only for one of the departments
Random r = new Random(); //to avoid possible deadlock we save the timestamp on random time
int hourly = 1000 * 60 * 60; //this is an hour
Timer TimerItem = new Timer(TimerCallBack, null, 0, r.Next(20000, 25000)); //alive timer
Timer purgeLogs = new Timer(TimerPurgeLogs, null,0,hourly); //purge logs timer
if (selDep == "planners")
{
Timer UpdatePlannes = new Timer(TimerUpdatePlanners, null, 0, r.Next(50000,60000)); //planners update
}
private static void TimerUpdatePlanners (Object o)
{
string conn = System.Configuration.ConfigurationManager.ConnectionStrings["Microsoft.Rtc.Collaboration.Sample.Properties.Settings.lynccallrosterConnectionString"].ToString();
if (selDep == "planners") //Only for planners - auto turn the forwading on for everyone except the current person
{
string updateCommand = "update employees set on_call = 1 where Department ='Planners' and gkey <> (Select gkey from currently_on_call where Department ='Planners')";
using (SqlConnection updatePlanners = new SqlConnection(conn))
{
SqlCommand executeUpdate = new SqlCommand(updateCommand, updatePlanners);
try
{
updatePlanners.Open();
executeUpdate.ExecuteNonQuery();
updatePlanners.Close();
}
catch { }
}
}
Unfortunately after I've done this, it caused a hell of a problems with other program that is trying to read the same database, because it somehow deadlocked the database (SQL 2005).
I can't see the reason for deadlock here: timer is executed only every 50-60s and the command execution last few millisecond, yet it seems that the connection is kept open all the time - in the activity monitor I can see dozen or so Update commands that keep clogging the data traffic.
What is also interesting, this issue only happens when the program is run as a service - topsehlf allows to run service as a console app and when its run in this mode, I can't see any deadlocks or unnecessary Update commands waiting in the queue.
What have I done wrong?
EDIT: I've tried as suggested to add Dispose() method in the try-catch
try
{
updatePlanners.Open();
executeUpdate.ExecuteNonQuery();
updatePlanners.Close();
}
catch { }
finally
{
executeUpdate.Dispose();
updatePlanners.Dispose();
}
But that still doesn't solve the issue with deadlock.
EDIT2: I think I solved it using try-and-error method by adding the delay to the timer start:
Timer UpdatePlanners = new Timer(TimerUpdatePlanners, null, r.Next(10000, 20000), r.Next(10000, 20000));
It still bugs me why there are so many UPDATE spawning, but at least now they are not deadlocking each other, because each one spawns at different time from the service start/timer tick.

Can this code have bottleneck or be resource-intensive?

It's code that will execute 4 threads in 15-min intervals. The last time that I ran it, the first 15-minutes were copied fast (20 files in 6 minutes), but the 2nd 15-minutes are much slower. It's something sporadic and I want to make certain that, if there's any bottleneck, it's in a bandwidth limitation with the remote server.
EDIT: I'm monitoring the last run and the 15:00 and :45 copied in under 8 minutes each. The :15 hasn't finished and neither has :30, and both began at least 10 minutes before :45.
Here's my code:
static void Main(string[] args)
{
Timer t0 = new Timer((s) =>
{
Class myClass0 = new Class();
myClass0.DownloadFilesByPeriod(taskRunDateTime, 0, cts0.Token);
Copy0Done.Set();
}, null, TimeSpan.FromMinutes(20), TimeSpan.FromMilliseconds(-1));
Timer t1 = new Timer((s) =>
{
Class myClass1 = new Class();
myClass1.DownloadFilesByPeriod(taskRunDateTime, 1, cts1.Token);
Copy1Done.Set();
}, null, TimeSpan.FromMinutes(35), TimeSpan.FromMilliseconds(-1));
Timer t2 = new Timer((s) =>
{
Class myClass2 = new Class();
myClass2.DownloadFilesByPeriod(taskRunDateTime, 2, cts2.Token);
Copy2Done.Set();
}, null, TimeSpan.FromMinutes(50), TimeSpan.FromMilliseconds(-1));
Timer t3 = new Timer((s) =>
{
Class myClass3 = new Class();
myClass3.DownloadFilesByPeriod(taskRunDateTime, 3, cts3.Token);
Copy3Done.Set();
}, null, TimeSpan.FromMinutes(65), TimeSpan.FromMilliseconds(-1));
}
public struct FilesStruct
{
public string RemoteFilePath;
public string LocalFilePath;
}
Private void DownloadFilesByPeriod(DateTime TaskRunDateTime, int Period, Object obj)
{
FilesStruct[] Array = GetAllFiles(TaskRunDateTime, Period);
//Array has 20 files for the specific period.
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
TransferOperationResult transferResult;
foreach (FilesStruct u in Array)
{
if (session.FileExists(u.RemoteFilePath)) //File exists remotely
{
if (!File.Exists(u.LocalFilePath)) //File does not exist locally
{
transferResult = session.GetFiles(u.RemoteFilePath, u.LocalFilePath);
transferResult.Check();
foreach (TransferEventArgs transfer in transferResult.Transfers)
{
//Log that File has been transferred
}
}
else
{
using (StreamWriter w = File.AppendText(Logger._LogName))
{
//Log that File exists locally
}
}
}
else
{
using (StreamWriter w = File.AppendText(Logger._LogName))
{
//Log that File exists remotely
}
}
if (token.IsCancellationRequested)
{
break;
}
}
}
}

Something is not quite right here. First thing is, you're setting 4 timers to run parallel. If you think about it, there is no need. You don't need 4 threads running parallel all the time. You just need to initiate tasks at specific intervals. So how many timers do you need? ONE.
The second problem is why TimeSpan.FromMilliseconds(-1)? What is the purpose of that? I can't figure out why you put that in there, but I wouldn't.
The third problem, not related to multi-programming, but I should point out anyway, is that you create a new instance of Class each time, which is unnecessary. It would be necessary if, in your class, you need to set constructors and your logic access different methods or fields of the class in some order. In your case, all you want to do is to call the method. So you don't need a new instance of the class every time. You just need to make the method you're calling static.
Here is what I would do:
Store the files you need to download in an array / List<>. Can't you spot out that you're doing the same thing every time? Why write 4 different versions of code for that? This is unnecessary. Store items in an array, then just change the index in the call!
Setup the timer at perhaps 5 seconds interval. When it reaches the 20 min/ 35 min/ etc. mark, spawn a new thread to do the task. That way a new task can start even if the previous one is not finished.
Wait for all threads to complete (terminate). When they do, check if they throw exceptions, and handle them / log them if necessary.
After everything is done, terminate the program.
For step 2, you have the option to use the new async keyword if you're using .NET 4.5. But it won't make a noticeable difference if you use threads manually.
And why is it so slow...why don't you check your system status using task manager? Is the CPU high and running or is the network throughput occupied by something else or what? You can easily tell the answer yourself from there.

The problem was the sftp client.
The purpose of the console application was to loop through a list<> and download the files. I tried with winscp and, even though, it did the job, it was very slow. I also tested sharpSSH and it was even slower than winscp.
I finally ended up using ssh.net which, at least in my particular case, was much faster than both winscp and sharpssh. I think the problem with winscp is that there was no evident way of disconnecting after I was done. With ssh.net I could connect/disconnect after every file download was made, something I couldn't do with winscp.

Move record insert to SQL Server CE database into another thread? C# Compact Framework 3.5

I had some trouble trying to place piece of code into another thread to increase performance.
I have following code below (with thread additions with comments), where I parse large XML file (final goal 100,000 rows) and then write it to a SQL Server CE 3.5 database file (.sdf) using record and insert (SqlCeResultSet/SqlCeUpdatableRecord).
Two lines of code in if statement inside the while loop,
xElem = (XElement)XNode.ReadFrom(xmlTextReader);
and
rs.Insert(record);
take about the same amount of time to execute. I was thinking to run rs.Insert(record); while I am parsing the next line of xml file. However, I still was unable to do it using either Thread or ThreadPool.
I have to make sure that the record that I pass to thread is not changed until I finish executing rs.Insert(record); in existing thread. Thus, I tried to place thread.Join() before writing new record (record.SetValue(i, values[i]);), but I still get conflict when I try to run the program - program crashes with bunch of errors due to trying to write identical row several times (especially for index).
Can anyone help me with some advise? How can I move rs.Insert(record); into another thread to increase performance?
XmlTextReader xmlTextReader = new XmlTextReader(modFunctions.InFName);
XElement xElem = new XElement("item");
using (SqlCeConnection cn = new SqlCeConnection(connectionString))
{
if (cn.State == ConnectionState.Closed)
cn.Open();
using (SqlCeCommand cmd = new SqlCeCommand())
{
cmd.Connection = cn;
cmd.CommandText = "item";
cmd.CommandType = CommandType.TableDirect;
using (SqlCeResultSet rs = cmd.ExecuteResultSet(ResultSetOptions.Updatable))
{
SqlCeUpdatableRecord record = rs.CreateRecord();
// Thread code addition
Thread t = new Thread(new ThreadStart(() => rs.Insert(record));
while (xmlTextReader.Read())
{
if (xmlTextReader.NodeType == XmlNodeType.Element &&
xmlTextReader.LocalName == "item" &&
xmlTextReader.IsStartElement() == true)
{
xElem = (XElement)XNode.ReadFrom(xmlTextReader);
values[0] = (string)xElem.Element("Index"); // 0
values[1] = (string)xElem.Element("Name"); // 1
~~~
values[13] = (string)xElem.Element("Notes"); // 13
// Thread code addition -- Wait until previous thread finishes
if (ThreadStartedS == 1)
{
t.Join()
}
// SetValues to record
for (int i = 0; i < values.Length; i++)
{
record.SetValue(i, values[i]); // 0 to 13
}
// Thread code addition -- Start thread to execute rs.Insert(record)
ThreadStartedS = 1;
t.Start();
// Original code without threads
// Insert Record
//rs.Insert(record);
}
}
}
}
}

If all of your processing is going to be done on the device (reading from the XML file on the device then parsing the data on the device), then you will see no performance increase from threading your work.
These Windows Mobile devices only have a single processor, so for them to multithread means one process works for a while, then another process works for a while. You will never have simultaneous processes running at the same time.
On the other hand, if the data from your XML file were located on a remote server, you could call the data in chunks. As a chunk arrives, you could process that data in another thread while waiting on the next chunk of data to arrive in the main thread.
If all of this work is being done on one device, you will not have good luck with multithreading.
You can still display a progress bar (from 0 to NumberOfRecords) with a cancel button so the person waiting for the data collection to complete does not go insane with anticipation.

Track dead WebDriver instances during parallel task

I am seeing some dead-instance weirdness running parallelized nested-loop web stress tests using Selenium WebDriver, simple example being, say, hit 300 unique pages with 100 impressions each.
I'm "successfully" getting 4 - 8 WebDriver instances going using a ThreadLocal<FirefoxWebDriver> to isolate them per task thread, and MaxDegreeOfParallelism on a ParallelOptions instance to limit the threads. I'm partitioning and parallelizing the outer loop only (the collection of pages), and checking .IsValueCreated on the ThreadLocal<> container inside the beginning of each partition's "long running task" method. To facilitate cleanup later, I add each new instance to a ConcurrentDictionary keyed by thread id.
No matter what parallelizing or partitioning strategy I use, the WebDriver instances will occasionally do one of the following:
Launch but never show a URL or run an impression
Launch, run any number of impressions fine, then just sit idle at some point
When either of these happen, the parallel loop eventually seems to notice that a thread isn't doing anything, and it spawns a new partition. If n is the number of threads allowed, this results in having n productive threads only about 50-60% of the time.
Cleanup still works fine at the end; there may be 2n open browsers or more, but the productive and unproductive ones alike get cleaned up.
Is there a way to monitor for these useless WebDriver instances and a) scavenge them right away, plus b) get the parallel loop to replace the task segment immediately, instead of lagging behind for several minutes as it often does now?

I was having a similar problem. It turns out that WebDriver doesn't have the best method for finding open ports. As described here it gets a system wide lock on ports, finds an open port, and then starts the instance. This can starve the other instances that you're trying to start of ports.
I got around this by specifying a random port number directly in the delegate for the ThreadLocal<IWebDriver> like this:
var ports = new List<int>();
var rand = new Random((int)DateTime.Now.Ticks & 0x0000FFFF);
var driver = new ThreadLocal<IWebDriver>(() =>
{
var profile = new FirefoxProfile();
var port = rand.Next(50) + 7050;
while(ports.Contains(port) && ports.Count != 50) port = rand.Next(50) + 7050;
profile.Port = port;
ports.Add(port);
return new FirefoxDriver(profile);
});
This works pretty consistently for me, although there's the issue if you end up using all 50 in the list that is unresolved.

Since there is no OnReady event nor an IsReady property, I worked around it by sleeping the thread for several seconds after creating each instance. Doing that seems to give me 100% durable, functioning WebDriver instances.
Thanks to your suggestion, I've implemented IsReady functionality in my open-source project Webinator. Use that if you want, or use the code outlined below.
I tried instantiating 25 instances, and all of them were functional, so I'm pretty confident in the algorithm at this point (I leverage HtmlAgilityPack to see if elements exist, but I'll skip it for the sake of simplicity here):
public void WaitForReady(IWebDriver driver)
{
var js = #"{ var temp=document.createElement('div'); temp.id='browserReady';" +
#"b=document.getElementsByTagName('body')[0]; b.appendChild(temp); }";
((IJavaScriptExecutor)driver).ExecuteScript(js);
WaitForSuccess(() =>
{
IWebElement element = null;
try
{
element = driver.FindElement(By.Id("browserReady"));
}
catch
{
// element not found
}
return element != null;
},
timeoutInMilliseconds: 10000);
js = #"{var temp=document.getElementById('browserReady');" +
#" temp.parentNode.removeChild(temp);}";
((IJavaScriptExecutor)driver).ExecuteScript(js);
}
private bool WaitForSuccess(Func<bool> action, int timeoutInMilliseconds)
{
if (action == null) return false;
bool success;
const int PollRate = 250;
var maxTries = timeoutInMilliseconds / PollRate;
int tries = 0;
do
{
success = action();
tries++;
if (!success && tries <= maxTries)
{
Thread.Sleep(PollRate);
}
}
while (!success && tries < maxTries);
return success;
}
The assumption is if the browser is responding to javascript functions and is finding elements, then it's probably a reliable instance and ready to be used.