We Keep Coding

Thread with Apartment State set to STA has issues sending many queries to postgres database

I recently solved an issue where my program was crashing while using a BackgroundWorker.
I do not fully understand why there was a problem in the first place. When the thread has it's apartment state set to STA the Process memory increases until the program throws the exception "Unable to allocate environment handle", at first I thought it was an issue with the Database not being able to process the queries quickly enough. The bottom chart on the picture below shows the program running with the thread set to STA. You can see a steady increase in usage until it drops off almost completely. That is when the exception is thrown.
When the thread is running with the ApartmentState set to MTA, the top chart shows it behaving as expected, there is a increase in usage, then it starts a pattern of using and free the memory.
TLDR:
Why does a thread using the Apartment state STA have an issue.
Below I have included some relevant code.
BackgroundWorker myWorkerBee;
List<Customers> AllCustomers
private void btnStartConversion_Click(object sender, RoutedEventArgs e)
{
myWorkerBee = new Thread(myWorkerBee_DoWork);
myWorkerBee.SetApartmentState(ApartmentState.STA);
myWorkerBee.Start();
{
private void myWorkerBee_DoWork()
{
GetOldData(); //Creates Customer object and fills AllCustomers list
AddCustomers();
}
There is no issue with the program if it doesn't use the AddNewCustomerConvert(); Method.
private void AddCustomers()
{
for (int i = 0; i < AllCustomers.Count; i++)
{
AllCustomers[i].AddNewCustomerConvert();
}
}
This method exclusively calls the RawQuery(); Method, or methods that only call the RawQuery(); method
//REFERENCED PROGRAM
AddNewCustomerConvert()
{
//40 or so insert statements.
databaseConnection.RawQuery("//INSERT STATEMENT");
}
Sends Queries to the database
//REFERENCE DLL
public OdbcDataReader RawQuery(string query_to_perform)
{
// This method executes a query in the specific database that you
// are connected to.
System.Data.Odbc.OdbcCommand command = null;
// holds the query sent to the database
System.Data.Odbc.OdbcDataReader result_reader = null;
// The query is put into an OdbcCommand object and sent to the database. The
// return result will then be given back to the caller.
try
{
if (loggingEnabled)
{
myLog = File.AppendText(loggingFileName);
myLog.WriteLine(query_to_perform);
myLog.Close();
myLog.Dispose();
}
command = new System.Data.Odbc.OdbcCommand(query_to_perform, this.database_connection);
result_reader = command.ExecuteReader();
this.successful_query = true;
this.error_message = "";
}
catch (System.Data.Odbc.OdbcException ex)
{
this.successful_query = false;
this.error_message = ex.Message;
//destroy the connection on a failure
database_connection = new OdbcConnection();
throw;
}
return result_reader;
}

Running Sql Server Agent Jobs from C#

While searching on above topic in the internet, I found two approaches,Both are working fine, But I need to know the difference between the two, Which one is suitable for what occasion etc... Our Jobs take some time and I need a way to wait till the Job finishes before the next C# line executes.
Approach One
var dbConn = new SqlConnection(myConString);
var execJob = new SqlCommand
{
CommandType = CommandType.StoredProcedure,
CommandText = "msdb.dbo.sp_start_job"
};
execJob.Parameters.AddWithValue("#job_name", p0);
execJob.Connection = dbConn;
using (dbConn)
{
dbConn.Open();
using (execJob)
{
execJob.ExecuteNonQuery();
Thread.Sleep(5000);
}
}
Approach Two
using System.Threading;
using Microsoft.SqlServer.Management.Smo;
using Microsoft.SqlServer.Management.Smo.Agent;
var server = new Server(#"localhost\myinstance");
var isStopped = false;
try
{
server.ConnectionContext.LoginSecure = true;
server.ConnectionContext.Connect();
var job = server.JobServer.Jobs[jobName];
job.Start();
Thread.Sleep(1000);
job.Refresh();
while (job.CurrentRunStatus == JobExecutionStatus.Executing)
{
Thread.Sleep(1000);
job.Refresh();
}
isStopped = true;
}
finally
{
if (server.ConnectionContext.IsOpen)
{
server.ConnectionContext.Disconnect();
}
}

sp_start_job - sample 1
Your first example calls your job via the sp_start_job system stored procedure.
Note that it kicks off the job asynchronously, and the thread sleeps for an arbitrary period of time (5 seconds) before continuing regardless of the job's success or failure.
SQL Server Management Objects (SMO) - sample 2
Your second example uses (and therefore has a dependency on) the SQL Server Management Objects to achieve the same goal.
In the second case, the job also commences running asynchronously, but the subsequent loop watches the Job Status until it is not longer Executing. Note that the "isStopped" flag appears to serve no purpose, and the loop could be refactored somewhat as:
job.Start();
do
{
Thread.Sleep(1000);
job.Refresh();
} while (job.CurrentRunStatus == JobExecutionStatus.Executing);
You'd probably want to add a break-out of that loop after a certain period of time.
Other Considerations
It seems the same permissions are required by each of your examples; essentially the solution using SMO is a wrapper around sp_start_job, but provides you with (arguably) more robust code which has a clearer purpose.
Use whichever suits you best, or do some profiling and pick the most efficient if performance is a concern.

Can this code have bottleneck or be resource-intensive?

It's code that will execute 4 threads in 15-min intervals. The last time that I ran it, the first 15-minutes were copied fast (20 files in 6 minutes), but the 2nd 15-minutes are much slower. It's something sporadic and I want to make certain that, if there's any bottleneck, it's in a bandwidth limitation with the remote server.
EDIT: I'm monitoring the last run and the 15:00 and :45 copied in under 8 minutes each. The :15 hasn't finished and neither has :30, and both began at least 10 minutes before :45.
Here's my code:
static void Main(string[] args)
{
Timer t0 = new Timer((s) =>
{
Class myClass0 = new Class();
myClass0.DownloadFilesByPeriod(taskRunDateTime, 0, cts0.Token);
Copy0Done.Set();
}, null, TimeSpan.FromMinutes(20), TimeSpan.FromMilliseconds(-1));
Timer t1 = new Timer((s) =>
{
Class myClass1 = new Class();
myClass1.DownloadFilesByPeriod(taskRunDateTime, 1, cts1.Token);
Copy1Done.Set();
}, null, TimeSpan.FromMinutes(35), TimeSpan.FromMilliseconds(-1));
Timer t2 = new Timer((s) =>
{
Class myClass2 = new Class();
myClass2.DownloadFilesByPeriod(taskRunDateTime, 2, cts2.Token);
Copy2Done.Set();
}, null, TimeSpan.FromMinutes(50), TimeSpan.FromMilliseconds(-1));
Timer t3 = new Timer((s) =>
{
Class myClass3 = new Class();
myClass3.DownloadFilesByPeriod(taskRunDateTime, 3, cts3.Token);
Copy3Done.Set();
}, null, TimeSpan.FromMinutes(65), TimeSpan.FromMilliseconds(-1));
}
public struct FilesStruct
{
public string RemoteFilePath;
public string LocalFilePath;
}
Private void DownloadFilesByPeriod(DateTime TaskRunDateTime, int Period, Object obj)
{
FilesStruct[] Array = GetAllFiles(TaskRunDateTime, Period);
//Array has 20 files for the specific period.
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
TransferOperationResult transferResult;
foreach (FilesStruct u in Array)
{
if (session.FileExists(u.RemoteFilePath)) //File exists remotely
{
if (!File.Exists(u.LocalFilePath)) //File does not exist locally
{
transferResult = session.GetFiles(u.RemoteFilePath, u.LocalFilePath);
transferResult.Check();
foreach (TransferEventArgs transfer in transferResult.Transfers)
{
//Log that File has been transferred
}
}
else
{
using (StreamWriter w = File.AppendText(Logger._LogName))
{
//Log that File exists locally
}
}
}
else
{
using (StreamWriter w = File.AppendText(Logger._LogName))
{
//Log that File exists remotely
}
}
if (token.IsCancellationRequested)
{
break;
}
}
}
}

Something is not quite right here. First thing is, you're setting 4 timers to run parallel. If you think about it, there is no need. You don't need 4 threads running parallel all the time. You just need to initiate tasks at specific intervals. So how many timers do you need? ONE.
The second problem is why TimeSpan.FromMilliseconds(-1)? What is the purpose of that? I can't figure out why you put that in there, but I wouldn't.
The third problem, not related to multi-programming, but I should point out anyway, is that you create a new instance of Class each time, which is unnecessary. It would be necessary if, in your class, you need to set constructors and your logic access different methods or fields of the class in some order. In your case, all you want to do is to call the method. So you don't need a new instance of the class every time. You just need to make the method you're calling static.
Here is what I would do:
Store the files you need to download in an array / List<>. Can't you spot out that you're doing the same thing every time? Why write 4 different versions of code for that? This is unnecessary. Store items in an array, then just change the index in the call!
Setup the timer at perhaps 5 seconds interval. When it reaches the 20 min/ 35 min/ etc. mark, spawn a new thread to do the task. That way a new task can start even if the previous one is not finished.
Wait for all threads to complete (terminate). When they do, check if they throw exceptions, and handle them / log them if necessary.
After everything is done, terminate the program.
For step 2, you have the option to use the new async keyword if you're using .NET 4.5. But it won't make a noticeable difference if you use threads manually.
And why is it so slow...why don't you check your system status using task manager? Is the CPU high and running or is the network throughput occupied by something else or what? You can easily tell the answer yourself from there.

The problem was the sftp client.
The purpose of the console application was to loop through a list<> and download the files. I tried with winscp and, even though, it did the job, it was very slow. I also tested sharpSSH and it was even slower than winscp.
I finally ended up using ssh.net which, at least in my particular case, was much faster than both winscp and sharpssh. I think the problem with winscp is that there was no evident way of disconnecting after I was done. With ssh.net I could connect/disconnect after every file download was made, something I couldn't do with winscp.

Move record insert to SQL Server CE database into another thread? C# Compact Framework 3.5

I had some trouble trying to place piece of code into another thread to increase performance.
I have following code below (with thread additions with comments), where I parse large XML file (final goal 100,000 rows) and then write it to a SQL Server CE 3.5 database file (.sdf) using record and insert (SqlCeResultSet/SqlCeUpdatableRecord).
Two lines of code in if statement inside the while loop,
xElem = (XElement)XNode.ReadFrom(xmlTextReader);
and
rs.Insert(record);
take about the same amount of time to execute. I was thinking to run rs.Insert(record); while I am parsing the next line of xml file. However, I still was unable to do it using either Thread or ThreadPool.
I have to make sure that the record that I pass to thread is not changed until I finish executing rs.Insert(record); in existing thread. Thus, I tried to place thread.Join() before writing new record (record.SetValue(i, values[i]);), but I still get conflict when I try to run the program - program crashes with bunch of errors due to trying to write identical row several times (especially for index).
Can anyone help me with some advise? How can I move rs.Insert(record); into another thread to increase performance?
XmlTextReader xmlTextReader = new XmlTextReader(modFunctions.InFName);
XElement xElem = new XElement("item");
using (SqlCeConnection cn = new SqlCeConnection(connectionString))
{
if (cn.State == ConnectionState.Closed)
cn.Open();
using (SqlCeCommand cmd = new SqlCeCommand())
{
cmd.Connection = cn;
cmd.CommandText = "item";
cmd.CommandType = CommandType.TableDirect;
using (SqlCeResultSet rs = cmd.ExecuteResultSet(ResultSetOptions.Updatable))
{
SqlCeUpdatableRecord record = rs.CreateRecord();
// Thread code addition
Thread t = new Thread(new ThreadStart(() => rs.Insert(record));
while (xmlTextReader.Read())
{
if (xmlTextReader.NodeType == XmlNodeType.Element &&
xmlTextReader.LocalName == "item" &&
xmlTextReader.IsStartElement() == true)
{
xElem = (XElement)XNode.ReadFrom(xmlTextReader);
values[0] = (string)xElem.Element("Index"); // 0
values[1] = (string)xElem.Element("Name"); // 1
~~~
values[13] = (string)xElem.Element("Notes"); // 13
// Thread code addition -- Wait until previous thread finishes
if (ThreadStartedS == 1)
{
t.Join()
}
// SetValues to record
for (int i = 0; i < values.Length; i++)
{
record.SetValue(i, values[i]); // 0 to 13
}
// Thread code addition -- Start thread to execute rs.Insert(record)
ThreadStartedS = 1;
t.Start();
// Original code without threads
// Insert Record
//rs.Insert(record);
}
}
}
}
}

If all of your processing is going to be done on the device (reading from the XML file on the device then parsing the data on the device), then you will see no performance increase from threading your work.
These Windows Mobile devices only have a single processor, so for them to multithread means one process works for a while, then another process works for a while. You will never have simultaneous processes running at the same time.
On the other hand, if the data from your XML file were located on a remote server, you could call the data in chunks. As a chunk arrives, you could process that data in another thread while waiting on the next chunk of data to arrive in the main thread.
If all of this work is being done on one device, you will not have good luck with multithreading.
You can still display a progress bar (from 0 to NumberOfRecords) with a cancel button so the person waiting for the data collection to complete does not go insane with anticipation.

Track dead WebDriver instances during parallel task

I am seeing some dead-instance weirdness running parallelized nested-loop web stress tests using Selenium WebDriver, simple example being, say, hit 300 unique pages with 100 impressions each.
I'm "successfully" getting 4 - 8 WebDriver instances going using a ThreadLocal<FirefoxWebDriver> to isolate them per task thread, and MaxDegreeOfParallelism on a ParallelOptions instance to limit the threads. I'm partitioning and parallelizing the outer loop only (the collection of pages), and checking .IsValueCreated on the ThreadLocal<> container inside the beginning of each partition's "long running task" method. To facilitate cleanup later, I add each new instance to a ConcurrentDictionary keyed by thread id.
No matter what parallelizing or partitioning strategy I use, the WebDriver instances will occasionally do one of the following:
Launch but never show a URL or run an impression
Launch, run any number of impressions fine, then just sit idle at some point
When either of these happen, the parallel loop eventually seems to notice that a thread isn't doing anything, and it spawns a new partition. If n is the number of threads allowed, this results in having n productive threads only about 50-60% of the time.
Cleanup still works fine at the end; there may be 2n open browsers or more, but the productive and unproductive ones alike get cleaned up.
Is there a way to monitor for these useless WebDriver instances and a) scavenge them right away, plus b) get the parallel loop to replace the task segment immediately, instead of lagging behind for several minutes as it often does now?

I was having a similar problem. It turns out that WebDriver doesn't have the best method for finding open ports. As described here it gets a system wide lock on ports, finds an open port, and then starts the instance. This can starve the other instances that you're trying to start of ports.
I got around this by specifying a random port number directly in the delegate for the ThreadLocal<IWebDriver> like this:
var ports = new List<int>();
var rand = new Random((int)DateTime.Now.Ticks & 0x0000FFFF);
var driver = new ThreadLocal<IWebDriver>(() =>
{
var profile = new FirefoxProfile();
var port = rand.Next(50) + 7050;
while(ports.Contains(port) && ports.Count != 50) port = rand.Next(50) + 7050;
profile.Port = port;
ports.Add(port);
return new FirefoxDriver(profile);
});
This works pretty consistently for me, although there's the issue if you end up using all 50 in the list that is unresolved.

Since there is no OnReady event nor an IsReady property, I worked around it by sleeping the thread for several seconds after creating each instance. Doing that seems to give me 100% durable, functioning WebDriver instances.
Thanks to your suggestion, I've implemented IsReady functionality in my open-source project Webinator. Use that if you want, or use the code outlined below.
I tried instantiating 25 instances, and all of them were functional, so I'm pretty confident in the algorithm at this point (I leverage HtmlAgilityPack to see if elements exist, but I'll skip it for the sake of simplicity here):
public void WaitForReady(IWebDriver driver)
{
var js = #"{ var temp=document.createElement('div'); temp.id='browserReady';" +
#"b=document.getElementsByTagName('body')[0]; b.appendChild(temp); }";
((IJavaScriptExecutor)driver).ExecuteScript(js);
WaitForSuccess(() =>
{
IWebElement element = null;
try
{
element = driver.FindElement(By.Id("browserReady"));
}
catch
{
// element not found
}
return element != null;
},
timeoutInMilliseconds: 10000);
js = #"{var temp=document.getElementById('browserReady');" +
#" temp.parentNode.removeChild(temp);}";
((IJavaScriptExecutor)driver).ExecuteScript(js);
}
private bool WaitForSuccess(Func<bool> action, int timeoutInMilliseconds)
{
if (action == null) return false;
bool success;
const int PollRate = 250;
var maxTries = timeoutInMilliseconds / PollRate;
int tries = 0;
do
{
success = action();
tries++;
if (!success && tries <= maxTries)
{
Thread.Sleep(PollRate);
}
}
while (!success && tries < maxTries);
return success;
}
The assumption is if the browser is responding to javascript functions and is finding elements, then it's probably a reliable instance and ready to be used.