We Keep Coding

ThreadPool race-condition, closure, locking or something else?

Still pretty new to threads so I'm sure it is one of those little gotchas and a repeat question, but I have been unable to find the answer browsing the threads.
I have a port scanner app in C#.
I'm using threadpools to spin up a new TcpClient for each port and probe if it's open.
After suffering through the concepts of closures and thread synchronization, I am having an issue where when multiple threads try to save their results to different indexes in the Orchestrator.hosts (List).
I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write, however I'm finding that on some updates, multiple entries are getting the same update.
IE, Thread #1 supposed to update Hosts[0].Ports[0].Status to "Open",
What happens:
Thread #1 updates multiple host with the port result despite passing a specific index for Hosts.
Hosts[0].Ports[0].Status to "Open",
Hosts[1].Ports[0].Status to "Open",
Hosts[2].Ports[0].Status to "Open",
Not sure where my problem is. The Static method I'm calling to perform a probe of a given port
public static void ScanTCPPorts()
{
// Create a list of portsToScan objects to send to thread workers
//List<ScanPortRequest> portsToScan = new List<ScanPortRequest>();
using (ManualResetEvent resetEvent = new ManualResetEvent(false))
{
int toProcess = 0;
for (var i = 0; i < hostCount; i++) // Starting at Begining
{
int currentHostId = i;
// To hold our current hosts ID (Assign outside of threaded function to avoid race-condition)
if (hosts[i].IsAlive || scanDefinition.isForced())
{
int portCount = hosts[i].Ports.Count;
for (int p = 0; p < portCount; p++)
{
// Thread-safe Increment our workQueue counter
Interlocked.Increment(ref toProcess);
int currentPortPosition = p;
// We need to send the arrayIndex in to the thread function
PortScanRequestResponse portRequestResponse = new PortScanRequestResponse(hosts[currentHostId], currentHostId, hosts[currentHostId].Ports[currentPortPosition], currentPortPosition);
ThreadPool.QueueUserWorkItem(
new WaitCallback(threadedRequestResponseInstance => {
PortScanRequestResponse portToScan = threadedRequestResponseInstance as PortScanRequestResponse;
PortScanRequestResponse threadResult = PortScanner.scanTCPPort(portToScan);
// Lock so Thread-safe update to result
lock (Orchestrator.hosts[portToScan.hostResultIndex])
{
if (threadResult.port.status == PortStatus.Open)
{
// Update result
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Open;
//Logger.Log(hosts[currentHostId].IPAddress + " " + hosts[currentHostId].Ports[currentPortPosition].type + " " + hosts[currentHostId].Ports[currentPortPosition].portNumber + " is open");
}
else
{
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Closed;
}
// Check if this was the last scan for the given host
if (Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete != true)
{
if (Orchestrator.hosts[portToScan.hostResultIndex].isCompleted())
{
Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete = true;
// Logger.Log(hosts[currentHostId].IPAddress + " has completed a port scan");
Orchestrator.hosts[portToScan.hostResultIndex].PrintPortSummery();
}
}
}
// Safely decrement the counter
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}), portRequestResponse); // Pass in our Port to scan
}
}
}
resetEvent.WaitOne();
}
}
Here is the worker process in a separate public static class.
public static PortScanRequestResponse scanTCPPort(object portScanRequest) {
PortScanRequestResponse portScanResponse = portScanRequest as PortScanRequestResponse;
HostDefinition host = portScanResponse.host;
ScanPort port = portScanResponse.port;
try
{
using (TcpClient threadedClient = new TcpClient())
{
try
{
IAsyncResult result = threadedClient.BeginConnect(host.IPAddress, port.portNumber, null, null);
Boolean success = result.AsyncWaitHandle.WaitOne(Orchestrator.scanDefinition.GetPortTimeout(), false);
if (threadedClient.Client != null)
{
if (success)
{
threadedClient.EndConnect(result);
threadedClient.Close();
portScanResponse.port.status = PortStatus.Open;
return portScanResponse;
}
}
} catch { }
}
}
catch
{ }
portScanResponse.port.status = PortStatus.Closed;
return portScanResponse;
}
Originally I was pulling the host index from a free variable, thinking this was the problem moved it to inside the delegate.
I tried locking the Hosts object everywhere there was a write.
I have tried different thread sync techniques (CountdownEvent and ManualResetEvent).
I think there is just some fundamental threading principal I have not been introduced to yet, or I have made a very simple logic mistake.

I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write.
I haven't studied your code, but the above statement alone is incorrect. When a List<T>, or any other non-thread-safe object , is used in a multithreaded environment, all interactions with the object must be synchronized. Only one thread at a time should be allowed to interact with the object. Both writes and reads must be enclosed in lock statements, using the same locker object. Even reading the Count must be synchronized. Otherwise the usage is erroneous, and the behavior of the program is undefined.

I was hyper-focused on it being a thread issue because this was my firsts threaded project. Turned out to be that I didn't realize copies of a List<> objects are references to their original object (reference type). I assumed my threads were accessing my save structure in an unpredictable way, but my arrays of ports were all referencing the same object.
This was a "reference type" vs "value type" issue on my List<> of ports.

Call Web API for 30K times within one hour of time period

I need to call Web API(Hosted on different network) from windows application from users machine for 30,000 times within one hour of time span.
I tried Multithreading to achieve the same but it is not working(giving system out of memory exceprion).
I used TreadPool as below
private static object threadLock = new object();
public delegate void BarDelegate();
int ThreadCount = dtExcel.Rows.Count;
private void button2_Click(object sender, EventArgs e)
{
for (int i = 0; i < ThreadCount - 1; i++)
{
ThreadPool.QueueUserWorkItem(output => CallAPI());
}
}
public void CallAPI()
{
string branchCode = "",
para1 = dtExcel.Rows[progressBar.Value]["para1"].ToString(),
para2 = "324",
para3 = "Test",
para4 = dtExcel.Rows[progressBar.Value]["para4"].ToString();
//Console.WriteLine(Thread.CurrentThread.Name + ": " + progressBar.Value);
var service = new APIService();
var resp = service.CallAPIService(para1, para2, para3, para4, para5);
if (resp.IsSuccess == true)
{
DataGridViewRow dtGrdVwR = dataGrid.Rows[progressBar.Value];
dtGrdVwR.Cells[3].Value = "Success";
}
else
{
DataGridViewRow dtGrdVwR = dataGrid.Rows[progressBar.Value];
dtGrdVwR.Cells[3].Value = "Failed: "+ resp.Message;
}
try
{
this.Invoke(new BarDelegate(UpdateBar));
}
catch
{
}
}
private void UpdateBar()
{
lblEndTime.Text = DateTime.Now.ToString();
progressBar.Value++;
if (progressBar.Value == progressBar.Maximum)
{
// We are finished and the progress bar is full.
}
}
Here dtExcel has 30,000 records(Uploaded by user from excel) which needs to processed within one hour and update the status of executed record in respective row in dataGrid.
The API call is made over network where single call takes approximate 1 to 2 seconds to execute.
service.CallAPIService(para1, para2, para3, para4, para5);
The above method internally performs heavy task like request encryption and digital signature and response decryption and digital signature verification.
Please help me with the best way possible where i can perform the task within time period and without getting SystemOutOfmemoryException.
Thanks in Advance.

Right now your code is horribly broken because of the race condition accessing progressBar.Value. It's pointless to discuss any other issues, because you are going to totally reorganize your code to fix the race condition, rendering other comments obsolete.
Fix it so that you don't have N threads all trying to process item #1, and then ask a new question with your new code.

Getting the C# BackgroundWorker process to invoke Pings

Read through most (all?) of the answered questions regarding the C# BackgroundWorker but none seemed to apply to this situation. If I missed one, please point me in that direction!
Anyway, I having troubles getting the Ping process to run as a background process. I made a simple form application to send pings and report back. That worked fine but it would only results results to the user after the pings were complete -- thus the need to a background process. I am somewhat new to C# and was unfamiliar with the particulars of BackgroundWorker. However found a helpful walkthrough from Microsoft here: http://msdn.microsoft.com/en-us/library/ywkkz4s1.aspx
I am now attempting to get the same process to apply to a System.Net.NetworkInformation object instead of a System.IO.StreamReader object. I think I am really close (read: I can get the app to build and run) but I consistently get an error at runtime (see below).
This is the Microsoft code for their sample app. It works like a champ:
The method in MainForm.cs that calls the Words.cs class referenced in the walkthrough
void backgroundWorker1DoWork(object sender, DoWorkEventArgs e)
{
System.ComponentModel.BackgroundWorker worker;
worker = (System.ComponentModel.BackgroundWorker)sender;
Words WC = (Words)e.Argument;
WC.CountWords(worker, e);
}
The relevant method in the 'Words.cs' class
public void CountWords(
System.ComponentModel.BackgroundWorker worker,
System.ComponentModel.DoWorkEventArgs e)
{
// Initialize the variables.
CurrentState state = new CurrentState();
string line = "";
int elapsedTime = 20;
DateTime lastReportDateTime = DateTime.Now;
if (CompareString == null ||
CompareString == System.String.Empty)
{
throw new Exception("CompareString not specified.");
}
// Open a new stream.
using (System.IO.StreamReader myStream = new System.IO.StreamReader(SourceFile))
{
// Process lines while there are lines remaining in the file.
while (!myStream.EndOfStream)
{
if (worker.CancellationPending)
{
e.Cancel = true;
break;
}
else
{
line = myStream.ReadLine();
WordCount += CountInString(line, CompareString);
LinesCounted += 1;
// Raise an event so the form can monitor progress.
int compare = DateTime.Compare(
DateTime.Now, lastReportDateTime.AddMilliseconds(elapsedTime));
if (compare > 0)
{
state.LinesCounted = LinesCounted;
state.WordsMatched = WordCount;
worker.ReportProgress(0, state);
lastReportDateTime = DateTime.Now;
}
}
// Uncomment for testing.
System.Threading.Thread.Sleep(5);
}
// Report the final count values.
state.LinesCounted = LinesCounted;
state.WordsMatched = WordCount;
worker.ReportProgress(0, state);
}
}
When I try a similar process (sending a Ping instead of a reading a file) I get this error:
Error: Object reference not set to an instance of an object.
Details: System.Collections.ListDictionaryInternal //This is defined in the MyApp namespace as: using System.Collections
Source: MyApp
StackTrack: at MyApp.MainForm.Bw01DoWork(Object sender, DoWorkEventArgs e) in
[path]\MainForm.cs:line 152
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
Target: Void Bw01DoWork(System.Object, System.ComponentModel.DoWorkEventArgs)
Here is my method. Line 152 referenced in the error is the very last line of the last method in MainForm.cs (the var names are different, but you get the idea):
void Bw01DoWork(object sender, DoWorkEventArgs e)
{
System.ComponentModel.BackgroundWorker worker;
worker = (System.ComponentModel.BackgroundWorker)sender;
PTResults PR = (PTResults)e.Argument;
PR.SendPings(worker, e); // Line 152
}
And the relevant portion of the PTResults.cs class:
using (Ping newPing = new Ping())
{
PingReply reply = newPing.Send([Target Site],[Timeout]);
if(reply.Status == IPStatus.Success)
{
state.PingOK = true;
}
else if(reply.Status == IPStatus.TimedOut)
{
state.PingOK = false;
state.PingUpdateState = " Timed Out";
}
else if(reply.Status != IPStatus.Success)
{
state.PingOK = false;
state.PingUpdateState = " FAILED";
}
else
{
state.PingOK = false;
state.PingUpdateState = " UNKNOWN";
}
worker.ReportProgress(0, state.PingOK);
}
I am thinking the System.Net.NetworkInformation.Ping component cannot be invoked the same way System.IO.StreamReader is. Thoughts?
I doubt it makes a difference but FWIW I am coding in SharpDevelop on a Windows 8.1 system.

Take a look at the Ping SendAsync, you may be able to eliminate most of your code - just call PingAsync, and handle the result being sure to dispatch it to the UI thread and then re-queue another call.
http://msdn.microsoft.com/en-us/library/ms144961(v=vs.110).aspx

Simple way to run methods asynchronously in .NET 4?

This is a simple .NET 4 application. Here's the code I want to run:
string username = "userfoo";
string password = "passwordfoo";
for (int i = 0; i < 2000; i++)
{
uint matchId;
if (!uint.TryParse(i.ToString(), out matchId))
{
Console.WriteLine("Invalid Match ID!");
return;
}
Client client = new Client (username, password, matchId);
// connect
client.Connect();
client.Wait();
if (client.Match != null)
{
Console.WriteLine("Inserting match: #{0}", client.Match.match_id);
Helpers.MatchHelper.AddMatchToDatabase(client.Match);
}
else
{
Console.WriteLine("Couldn't get match: #{0}", 1);
}
}
Instead doing this one by one (it would take forever - 415 days nonstop according to my calculations), what's the easiest way to invoke each iteration of this for loop asynchronously?
Most questions and articles are very old (circa 2001!) surely there must be a more modern approach?
http://msdn.microsoft.com/en-us/magazine/cc301332.aspx

You can find information here: http://msdn.microsoft.com/en-us/library/ff963552.aspx. Basically, you just use Parallel.For(0, n, x => doSomething). That takes care of parallelization. This is a functionality of PLINQ that is extremely easy to use an in my experience works quite well.
Your sample would look like this:
string username = "userfoo";
string password = "passwordfoo";
Parallel.For(0, 2000, i =>
{
uint matchId;
if (!uint.TryParse(i.ToString(), out matchId))
{
Console.WriteLine("Invalid Match ID!");
return;
}
Client client = new Client (username, password, matchId);
// connect
client.Connect();
client.Wait();
if (client.Match != null)
{
Console.WriteLine("Inserting match: #{0}", client.Match.match_id);
Helpers.MatchHelper.AddMatchToDatabase(client.Match);
}
else
{
Console.WriteLine("Couldn't get match: #{0}", 1);
}
});

I think this is what you are looking for:
http://www.codeproject.com/Articles/71285/Introducing-NET-4-0-Parallel-Programming

You should look at the task parallel library

If i understand you correctly, you want to run these in a separate thread. Here's one way to do this:
You need to move the code from the loop into a void function:
void MyThreadInsteadOfLoop(object parameter)
{
int i = (int)parameter;
uint matchId;
if (!uint.TryParse(i.ToString(), out matchId))
{
Console.WriteLine("Invalid Match ID!");
return;
}
Client client = new Client (username, password, matchId);
// connect
client.Connect();
client.Wait();
if (client.Match != null)
{
Console.WriteLine("Inserting match: #{0}", client.Match.match_id);
Helpers.MatchHelper.AddMatchToDatabase(client.Match);
}
else
{
Console.WriteLine("Couldn't get match: #{0}", 1);
}
}
In your main thread, you need to prepare threads to run, start them, and wait them to finish, if you want to. Here's the code:
//Create threads
List<Thread> threads = new List<Thread>();
for(int i=0;i<2000;i++)
{
threads.Add(new Thread(new ParameterizedThreadStart(MyThreadInsteadOfLoop)));
}
//Start threads
int x = 0;
foreach(var t in threads)
{
t.Start(x);
x++;
}
//wait for the threads to finish
foreach(var t in threads)
{
t.Join();
}
Be aware, that you have to make the MatchHelper class, and other classes that exchange data with your threads thread safe, and that tends to add lots of overhead to your program. Also, you can possibly run into trouble with the network connections.
Only [NumberOfCpuCores]*2 threads will actively work (*2 because of hyper-threading) at a time, but since you have to wait for the client (I really hope that's not a while(true) cycle cloaked) that might get concealed at least partly.

HttpWebRequest Limitations? Or bad implementation

I am trying to build a c# console app that will monitor about 3000 urls (Just need to know that HEAD request returned 200, not necessarily content, etc.)
My attempt here was to build a routine the checks the web URLS, looping and creating threads each executing the routine. What's happening is if i run with <20 threads, it executes ok most of the time, but if i use >20 threads, some of the url's time out. I tried increasing the Timeout to 30 seconds, same occurs. The network I am running this on is more than capable of executing 50 HTTP HEAD requests (10MBIT connection at ISP), and both the CPU and network run very low when executing the routine.
When a timeout occurs, i test the same IP on a browser and it works fine, I tested this repeatedly and there was never a case during testing that a "timed out" url was actually timing out.
The reason i want to run >20 threads is that i want to perform this test every 5 minutes, with some of the URL's taking a full 10sec (or higher if the timeout is set higher), i want to make sure that its able to run through all URLs within 2-3 minutes.
Is there a better way to go about checking if a URL is available, or, should I be looking at the system/network for an issue.
MAIN
while (rdr.Read())
{
Thread t = new Thread(new ParameterizedThreadStart(check_web));
t.Start(rdr[0]);
}
static void check_web(object weburl)
{
bool isok;
isok = ConnectionAvailable(weburl.ToString());
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
HttpWebRequest reqFP = (HttpWebRequest)HttpWebRequest.Create(strServer);
reqFP.Timeout = 10000;
reqFP.Method = "HEAD";
HttpWebResponse rspFP = (HttpWebResponse)reqFP.GetResponse();
if (HttpStatusCode.OK == rspFP.StatusCode)
{
Console.WriteLine(strServer + " - OK");
rspFP.Close();
return true;
}
else
{
Console.WriteLine(strServer + " Server returned error..");
rspFP.Close();
return false;
}
}
catch (WebException x)
{
if (x.ToString().Contains("timed out"))
{
Console.WriteLine(strServer + " - Timed out");
}
else
{
Console.WriteLine(x.Message.ToString());
}
return false;
}
}

Just remember, you asked.
Very bad implementation.
Do not go creating threads like that. It does very little good to have more threads than processor cores. The extra threads will pretty much just compete with each other, especially since they're all running the same code.
You need to implement using blocks. If you throw an exception (and chances are you will), then you will be leaking resources.
What is the purpose in returning a bool? Do you check it somewhere? In any case, your error and exception processing are a mess.
When you get a non-200 response, you don't display the error code.
You're comparing against the Message property to decide if it's a timeout. Microsoft should put a space between the "time" and "out" just to spite you.
When it's not a timeout, you display only the Message property, not the entire exception, and the Message property is already a string and doesn't need you to call ToString() on it.
Next Batch of Changes
This isn't finished, I don't think, but try this one:
public static void Main()
{
// Don't mind the interpretation. I needed an excuse to define "rdr"
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("SELECT Url FROM UrlsToCheck", conn))
{
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
// Use the thread pool. Please.
ThreadPool.QueueUserWorkItem(
delegate(object weburl)
{
// I invented a reason for you to return bool
if (!ConnectionAvailable(weburl.ToString()))
{
// Console would be getting pretty busy with all
// those threads
Debug.WriteLine(
String.Format(
"{0} was not available",
weburl));
}
},
rdr[0]);
}
}
}
}
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
var reqFp = (HttpWebRequest)WebRequest.Create(strServer);
reqFp.Timeout = 10000;
reqFp.Method = "HEAD";
// BTW, what's an "FP"?
using (var rspFp = (HttpWebResponse) reqFp.GetResponse()) // IDisposable
{
if (HttpStatusCode.OK == rspFp.StatusCode)
{
Debug.WriteLine(string.Format("{0} - OK", strServer));
return true; // Dispose called when using is exited
}
// Include the error because it's nice to know these things
Debug.WriteLine(String.Format(
"{0} Server returned error: {1}",
strServer, rspFp.StatusCode));
return false;
}
}
catch (WebException x)
{
// Don't tempt fate and don't let programs read human-readable messages
if (x.Status == WebExceptionStatus.Timeout)
{
Debug.WriteLine(string.Format("{0} - Timed out", strServer));
}
else
{
// The FULL exception, please
Debug.WriteLine(x.ToString());
}
return false;
}
}
Almost Done - Not Tested Late Night Code
public static void Main()
{
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("", conn))
{
using (var rdr = cmd.ExecuteReader())
{
if (rdr == null)
{
return;
}
while (rdr.Read())
{
ThreadPool.QueueUserWorkItem(
CheckConnectionAvailable, rdr[0]);
}
}
}
}
}
private static void CheckConnectionAvailable(object weburl)
{
try
{
// If this works, it's a lot simpler
var strServer = new Uri("http://" + weburl);
using (var client = new WebClient())
{
client.UploadDataCompleted += ClientOnUploadDataCompleted;
client.UploadDataAsync(
strServer, "HEAD", new byte[] {}, strServer);
}
}
catch (WebException x)
{
Debug.WriteLine(x);
}
}
private static void ClientOnUploadDataCompleted(
object sender, UploadDataCompletedEventArgs args)
{
if (args.Error == null)
{
Debug.WriteLine(string.Format("{0} - OK", args.UserState));
}
else
{
Debug.WriteLine(string.Format("{0} - Error", args.Error));
}
}

Use ThreadPool class. Don't spawn hundreds of threads like this. Threads have such a huge overhead and what happens in your case is that your CPU will spend 99% time on context switching and 1% doing real work.

Don't use threads.
Asynch Call backs and queues. Why create a thread when the resource that they are all wanting is access to the outside world. Limit your threads to about 5, and then implement a class that uses a queue. split the code into two parts, the fetch and the process. One controls the flow of data while the other controls access to the outside world.
Use whatever language you like but you won't got wrong if you think that threads are for processing and number crunching and async call backs are for resource management.