We Keep Coding

ThreadPool race-condition, closure, locking or something else?

Still pretty new to threads so I'm sure it is one of those little gotchas and a repeat question, but I have been unable to find the answer browsing the threads.
I have a port scanner app in C#.
I'm using threadpools to spin up a new TcpClient for each port and probe if it's open.
After suffering through the concepts of closures and thread synchronization, I am having an issue where when multiple threads try to save their results to different indexes in the Orchestrator.hosts (List).
I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write, however I'm finding that on some updates, multiple entries are getting the same update.
IE, Thread #1 supposed to update Hosts[0].Ports[0].Status to "Open",
What happens:
Thread #1 updates multiple host with the port result despite passing a specific index for Hosts.
Hosts[0].Ports[0].Status to "Open",
Hosts[1].Ports[0].Status to "Open",
Hosts[2].Ports[0].Status to "Open",
Not sure where my problem is. The Static method I'm calling to perform a probe of a given port
public static void ScanTCPPorts()
{
// Create a list of portsToScan objects to send to thread workers
//List<ScanPortRequest> portsToScan = new List<ScanPortRequest>();
using (ManualResetEvent resetEvent = new ManualResetEvent(false))
{
int toProcess = 0;
for (var i = 0; i < hostCount; i++) // Starting at Begining
{
int currentHostId = i;
// To hold our current hosts ID (Assign outside of threaded function to avoid race-condition)
if (hosts[i].IsAlive || scanDefinition.isForced())
{
int portCount = hosts[i].Ports.Count;
for (int p = 0; p < portCount; p++)
{
// Thread-safe Increment our workQueue counter
Interlocked.Increment(ref toProcess);
int currentPortPosition = p;
// We need to send the arrayIndex in to the thread function
PortScanRequestResponse portRequestResponse = new PortScanRequestResponse(hosts[currentHostId], currentHostId, hosts[currentHostId].Ports[currentPortPosition], currentPortPosition);
ThreadPool.QueueUserWorkItem(
new WaitCallback(threadedRequestResponseInstance => {
PortScanRequestResponse portToScan = threadedRequestResponseInstance as PortScanRequestResponse;
PortScanRequestResponse threadResult = PortScanner.scanTCPPort(portToScan);
// Lock so Thread-safe update to result
lock (Orchestrator.hosts[portToScan.hostResultIndex])
{
if (threadResult.port.status == PortStatus.Open)
{
// Update result
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Open;
//Logger.Log(hosts[currentHostId].IPAddress + " " + hosts[currentHostId].Ports[currentPortPosition].type + " " + hosts[currentHostId].Ports[currentPortPosition].portNumber + " is open");
}
else
{
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Closed;
}
// Check if this was the last scan for the given host
if (Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete != true)
{
if (Orchestrator.hosts[portToScan.hostResultIndex].isCompleted())
{
Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete = true;
// Logger.Log(hosts[currentHostId].IPAddress + " has completed a port scan");
Orchestrator.hosts[portToScan.hostResultIndex].PrintPortSummery();
}
}
}
// Safely decrement the counter
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}), portRequestResponse); // Pass in our Port to scan
}
}
}
resetEvent.WaitOne();
}
}
Here is the worker process in a separate public static class.
public static PortScanRequestResponse scanTCPPort(object portScanRequest) {
PortScanRequestResponse portScanResponse = portScanRequest as PortScanRequestResponse;
HostDefinition host = portScanResponse.host;
ScanPort port = portScanResponse.port;
try
{
using (TcpClient threadedClient = new TcpClient())
{
try
{
IAsyncResult result = threadedClient.BeginConnect(host.IPAddress, port.portNumber, null, null);
Boolean success = result.AsyncWaitHandle.WaitOne(Orchestrator.scanDefinition.GetPortTimeout(), false);
if (threadedClient.Client != null)
{
if (success)
{
threadedClient.EndConnect(result);
threadedClient.Close();
portScanResponse.port.status = PortStatus.Open;
return portScanResponse;
}
}
} catch { }
}
}
catch
{ }
portScanResponse.port.status = PortStatus.Closed;
return portScanResponse;
}
Originally I was pulling the host index from a free variable, thinking this was the problem moved it to inside the delegate.
I tried locking the Hosts object everywhere there was a write.
I have tried different thread sync techniques (CountdownEvent and ManualResetEvent).
I think there is just some fundamental threading principal I have not been introduced to yet, or I have made a very simple logic mistake.

I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write.
I haven't studied your code, but the above statement alone is incorrect. When a List<T>, or any other non-thread-safe object , is used in a multithreaded environment, all interactions with the object must be synchronized. Only one thread at a time should be allowed to interact with the object. Both writes and reads must be enclosed in lock statements, using the same locker object. Even reading the Count must be synchronized. Otherwise the usage is erroneous, and the behavior of the program is undefined.

I was hyper-focused on it being a thread issue because this was my firsts threaded project. Turned out to be that I didn't realize copies of a List<> objects are references to their original object (reference type). I assumed my threads were accessing my save structure in an unpredictable way, but my arrays of ports were all referencing the same object.
This was a "reference type" vs "value type" issue on my List<> of ports.

IP Protection using loops, C#

So today I was messing around with some IP protection for a program I am making.
The whitelist for IPs is hosted on pastebin.
My program downloads the IPs, splits it into an array of strings.
My program also checks the ip using https://wtfismyip.com/text
What I want to do is compare each string from the array and check if it is the current IP.
If the IP is not the current IP by the end, then it will close.
How would I do this?
Example of the code:
for (int i = 0; i < iplist.Length; i++)
{
if(iplist[i] == WebIP)
{
MessageBox.Show("Passed");
}
else
{
this.Close();
}
}

You can use some LINQ to make the code more readable:
if (iplist.Any(ip => ip == WebIP))
{
MessageBox.Show("Passed");
}
else
{
this.Close();
}

assuming you're going to test lots of times, create a HashSet<T> of the whitelist (or similarly: blacklist); assuming it is a string:
var whiteList = new HashSet<string>(iplist);
(do this once, not every time you need to check)
then just check .Contains:
bool isOK = whiteList.Contains(WebIP);
job done, and very efficiently

You can have a flag and set it accordingly like
bool flag = false;
for (int i = 0; i < iplist.Length; i++)
{
if(iplist[i] == WebIP)
{
flag = true;
break;
}
}
if(flag)
MessageBox.Show("Passed");
else
this.Close();

Redis Booksleeve client, ResultCompletionMode.PreserveOrder not working

When I print out received messages on the Console the displayed messages are all messed up, each message containing 5 string sub-messages that are printed on the Console before control reverts back to the incoming message callback. I strongly assume this is because the incoming message event is raised async in Booksleeve?
I refer to the following post, How does PubSub work in BookSleeve/ Redis?, where the author, Marc Gravell, pointed to the ability to force sync reception by setting Completion Mode to "PreserveOrder". I have done that, tried before and after connecting the client. Neither seems to work.
Any ideas how I can receive messages and print them on the console in the exact order they were sent? I only have one single publisher in this case.
Thanks
Edit:
Below some code snippets to show how I send messages and the Booksleeve wrapper I quickly wrote.
Here the client (I have a similar Client2 that receives the messages and checks order, but I omitted it as it seems trivial).
class Client1
{
const string ClientId = "Client1";
private static Messaging Client { get; set; }
private static void Main(string[] args)
{
var settings = new MessagingSettings("127.0.0.1", 6379, -1, 60, 5000, 1000);
Client = new Messaging(ClientId, settings, ReceiveMessage);
Client.Connect();
Console.WriteLine("Press key to start sending messages...");
Console.ReadLine();
for (int index = 1; index <= 100; index++)
{
//I turned this off because I want to preserve
//the order even if messages are sent in rapit succession
//Thread.Sleep(5);
var msg = new MessageEnvelope("Client1", "Client2", index.ToString());
Client.SendOneWayMessage(msg);
}
Console.WriteLine("Press key to exit....");
Console.ReadLine();
Client.Disconnect();
}
private static void ReceiveMessage(MessageEnvelope msg)
{
Console.WriteLine("Message Received");
}
}
Here the relevant code snippets of the library:
public void Connect()
{
RequestForReplyMessageIds = new ConcurrentBag<string>();
Connection = new RedisConnection(Settings.HostName, Settings.Port, Settings.IoTimeOut);
Connection.Closed += OnConnectionClosed;
Connection.CompletionMode = ResultCompletionMode.PreserveOrder;
Connection.SetKeepAlive(Settings.PingAliveSeconds);
try
{
if (Connection.Open().Wait(Settings.RequestTimeOutMilliseconds))
{
//Subscribe to own ClientId Channel ID
SubscribeToChannel(ClientId);
}
else
{
throw new Exception("Could not connect Redis client to server");
}
}
catch
{
throw new Exception("Could not connect Redis Client to Server");
}
}
public void SendOneWayMessage(MessageEnvelope message)
{
SendMessage(message);
}
private void SendMessage(MessageEnvelope msg)
{
//Connection.Publish(msg.To, msg.GetByteArray());
Connection.Publish(msg.To, msg.GetByteArray()).Wait();
}
private void IncomingChannelSubscriptionMessage(string channel, byte[] body)
{
var msg = MessageEnvelope.GetMessageEnvelope(body);
//forward received message
ReceivedMessageCallback(msg);
//release requestMessage if returned msgId matches
string msgId = msg.MessageId;
if (RequestForReplyMessageIds.Contains(msgId))
{
RequestForReplyMessageIds.TryTake(out msgId);
}
}
public void SubscribeToChannel(string channelName)
{
if (!ChannelSubscriptions.Contains(channelName))
{
var subscriberChannel = Connection.GetOpenSubscriberChannel();
subscriberChannel.Subscribe(channelName, IncomingChannelSubscriptionMessage).Wait();
ChannelSubscriptions.Add(channelName);
}
}

Without seeing exactly how you are checking for this, it is hard to comment, but what I can say is that any threading oddity is going to be hard to track down and fix, and is therefore very unlikely to be addressed in BookSleeve, given that it has been succeeded. However! It will absolutely be checked in StackExchange.Redis. Here's the a rig I've put together in SE.Redis (and, embarrassingly, it did highlight a slight bug, fixed in next release, so .222 or later); output first:
Subscribing...
Sending (preserved order)...
Allowing time for delivery etc...
Checking...
Received: 500 in 2993ms
Out of order: 0
Sending (any order)...
Allowing time for delivery etc...
Checking...
Received: 500 in 341ms
Out of order: 306
(keep in mind that 500 x 5ms is 2500, so we should not be amazed by the 2993ms number, or the 341ms - this is mainly the cost of the Thread.Sleep we have added to nudge the thread-pool into overlapping them; if we remove that, both loops take 0ms, which is awesome - but we can't see the overlapping issue so convincingly)
As you can see, the first run has the correct order output; the second run has mixed order, but it ten times faster. And that is when doing trivial work; for real work it would be even more noticeable. As always, it is a trade-off.
Here's the test rig:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using StackExchange.Redis;
static class Program
{
static void Main()
{
using (var conn = ConnectionMultiplexer.Connect("localhost"))
{
var sub = conn.GetSubscriber();
var received = new List<int>();
Console.WriteLine("Subscribing...");
const int COUNT = 500;
sub.Subscribe("foo", (channel, message) =>
{
lock (received)
{
received.Add((int)message);
if (received.Count == COUNT)
Monitor.PulseAll(received); // wake the test rig
}
Thread.Sleep(5); // you kinda need to be slow, otherwise
// the pool will end up doing everything on one thread
});
SendAndCheck(conn, received, COUNT, true);
SendAndCheck(conn, received, COUNT, false);
}
Console.WriteLine("Press any key");
Console.ReadLine();
}
static void SendAndCheck(ConnectionMultiplexer conn, List<int> received, int quantity, bool preserveAsyncOrder)
{
conn.PreserveAsyncOrder = preserveAsyncOrder;
var sub = conn.GetSubscriber();
Console.WriteLine();
Console.WriteLine("Sending ({0})...", (preserveAsyncOrder ? "preserved order" : "any order"));
lock (received)
{
received.Clear();
// we'll also use received as a wait-detection mechanism; sneaky
// note: this does not do any cheating;
// it all goes to the server and back
for (int i = 0; i < quantity; i++)
{
sub.Publish("foo", i);
}
Console.WriteLine("Allowing time for delivery etc...");
var watch = Stopwatch.StartNew();
if (!Monitor.Wait(received, 10000))
{
Console.WriteLine("Timed out; expect less data");
}
watch.Stop();
Console.WriteLine("Checking...");
lock (received)
{
Console.WriteLine("Received: {0} in {1}ms", received.Count, watch.ElapsedMilliseconds);
int wrongOrder = 0;
for (int i = 0; i < Math.Min(quantity, received.Count); i++)
{
if (received[i] != i) wrongOrder++;
}
Console.WriteLine("Out of order: " + wrongOrder);
}
}
}
}

How to make multithreaded app with foreach loop

Have an app which sort large txt file by some conditions.
I need to start for example 5 threads but i read line by line from file with foreach loop.
And if I start 5 threads with my code, all threads will take same lines.
Here is my code where i start 1 thread:
Thread[] thr;
private void button1_Click(object sender, EventArgs e)
{
button1.Enabled = false;
button4.Enabled = true;
decimal value = 1;
int i = 0;
int j = (int)(value);
thr = new Thread[j];
for (; i < j; i++)
{
thr[i] = new Thread(new ThreadStart(go));
thr[i].IsBackground = true;
thr[i].Start();
}
}
private static IEnumerable<string> ReadLineFromFile(TextReader fileReader)
{
using (fileReader)
{
string currentLine;
while ((currentLine = fileReader.ReadLine()) != null)
{
yield return currentLine;
}
}
}
public void go()
{
while (true)
{
TextReader readFile = new StreamReader(file_path, System.Text.Encoding.UTF8, true);
foreach (string line in ReadLineFromFile(readFile))
{
if (line.Split(':')[0].Contains("#"))
{
string out_line = line.Split(':')[0].Replace("+", "") + ":" + line.Split(':')[1];
lock (locker)
{
mail_count++;
log_mail(mail_count);
mail.Add(out_line.Trim().Replace(";", ":"));
}
}
else
{
string out_line = line.Split(':')[0].Replace("+", "") + ":" + line.Split(':')[1];
lock (locker)
{
rubbish_count++;
log_rubbish(rubbish_count);
rubbish.Add(out_line.Trim());
}
}
}
MessageBox.Show("Ready");
BeginInvoke(
new MethodInvoker(() =>
{
button1.Enabled = true;
button4.Enabled = false;
}));
break;
}
}

It's no use for all threads to read the same file and reading from a shared file is difficult and inefficient.
In your main function, you would need something like:
Parallel.ForEach(System.IO.File.ReadLines(file_path, System.Text.Encoding.UTF8),
line => ProcessOneLine(line)
);
and then ProcessOneLine would do the .Split(':') etc.

Why not going with usual producer-consumer pattern? Make one thread read file, put lines in some shared collection, and other threads just pick data from the collection and process it.
More - you can read from file and for each line create Task that will take care of processing this line and put the result in output collection.
This seems better than 5 threads trying to read the same file and not reading the same line multiple times.

I would like to confirm and expand on what Pako said. The other threads should use data from a shared collection containing the data and process it.
Having multiple threads access a text file sounds like a potential for race conditions to occur. Essentially unpredictable outcomes can occur if a thread is changing a file whilst another thread is reading from it.
I've also experienced BSODs in the past when using multiple threads which access the same text file and can recommend against it. If you are insistent on doing this, however, I'd recommend you take a look at the "lock" keyword and the singleton design pattern. This will allow you to make sure that only a single thread is accessing the file at a time.
Related links:
http://msdn.microsoft.com/en-us/library/c5kehkcz(v=vs.80).aspx
http://en.wikipedia.org/wiki/Singleton_pattern
http://en.wikipedia.org/wiki/Double-checked_locking

HttpWebRequest Limitations? Or bad implementation

I am trying to build a c# console app that will monitor about 3000 urls (Just need to know that HEAD request returned 200, not necessarily content, etc.)
My attempt here was to build a routine the checks the web URLS, looping and creating threads each executing the routine. What's happening is if i run with <20 threads, it executes ok most of the time, but if i use >20 threads, some of the url's time out. I tried increasing the Timeout to 30 seconds, same occurs. The network I am running this on is more than capable of executing 50 HTTP HEAD requests (10MBIT connection at ISP), and both the CPU and network run very low when executing the routine.
When a timeout occurs, i test the same IP on a browser and it works fine, I tested this repeatedly and there was never a case during testing that a "timed out" url was actually timing out.
The reason i want to run >20 threads is that i want to perform this test every 5 minutes, with some of the URL's taking a full 10sec (or higher if the timeout is set higher), i want to make sure that its able to run through all URLs within 2-3 minutes.
Is there a better way to go about checking if a URL is available, or, should I be looking at the system/network for an issue.
MAIN
while (rdr.Read())
{
Thread t = new Thread(new ParameterizedThreadStart(check_web));
t.Start(rdr[0]);
}
static void check_web(object weburl)
{
bool isok;
isok = ConnectionAvailable(weburl.ToString());
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
HttpWebRequest reqFP = (HttpWebRequest)HttpWebRequest.Create(strServer);
reqFP.Timeout = 10000;
reqFP.Method = "HEAD";
HttpWebResponse rspFP = (HttpWebResponse)reqFP.GetResponse();
if (HttpStatusCode.OK == rspFP.StatusCode)
{
Console.WriteLine(strServer + " - OK");
rspFP.Close();
return true;
}
else
{
Console.WriteLine(strServer + " Server returned error..");
rspFP.Close();
return false;
}
}
catch (WebException x)
{
if (x.ToString().Contains("timed out"))
{
Console.WriteLine(strServer + " - Timed out");
}
else
{
Console.WriteLine(x.Message.ToString());
}
return false;
}
}

Just remember, you asked.
Very bad implementation.
Do not go creating threads like that. It does very little good to have more threads than processor cores. The extra threads will pretty much just compete with each other, especially since they're all running the same code.
You need to implement using blocks. If you throw an exception (and chances are you will), then you will be leaking resources.
What is the purpose in returning a bool? Do you check it somewhere? In any case, your error and exception processing are a mess.
When you get a non-200 response, you don't display the error code.
You're comparing against the Message property to decide if it's a timeout. Microsoft should put a space between the "time" and "out" just to spite you.
When it's not a timeout, you display only the Message property, not the entire exception, and the Message property is already a string and doesn't need you to call ToString() on it.
Next Batch of Changes
This isn't finished, I don't think, but try this one:
public static void Main()
{
// Don't mind the interpretation. I needed an excuse to define "rdr"
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("SELECT Url FROM UrlsToCheck", conn))
{
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
// Use the thread pool. Please.
ThreadPool.QueueUserWorkItem(
delegate(object weburl)
{
// I invented a reason for you to return bool
if (!ConnectionAvailable(weburl.ToString()))
{
// Console would be getting pretty busy with all
// those threads
Debug.WriteLine(
String.Format(
"{0} was not available",
weburl));
}
},
rdr[0]);
}
}
}
}
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
var reqFp = (HttpWebRequest)WebRequest.Create(strServer);
reqFp.Timeout = 10000;
reqFp.Method = "HEAD";
// BTW, what's an "FP"?
using (var rspFp = (HttpWebResponse) reqFp.GetResponse()) // IDisposable
{
if (HttpStatusCode.OK == rspFp.StatusCode)
{
Debug.WriteLine(string.Format("{0} - OK", strServer));
return true; // Dispose called when using is exited
}
// Include the error because it's nice to know these things
Debug.WriteLine(String.Format(
"{0} Server returned error: {1}",
strServer, rspFp.StatusCode));
return false;
}
}
catch (WebException x)
{
// Don't tempt fate and don't let programs read human-readable messages
if (x.Status == WebExceptionStatus.Timeout)
{
Debug.WriteLine(string.Format("{0} - Timed out", strServer));
}
else
{
// The FULL exception, please
Debug.WriteLine(x.ToString());
}
return false;
}
}
Almost Done - Not Tested Late Night Code
public static void Main()
{
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("", conn))
{
using (var rdr = cmd.ExecuteReader())
{
if (rdr == null)
{
return;
}
while (rdr.Read())
{
ThreadPool.QueueUserWorkItem(
CheckConnectionAvailable, rdr[0]);
}
}
}
}
}
private static void CheckConnectionAvailable(object weburl)
{
try
{
// If this works, it's a lot simpler
var strServer = new Uri("http://" + weburl);
using (var client = new WebClient())
{
client.UploadDataCompleted += ClientOnUploadDataCompleted;
client.UploadDataAsync(
strServer, "HEAD", new byte[] {}, strServer);
}
}
catch (WebException x)
{
Debug.WriteLine(x);
}
}
private static void ClientOnUploadDataCompleted(
object sender, UploadDataCompletedEventArgs args)
{
if (args.Error == null)
{
Debug.WriteLine(string.Format("{0} - OK", args.UserState));
}
else
{
Debug.WriteLine(string.Format("{0} - Error", args.Error));
}
}

Use ThreadPool class. Don't spawn hundreds of threads like this. Threads have such a huge overhead and what happens in your case is that your CPU will spend 99% time on context switching and 1% doing real work.

Don't use threads.
Asynch Call backs and queues. Why create a thread when the resource that they are all wanting is access to the outside world. Limit your threads to about 5, and then implement a class that uses a queue. split the code into two parts, the fetch and the process. One controls the flow of data while the other controls access to the outside world.
Use whatever language you like but you won't got wrong if you think that threads are for processing and number crunching and async call backs are for resource management.