We Keep Coding

Maximize Task Concurrency - using TCP/IP with specific Port Filtering

This is a follow-up question to this question. On the next level, I now want to use maximal task concurrency to connect to expected hosts on a large set of IP addresses, using TCP/IP on a specific port.
My own research, as well as community reference, has lead me to key articles, for example:
How to check TCP/IP port availability using C# (Socket Communication)
Checking if ip with port is available?
How to set the timeout for a TcpClient?
A very impressive solution for large-scale pinging: Multithreading C# GUI ping example
And of course the precursor to this question: C#, Maximize Thread Concurrency
This allowed me to set up my own code, which works fine, but currently takes a full 30 seconds to finish scanning 255 IPs, using only one specific port. Given the test, machine has 8 logical cores this observation suggests that my construct actually spawns at maximum 8 concurrent tasks (255/8=31.85).
The function I wrote returns a list of responding IPs {IPs} which is a subset of the List of all IPs {IP_Ports} to be checked. This is my current code, working fine but not yet suitable for use on larger networks due to what I suspect is lack of efficient task concurrency:
// Check remote host connectivity
public static class CheckRemoteHost
{
// Private Class members
private static bool AllDone = false;
private static object lockObj = new object();
private static List<string> IPs;
// Wrapper: manage async method <TCP_check>
public static List<string> TCP(Dictionary<string, int> IP_Ports, int TimeoutInMS = 100)
{
// Locals
IPs = new List<string>();
// Perform remote host check
AllDone = false;
TCP_check(IP_Ports, TimeoutInMS);
while (!AllDone) { Thread.Sleep(50); }
// Finish
return IPs;
}
private static async void TCP_check(Dictionary<string, int> IP_Ports, int timeout)
{// async worker method: check remote host via TCP-IP
// Build task-set for parallel IP queries
var tasks = IP_Ports.Select(host => TCP_IPAndUpdateAsync(host.Key, host.Value, timeout));
// Start execution queue
await Task.WhenAll(tasks).ContinueWith(t =>
{
AllDone = true;
});
}
private static async Task TCP_IPAndUpdateAsync(string ip, int port, int timeout)
{// method to call IP-check
// Run method asynchronously
await Task.Run(() =>
{
// Locals
TcpClient client;
IAsyncResult result;
bool success;
try
{
client = new TcpClient();
result = client.BeginConnect(ip, port, null, null);
success = result.AsyncWaitHandle.WaitOne(TimeSpan.FromMilliseconds(timeout));
if (success)
{
lock (lockObj)
{
IPs.Add(ip);
}
}
}
catch (Exception e)
{
// do nothing
}
});
}
}// end public static class CheckRemoteHost
So my question is: how can I maximize the task concurrency of requesting a response using TCP/IP at Port X such that I can obtain very fast IP-Port network scans on large internal networks?

Details
The default task scheduler is usually the ThreadPool scheduler. That means the number of concurrent tasks will be limited by the available threads in the pool.
Remarks
The thread pool provides new worker threads or I/O completion threads on demand until it reaches the minimum for each category. By default, the minimum number of threads is set to the number of processors on a system. When the minimum is reached, the thread pool can create additional threads in that category or wait until some tasks complete. Beginning with the .NET Framework 4, the thread pool creates and destroys threads in order to optimize throughput, which is defined as the number of tasks that complete per unit of time. Too few threads might not make optimal use of available resources, whereas too many threads could increase resource contention.
(Source: https://msdn.microsoft.com/en-us/library/system.threading.threadpool.getminthreads(v=vs.110).aspx)
You are likely just under the threshold where the threadpool would spin up new threads since tasks are being completed. Hence why you only have 8 concurrent tasks running at once.
Solutions
1. Use ConnectAsync with a timeout.
Instead of creating a separate task which blocks waiting for the connect. You can call ConnectAsync and join it with a delay to create a timeout. ConnectAsync doesn't seem to block the threadpool threads.
public static async Task<bool> ConnectAsyncWithTimeout(this Socket socket, string host, int port, int timeout = 0)
{
if (timeout < 0)
throw new ArgumentOutOfRangeException("timeout");
try
{
var connectTask = socket.ConnectAsync(host, port);
var res = await Task.WhenAny(connectTask, Task.Delay(timeout));
await res;
return connectTask == res && connectTask.IsCompleted && !connectTask.IsFaulted;
}
catch(SocketException se)
{
return false;
}
}
Example usage
private static async Task TCP_IPAndUpdateAsync(string ip, int port, int timeout)
{// method to call IP-check
client = new TcpClient();
var success = await client.Client.ConnectAsyncWithTimeout(ip, port, timeout);
if (success)
{
lock (lockObj)
{
IPs.Add(ip);
}
}
}
2. Use long running tasks.
Using Task.Factor.StartNew you can specify that the task is LongRunning. The threadpool task scheduler specifically will create a new thread for the task instead of using the threadpool. This will get around the 8 thread limit you are hitting. However, it should be noted that this is not a good solution if you plan to naively create thousands of tasks. Since at that point, the bottle neck will be thread context switches. You could however split all of the work between, for example, 100 tasks.
3. Use non-blocking connect
This method doesn't require creating multiple tasks. Instead you can call multiple connects on a single thread and check the status of multiple sockets at once. This method is a bit more involved though. If you rather go with this approach and want a more complete example then comment letting me know. Here is a quick snippet on how to use the API.
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
socket.Blocking = false;
try
{
socket.Connect("127.0.0.1", 12345);
}
catch(SocketException se)
{
//Ignore the "A non-blocking socket operation could not be completed immediately" error
if (se.ErrorCode != 10035)
throw;
}
//Check the connection status of the socket.
var writeCheck = new List<Socket>() { socket };
var errorCheck = new List<Socket>() { socket };
Socket.Select(null, writeCheck, errorCheck, 0);
if (writeCheck.Contains(socket))
{
//Connection opened successfully.
}
else if (errorCheck.Contains(socket))
{
//Connection refused
}
else
{
//Connect still pending
}

Lock on dedicated lock-object causes deadlock

Yesterday I came accross a strange problem, which gave me quite some headaches. I have a server application with a Server class, which in turn is derived from a Connection class. The Connection class provides information about the connection state and the possibility to close the connection
public bool Connected
{
get
{
if (connection != null)
{
lock (lockObject)
{
bool blockingState = connection.Blocking;
try
{
connection.Blocking = false;
connection.Send(new byte[1], 1, 0);
}
catch (SocketException e)
{
if (!e.NativeErrorCode.Equals(10035))
{
return false;
}
//is connected, but would block
}
finally
{
connection.Blocking = blockingState;
}
return connection.Connected;
}
}
return false;
}
}
public virtual void CloseConnection()
{
if (Connected)
{
lock (lockObject)
{
connection.Close();
}
}
}
The Server class is resonsible for actually sending data
private void ConnectAndPollForData()
{
try
{
TcpListener listener = new TcpListener(Port);
listener.Start();
while (true)
{
connection = listener.AcceptSocket();
string currentBuffr = string.Empty;
const int READ_BUFFER_SIZE = 1024;
byte[] readBuffr = new byte[READ_BUFFER_SIZE];
while (Connected)
{
int bytesReceived;
lock (lockObject)
{
bytesReceived = connection.Receive(readBuffr, READ_BUFFER_SIZE, SocketFlags.None);
}
currentBuffr += ASCIIEncoding.ASCII.GetString(readBuffr, 0, bytesReceived);
//do stuff
}
}
catch(ThreadAbortException)
{
Thread.ResetAbort();
}
finally
{
}
}
public void SendString(string stringToSend)
{
stringToSend += "\r\n";
if(Connected)
{
lock(lockObject)
{
connection.Send(ASCIIEncoding.UTF7.GetBytes(stringToSend));
}
}
}
There is no other explicit access to the connection object. The ConnectAndPollForData function executes in a seperate thread. Whenever I ran the host in this version (I am currently using a non thread-safe version, which causes other problems) it hang after quite a few lines received via TCP. Pausing the debugger showed me, that one thread tried to execute the code withing the lock of Connected, while the other tried to receive data in the lock of ConnectAndPollForData. This behavior seems strange to me, for I would expect to execute the code within the first lock and then do the second. There do seem to be similar problems when using callbacks like in Deadlocking lock() method or 'Deadlock' with only one locked object? but the situation here is a bit different, for in my situation (I think) the code within the locks should not emit any events that themselves try to obtain a lock on the object.

Let's assume it gets the lock in the second method first. So it is holding the lock, and waiting for data. It is unclear whether this is directly receiving the data sent by the first method, or whether this is looking for a reply from an unrelated server - a reply to the message sent in the first method. But either way, I'm assuming that there won't be data incoming until the outbound message is sent.
Now consider: the outbound message can't be sent, because you are holding an exclusive lock.
So yes, you've deadlocked yourself. Basically, don't do that. There is no need to synchronize between inbound and outbound socket operations, even on the same socket. And since it makes very little sense to have concurrent readers on the same socket, or concurrent writers, I'm guessing you don't actually need those locks at all.

abort a thread or just let it to be cleaned by the computer?

I have the following code, I want to abort a thread if it is unfinished in 2 secs.
You can see from the first code that i create a new myThread evertyime in the while loop and do not abort it. Well, I dont want to it to be like this, but if i take the myThread outside the loop and use the abort() function as the second code does. there will be error in aborting.
while (true)
{
try
{
m_socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);//reset up socket
myThread = new System.Threading.Thread(new System.Threading.ThreadStart(socket_connect));
myThread.Start();
if (!myThread.Join(2000))
{
throw new SocketException(SocketError.AccessDenied);
}
}
catch (Exception ex)
{
m_socket.Close();
}
}
}
private static void socket_connect()
{
m_socket.Connect(remoteEndPoint);//Connect to remote device
}
I was trying the following code at first, however it give threadabortexceptions.
myThread = new System.Threading.Thread(new System.Threading.ThreadStart(socket_connect));
while (true)
{ try
{
m_socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);//reset up socket
myThread.Start();
if (!myThread.Join(2000))
{
myThread.Abort();
throw new SocketException(SocketError.AccessDenied);
}
}
catch (Exception ex)
{
m_socket.Close();
}
}
}
private static void socket_connect()
{
m_socket.Connect(remoteEndPoint);//Connect to remote device
}
I know abort() is not a good idea so I turn to let the threads stay and let C#(.Net? I dont know who actually does that) dealing with the garbage collection. And can anyone tell if it is a good idea since this program will be run on a board which does not have lots of memory for holding bunches of threads. And can someone tell me how exactly the garbage collection is done in C#? for example the threads.
Another thing need to be mention is that i dont have the Task class, or the socket.beginconnect() method, I guess its because i'm building a program that is going to run on a small board, not a pc. The board is a netduido plus and I'm building my project on a netduino plus platform.

I have the following code, I want to abort a thread if it is unfinished in 2 secs.
Reading into the code, it looks like you actually want to attempt to connect a socket to a resource within two seconds. If more than two seconds elapses, you want to move on.
I'm mostly reproducing the code in this answer, I believe this is approximately what you should be doing to achieve your goal, rather than spinning up a thread and aborting it:
Socket socket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp);
// Connect using a timeout (2 seconds)
IAsyncResult result = socket.BeginConnect( sIP, iPort, null, null );
bool success = result.AsyncWaitHandle.WaitOne( 2000, true );
if ( !success )
{
// NOTE, MUST CLOSE THE SOCKET
socket.Close();
throw new ApplicationException("Failed to connect server.");
}
// Success
//...

[edit: massive copypasta fail when untabbing/spacing]
Oh, please use the Task library, it's way easier to deal with these situations:
(LINQPad-friendly blob)
void Main()
{
var canceller = new CancellationTokenSource();
var task = Task.Factory.StartNew(() => DoStuff(canceller.Token), canceller.Token);
if(!task.Wait(2000, canceller.Token))
{
canceller.Cancel();
task.Wait(2);
}
sw.Elapsed.Dump();
}
private Stopwatch sw;
private void DoStuff(CancellationToken token)
{
try
{
sw = Stopwatch.StartNew();
while(!token.IsCancellationRequested)
{
}
}
// no catch - rethrown exceptions must be checked on Task
finally
{
sw.Stop();
}
}
Alternatively, you can use some "exit flag" condition - a bool that both your thread-starter and your thread-runner can see/alter, and use that in your while condition.

Why checking proxies with ThreadPool takes much more time than doing the same without ThreadPool?

Using this post I wrote code that checks 200 proxies for example. The timeout for a socket is 2sec. Everything is working, but the problem that Code #1 takes more than 2minutes to check 200 proxies limited to 2sec timeout. But with Code #2 it takes 2sec to check 200 proxies and it would take also 2sec to check 1000 proxies with Code #2.
Code #1 uses ThreadPool.
Code #1 opens proxyCount sockets, goes to Sleep for 2sec and than checks what succeeded. It takes 2sec exactly.
So where is the problem in Code #1? Why ThreadPool with minimum 20 threads are much much slower than doing it without threads?
Code #1
int proxyCount = 200;
CountdownEvent cde = new CountdownEvent(proxyCount);
private void RefreshProxyIPs(object obj)
{
int workerThreads, ioThreads;
ThreadPool.GetMinThreads(out workerThreads, out ioThreads);
ThreadPool.SetMinThreads(20, ioThreads);
var proxies = GetServersIPs(proxyCount);
watch.Start();
for (int i = 0; i < proxyCount; i++)
{
var proxy = proxies[i];
ThreadPool.QueueUserWorkItem(CheckProxy, new IPEndPoint(IPAddress.Parse(proxy.IpAddress), proxy.Port));
}
cde.Wait();
cde.Dispose();
watch.Stop();
}
private List<IPEndPoint> list = new List<IPEndPoint>();
private void CheckProxy(object o)
{
var proxy = o as IPEndPoint;
using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
{
var asyncResult = socket.BeginConnect(proxy.Address, proxy.Port, null, null);
if (asyncResult.AsyncWaitHandle.WaitOne(2000))
{
try
{
socket.EndConnect(asyncResult);
}
catch (SocketException)
{
}
catch (ObjectDisposedException)
{
}
}
if (socket.Connected)
{
list.Add(proxy);
socket.Close();
}
}
cde.Signal();
}
Code #2
int proxyCount = 200;
var sockets = new Socket[proxyCount];
var socketsResults = new IAsyncResult[proxyCount];
var proxies = GetServersIPs(proxyCount);
for (int i = 0; i < proxyCount; i++)
{
var proxy = proxies[i];
sockets[i] = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socketsResults[i] = sockets[i].BeginConnect(IPAddress.Parse(proxy.IpAddress), proxy.Port, null, proxy);
}
Thread.Sleep(2000);
for (int i = 0; i < proxyCount; i++)
{
var success = false;
try
{
if (socketsResults[i].IsCompleted)
{
sockets[i].EndConnect(socketsResults[i]);
success = sockets[i].Connected;
sockets[i].Close();
}
sockets[i].Dispose();
}
catch { }
var proxy = socketsResults[i].AsyncState as Proxy;
if (success) { _validProxies.Add(proxy); }
}

The threadpool threads you start are just not very good tp threads. They don't perform any real work but just block on the WaitOne() call. So 20 of them start executing right away and don't complete for 2 seconds. the threadpool scheduler only allows another thread to start when one of them completes or none of them complete within 0.5 seconds. It then allow an extra one to run. So it takes a while before all the requests are completed.
You could fix it by calling SetMinThreads() and setting the minimum to 200. But that's incredibly wasteful of system resources. You might as well call Socket.BeginConnect() 200 times and find out what happened 2 seconds later. Your fast version.

Looks like in the first example, you're waiting for each proxy connection to timeout, or 2 seconds, whichever comes first. Plus, you're queuing up 200 separate work requests. Your thread pool size is probably going to be way less than this. Check it with GetMaxThreads. You're only going to have that number of work requests running concurrently, and the next request has to wait on a previous item to timeout.

fast parallel TcpClients with await BeginConnect new Thread Pulse

I am trying to establish a TCP connection with a number of IPs in parallel, and do that as fast as possible. I have converted some older code to use AsyncCTP for that purpose, introducing the parallelism.
Changes to Design and Speed, and Accessing Successful Connections?
My question is three-fold:
How bad is the following flow / what should I change?
i.e. the await starts a bunch of parallel TcpRequest threads,
but within each TcpRequest there is a tcpClient.BeginConnect
as well as another thread being spawn for reading (if connection is successful)
and the writing to the stream is done with a Wait / Pulse mechanism in a while loop.
Secondly, how could i make the process of connecting to a number of targets faster?
Currently, if the ip:port targets are not actually running any servers, then i get the "All Done" printed after about 18 seconds from the start, when trying to connect to about 500 local targets (that are not listening, and thus fail, on those ports).
How could i access the WriteToQueue method of successful connections, from the mothership?
Async Mothership Trying to Connect to All Targets in Parallel
// First get a bunch of IPAddress:Port targets
var endpoints = EndPointer.Get();
// Try connect to all those targets
var tasks = from t in topList select TcpRequester.ConnectAsync(t);
await TaskEx.WhenAll(tasks);
Debug.WriteLine("All Done");
Static Accessor for Individual TcpRequest Tasks
public static Task<TcpRequester> ConnectAsync(IPEndPoint endPoint)
{
var tcpRequester = Task<TcpRequester>.Factory.StartNew(() =>
{
var request = new TcpRequester();
request.Connect(endPoint);
return request;
}
);
return tcpRequester;
}
TcpRequester with BeginConnect TimeOut and new Thread for Reading
public void Connect(IPEndPoint endPoint)
{
TcpClient tcpClient = null;
Stream stream = null;
using (tcpClient = new TcpClient())
{
tcpClient.ReceiveTimeout = 1000;
tcpClient.SendTimeout = 1000;
IAsyncResult ar = tcpClient.BeginConnect(endPoint.Address, endPoint.Port, null, null);
WaitHandle wh;
wh = ar.AsyncWaitHandle;
try
{
if (!ar.AsyncWaitHandle.WaitOne(TimeSpan.FromMilliseconds(1000), false))
{
throw new TimeoutException();
}
if (tcpClient.Client != null)
{
// Success
tcpClient.EndConnect(ar);
}
if (tcpClient.Connected)
{
stream = tcpClient.GetStream();
}
// Start to read stream until told to close or remote close
ThreadStart reader = () => Read(stream);
// Reading is done in a separate thread
var thread = new Thread(reader);
thread.Start();
// See Writer method below
Writer(stream);
} finally
{
wh.Close();
}
}
} catch (Exception ex)
{
if (tcpClient != null)
tcpClient.Close();
}
}
}
Writing to Stream with Wait and Pulse
readonly Object _writeLock = new Object();
public void WriteToQueue(String message)
{
_bytesToBeWritten.Add(Convert(message));
lock (_writeLock)
{
Monitor.Pulse(_writeLock);
}
}
void Writer(Stream stream)
{
while (!_halt)
{
while (_bytesToBeWritten.Count > 0 && !_halt)
{
// Write method does the actual writing to the stream:
if (Write(stream, _bytesToBeWritten.ElementAt(0)))
{
_bytesToBeWritten.RemoveAt(0);
} else
{
Discontinue();
}
}
if (!(_bytesToBeWritten.Count > 0) && !_halt)
{
lock (_writeLock)
{
Monitor.Wait(_writeLock);
}
}
}
Debug.WriteLine("Discontinuing Writer and TcpRequester");
}

There are a few red flags that pop out at a cursory glance.
You have this Stream that is accepting reads and writes, but there is no clear indication that the operations have been synchronized appropriately. The documentation does state that a Stream's instance methods are not safe for multithreaded operations.
There does not appear to be synchronization around operations involving _bytesToBeWritten.
Acquiring a lock solely to execute Monitor.Wait and Monitor.Pulse is a little weird, if not downright incorrect. It is basically equivalent to using a ManualResetEvent.
It is almost never correct to use Monitor.Wait without a while loop. To understand why you have to understand the purpose of pulsing and waiting on a lock. That is really outside the scope of this answer.
It appears like the Writer and WriteToQueue methods are an attempt to generate a producer-consumer queue. The .NET BCL already contains the innards for this via the BlockingCollection class.
For what it is worth I see nothing flagrantly wrong with the general approach and usage of the await keyword.