We Keep Coding

Why is the Completed callback from SocketAsyncEventArgs frequently executed in newly created threads instead of using a bounded thread pool?

I have a simple client application that receives byte buffers from the network with a low throughput. Here is the code:
private static readonly HashSet<int> _capturedThreadIds = new HashSet<int>();
private static void RunClient(Socket socket)
{
var e = new SocketAsyncEventArgs();
e.SetBuffer(new byte[10000], 0, 10000);
e.Completed += SocketAsyncEventsArgsCompleted;
Receive(socket, e);
}
private static void Receive(Socket socket, SocketAsyncEventArgs e)
{
var isAsynchronous = socket.ReceiveAsync(e);
if (!isAsynchronous)
SocketAsyncEventsArgsCompleted(socket, e);
}
private static void SocketAsyncEventsArgsCompleted(object sender, SocketAsyncEventArgs e)
{
if (e.LastOperation != SocketAsyncOperation.Receive || e.SocketError != SocketError.Success || e.BytesTransferred <= 0)
{
Console.WriteLine("Operation: {0}, Error: {1}, BytesTransferred: {2}", e.LastOperation, e.SocketError, e.BytesTransferred);
return;
}
var thread = Thread.CurrentThread;
if (_capturedThreadIds.Add(thread.ManagedThreadId))
Console.WriteLine("New thread, ManagedId: " + thread.ManagedThreadId + ", NativeId: " + GetCurrentThreadId());
//Console.WriteLine(e.BytesTransferred);
Receive((Socket)sender, e);
}
The threading behavior of the application is quite curious:
The SocketAsyncEventsArgsCompleted method is frequently run in new threads. I would have expected that after some time no new thread would be created. I would have expected the threads to be reused, because of the thread pool (or IOCP thread pool) and because the throughput is very stable.
The number of threads stays low, but I can see in the process explorer that threads are frequently created and destroyed. Likewise, I would not have expected threads to be created or destroyed.
Can you explain the application behavior?
Edit: The "low" throughput is 20 messages per second (roughly 200 KB/s). If I increase the throughput to more than 1000 messages per second (50 MB/s), the application behavior does not change.

The low application throughput itself cannot explain the thread creation and destruction. The socket receives 20 messages per seconds, which is more than enough to keep a thread alive (the waiting threads are being destroyed after spending 10 seconds idle).
This problem is related to the thread pool thread injection, i.e. the threads creation and destruction strategy. Thread pool threads are regularly injected and destroyed in order to measure the impact of new threads on the thread pool throughput.
This is called thread probing. It is clearly explained in the Channel 9 video CLR 4 - Inside the Thread Pool (jump to 26:30).
It seems like thread probing is always done with newly created threads instead of moving a thread in and out of the pool. I suppose it works better like this for most applications because it avoids to keep an unused thread alive.

From MSDN
Beginning with the .NET Framework 4, the thread pool creates and
destroys worker threads in order to optimize throughput, which is
defined as the number of tasks that complete per unit of time. Too few
threads might not make optimal use of available resources, whereas too
many threads could increase resource contention.
Note
When demand is low, the actual number of thread pool threads can
fall below the minimum values.
Basically it sounds like your low throughput is causing the thread pool to destroy threads since they are not required, and are just sat taking up resources. I wouldn't worry about it. As MS explicitly state:
In most cases the thread pool will perform better with its own
algorithm for allocating threads.
If you're really bothered, you could always poll ThreadPool.GetAvailableThreads() to watch the pool, and see how different network throughputs affect it.

Parallel.ForEach keeps spawning new threads

While I was using Parallel.ForEach in my program, I found that some threads never seemed to finish. In fact, it kept spawning new threads over and over, a behaviour that I wasn't expecting and definitely don't want.
I was able to reproduce this behaviour with the following code which, just like my 'real' program, both uses processor and memory a lot (.NET 4.0 code):
public class Node
{
public Node Previous { get; private set; }
public Node(Node previous)
{
Previous = previous;
}
}
public class Program
{
public static void Main(string[] args)
{
DateTime startMoment = DateTime.Now;
int concurrentThreads = 0;
var jobs = Enumerable.Range(0, 2000);
Parallel.ForEach(jobs, delegate(int jobNr)
{
Interlocked.Increment(ref concurrentThreads);
int heavyness = jobNr % 9;
//Give the processor and the garbage collector something to do...
List<Node> nodes = new List<Node>();
Node current = null;
for (int y = 0; y < 1024 * 1024 * heavyness; y++)
{
current = new Node(current);
nodes.Add(current);
}
TimeSpan elapsed = DateTime.Now - startMoment;
int threadsRemaining = Interlocked.Decrement(ref concurrentThreads);
Console.WriteLine("[{0:mm\\:ss}] Job {1,4} complete. {2} threads remaining.",
elapsed, jobNr, threadsRemaining);
});
}
}
When run on my quad-core, it initially starts of with 4 concurrent threads, just as you would expect. However, over time more and more threads are being created. Eventually, this program then throws an OutOfMemoryException:
[00:00] Job 0 complete. 3 threads remaining.
[00:01] Job 1 complete. 4 threads remaining.
[00:01] Job 2 complete. 4 threads remaining.
[00:02] Job 3 complete. 4 threads remaining.
[00:05] Job 9 complete. 5 threads remaining.
[00:05] Job 4 complete. 5 threads remaining.
[00:05] Job 5 complete. 5 threads remaining.
[00:05] Job 10 complete. 5 threads remaining.
[00:08] Job 11 complete. 5 threads remaining.
[00:08] Job 6 complete. 5 threads remaining.
...
[00:55] Job 67 complete. 7 threads remaining.
[00:56] Job 81 complete. 8 threads remaining.
...
[01:54] Job 107 complete. 11 threads remaining.
[02:00] Job 121 complete. 12 threads remaining.
..
[02:55] Job 115 complete. 19 threads remaining.
[03:02] Job 166 complete. 21 threads remaining.
...
[03:41] Job 113 complete. 28 threads remaining.
<OutOfMemoryException>
The memory usage graph for the experiment above is as follows:
(The screenshot is in Dutch; the top part represents processor usage, the bottom part memory usage.) As you can see, it looks like a new thread is being spawned almost every time the garbage collector gets in the way (as can be seen in the dips of memory usage).
Can anyone explain why this is happening, and what I can do about it? I just want .NET to stop spawning new threads, and finish the existing threads first...

You can limit the maximum number of threads that get created by specifying a ParallelOptions instance with the MaxDegreeOfParallelism property set:
var jobs = Enumerable.Range(0, 2000);
ParallelOptions po = new ParallelOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount
};
Parallel.ForEach(jobs, po, jobNr =>
{
// ...
});
As to why you're getting the behaviour you're observing: The TPL (which underlies PLINQ) is, by default, at liberty to guess the optimal number of threads to use. Whenever a parallel task blocks, the task scheduler may create a new thread in order to maintain progress. In your case, the blocking might be happening implicitly; for example, through the Console.WriteLine call, or (as you observed) during garbage collection.
From Concurrency Levels Tuning with Task Parallel Library (How Many Threads to Use?):
Since the TPL default policy is to use one thread per processor, we can conclude that TPL initially assumes that the workload of a task is ~100% working and 0% waiting, and if the initial assumption fails and the task enters a waiting state (i.e. starts blocking) - TPL with take the liberty to add threads as appropriate.

You should probably read a bit about the how the task scheduler works.
Parallel Programming with Microsoft .NET - Parallel Tasks
(latter half of the page)
"The .NET thread pool automatically manages the number of worker
threads in the pool. It adds and removes threads according to built-in
heuristics. The .NET thread pool has two main mechanisms for injecting
threads: a starvation-avoidance mechanism that adds worker threads if
it sees no progress being made on queued items and a hill-climbing
heuristic that tries to maximize throughput while using as few threads
as possible.
The goal of starvation avoidance is to prevent deadlock. This kind of
deadlock can occur when a worker thread waits for a synchronization
event that can only be satisfied by a work item that is still pending
in the thread pool's global or local queues. If there were a fixed
number of worker threads, and all of those threads were similarly
blocked, the system would be unable to ever make further progress.
Adding a new worker thread resolves the problem.
A goal of the hill-climbing heuristic is to improve the utilization of
cores when threads are blocked by I/O or other wait conditions that
stall the processor. By default, the managed thread pool has one
worker thread per core. If one of these worker threads becomes
blocked, there's a chance that a core might be underutilized,
depending on the computer's overall workload. The thread injection
logic doesn't distinguish between a thread that's blocked and a thread
that's performing a lengthy, processor-intensive operation. Therefore,
whenever the thread pool's global or local queues contain pending work
items, active work items that take a long time to run (more than a
half second) can trigger the creation of new thread pool worker
threads."
You can mark a task as LongRunning but this has the side effect of allocating a thread for it from outside the thread pool which means that the task cannot be inlined.
Remember that the ParallelFor treats the work it is given as blocks so even if the work in one loop is fairly small the overall work done by the task invoked by the look may appear longer to the scheduler.
Most calls to the GC in and of them selves aren't blocking (it runs on a separate thread) but if you wait for GC to complete then this does block. Remember also that the GC is rearranging memory so this may have some side effects (and blocking) if you are trying to allocate memory while running GC. I don't have specifics here but I know the PPL has some memory allocation features specifically for concurrent memory management for this reason.
Looking at your code's output it seems that things are running for many seconds. So I'm not surprised that you are seeing thread injection. However I seem to remember that the default thread pool size is roughly 30 threads (probably depending on the number of cores on your system). A thread takes up roughly a MB of memory before your code allocates any more so I'm not clear why you could get an out of memory exception here.

I've posted the follow-up question "How to count the amount of concurrent threads in .NET application?"
If to count the threads directly, their number in Parallel.For() mostly ((very rarely and insignificantly decreasing) only increases and is not releleased after loop completion.
Checked this in both Release and Debug mode, with
ParallelOptions po = new ParallelOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount
};
and without
The digits vary but conclusions are the same.
Here is the ready code I was using, if someone wants to play with:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
namespace Edit4Posting
{
public class Node
{
public Node Previous { get; private set; }
public Node(Node previous)
{
Previous = previous;
}
}
public class Edit4Posting
{
public static void Main(string[] args)
{
int concurrentThreads = 0;
int directThreadsCount = 0;
int diagThreadCount = 0;
var jobs = Enumerable.Range(0, 160);
ParallelOptions po = new ParallelOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount
};
Parallel.ForEach(jobs, po, delegate(int jobNr)
//Parallel.ForEach(jobs, delegate(int jobNr)
{
int threadsRemaining = Interlocked.Increment(ref concurrentThreads);
int heavyness = jobNr % 9;
//Give the processor and the garbage collector something to do...
List<Node> nodes = new List<Node>();
Node current = null;
//for (int y = 0; y < 1024 * 1024 * heavyness; y++)
for (int y = 0; y < 1024 * 24 * heavyness; y++)
{
current = new Node(current);
nodes.Add(current);
}
//*******************************
directThreadsCount = Process.GetCurrentProcess().Threads.Count;
//*******************************
threadsRemaining = Interlocked.Decrement(ref concurrentThreads);
Console.WriteLine("[Job {0} complete. {1} threads remaining but directThreadsCount == {2}",
jobNr, threadsRemaining, directThreadsCount);
});
Console.WriteLine("FINISHED");
Console.ReadLine();
}
}
}

Multiple BackgroundWorkers do not start work immediately

I'm trying to use BackgroundWorker to load-test a web service,
Code is:
private void button1_Click(object sender, EventArgs e)
{
CheckWS("11111");
CheckWS("22222");
CheckWS("3344111");
// .. and some more of those, approx 1000
}
public void CheckWS(string sUser)
{
BackgroundWorker bgWork = new BackgroundWorker();
bgWork.DoWork += new DoWorkEventHandler(bgWork_DoWork);
sObject sData = new sObject();
sData.iNum = iCount++;
sData.sId = sUser;
bgWork.RunWorkerAsync(sData);
}
public void bgWork_DoWork(object sender, DoWorkEventArgs e)
{
// Update a textbox that this worker started
...
// Run a web service
...
// Update the textbox as worker ended
}
But according to the time taken for the job the get done, and according to the output of the workers (worker begin&end in the textbox), it takes way too long for each worker to start -
I would expect everything to run simultaneously, but instead, it executes only 2-3 background workers at a time...
Any ideas on this issue?

Backgroundworkers run on top of the ThreadPool and the TP purposely limits the number of threads.
Since this is a Test(-driver) app there is no problem in changing the config of the Pool. You can set the Minimum number of threads to something sensible.
ThreadPool.SetMinThreads(20, 20);

Giving some background to #Henk Holterman's correct answer that states the BackgroundWorker class uses the thread pool...
The thread pool starts out with one thread in it pool and the pool manager ‘injects' new threads to cope with extra asynchronous workload, upto some limiting maximum. After a set period of inactivity the pool manager may 'retire' threads if it 'thinks' that doing so will lead to a better throughput. In your case above the pool manager is limiting the number of concurrent threads.
You can set the upper limit on the number of threads the pool will create by calling Thread.Pool.SetMaxThread;, the defaults are:
1023 in Framework 4.0 in a 32-bit environment.
32768 in Framework 4.0 in a 64-bit environment.
250 per core in Framework 3.5.
25 per core in Framework 2.0.
[_these figure may vary according to hardware and OS]. The reason for these vast number (at least in the case of .NET 4.0) is to ensure that progress is made even when some threads are blocked (running some intense work etc.)
You can set the lower limit via ThreadPool.SetMinThreads. The role of this limiter is more subtle than that of the max limiter: this instructs the pool manager not to delay the creation of threads until reaching this lower limit on number - setting this number as #Henk pointed out quite rightly will improve you concurrency when there are blocked threads.
I hope this helps.

Limiting the number of threadpool threads

I am using ThreadPool in my application. I have first set the limit of the thread pool by using the following:
ThreadPool.SetMaxThreads(m_iThreadPoolLimit,m_iThreadPoolLimit);
m_Events = new ManualResetEvent(false);
and then I have queued up the jobs using the following
WaitCallback objWcb = new WaitCallback(abc);
ThreadPool.QueueUserWorkItem(objWcb, m_objThreadData);
Here abc is the name of the function that I am calling.
After this I am doing the following so that all my threads come to 1 point and the main thread takes over and continues further
m_Events.WaitOne();
My thread limit is 3. The problem that I am facing is, inspite of the thread pool limit set to 3, my application is processing more than 3 files at the same time, whereas it was supposed to process only 3 files at a time. Please help me solve this issue.

What kind of computer are you using?
From MSDN
You cannot set the number of worker
threads or the number of I/O
completion threads to a number smaller
than the number of processors in the
computer.
If you have 4 cores, then the smallest you can have is 4.
Also note:
If the common language runtime is
hosted, for example by Internet
Information Services (IIS) or SQL
Server, the host can limit or prevent
changes to the thread pool size.
If this is a web site hosted by IIS then you cannot change the thread pool size either.

A better solution involves the use of a Semaphore which can throttle the concurrent access to a resource1. In your case the resource would simply be a block of code that processes work items.
var finished = new CountdownEvent(1); // Used to wait for the completion of all work items.
var throttle = new Semaphore(3, 3); // Used to throttle the processing of work items.
foreach (WorkItem item in workitems)
{
finished.AddCount();
WorkItem capture = item; // Needed to safely capture the loop variable.
ThreadPool.QueueUserWorkItem(
(state) =>
{
throttle.WaitOne();
try
{
ProcessWorkItem(capture);
}
finally
{
throttle.Release();
finished.Signal();
}
}, null);
}
finished.Signal();
finished.Wait();
In the code above WorkItem is a hypothetical class that encapsulates the specific parameters needed to process your tasks.
The Task Parallel Library makes this pattern a lot easier. Just use the Parallel.ForEach method and specify a ParallelOptions.MaxDegreesOfParallelism that throttles the concurrency.
var options = new ParallelOptions();
options.MaxDegreeOfParallelism = 3;
Parallel.ForEach(workitems, options,
(item) =>
{
ProcessWorkItem(item);
});
1I should point out that I do not like blocking ThreadPool threads using a Semaphore or any blocking device. It basically wastes the threads. You might want to rethink your design entirely.

You should use Semaphore object to limit concurent threads.

You say the files are open: are they actually being actively processed, or just left open?
If you're leaving them open: Been there, done that! Relying on connections and resources (it was a DB connection in my case) to close at end of scope should work, but it can take for the dispose / garbage collection to kick in.

Why Thread.Sleep() is so CPU intensive?

I have an ASP.NET page with this pseduo code:
while (read)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
}
Any client who requests this page will start to download a binary file. Everything is OK at this point but clients had no limit in download speed so changed the above code to this:
while (read)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
Thread.Sleep(500);
}
Speed problem is solved now, but under test with 100 concurrent clients who connect one after another (3 seconds lag between each new connection) the CPU usage increases when the number of clients increases and when there are 70 ~ 80 concurrent clients CPU reaches 100% and any new connection is refused. Numbers may be different on other machines but the question is why Thread.Sleep() is so CPU intensive and is there any way to speed done the client without CPU rising ?
I can do it at IIS level but I need more control from inside of my application.

Let's take a look at whether Michael's answer seems reasonable.
Now, Michael wisely points out that Thread.Sleep(500) shouldn't cost much in the way of CPU. That's all well and good in theory, but let's see if that pans out in practice.
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
Thread.Sleep(500);
}
}
Running this, the CPU use of the application hovers around the 0% mark.
Michael also points out that since all the threads that ASP.NET has to use are sleeping, it will have to spawn new threads, and offers that this is expensive. Let's try not sleeping, but doing lots of spawning:
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
new Thread(o => {}).Start();
}
}
We create lots of threads, but they just execute a null operation. That uses a lot of CPU, even though the threads aren't doing anything.
The total number of threads never gets very high though, because each lives for such a short time. Lets combine the two:
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
new Thread(o => {Thread.Sleep(500);}).Start();
}
}
Adding this operation that we have shown to be low in CPU use to each thread increases CPU use even more, as the threads mount up. If I run it in a debugger it pushes up to near 100% CPU. If I run it outside of a debugger, it performs a bit better, but only because it throws an out of memory exception before it gets a chance to hit 100%.
So, it isn't Thread.Sleep itself that is the problem, but the side-effect that having all available threads sleep forces more and more threads to be created to handle other work, just as Michael said.

Just a guess:
I don't think it's Thread.Sleep() that's tying up the CPU - it's the fact that you're causing threads to be tied up responding to a request for so long, and the system needs to spin up new threads (and other resources) to respond to new requests since those sleeping threads are no longer available in the thread pool.

Rather than an ASP.NET page you should implement an IHttpAsyncHandler. ASP.NET page code puts many things between your code and the browser that would not be appropriate for transferring binary files. Also, since you're attempting to perform rate limitation, you should use asynchronous code to limit resource usage, which would be difficult in an ASP.NET page.
Creating an IHttpAsyncHandler is fairly simple. Just trigger some asynchronous operations in the BeginProcessRequest method, and don't forget to properly close the context to show you have reached the end of the file. IIS won't be able to close it for you here.
The following is my really bad example of how to perform an an asynchronous operation consisting of a series of steps, counting from 0 to 10, each performed at a 500ms interval.
using System;
using System.Threading;
namespace ConsoleApplication1 {
class Program {
static void Main() {
// Create IO instances
EventWaitHandle WaitHandle = new EventWaitHandle(false, EventResetMode.AutoReset); // We don't actually fire this event, just need a ref
EventWaitHandle StopWaitHandle = new EventWaitHandle(false, EventResetMode.AutoReset);
int Counter = 0;
WaitOrTimerCallback AsyncIOMethod = (s, t) => { };
AsyncIOMethod = (s, t) => {
// Handle IO step
Counter++;
Console.WriteLine(Counter);
if (Counter >= 10)
// Counter has reaced 10 so we stop
StopWaitHandle.Set();
else
// Register the next step in the thread pool
ThreadPool.RegisterWaitForSingleObject(WaitHandle, AsyncIOMethod, null, 500, true);
};
// Do initial IO
Console.WriteLine(Counter);
// Register the first step in the thread pool
ThreadPool.RegisterWaitForSingleObject(WaitHandle, AsyncIOMethod, null, 500, true);
// We force the main thread to wait here so that the demo doesn't close instantly
StopWaitHandle.WaitOne();
}
}
}
You'll also need to register your IHttpAsyncHandler implementation with IIS in whichever way is appropriate for your situation.

Its because the thread gets a priority boost every time it yields its time slice. Avoid calling sleep often ( particularly with low values ).