Is there a magic number or formula for setting the values of SetMaxThreads and SetMinThreads for ThreadPool? I have thousands of long-running methods that need execution but just can't find the perfect match for setting these values. Any advise would be greatly appreciated.
The default minimum number of threads is the number of cores your machine has. That's a good number, it doesn't generally make sense to run more threads than you have cores.
The default maximum number of threads is 250 times the number of cores you have on .NET 2.0 SP1 and up. There is an enormous amount of breathing room here. On a four core machine, it would take 499 seconds to reach that maximum if none of the threads complete in a reasonable amount of time.
The threadpool scheduler tries to limit the number of active threads to the minimum, by default the number of cores you have. Twice a second it allows one more thread to start if the active threads do not complete. Threads that run for a very long time or do a lot of blocking that is not caused by I/O are not good candidates for the threadpool. You should use a regular Thread instead.
Getting to the maximum isn't healthy. On a four core machine, just the stacks of those threads will consume a gigabyte of virtual memory space. Getting OOM is very likely. Consider lowering the max number of threads if that's your problem. Or consider starting just a few regular Threads that receive packets of work from a thread-safe queue.
Typically, the magic number is to leave it alone. The ThreadPool does a good job of handling this.
That being said, if you're doing a lot of long running services, and those services will have large periods where they're waiting, you may want to increase the maximum threads to handle more options. (If the processes aren't blocking, you'll probably just slow things down if you increase the thread count...)
Profile your application to find the correct number.
If you want better control, you might want to consider NOT using the built-in ThreadPool. There is a nice replacement at http://www.codeproject.com/KB/threads/smartthreadpool.aspx.
Related
I'm doing heavy mathematical computations using Math.Net Numerics parallely inside Parallel.For block.
When I run code in my local system with 4 cores(2*2), it's using all 4 cores.
But when I run same code in our dev server with 8 cores(4*2), it's using only 4 cores.
I've tried setting MaxDegreeOfParallism,but couldn't help.
Any idea why all cores are not being utilised.
Below is sample code.
Parallel.For(0,10000,(i)=>
{
// heavy math computations using matrices
});
From MSDN
By default, For and ForEach will utilize however many threads the underlying scheduler provides, so changing MaxDegreeOfParallelism from the default only limits how many concurrent tasks will be used.
The way I read the documentation: if the underlying scheduler only offers a single thread, then setting MaxDegreeOfParallelism > 1 will still result in a single thread.
Parallelization is done runtime, based on the current conditions and a lots of other circumstances. You cannot force .NET to use all the cores (in managed code at least).
From MSDN:
Conversely, by default, the Parallel.ForEach and Parallel.For methods can use a variable number of tasks. That's why, for example, the ParallelOptions class has a MaxDegreeOfParallelism property instead of a "MinDegreeOfParallelism" property. The idea is that the system can use fewer threads than requested to process a loop.
The .NET thread pool adapts dynamically to changing workloads by allowing the number of worker threads for parallel tasks to change over time. At run time, the system observes whether increasing the number of threads improves or degrades overall throughput and adjusts the number of worker threads accordingly.
Be careful if you use parallel loops with individual steps that take several seconds or more. This can occur with I/O-bound workloads as well as lengthy calculations. If the loops take a long time, you may experience an unbounded growth of worker threads due to a heuristic for preventing thread starvation that's used by the .NET ThreadPool class's thread injection logic.
I have a dual core processor, now let's say that I want to make a spam bot program, which will spam messages such as "Hey, how are you?".
My question is, what number of threads would be able to pop up these messages the fastest, running 5 threads or 100 threads botting the messages? (Of course, these numbers aren't special, just for the example). All of the threads will run in thread-safe.
EDIT: As for the down votes before, I'm not really writing a spambot program, I just mentioned it as an example for my question, sorry for the misunderstanding
The ideal number of threads depends on your hardware (in this case a dual core processor), and on what those threads are doing. If they are CPU intensive, more than 1 thread per core will probably slow things down.
If the threads do some IO, you will see an overall increase in performance by adding threads. The point of diminishing returns depends entirely on the nature of the non-CPU tasks and on the specific hardware.
To find that point, you will have to test various thread totals.
You can design your system to self-tune the number of threads in use. I once designed a system that ran best (most total throughput) when the total CPU load was about 70%. To optimize for that value, I added threads (with a delay between threads) until the CPU was at 70%, +/- 5%. If it went above 80%, I signaled one or more threads to finish their current work and terminate. If it went below 60%, I gradually added threads. Worked like a charm.
Deliberately creating more threads than processors is a standard technique used to make use of "spare cycles" where a thread is blocked waiting for something, whether that's I/O, a mutex, or something else by providing some other useful work for the processor to do.
If your threads are doing I/O then this is a strong contender for the speed-up: as each thread blocks waiting for the I/O, the processor can run the other threads until they too block for I/O, hopefully by which time the data for the first thread is ready, and so forth.
Source: Anthony Williams
How to auto scale worker tasks?
I have an application which I want to automatically scale the number of worker tasks to accommodate for the throughput of items to process (with a maximum limit to the number of workers obviously).
All items to process are routed through a single point, from which they are distributed among the worker tasks (right now I find the worker with the shortest queue, and enqueue the item to it).
What would be a good pattern or technique to use to make some intelligent decisions as to how many workers I should spin up to handle the items? This logic would need to include shutting workers down if the items are able to be processed in a timely fashion by less workers.
I realize that adding more workers won't scale infinitely as ultimately other resources will be bottlenecked, and at some point adding more workers will hurt more than help. If I could account for this, and decide to scale DOWN the number of workers to find the 'sweet spot' automatically, that would be fantastic, however at this point I'd be happy if my system could just increase the number of workers as more items are added, and then decrease the number as less items are needed.
One idea I've toyed around with is measuring the average time an item sits in the queue. If this average time is greater than a couple seconds, I should spin up more workers (until the set max limit is reached). If the average time is less than 1 second, I should spin down more workers (until there's only one left of course).
Does anyone have any suggestions on the best way to approach this?
You won't be able to scale unless you can break the work down into chunks that can be distributed amongst workers. How you distribute that work really depends on how you break down the work. If you break down the work into smaller chunks whose results get merged when done, then the map/reduce pattern may fit the bill...
One thing to keep in mind, regardless of what you do, is that as soon as you have more threads running than CPUs your performance will degrade dramatically. this is because you increase/introduce context switching which is very expensive. Using things like TPL and PLINQ will allow you to schedule tasks and avoid this problem.
You could also compute a global running average of number of queued task per seconds, and divide by maximum task per seconds one worker can process. This will give you the number of worker you need to not fall behind.
You also want to introduce a delay, (if for x seconds there is less worker then needed increase) (if for x seconds there is more worker then needed decrease).
This assume all task take the same amount of time to process and that all your worker have the same throughput.
I've got some trouble with .NET's ThreadPool (.NET 4).
I've read that by default .NET has a limit of 25 threads per processor, but according to forum posts on SO and on other places, I can increase the limit with the below code.
void SetThreads(int threads)
{
ThreadPool.SetMaxThreads(threads, threads);
ThreadPool.SetMinThreads(threads, threads);
}
However, when I set the above to some arbitrarily high number, for example, 2000, and queue ~1000 items, I still only have ~33 threads running (.NET CLR takes ~5 threads), and ThreadPool.GetAvailableThreads() returns 1971 threads remaining.
Why doesn't the code above work?
Firstly, your "knowledge" of the defaults is incorrect. The limit of 25 threads per processor was back from .NET 1.1. It was increased in .NET 2, and now:
Beginning with the .NET Framework version 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads.
However, there's something else at play: the thread pool doesn't immediately create new threads in all situations. In order to cope with bursts of small tasks, it limits how quickly it creates new threads. IIRC, it will create one thread every 0.5 seconds if there are outstanding tasks, up to the maximum number of threads. I can't immediately see that figure documented though, so it may well change. I strongly suspect that's what you're seeing though. Try queuing a lot of items and then monitor the number of threads over time.
From the MSDN :
When demand is low, the actual number of thread pool threads can fall
below the minimum values.
Read this too: Patterns for Parallel Programming: Understanding and Applying Parallel Patterns with the .NET Framework 4
Firstly check this link, especially this remark:
If the common language runtime is hosted, for example by Internet
Information Services (IIS) or SQL Server, the host can limit or
prevent changes to the thread pool size.
Then you should check the return value of ThreadPool.SetMaxThreads(threads, threads) method. Maybe it returns false?
Are there any benefits to limiting the number of concurrent threads doing a given task to equal the number of processors on the host system? Or better to simply trust libraries such as .NET's ThreadPool to do the right thing ... even if there are 25 different concurrent threads happening at any one given moment?
Most threads are not CPU bound, they end up waiting on IO or other events. If you look at your system now, I imagine you have 100's (if not 1000's) of threads executing with no problems. By that measure, you're probably best just leaving the .NET thread pool to do the right thing!
However, if the threads were all CPU bound (e.g. something like ray tracing) then it would be a good idea to limit the number of threads to the number of cores, otherwise chances are that context switching will begin to hurt performance.
The threadpool already does a reasonably good job at this. It tries to limit the number of running threads to the number of CPU cores in your machine. When one thread ends, it immediately schedules another eligible thread for execution.
Every 0.5 seconds, it evaluates what is going on with the running threads. When the threads have been running too long, it assumes they are stalled and allows another thread to start executing. You'll now have more threads running than you have CPU cores. This can go up to the maximum number of allowed thread, as set by ThreadPool.SetMaxThreads().
Starting around .NET 2.0 SP1, the default maximum number of threads was increased considerably to 250 times the number of cores. You should never ever get there. If you do, you would have wasted about 2 minutes of time where a possibly non-optimal number of threads were running. Those threads however would all have to be blocking for that long, not exactly a typical execution pattern for a thread. On the other hand, if these threads are all waiting on the same kind of resource they are likely to just take turns, adding more threads cannot improve throughput.
Long story short, the thread pool will work well if you run threads that execute quickly (seconds at most) and don't block for a long time. You probably ought to consider creating your own Thread objects when your code doesn't match that pattern.
Well, if your bottleneck is ONLY processors, then it might make sense, but that would ignore all memory and other i/o bottlenecks, and chances are at least your cache memory is throwing page faults and other events that would slow the threads.
I'd trust the library myself. Threads wait for all kinds of things, and you don't want your application to slow down because it can't spawn a new thread, even though most of the rest are just sleeping, waiting for some event or resource.
Measure your application under a variety of thread:processor ratios. Come to conclusions based on hard data about your application. Accept no arguments from first principles about what performance you should get, only what you do get matters.