We Keep Coding

Azure Service Bus ReceiveBatch() getting slow over time

I'm using retrieving message from Azure Service Bus Topic currently and running into a performance issue. I'm requesting messages in batches of 1000 from ASB. The code fetches anywhere between 200-300 messages in first few iterations and then the number of messages retrieved comes down to 1 per iteration. I pull approximately 50K messages which the application is able to retrieve about 2 hours. Every time I stop the application and re-run it, it behaves the same. First 4-5K records get processed in 1-2 mins, but the rest take lot longer. How can I make my code to retrieve more messages in every batch?
Sample Code with batch size set to 50
using Microsoft.ServiceBus.Messaging;
public static Microsoft.ServiceBus.Messaging.SubscriptionClient subscriptionClient;
public async Task startProcess()
{
await Run();
}
public async Task Run()
{
subscriptionClient = Microsoft.ServiceBus.Messaging.SubscriptionClient.CreateFromConnectionString(ServiceBusConnectionString, TopicName, subscriptionname);
for(int counter=1;counter<= 5;)
{
tasks.Add(Task.Factory.StartNew(() => PullMessages(counter))) ;
taskCounter++;
}
var arr = tasks.ToArray();
Task.WaitAll(arr);
}
public async Task PullMessages(object ThreadNumber)
{
bool retry = true;
do
{
subscriptionClient.PrefetchCount = 1000;
iterationCount++;
var messages = await subscriptionClient.ReceiveBatchAsync(50);
logger.Info("{0} messages retrieved for iteration {1} for thread {2}", messages.Count().ToString(), currentiteration.ToString(),ThreadNumber);
await subscriptionClient.CompleteBatchAsync(messages.Select(m => m.LockToken));
logger.Info("Completed iteration {0} for thread {1}", currentiteration.ToString(), ThreadNumber);
if (messages != null && messages.Count() > 0)
{
retry = true;
}
else
{
retry = false;
}
} while (retry);
}
Log:
2020-07-21 19:06:48.3177 Info 50 messages retrieved for iteration 2 for thread 1
2020-07-21 19:06:48.3177 Info 50 messages retrieved for iteration 3 for thread 3
2020-07-21 19:06:48.3177 Info Completed iteration 4 for thread 5
2020-07-21 19:06:48.3177 Info 1 messages retrieved for iteration 6 for thread 5
2020-07-21 19:06:48.3177 Info 50 messages retrieved for iteration 5 for thread 4
2020-07-21 19:06:48.3177 Info 50 messages retrieved for iteration 1 for thread 2
2020-07-21 19:06:48.6010 Info Completed iteration 3 for thread 3
2020-07-21 19:06:48.6010 Info Completed iteration 5 for thread 4
2020-07-21 19:06:48.6010 Info Completed iteration 2 for thread 1
2020-07-21 19:06:48.6010 Info Completed iteration 6 for thread 5
2020-07-21 19:06:48.6225 Info Completed iteration 1 for thread 2
2020-07-21 19:06:48.8367 Info 1 messages retrieved for iteration 7 for thread 3
2020-07-21 19:06:49.0683 Info 10 messages retrieved for iteration 8 for thread 4
2020-07-21 19:06:49.0934 Info Completed iteration 7 for thread 3
2020-07-21 19:06:49.3005 Info 1 messages retrieved for iteration 9 for thread 1
2020-07-21 19:06:49.3491 Info Completed iteration 8 for thread 4
2020-07-21 19:06:49.5301 Info 1 messages retrieved for iteration 10 for thread 5
2020-07-21 19:06:49.5612 Info Completed iteration 9 for thread 1
2020-07-21 19:06:49.7868 Info Completed iteration 10 for thread 5
2020-07-21 19:06:49.7868 Info 1 messages retrieved for iteration 11 for thread 2
2020-07-21 19:06:50.0586 Info Completed iteration 11 for thread 2
2020-07-21 19:06:50.2511 Info 1 messages retrieved for iteration 12 for thread 3
2020-07-21 19:06:50.5141 Info Completed iteration 12 for thread 3
2020-07-21 19:06:50.5838 Info 1 messages retrieved for iteration 13 for thread 4
2020-07-21 19:06:50.8426 Info Completed iteration 13 for thread 4
2020-07-21 19:06:50.9471 Info 1 messages retrieved for iteration 14 for thread 1
2020-07-21 19:06:51.2014 Info Completed iteration 14 for thread 1
2020-07-21 19:06:51.2475 Info 1 messages retrieved for iteration 15 for thread 5
2020-07-21 19:06:51.5003 Info Completed iteration 15 for thread 5
2020-07-21 19:06:51.6126 Info 1 messages retrieved for iteration 16 for thread 2
2020-07-21 19:06:51.8709 Info Completed iteration 16 for thread 2
2020-07-21 19:06:51.9732 Info 1 messages retrieved for iteration 17 for thread 3
2020-07-21 19:06:52.2269 Info Completed iteration 17 for thread 3
2020-07-21 19:06:52.3523 Info 1 messages retrieved for iteration 18 for thread 4
2020-07-21 19:06:52.6095 Info Completed iteration 18 for thread 4
2020-07-21 19:06:52.7559 Info 1 messages retrieved for iteration 19 for thread 1
2020-07-21 19:06:53.0392 Info Completed iteration 19 for thread 1
2020-07-21 19:06:53.1315 Info 1 messages retrieved for iteration 20 for thread 5
2020-07-21 19:06:53.3902 Info Completed iteration 20 for thread 5
2020-07-21 19:06:53.4929 Info 1 messages retrieved for iteration 21 for thread 2
2020-07-21 19:06:53.9374 Info Completed iteration 21 for thread 2
2020-07-21 19:06:53.9374 Info 1 messages retrieved for iteration 22 for thread 3
2020-07-21 19:06:54.1891 Info 1 messages retrieved for iteration 23 for thread 4
2020-07-21 19:06:54.5142 Info Completed iteration 22 for thread 3
2020-07-21 19:06:54.5142 Info Completed iteration 23 for thread 4
2020-07-21 19:06:54.5735 Info 1 messages retrieved for iteration 24 for thread 1
2020-07-21 19:06:54.8614 Info Completed iteration 24 for thread 1
2020-07-21 19:06:55.0985 Info 1 messages retrieved for iteration 25 for thread 5
2020-07-21 19:06:55.3322 Info 1 messages retrieved for iteration 26 for thread 2
2020-07-21 19:06:55.3994 Info Completed iteration 25 for thread 5
2020-07-21 19:06:55.7143 Info Completed iteration 26 for thread 2
2020-07-21 19:06:55.7143 Info 1 messages retrieved for iteration 27 for thread 3
2020-07-21 19:06:56.0099 Info Completed iteration 27 for thread 3
2020-07-21 19:06:56.3277 Info 1 messages retrieved for iteration 28 for thread 4
2020-07-21 19:06:56.5589 Info 1 messages retrieved for iteration 29 for thread 1
2020-07-21 19:06:56.6247 Info Completed iteration 28 for thread 4
2020-07-21 19:06:56.9426 Info Completed iteration 29 for thread 1
2020-07-21 19:06:56.9426 Info 1 messages retrieved for iteration 30 for thread 5
2020-07-21 19:06:57.2370 Info Completed iteration 30 for thread 5
2020-07-21 19:06:57.2957 Info 1 messages retrieved for iteration 31 for thread 2
2020-07-21 19:06:57.5569 Info Completed iteration 31 for thread 2
2020-07-21 19:06:57.6317 Info 1 messages retrieved for iteration 32 for thread 3
2020-07-21 19:06:58.1697 Info Completed iteration 32 for thread 3
2020-07-21 19:06:58.1697 Info 1 messages retrieved for iteration 33 for thread 4
2020-07-21 19:06:58.3963 Info 1 messages retrieved for iteration 34 for thread 1
2020-07-21 19:06:58.4675 Info Completed iteration 33 for thread 4
2020-07-21 19:06:58.7874 Info Completed iteration 34 for thread 1
2020-07-21 19:06:58.7874 Info 1 messages retrieved for iteration 35 for thread 5
2020-07-21 19:06:59.0824 Info Completed iteration 35 for thread 5
2020-07-21 19:06:59.1997 Info 1 messages retrieved for iteration 36 for thread 2
2020-07-21 19:06:59.4628 Info Completed iteration 36 for thread 2
2020-07-21 19:06:59.5767 Info 1 messages retrieved for iteration 37 for thread 3
2020-07-21 19:06:59.8384 Info Completed iteration 37 for thread 3
2020-07-21 19:06:59.9965 Info 1 messages retrieved for iteration 38 for thread 4
2020-07-21 19:07:00.2579 Info Completed iteration 38 for thread 4
2020-07-21 19:07:00.4314 Info 1 messages retrieved for iteration 39 for thread 1
2020-07-21 19:07:00.6886 Info Completed iteration 39 for thread 1
2020-07-21 19:07:00.7793 Info 1 messages retrieved for iteration 40 for thread 5
2020-07-21 19:07:01.0557 Info Completed iteration 40 for thread 5
2020-07-21 19:07:01.1719 Info 1 messages retrieved for iteration 41 for thread 2
2020-07-21 19:07:01.4312 Info Completed iteration 41 for thread 2
2020-07-21 19:07:01.5550 Info 1 messages retrieved for iteration 42 for thread 3
2020-07-21 19:07:01.8257 Info Completed iteration 42 for thread 3
2020-07-21 19:07:01.9706 Info 1 messages retrieved for iteration 43 for thread 4
2020-07-21 19:07:02.2463 Info Completed iteration 43 for thread 4
2020-07-21 19:07:02.3205 Info 1 messages retrieved for iteration 44 for thread 1
2020-07-21 19:07:02.5932 Info Completed iteration 44 for thread 1
2020-07-21 19:07:02.6965 Info 1 messages retrieved for iteration 45 for thread 5
2020-07-21 19:07:02.9570 Info Completed iteration 45 for thread 5
2020-07-21 19:07:03.0397 Info 1 messages retrieved for iteration 46 for thread 2
2020-07-21 19:07:03.2964 Info Completed iteration 46 for thread 2
2020-07-21 19:07:03.4604 Info 1 messages retrieved for iteration 47 for thread 3
2020-07-21 19:07:03.7161 Info Completed iteration 47 for thread 3
2020-07-21 19:07:03.8770 Info 1 messages retrieved for iteration 48 for thread 4
2020-07-21 19:07:04.1327 Info Completed iteration 48 for thread 4
2020-07-21 19:07:04.2976 Info 1 messages retrieved for iteration 49 for thread 1
2020-07-21 19:07:04.5538 Info Completed iteration 49 for thread 1
2020-07-21 19:07:04.6536 Info 1 messages retrieved for iteration 50 for thread 5
2020-07-21 19:07:04.9064 Info Completed iteration 50 for thread 5
2020-07-21 19:07:05.0138 Info 1 messages retrieved for iteration 51 for thread 2
2020-07-21 19:07:05.2715 Info Completed iteration 51 for thread 2
2020-07-21 19:07:05.3778 Info 1 messages retrieved for iteration 52 for thread 3
2020-07-21 19:07:05.6295 Info Completed iteration 52 for thread 3
2020-07-21 19:07:05.7348 Info 1 messages retrieved for iteration 53 for thread 4
2020-07-21 19:07:05.9970 Info Completed iteration 53 for thread 4
2020-07-21 19:07:06.1308 Info 1 messages retrieved for iteration 54 for thread 1
2020-07-21 19:07:06.3863 Info Completed iteration 54 for thread 1
2020-07-21 19:07:06.4485 Info 1 messages retrieved for iteration 55 for thread 5
2020-07-21 19:07:06.7028 Info Completed iteration 55 for thread 5
2020-07-21 19:07:06.8388 Info 1 messages retrieved for iteration 56 for thread 2
2020-07-21 19:07:07.1086 Info Completed iteration 56 for thread 2
2020-07-21 19:07:07.2214 Info 1 messages retrieved for iteration 57 for thread 3
2020-07-21 19:07:07.4797 Info Completed iteration 57 for thread 3
2020-07-21 19:07:07.6303 Info 1 messages retrieved for iteration 58 for thread 4
2020-07-21 19:07:07.8891 Info Completed iteration 58 for thread 4
2020-07-21 19:07:07.9782 Info 1 messages retrieved for iteration 59 for thread 1
2020-07-21 19:07:08.2421 Info Completed iteration 59 for thread 1
2020-07-21 19:07:08.3398 Info 1 messages retrieved for iteration 60 for thread 5
2020-07-21 19:07:08.5981 Info Completed iteration 60 for thread 5
2020-07-21 19:07:08.8066 Info 1 messages retrieved for iteration 61 for thread 2
2020-07-21 19:07:09.0616 Info Completed iteration 61 for thread 2
2020-07-21 19:07:09.2397 Info 1 messages retrieved for iteration 62 for thread 3
2020-07-21 19:07:09.4960 Info Completed iteration 62 for thread 3
2020-07-21 19:07:09.6018 Info 1 messages retrieved for iteration 63 for thread 4
2020-07-21 19:07:09.9001 Info Completed iteration 63 for thread 4
2020-07-21 19:07:10.0836 Info 1 messages retrieved for iteration 64 for thread 1
2020-07-21 19:07:10.3393 Info Completed iteration 64 for thread 1
2020-07-21 19:07:10.4576 Info 1 messages retrieved for iteration 65 for thread 5
2020-07-21 19:07:10.7138 Info Completed iteration 65 for thread 5
2020-07-21 19:07:10.8472 Info 1 messages retrieved for iteration 66 for thread 2
2020-07-21 19:07:11.1566 Info Completed iteration 66 for thread 2
2020-07-21 19:07:11.2157 Info 1 messages retrieved for iteration 67 for thread 3
2020-07-21 19:07:11.4675 Info Completed iteration 67 for thread 3
2020-07-21 19:07:11.5828 Info 1 messages retrieved for iteration 68 for thread 4
2020-07-21 19:07:11.8385 Info Completed iteration 68 for thread 4
2020-07-21 19:07:11.9688 Info 1 messages retrieved for iteration 69 for thread 1
2020-07-21 19:07:12.2347 Info Completed iteration 69 for thread 1
2020-07-21 19:07:12.3906 Info 1 messages retrieved for iteration 70 for thread 5
2020-07-21 19:07:12.6533 Info Completed iteration 70 for thread 5
2020-07-21 19:07:12.6990 Info 1 messages retrieved for iteration 71 for thread 2
2020-07-21 19:07:12.9680 Info Completed iteration 71 for thread 2
2020-07-21 19:07:13.1004 Info 1 messages retrieved for iteration 72 for thread 3
2020-07-21 19:07:13.3642 Info Completed iteration 72 for thread 3
2020-07-21 19:07:13.4772 Info 1 messages retrieved for iteration 73 for thread 4
2020-07-21 19:07:13.7345 Info Completed iteration 73 for thread 4
2020-07-21 19:07:13.8477 Info 1 messages retrieved for iteration 74 for thread 1
2020-07-21 19:07:14.1009 Info Completed iteration 74 for thread 1
2020-07-21 19:07:14.1812 Info 1 messages retrieved for iteration 75 for thread 5
2020-07-21 19:07:14.4479 Info Completed iteration 75 for thread 5
2020-07-21 19:07:14.5517 Info 1 messages retrieved for iteration 76 for thread 2
2020-07-21 19:07:14.8190 Info Completed iteration 76 for thread 2
2020-07-21 19:07:14.9361 Info 1 messages retrieved for iteration 77 for thread 3
2020-07-21 19:07:15.1989 Info Completed iteration 77 for thread 3
2020-07-21 19:07:15.3167 Info 1 messages retrieved for iteration 78 for thread 4
2020-07-21 19:07:15.5961 Info Completed iteration 78 for thread 4
2020-07-21 19:07:15.6242 Info 1 messages retrieved for iteration 79 for thread 1

Is ParallelOptions.MaxDegreeOfParallelism applied globally over multiple concurrent Parallel calls?

Consider this code run on a CPU with 32 cores:
ParallelOptions po = new ParallelOptions();
po.MaxDegreeOfParallelism = 8;
Parallel.For(0, 4, po, (i) =>
{
Parallel.For(0, 4, po, (j) =>
{
WorkMethod(i, j); // assume a long-running method
});
}
);
My question is what is the actual maximum possibly concurrency of WorkMethod(i, j)? Is it 4, 8, or 16?

ParallelOptions.MaxDegreeOfParallelism is not applied globally. If you have enough cores, and the scheduler sees fit you will get a multiplication of the nested MPD values with each For able to spin up that many tasks (if the workloads are unconstrained).
Consider this example, 3 tasks can start 3 more tasks. This is limited by the MDP option of 3.
int k = 0;
ParallelOptions po = new ParallelOptions();
po.MaxDegreeOfParallelism = 3;
Parallel.For(0, 10, po, (i) =>
{
Parallel.For(0, 10, po, (j) =>
{
Interlocked.Increment(ref k);
Console.WriteLine(k);
Thread.Sleep(2000);
Interlocked.Decrement(ref k);
});
Thread.Sleep(2000);
});
Output
1
2
3
4
7
5
6
8
9
9
5
6
7
9
9
8
8
9
...
If MDP was global you would only get 3 I guess, since it's not you get 9s.

ParallelOptions.MaxDegreeOfParallelism is not global, it is per parallel loop. And more specifically, it sets the max number of tasks that can run in parallel, not the max number of cores or threads that will run those tasks in parallel.
Some demo tests
note: i have 4 cores, 8 threads
What's happening in the code
We're running 2 async methods; each one kicks off nested parallel loops.
We're setting max degrees of parallelism to 2 and a sleep time of 2 seconds to simulate the work each task does
So, due to setting MaxDegreeOfParallelism to 2, we would expect to reach up to 12 concurrent tasks before the 40 tasks complete (i'm only counting tasks kicked off by the nested parallel loops)
how do i get 12?
2 max concurrent tasks started in the outer loop
+4 max concurrent tasks from inner loop (2 started per task started in outer loop)
that's 6 (per asynchronous task kicked off in Main)
12 total
test code
using System;
using System.Threading;
using System.Threading.Tasks;
namespace forfun
{
class Program
{
static void Main(string[] args)
{
var taskRunner = new TaskRunner();
taskRunner.RunTheseTasks();
taskRunner.RunTheseTasksToo();
Console.ReadLine();
}
private class TaskRunner
{
private int _totalTasks = 0;
private int _runningTasks = 0;
public async void RunTheseTasks()
{
await Task.Run(() => ProcessThingsInParallel());
}
public async void RunTheseTasksToo()
{
await Task.Run(() => ProcessThingsInParallel());
}
private void ProcessThingsInParallel()
{
ParallelOptions po = new ParallelOptions();
po.MaxDegreeOfParallelism = 2;
Parallel.For(0, 4, po, (i) =>
{
Interlocked.Increment(ref _totalTasks);
Interlocked.Increment(ref _runningTasks);
Console.WriteLine($"{_runningTasks} currently running of {_totalTasks} total tasks");
Parallel.For(0, 4, po, (j) =>
{
Interlocked.Increment(ref _totalTasks);
Interlocked.Increment(ref _runningTasks);
Console.WriteLine($"{_runningTasks} currently running of {_totalTasks} total tasks");
WorkMethod(i, j); // assume a long-running method
Interlocked.Decrement(ref _runningTasks);
});
Interlocked.Decrement(ref _runningTasks);
}
);
}
private static void WorkMethod(int i, int l)
{
Thread.Sleep(2000);
}
}
}
}
Spoiler, the output shows that setting MaxDegreeOfParallelism is not global, is not limited to core or thread count, and is specifically setting a max on concurrent running tasks.
output with max set to 2:
1 currently running of 1 total tasks
3 currently running of 3 total tasks
2 currently running of 2 total tasks
4 currently running of 4 total tasks
5 currently running of 5 total tasks
7 currently running of 7 total tasks
[ ... snip ...]
11 currently running of 33 total tasks
12 currently running of 34 total tasks
11 currently running of 35 total tasks
12 currently running of 36 total tasks
11 currently running of 37 total tasks
12 currently running of 38 total tasks
11 currently running of 39 total tasks
12 currently running of 40 total tasks
(output will vary, but each time, the max concurrent should be 12)
output without max set:
1 currently running of 1 total tasks
3 currently running of 3 total tasks
4 currently running of 4 total tasks
2 currently running of 2 total tasks
5 currently running of 5 total tasks
7 currently running of 7 total tasks
[ ... snip ...]
19 currently running of 28 total tasks
19 currently running of 29 total tasks
18 currently running of 30 total tasks
13 currently running of 31 total tasks
13 currently running of 32 total tasks
16 currently running of 35 total tasks
16 currently running of 36 total tasks
14 currently running of 33 total tasks
15 currently running of 34 total tasks
15 currently running of 37 total tasks
16 currently running of 38 total tasks
16 currently running of 39 total tasks
17 currently running of 40 total tasks
notice how without setting the max, we get up to 19 concurrent tasks
- now the 2 second sleep time is limiting the number of tasks that could kick off before others finished
output after increasing sleep time to 12 seconds
1 currently running of 1 total tasks
2 currently running of 2 total tasks
3 currently running of 3 total tasks
4 currently running of 4 total tasks
[ ... snip ...]
26 currently running of 34 total tasks
26 currently running of 35 total tasks
27 currently running of 36 total tasks
28 currently running of 37 total tasks
28 currently running of 38 total tasks
28 currently running of 39 total tasks
28 currently running of 40 total tasks
got up to 28 concurrent tasks
now setting loops to 10 nested in 10 and setting sleep time back to 2 seconds - again no max set
1 currently running of 1 total tasks
3 currently running of 3 total tasks
2 currently running of 2 total tasks
4 currently running of 4 total tasks
[ ... snip ...]
38 currently running of 176 total tasks
38 currently running of 177 total tasks
38 currently running of 178 total tasks
37 currently running of 179 total tasks
38 currently running of 180 total tasks
38 currently running of 181 total tasks
[ ... snip ...]
35 currently running of 216 total tasks
35 currently running of 217 total tasks
32 currently running of 218 total tasks
32 currently running of 219 total tasks
33 currently running of 220 total tasks
got up to 38 concurrent tasks before all 220 finished
More related information
ParallelOptions.MaxDegreeOfParallelism Property
The MaxDegreeOfParallelism property affects the number of concurrent operations run by Parallel method calls that are passed this ParallelOptions instance. A positive property value limits the number of concurrent operations to the set value. If it is -1, there is no limit on the number of concurrently running operations.
By default, For and ForEach will utilize however many threads the underlying scheduler provides, so changing MaxDegreeOfParallelism from the default only limits how many concurrent tasks will be used.
to get the max degree of parallelism, don't set it, rather allow the TPL and its scheduler handle it
setting the max degree of parallelism only affects the number of concurrent tasks, not threads used
the maximum number of concurrent tasks is not equal to the number of threads available--threads will still be able to juggle multiple tasks; and even if your app is using all threads, it is still sharing those threads with the other processes that the machine is hosting
Environment.ProcessorCount
Gets the number of processors on the current machine.
What if we say MaxDegreeOfParallelism = Environment.ProcessorCount?
Even setting max degree of parallism to Environment.ProcessorCount does not dynamically ensure that you get the maximum concurrency regardless of the system your app is running on. Doing this still limits the degree of parallelism, because any given thread can switch between many tasks--so this would just limit the number of concurrent tasks to equal the number of available threads--and this does not necessarily mean that each concurrent task will be assigned neatly to each thread in a one-to-one relationship.

Does System.Threading.Timer actually run in two different threads?

The code sample below
using System.Threading;
namespace TimerApp
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("***** Timer Application *****\n");
Console.WriteLine("In the thread #{0}", Thread.CurrentThread.ManagedThreadId);
// Create the delegate for the Timer type.
TimerCallback timerCB = new TimerCallback(ShowTime);
// Establish timer settings.
Timer t = new Timer(
timerCB, // The TimerCallback delegate object.
"Hello from Main()", // Any info to pass into the called method (null for no info).
0, // Amount of time to wait before starting (in milliseconds).
1000); // Interval of time between calls (in milliseconds).
Console.WriteLine("Hit key to terminate...");
Console.ReadLine();
}
// Method to show current time...
public static void ShowTime(object state)
{
Console.WriteLine("From the thread #{0}, it is background?{1}: time is {2}, param is {3}",
Thread.CurrentThread.ManagedThreadId,
Thread.CurrentThread.IsBackground,
DateTime.Now.ToLongTimeString(),
state.ToString());
}
}
}
produces the following output
***** Timer Application *****
In the thread #1
Hit key to terminate...
From the thread #4, it is background?True: time is 10:37:54 PM, param is Hello from Main()
From the thread #4, it is background?True: time is 10:37:55 PM, param is Hello from Main()
From the thread #5, it is background?True: time is 10:37:56 PM, param is Hello from Main()
From the thread #4, it is background?True: time is 10:37:57 PM, param is Hello from Main()
From the thread #5, it is background?True: time is 10:37:58 PM, param is Hello from Main()
From the thread #4, it is background?True: time is 10:37:59 PM, param is Hello from Main()
From the thread #5, it is background?True: time is 10:38:00 PM, param is Hello from Main()
...
Press any key to continue . . .
Does the System.Threading.Timer make callbacks using several threads at a time?

It makes use of the thread pool, using the first thread that it finds available at each time interval. The timer simply triggers the firing of these threads.
void Main()
{
System.Threading.Timer timer = new Timer((x) =>
{
Console.WriteLine($"{DateTime.Now.TimeOfDay} - Is Thread Pool Thread: {Thread.CurrentThread.IsThreadPoolThread} - Managed Thread Id: {Thread.CurrentThread.ManagedThreadId}");
Thread.Sleep(5000);
}, null, 1000, 1000);
Console.ReadLine();
}
Output
07:19:44.2628607 - Is Thread Pool Thread: True - Managed Thread Id: 10
07:19:45.2639080 - Is Thread Pool Thread: True - Managed Thread Id: 13
07:19:46.2644998 - Is Thread Pool Thread: True - Managed Thread Id: 9
07:19:47.2649563 - Is Thread Pool Thread: True - Managed Thread Id: 8
07:19:48.2660500 - Is Thread Pool Thread: True - Managed Thread Id: 12
07:19:49.2664012 - Is Thread Pool Thread: True - Managed Thread Id: 14
07:19:50.2669635 - Is Thread Pool Thread: True - Managed Thread Id: 15
07:19:51.2679269 - Is Thread Pool Thread: True - Managed Thread Id: 10
07:19:52.2684307 - Is Thread Pool Thread: True - Managed Thread Id: 9
07:19:53.2693090 - Is Thread Pool Thread: True - Managed Thread Id: 13
07:19:54.2839838 - Is Thread Pool Thread: True - Managed Thread Id: 8
07:19:55.2844800 - Is Thread Pool Thread: True - Managed Thread Id: 12
07:19:56.2854568 - Is Thread Pool Thread: True - Managed Thread Id: 15
In the code above we are setting the thread to wait 5 seconds, so after printing out to the console, the thread is kept alive for an additional 5 seconds before completing execution and returning to the Thread Pool.
The timer carries on firing on each second regardless, it's not waiting on the thread it triggered to complete.

blocking collection process n items at a time - continuing as soon as 1 is done

I have the following Scenario.
I take 50 jobs from the database into a blocking collection.
Each job is a long running one. (potentially could be). So I want to run them in a separate thread. (I know - it may be better to run them as Task.WhenAll and let the TPL figure it out - but I want to control how many runs simultaneously)
Say I want to run 5 of them simultaneously (configurable)
I create 5 tasks (TPL), one for each job and run them in parallel.
What I want to do is to pick up the next Job in the blocking collection as soon as one of the jobs from step 4 is complete and keep going until all 50 are done.
I am thinking of creating a Static blockingCollection and a TaskCompletionSource which will be invoked when a job is complete and then it can call the consumer again to pick one job at a time from the queue. I would also like to call async/await on each job - but that's on top of this - not sure if that has an impact on the approach.
Is this the right way to accomplish what I'm trying to do?
Similar to this link, but catch is that I want to process the next Job as soon as one of the first N items are done. Not after all N are done.
Update :
Ok, I have this code snippet doing exactly what I want, if someone wants to use it later. As you can see below, 5 threads are created and each thread starts the next job when it is done with current. Only 5 threads are active at any given time. I understand this may not work 100% like this always, and will have performance issues of context switching if used with one cpu/core.
var block = new ActionBlock<Job>(
job => Handler.HandleJob(job),
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 5 });
foreach (Job j in GetJobs())
block.SendAsync(j);
Job 2 started on thread :13. wait time:3600000ms. Time:8/29/2014
3:14:43 PM
Job 4 started on thread :14. wait time:15000ms. Time:8/29/2014
3:14:43 PM
Job 0 started on thread :7. wait time:600000ms. Time:8/29/2014
3:14:43 PM
Job 1 started on thread :12. wait time:900000ms. Time:8/29/2014
3:14:43 PM
Job 3 started on thread :11. wait time:120000ms. Time:8/29/2014
3:14:43 PM
job 4 finished on thread :14. 8/29/2014 3:14:58 PM
Job 5 started on thread :14. wait time:1800000ms. Time:8/29/2014
3:14:58 PM
job 3 finished on thread :11. 8/29/2014 3:16:43 PM
Job 6 started on thread :11. wait time:1200000ms. Time:8/29/2014
3:16:43 PM
job 0 finished on thread :7. 8/29/2014 3:24:43 PM
Job 7 started on thread :7. wait time:30000ms. Time:8/29/2014 3:24:43
PM
job 7 finished on thread :7. 8/29/2014 3:25:13 PM
Job 8 started on thread :7. wait time:100000ms. Time:8/29/2014
3:25:13 PM
job 8 finished on thread :7. 8/29/2014 3:26:53 PM
Job 9 started on thread :7. wait time:900000ms. Time:8/29/2014
3:26:53 PM
job 1 finished on thread :12. 8/29/2014 3:29:43 PM
Job 10 started on thread :12. wait time:300000ms. Time:8/29/2014
3:29:43 PM
job 10 finished on thread :12. 8/29/2014 3:34:43 PM
Job 11 started on thread :12. wait time:600000ms. Time:8/29/2014
3:34:43 PM
job 6 finished on thread :11. 8/29/2014 3:36:43 PM
Job 12 started on thread :11. wait time:300000ms. Time:8/29/2014
3:36:43 PM
job 12 finished on thread :11. 8/29/2014 3:41:43 PM
Job 13 started on thread :11. wait time:100000ms. Time:8/29/2014
3:41:43 PM
job 9 finished on thread :7. 8/29/2014 3:41:53 PM
Job 14 started on thread :7. wait time:300000ms. Time:8/29/2014
3:41:53 PM
job 13 finished on thread :11. 8/29/2014 3:43:23 PM
job 11 finished on thread :12. 8/29/2014 3:44:43 PM
job 5 finished on thread :14. 8/29/2014 3:44:58 PM
job 14 finished on thread :7. 8/29/2014 3:46:53 PM
job 2 finished on thread :13. 8/29/2014 4:14:43 PM

You can easily achieve what you need using TPL Dataflow.
What you can do is use BufferBlock<T>, which is a buffer for storing you data, and link it together with an ActionBlock<T> which will consume those requests as they're coming in from the BufferBlock<T>.
Now, the beauty here is that you can specify how many requests you want the ActionBlock<T> to handle concurrently using the ExecutionDataflowBlockOptions class.
Here's a simplified console version, which processes a bunch of numbers as they're coming in, prints their name and Thread.ManagedThreadID:
private static void Main(string[] args)
{
var bufferBlock = new BufferBlock<int>();
var actionBlock =
new ActionBlock<int>(i => Console.WriteLine("Reading number {0} in thread {1}",
i, Thread.CurrentThread.ManagedThreadId),
new ExecutionDataflowBlockOptions
{MaxDegreeOfParallelism = 5});
bufferBlock.LinkTo(actionBlock);
Produce(bufferBlock);
Console.ReadKey();
}
private static void Produce(BufferBlock<int> bufferBlock)
{
foreach (var num in Enumerable.Range(0, 500))
{
bufferBlock.Post(num);
}
}
You can also post them asynchronously if needed, using the awaitable BufferBlock.SendAsync
That way, you let the TPL handle all the throttling for you without needing to do it manually.

You can use BlockingCollection and it will work just fine, but it was built before async-await so it blocks synchronously which could be less scalable in most cases.
You're better off using async ready TPL Dataflow as Yuval Itzchakov suggested. All you need is an ActionBlock that processes each item concurrently with a MaxDegreeOfParallelism of 5 and you post your work to it synchronously (block.Post(item)) or asynchronously (await block.SendAsync(item)):
private static void Main()
{
var block = new ActionBlock<Job>(
async job => await job.ProcessAsync(),
new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = 5});
for (var i = 0; i < 50; i++)
{
block.Post(new Job());
}
Console.ReadKey();
}

You could do this with a SemaphoreSlim like in this answer, or using ForEachAsync like in this answer.

Thread count growth when using Task Parallel Library

I'm using C# TPL and I'm having a problem with a producer/consumer code... for some reason, TPL doesn't reuse threads and keeps creating new ones without stopping
I made a simple example to demonstrate this behavior:
class Program
{
static BlockingCollection<int> m_Buffer = new BlockingCollection<int>(1);
static CancellationTokenSource m_Cts = new CancellationTokenSource();
static void Producer()
{
try
{
while (!m_Cts.IsCancellationRequested)
{
Console.WriteLine("Enqueuing job");
m_Buffer.Add(0);
Thread.Sleep(1000);
}
}
finally
{
m_Buffer.CompleteAdding();
}
}
static void Consumer()
{
Parallel.ForEach(m_Buffer.GetConsumingEnumerable(), Run);
}
static void Run(int i)
{
Console.WriteLine
("Job Processed\tThread: {0}\tProcess Thread Count: {1}",
Thread.CurrentThread.ManagedThreadId,
Process.GetCurrentProcess().Threads.Count);
}
static void Main(string[] args)
{
Task producer = new Task(Producer);
Task consumer = new Task(Consumer);
producer.Start();
consumer.Start();
Console.ReadKey();
m_Cts.Cancel();
Task.WaitAll(producer, consumer);
}
}
This code creates 2 tasks, producer and consumer. Produces adds 1 work item every second, and Consumer only prints out a string with information. I would assume that 1 consumer thread is enough in this situation, because tasks are processed much faster than they are being added to the queue, but what actually happens is that every second number of threads in the process grows by 1... as if TPL is creating new thread for every item
after trying to understand what's happening I also noticed another thing: even though BlockingCollection size is 1, after a while Consumer starts getting called in bursts, for example, this is how it starts:
Enqueuing job
Job Processed Thread: 4 Process Thread Count: 9
Enqueuing job
Job Processed Thread: 6 Process Thread Count: 9
Enqueuing job
Job Processed Thread: 5 Process Thread Count: 10
Enqueuing job
Job Processed Thread: 4 Process Thread Count: 10
Enqueuing job
Job Processed Thread: 6 Process Thread Count: 11
and this is how it's processing items less than a minute later:
Enqueuing job
Job Processed Thread: 25 Process Thread Count: 52
Enqueuing job
Enqueuing job
Job Processed Thread: 5 Process Thread Count: 54
Job Processed Thread: 5 Process Thread Count: 54
and because threads get disposed after finishing Parallel.ForEach loop (I don't show it in this example, but it was in the real project) I assumed that it has something to do with ForEach specifically... I found this artice http://reedcopsey.com/2010/01/26/parallelism-in-net-part-5-partitioning-of-work/, and I thought that my problem was caused by this default partitioner, so I took custom partitioner from TPL Examples that is feeding Consumer threads item one by one, and although it fixed the order of execution (got rid of delay)...
Enqueuing job
Job Processed Thread: 71 Process Thread Count: 140
Enqueuing job
Job Processed Thread: 12 Process Thread Count: 141
Enqueuing job
Job Processed Thread: 72 Process Thread Count: 142
Enqueuing job
Job Processed Thread: 38 Process Thread Count: 143
Enqueuing job
Job Processed Thread: 73 Process Thread Count: 143
Enqueuing job
Job Processed Thread: 21 Process Thread Count: 144
Enqueuing job
Job Processed Thread: 74 Process Thread Count: 145
...it didn't stop threads from growing
I know about ParallelOptions.MaxDegreeOfParallelism, but I still want to understand what's happening with TPL and why it creates hundreds of threads for no reason
in my project I a code that has to run for hours and read new data from database, put it into a BlockingCollections and have has data processed by other code, there's 1 new item about every 5 seconds and it takes from several milliseconds to almost a minute to process it, and after running for about 10 minutes, thread count reached over a 1000 threads

There are two things that together cause this behavior:
ThreadPool tries to use the optimal number of threads for your situation. But if one of the threads in the pool blocks, the pool sees this as if that thread wasn't doing any useful work and so it tends to create another thread soon after that. What this means is that if you have a lot of blocking, ThreadPool is really bad at guessing the optimal number of threads and it tends to create new threads until it reaches the limit.
Parallel.ForEach() trusts the ThreadPool to guess the correct number of threads, unless you set the maximum number of threads explicitly. Parallel.ForEach() was also primarily meant for bounded collections, not streams of data.
When you combine these two things with GetConsumingEnumerable(), what you get is that Parallel.ForEach() creates threads that are almost always blocked. The ThreadPool sees this, and, to try to keep the CPU utilized, creates more and more threads.
The correct solution here is to set MaxDegreeOfParallelism. If your computations are CPU-bound, the best value is most likely Environment.ProcessorCount. If they are IO-bound, you will have to find out the best value experimentally.
Another option, if you can use .Net 4.5, is to use TPL Dataflow. This library was made specifically to process streams of data, like you have, so it doesn't have the problems your code has. It's actually even better than that and doesn't use any threads at all when it's not processing anything currently.
Note: There is also a good reason why is a new thread created for each new item, but explaining that would require me to explain how Parallel.ForEach() works in more detail and I feel that's not necessary here.