We Keep Coding

Why do I seem to have so few threads

I am trying to understand some code (for performance reasons) that is processing tasks from a queue. The code is C# .NET Framework 4.8 (And I didn't write this stuff)
I have this code creating a timer that from what I can tell should use a new thread every 10 seconds
_myTimer = new Timer(new TimerCallback(OnTimerGo), null, 0, 10000 );
Inside the onTimerGo it calls DoTask() inside of DoTask() it grabs a task off a queue and then does this
System.Threading.Tasks.Task.Factory.StartNew(ProcessTask, task).ContinueWith(c => DoTask());
My reading of this is that a new thread should start running OnTimerGo every 10 seconds, and that thread should in parralel run ProcessTask on tasks as fast as it can get them from the queue.
I inserted some code to call ThreadPool.GetMaxThreads and ThreadPool.GetAvailableThreads to figure out how many threads were in use. Then I queued up 10,000 things for it to do and let it loose.
I never see more then 4 threads in use at a time. This is running on a c4.4xlarge ec2 instance... so 16 vCPU 30 gb mem. The get max and available return over 2k. So I would expect more threads. By looking at the logging I can see that a total of 50ish different threads (by thread id) end up doing the work over the course of 20 minutes. Since the timer is set to every 10 seconds, I would expect 100 threads to be doing the work (or for it to finish sooner).
Looking at the code, the only time a running thread should stop is if it asks for a task from the queue and doesn't get one. Some other logging shows that there are never more than 2 tasks running in a thread. This is probably because they work is pretty fast. So the threads shouldn't be exiting, and I can even see from the logs that many of them end up doing as many as 500 tasks over the 20 minutes.
so... what am I missing here. Are the ThreadPool.GetMaxThreads and ThreadPool.GetAvailableThreads not accurate if run from inside a thread? Is something shutting down some of the threads while letting others keep going?
EDIT: adding more code
public static void StartScheduler()
{
lock (TimerLock)
{
if (_timerShutdown == false)
{
_myTimer = new Timer(new TimerCallback(OnTimerGo), null, 0, 10 );
const int numberOfSecondsPerMinute = 60;
const int margin = 1;
var pollEventsPerMinute = (numberOfSecondsPerMinute/SystemPreferences.TaskPollingIntervalSeconds);
_numberOfTimerCallsForHeartbeat = pollEventsPerMinute - margin;
}
}
}
private static void OnTimerGo(object state)
{
try
{
_lastTimer = DateTime.UtcNow;
var currentTickCount = Interlocked.Increment(ref _timerCallCount);
if (currentTickCount == _numberOfTimerCallsForHeartbeat)
{
Interlocked.Exchange(ref _timerCallCount, 0);
MonitoringTools.SendHeartbeatMetric(Heartbeat);
}
CheckForTasks();
}
catch (Exception e)
{
Log.Warn("Scheduler: OnTimerGo exception", e);
}
}
public static void CheckForTasks()
{
try
{
if (DoTask())
_lastStart = DateTime.UtcNow;
_lastStartOrCheck = DateTime.UtcNow;
}
catch (Exception e)
{
Log.Error("Unexpected exception checking for tasks", e);
}
}
private static bool DoTask()
{
Func<DataContext, bool> a = db =>
{
var mtid = Thread.CurrentThread.ManagedThreadId;
int totalThreads = Process.GetCurrentProcess().Threads.Count;
int maxWorkerThreads;
int maxPortThreads;
ThreadPool.GetMaxThreads(out maxWorkerThreads, out maxPortThreads);
int AvailableWorkerThreads;
int AvailablePortThreads;
ThreadPool.GetAvailableThreads(out AvailableWorkerThreads, out AvailablePortThreads);
int usedWorkerThreads = maxWorkerThreads - AvailableWorkerThreads;
string usedThreadMessage = $"Thread {mtid}: Threads in Use count: {usedWorkerThreads}";
Log.Info(usedThreadMessage);
var taskTypeAndTasks = GetTaskListTypeAndTasks();
var task = GetNextTask(db, taskTypeAndTasks.Key, taskTypeAndTasks.Value);
if (_timerShutdown)
{
Log.Debug("Task processing stopped.");
return false;
}
if (task == null)
{
Log.DebugFormat("DoTask: Idle in thread {0} ({1} tasks running)", mtid, _processingTaskLock);
return false;
}
Log.DebugFormat("DoTask: starting task {2}:{0} on thread {1}", task.Id, mtid, task.Class);
System.Threading.Tasks.Task.Factory.StartNew(ProcessTask, task).ContinueWith(c => DoTask());
Log.DebugFormat("DoTask: done ({0})", mtid);
return true;
};
return DbExtensions.WithDbWrite(ctx => a(ctx));
}

The Task.Factory.StartNew by default doesn't create a new thread. It borrows a thread from the ThreadPool instead.
The ThreadPool is intended as a small pool of reusable threads, to help amortize the cost of running frequent and lightweight operations like callbacks, continuations, event handers etc. Depleting the ThreadPool from available workers by scheduling too much work on it, results in a situation that is called saturation or starvation. And as you've already figured out, it's not a happy situation to be.
You can prevent the saturation of the ThreadPool by running your long-running work on dedicated threads instead of ThreadPool threads. This can be done by passing the TaskCreationOptions.LongRunning as argument to the Task.Factory.StartNew:
_ = Task.Factory.StartNew(ProcessTask, task, CancellationToken.None,
TaskCreationOptions.LongRunning,
TaskScheduler.Default).ContinueWith(t => DoTask(), CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
The above code schedules the ProcessTask(task) on a new thread, and after the invocation is completed either successfully or unsuccessfully, the DoTask will be invoked on the same thread. Finally the thread will be terminated. The discard _ signifies that the continuation Task (the task returned by the ContinueWith) is fire-and-forget. Which, to put it mildly, is architecturally suspicious. 😃
In case you are wondering why I pass the TaskScheduler.Default explicitly as argument to StartNew and ContinueWith, check out this link.

My reading of this is that a new thread should start running OnTimerGo every 10 seconds, and that thread should in parralel run ProcessTask on tasks as fast as it can get them from the queue.
Well, that is definitely not what's happening. It's a lot of uncertainty about your code, but it's clear that another DoTask is starting AFTER ProcessTask completes. And that is not parallel execution. Your line of code is this
System.Threading.Tasks.Task.Factory.StartNew(ProcessTask, task).ContinueWith(c => DoTask());
I suggest you to start another DoTask right there like this:
System.Threading.Tasks.Task.Factory.StartNew(ProcessTask, task);
DoTask();
Make sure your code is ready for parallel execution, though.

Stop a line execution after a stipulated amount of time in C#

I have a Windows service, developed in C#, which does some calculation on data at equal intervals of time say 30 mins. It fetches the data from database and calls a method CalcData() which does some business logic calculations.
class Program
{
static void Main(string[] args)
{
try
{
AutoCalExecution ae = new AutoCalExecution();
ae.FetchData();
ae.CalData();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
Console.ReadLine();
}
}
class AutoCalExecution
{
public void FetchData()
{
// fetch data from db
}
public void CalData()
{
line1;
line2;
line3;
line4; // this line has som expression which actually does the calculation.
line5;
}
}
I have given the template of the code which I'm using. In CalData(), line4 is where the calculation is happening. The calculation typically takes 10 mins to be done. So line4 is executed for 10mins.
There are some scenarios where the calculation might take more than 10 mins. In that case I want to cancel the execution and go to line5 after certain amount of time, say 15 mins.
To summarize I want to set a timeout time for line4 for 15 mins(which can be configured based in requirement), If it doesn't finish with in 15 mins, it has to stop and come to line5.
public void CalData()
{
line1;
line2;
line3;
if ( set time to 15 mins exceeds){
line4; // this line has some expression which actually does the calculation.
}
else
{
Log (Line4 execution did not complete in stipulated time);
}
line5;
}
How do I set that condition in C#?
Update
This is something I tried:
var task = Task.Run(() => CalData());
if (task.Wait(TimeSpan.FromMinutes(Convert.ToDouble(timeout))))
{
if (task.Result)
{
log.Info("Completed");
}
else
{
log.Error("Not successful");
}
}
But the problem here is I want line5 to get executed in this method if line 4 doesn't finish. Is there a way I can write this similar code for a piece of code/snippet, instead of whole method?

Make line4 into a task.
Make the task cancellable by a cancellation token.
Use a cancellation token which cancels itself after 10mins (configured time).
https://learn.microsoft.com/en-us/dotnet/api/system.threading.cancellationtokensource.cancelafter?view=netframework-4.8
https://binary-studio.com/2015/10/23/task-cancellation-in-c-and-things-you-should-know-about-it/

I think you want something like this:
var work = Task.Run(() => line4);
if (work.Wait(TimeSpan.FromMinutes(10)))
{
// Work completed within the timeout
}
else
{
// Work did not complete within the timeout
}
Note that this will not actually stop the 'DoWork' code from running, it will continue on a worker thread until it is done. Also note that using 'Wait' risks deadlocks if used improperly, see Don't Block on Async Code.
If you actually want to cancel the processing you should give DoWork a cancellationToken and make the processing abort when the token is cancelled. There are also solutions to abort a running thread, but this is not recommended.

Call C# Task Repeatedly on conclusion

I have a C# app that must run blocks of code in parallel. Here is the basic structure of two of those blocks of code. In reality, there will be many more.
private async Task MyFirstTask()
{
// do stuff
await FirstTaskImplementation();
// cleanup
}
private async Task MySecondTask()
{
// do stuff
await SecondTaskImplementation();
// cleanup
}
Some of these blocks of code will run on a timer. Some will run repeatedly. In an attempt to accomplish, I have the following:
Task.Run(() => MyFirstTask());
Task.Run(() => MySecondTask());
When MyFirstTask has completed, I want to run it again. In fact, I want to run it over-and-over again until the program stops. Yet, I want MySecondTask to run in parallel of MyFirstTask. My question is, how do I execute MyFirstTask repeatedly, while still being parallel to MySecondTask?
I reviewed several of the related SO questions. I also do not see a Complete kind of event handler. So, I'm kind of lost in terms of how to implement this. I appreciate your help!

The beauty of async/await is that you can write asynchronous code in much the same way you'd write synchronous code. How would you repeat a synchronous operation? You could use a loop. You could do the same here, e.g.:
private async Task MyFirstTask(CancellationToken token) {
while (!token.IsCancellationRequested) {
// do stuff
await FirstTaskImplementation();
// cleanup
}
}
You can embed the loop in your current method or lift it into a wrapper method, but it should work either way.
You can continue scheduling your tasks the same way you're doing it now, though you really should await your async methods:
CancellationTokenSource cts = new CancellationTokenSource();
Task.Run(async () => await MyFirstTask(cts.Token));
Task.Run(async () => await MySecondTask());
// Request cancellation via `cts` when you want the looping to end.
And although you didn't ask about it, if you wanted to insert a delay between each iteration, you could simply place an await Task.Delay(...) statement at the end of the loop body.

You don't necessarily need to use Task.Run for the first task. You can use a Timer for the first task, with AutoReset set to true. That way it'll run forever and you don't have to worry about it anymore.
private static System.Timers.Timer aTimer;
public static void Main()
{
SetTimer(); //MyFirstTask starts running over and over in another thread
Task.Run(() => MySecondTask());
}
private static void SetTimer()
{
// Create a timer with a 1ms interval (has to be > 0)
aTimer = new System.Timers.Timer(1);
// Hook up the Elapsed event for the timer.
aTimer.Elapsed += MyFirstTask;
aTimer.AutoReset = true;
aTimer.Enabled = true;
}
private static async void MyFirstTask(Object source, ElapsedEventArgs e)
{
// do stuff
await FirstTaskImplementation();
// cleanup
}
private async Task MySecondTask()
{
// do stuff
await SecondTaskImplementation();
// cleanup
}

Another approach would be something like this:
var first=MyFirstTask().ToObservable();
var second=MySecondTask().ToObservable();
first.Repeat().Merge(second).Subscribe(x=>Console.WriteLine("Task completing."));
That's illustrative, not tested code. If you expect MySecondTask to complete, then perhaps this:
first.Repeat().TakeUntil(second).Subscribe(x=>Console.WriteLine("Task completing."));
If you want to add timeouts to second you could do this:
first.Repeat().TakeUntil(second.Timeout(TimeSpan.FromMilliseconds(100))).Subscribe(...)
If you want to show something on each task completion, declare the observables as:
var first=MyFirstTask().ToObservable().Do(Console.WriteLine("First completing"));
The above requires System.Reactive.Linq namespaces. I find these Rx based solutions to concurrency generally more elegant than the TPL, but that's just subjective.
Finally, if you do not want to start the tasks until the subscription is called, or something is ready to start watching, you can use Observable.FromAsync , as per info here

Using Task.Delay and a recursive call

I'm trying to figure out the best way to implement a delay into Task such that after the delay it calls itself again to attempt the same work.
My application is a server that generates reports from the database after the mobile devices sync their data with the server, however If another user has called the report generation method recently, I want it to pause for a period of time and then attempt to run again.
This is my current attempt
private static DateTime _lastRequest = Datetime.MinValue;
public async void IssueReports()
{
await Task.Run(() =>
{
if (DateTime.Now < _lastRequest + TimeSpan.FromMinutes(3)) //checks to see when a user last completed this method
{
Task.Delay(TimeSpan.FromMinutes(2));
IssueReports(); //calls itself again after the delay
return;
}
});
//code to generate reports goes here
_lastRequest = DateTime.Now; //updates last request into the static variable after it has finished running
}
Initially if it failed the check then the task would just end. This prevented 2 users hitting the database at the same time and it causing duplicate reports to be generated. However, the problem is that if 2 users sync within that same window then the second user reports wouldn't be sent until another sync call is done.
The delay is supposed to give the server time to finish generating the reports and updating the database before the next batch is requested by calling itself.
Am I overcomplicating things? I'm worried about it potentially hammering system resources with multiple loops in the event the reports take a long time to process

Following example run background service every 10 seconds recursively. This method is recommended only if you believe your task will complete within 10 seconds.
public frm_testform()
{
InitializeComponent();
dispatcherTimer_Tick().DoNotAwait();
}
private async Task dispatcherTimer_Tick()
{
DispatcherTimer timer = new DispatcherTimer();
TaskCompletionSource<bool> tcs = null;
EventHandler tickHandler = (s, e) => tcs.TrySetResult(true);
timer.Interval = TimeSpan.FromSeconds(10);
timer.Tick += tickHandler;
timer.Start();
while (true)
{
tcs = new TaskCompletionSource<bool>();
await Task.Run(() =>
{
// Run your background service and UI update here
await tcs.Task;
}
}

Load Test using C# Async Await

I am creating a console program, which can test read / write to a Cache by simulating multiple clients, and have written following code. Please help me understand:
Is it correct way to achieve the multi client simulation
What can I do more to make it a genuine load test
void Main()
{
List<Task<long>> taskList = new List<Task<long>>();
for (int i = 0; i < 500; i++)
{
taskList.Add(TestAsync());
}
Task.WaitAll(taskList.ToArray());
long averageTime = taskList.Average(t => t.Result);
}
public static async Task<long> TestAsync()
{
// Returns the total time taken using Stop Watch in the same module
return await Task.Factory.StartNew(() => // Call Cache Read / Write);
}

Adjusted your code slightly to see how many threads we have at a particular time.
static volatile int currentExecutionCount = 0;
static void Main(string[] args)
{
List<Task<long>> taskList = new List<Task<long>>();
var timer = new Timer(Print, null, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1));
for (int i = 0; i < 1000; i++)
{
taskList.Add(DoMagic());
}
Task.WaitAll(taskList.ToArray());
timer.Change(Timeout.Infinite, Timeout.Infinite);
timer = null;
//to check that we have all the threads executed
Console.WriteLine("Done " + taskList.Sum(t => t.Result));
Console.ReadLine();
}
static void Print(object state)
{
Console.WriteLine(currentExecutionCount);
}
static async Task<long> DoMagic()
{
return await Task.Factory.StartNew(() =>
{
Interlocked.Increment(ref currentExecutionCount);
//place your code here
Thread.Sleep(TimeSpan.FromMilliseconds(1000));
Interlocked.Decrement(ref currentExecutionCount);
return 4;
}
//this thing should give a hint to scheduller to use new threads and not scheduled
, TaskCreationOptions.LongRunning
);
}
The result is: inside a virtual machine I have from 2 to 10 threads running simultaneously if I don't use the hint. With the hint — up to 100. And on real machine I can see 1000 threads at once. Process explorer confirms this. Some details on the hint that would be helpful.

If it is very busy, then apparently your clients have to wait a while before their requests are serviced. Your program does not measure this, because your stopwatch starts running when the service request starts.
If you also want to measure what happen with the average time before a request is finished, you should start your stopwatch when the request is made, not when the request is serviced.
Your program takes only threads from the thread pool. If you start more tasks then there are threads, some tasks will have to wait before TestAsync starts running. This wait time would be measured if you remember the time Task.Run is called.
Besides the flaw in time measurements, how many service requests do you expect simultaneously? Are there enough free threads in your thread pool to simulate this? If you expect about 50 service requests at the same time, and the size of your thread pool is only 20 threads, then you'll never run 50 service requests at the same time. Vice versa: if your thread pool is way bigger than your number of expected simultaneous service requests, then you'll measure longer times than are actual the case.
Consider changing the number of threads in your thread pool, and make sure no one else uses any threads of the pool.