We Keep Coding

Task based vs. thread based Watchdog - but async needed

We're using watchdogs to determine whether a connected system is still alive or not.
In the previous code we used TCP directly and treated the watchdog in a separate thread. Now is a new service used that provides it's data using gRPC.
For that we tried using the async interface with tasks but a task based watchdog will fail.
I wrote a small DEMO that abstracts the code and illustrates the problem. You can switch between task based watchdog and thread based watchdog by commenting out line 18 with //.
The demo contains this code that causes the problem:
async Task gRPCSendAsync(CancellationToken cancellationToken = default) => await Task.Yield();
async Task gRPCReceiveAsync(CancellationToken cancellationToken = default) => await Task.Yield();
var start = DateTime.UtcNow;
await gRPCSendAsync(cancellationToken).ConfigureAwait(false);
await gRPCReceiveAsync(cancellationToken).ConfigureAwait(false);
var end = DateTime.UtcNow;
if ((end - start).TotalMilliseconds >= 100)
// signal failing
If this code is used in Task.Run it will signal failing if the application has a lot cpu-work to do in other tasks.
If a dedicated thread is used the watchdog works as expected and no problem is raise.
I do understand the problem: All code after await may be (if not finished already or does not contain a "real" await) queued to the thread pool. But the thread pool has other things to do so that it took too long to finish the method.
Yes the simple answer is: USE THREAD.
But using a thread limits us to only use synchronous methods. There is no way to call an async method out of a thread. I created another sample that shows that all code after first await will be queued to thread bool so that CallAsync().Wait() will not work. (Btw. that issue is much more handled here.)
We're having a lot of async code that may be used within such time critical operations.
So the question is: Is there any way to perform that that operations using tasks with async/await?
Maybe I'm completely wrong and creating an task based watchdog should be done very differently.
thoughts
I was thinking about System.Threading.Timer but the problem of async sending and async receiving will cause that problem anyways.

Here is how you could use Stephen Cleary's AsyncContext class from the Nito.AsyncEx.Context package, in order to constrain an asynchronous workflow to a dedicated thread:
await Task.Factory.StartNew(() =>
{
AsyncContext.Run(async () =>
{
await DoTheWatchdogAsync(watchdogCts.Token);
});
}, TaskCreationOptions.LongRunning);
The call to AsyncContext.Run will block until the supplied asynchronous operation is completed. All asynchronous continuations created by the DoTheWatchdogAsync will be processed internally by the AsyncContext on the current thread. In the above example the current thread is not a ThreadPool thread, because of the flag TaskCreationOptions.LongRunning used in the construction of the wrapper Task. You could confirm this by querying the property Thread.CurrentThread.IsThreadPoolThread.
If you prefer you could use a traditional Thread constructor instead of the somewhat unconventional Task.Factory.StartNew+LongRunning.

Task.WhenAny does not exist when task is finished

We have a logic for a background job to keep running, either every 20 minuts, or when a task if finished.
A simplified version of what I want to do is as followed:
Task to control if we need to exit:
private static TaskCompletionSource<bool> forceSyncTask = new TaskCompletionSource<bool>();
Background job:
Task.Factory.StartNew(
async () =>
{
do
{
await Dosomething();
await Task.WhenAny(Task.Delay(TimeSpan.FromMinutes(20)), forceSyncTask.Task);
// Always reset the force sync property
forceSyncTask = new TaskCompletionSource<bool>();
}
while (true);
});
Then everytime there is a notification comes, I run the following to force to exit the Task.WhenAny
if (!forceSyncTask.Task.IsCompleted)
{
forceSyncTask.TrySetResult(true);
}
I tested it in dev box and it works. However after I deployed it to our webervice in prod environment,
even if I successfully SetResult (I have logging to know if TrySetResult returns true or not), the Task.WhenAny does not exit as expected.
Anyone has any idea why?

First, I recommend you use an established solution for pausing asynchronous methods, such as Stephen Toub's PauseToken or the PauseToken from my AsyncEx library. There's some red flags in the code as it currently stands: StartNew with an async delegate and without a TaskScheduler, and TaskCompletionSource<T> being used without the RunContinuationsAsynchronously option. It's better to stick to higher-level constructs (Task.Run and PauseToken, respectively) because there are lots of sharp corners on the low-level constructs.
As far as what exactly the problem is, that's difficult to tell, especially since you (and we) cannot reproduce it locally. Here's my top guesses:
You're running into a problem caused by the fact that continuations run synchronously if possible - i.e., TrySetResult ends up directly invoking some code within Task.WhenAny.
You're experiencing thread exhaustion on your production server.

Synchronous I/O within an async/await-based Windows Service

Let's say I have a Windows Service which is doing some bit of work, then sleeping for a short amount of time, over and over forever (until the service is shut down). So in the service's OnStart, I could start up a thread whose entry point is something like:
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
}
And in the service's OnStop, I somehow set that shuttingDown flag and then join the thread. Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them. If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
DoSomething();
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
Assuming that's right, now let's say that DoSomething() call is (or includes) a synchronous write I/O to some piece of hardware. If I'm understanding correctly:
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
I am not sure I'm understanding that correctly - I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run. I'm not sure if it's also bad if I'm just doing it directly, as in the "private async Task Work()" example above?
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist), and then:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await DoSomethingAsync(cancel).ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Maybe I wrap it and use TaskCompletionSource? Something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await WrappedDoSomething().ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
private Task WrappedDoSomething()
{
var tcs = new TaskCompletionSource<object>();
DoSomething();
tcs.SetResult(null);
return tcs.Task;
}
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?

Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
On a side note, it's usually simpler to have a single "master" thread that will start/join all the others. Then OnStart/OnStop just deals with the master thread.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them.
That's a perfectly acceptable approach.
If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
Probably want to pass the CancellationToken down; cancellation can be used by synchronous code, too:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
DoSomething(cancel);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
It's not wrong, but it only saves you one thread on a Win32 service, which doesn't do much for you.
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
Dozens of threads are not a lot. Generally, asynchronous I/O is better because it doesn't use any threads at all, but in this case you're on the desktop, so threads are not a highly limited resource. async is most beneficial on UI apps (where the UI thread is special and needs to be freed), and ASP.NET apps that need to scale (where the thread pool limits scalability).
Bottom line: calling a blocking method from an asynchronous method is not bad but it's not the best, either. If there is an asynchronous method, call that instead. But if there isn't, then just keep the blocking call and document it in the XML comments for that method (because an asynchronous method blocking is rather surprising behavior).
I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run.
Yes, that is specifically about using Task.Run to wrap synchronous methods and pretend they're asynchronous. It's a common mistake; all it does is trade one thread pool thread for another.
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist)
Asynchronous is better (in terms of resource utilization - that is, fewer threads used), so if you want/need to reduce the number of threads, you should use async.
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Then just call it directly.
Maybe I wrap it and use TaskCompletionSource?
No, that doesn't do anything useful. That just calls it synchronously and then returns an already-completed task.
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Yup.
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
Assuming you already have a blocking Win32 service, it's probably fine to just keep it as it is. If you are writing a new one, personally I would make it async to reduce threads and allow asynchronous APIs, but you don't have to do it either way. I prefer Tasks over Threads in general, since it's much easier to get results from Tasks (including exceptions).
The "async all the way" rule only goes one way. That is, once you call an async method, then its caller should be async, and its caller should be async, etc. It does not mean that every method called by an async method must be async.
So, one good reason to have an async Win32 service would be if there's an async-only API you need to consume. That would cause your DoSomething method to become async DoSomethingAsync.
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?
No. You can always just block from an async method. With proper documentation so when you are reusing/maintaining this code a year from now, you don't swear at your past self. :)

If you still spawn your threads, well, yes, it's bad. Because it will not give you any benefit as the thread is still allocated and consuming resources for the specific purpose of running your worker function. Running a few threads to be able to do work in parallel within a service has a minimal impact on your application.
If DoSomething() is synchronous, you could switch to the Timer class instead. It allows multiple timers to use a smaller amount of threads.
If it's important that the jobs can complete, you can modify your worker classes like this:
SemaphoreSlim _shutdownEvent = new SemaphoreSlim(0,1);
public async Task Stop()
{
return await _shutdownEvent.WaitAsync();
}
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
_shutdownEvent.Release();
}
.. which means that during shutdown you can do this:
var tasks = myServices.Select(x=> x.Stop());
Task.WaitAll(tasks);

A thread can only do one thing at a time. While it is working on your DoSomething it can't do anything else.
In an interview Eric Lippert described async-await in a restaurant metaphor. He suggests to use async-await only for functionality where your thread can do other things instead of waiting for a process to complete, like respond to operator input.
Alas, your thread is not waiting, it is doing hard work in DoSomething. And as long as DoSomething is not awaiting, your thread will not return from DoSomething to do the next thing.
So if your thread has something meaningful to do while procedure DoSomething is executing, it's wise to let another thread do the DoSomething, while your original thread is doing the meaningful stuff. Task.Run( () => DoSomething()) could do this for you. As long as the thread that called Task.Run doesn't await for this task, it is free to do other things.
You also want to cancel your process. DoSomething can't be cancelled. So even if cancellation is requested you'll have to wait until DoSomething is completed.
Below is your DoSomething in a form with a Start button and a Cancel button. While your thread is DoingSomething, one of the meaningful things your GUI thread may want to do is respond to pressing the cancel button:
void CancellableDoSomething(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
DoSomething()
}
}
async Task DoSomethingAsync(CancellationToken token)
{
var task = Task.Run(CancellableDoSomething(token), token);
// if you have something meaningful to do, do it now, otherwise:
return Task;
}
CancellationTokenSource cancellationTokenSource = null;
private async void OnButtonStartSomething_Clicked(object sender, ...)
{
if (cancellationTokenSource != null)
// already doing something
return
// else: not doing something: start doing something
cancellationTokenSource = new CancellationtokenSource()
var task = AwaitDoSomethingAsync(cancellationTokenSource.Token);
// if you have something meaningful to do, do it now, otherwise:
await task;
cancellationTokenSource.Dispose();
cancellationTokenSource = null;
}
private void OnButtonCancelSomething(object sender, ...)
{
if (cancellationTokenSource == null)
// not doing something, nothing to cancel
return;
// else: cancel doing something
cancellationTokenSource.Cancel();
}

what are the circumstances of using .Wait() when calling async methods

I have the following async long running method inside my asp.net mvc-5 web application :-
public async Task<ScanResult> ScanAsync(string FQDN)
{
// sample of the operation i am doing
var c = await context.SingleOrDefaultAsync(a=>a.id == 1);
var list = await context.Employees.ToListAsync();
await context.SaveChangesAsync();
//etc..
}
and i am using Hangfire tool which support running background jobs to call this async method on timely basis, but un-fortuntly the hangefire tool does not support calling async methods directly . so to overcome this problem i created a sync version of the above method , as follow:-
public void Scan()
{
ScanAsync("test").Wait();
}
then from the HangFire scheduler i am calling the sync method as follow:-
RecurringJob.AddOrUpdate(() => ss.Scan(), Cron.Minutely);
so i know that using .Wait() will mainly occupy the iis thread during the method execution ,, but as i mentioned i need to do it in this way as i can not directly call an async TASK inside the hangefire scheduler .
so what will happen when i use .Wait() to call the async method ?, will the whole method's operations be done in a sync way ? for example as shown above i have three async operations inside the ScanAsync() ;SingleOrDefualtAsync,ToListAsync & SaveChangesAsync, so will they be executed in sync way because i am calling the ScanAsync method using .wait() ?

so what will happen when i use .Wait() to call the async method ?,
will the whole method's operations be done in a sync way ? for example
as shown above i have three async operations inside the ScanAsync()
;SingleOrDefualtAsync,ToListAsync & SaveChangesAsync, so will they be
executed in sync way because i am calling the ScanAsync method using
.wait() ?
The methods querying the database will still be executed asynchronously, but the fact that you're calling Wait means that even though you're releasing the thread, it wont return to the ASP.NET ThreadPool as you're halting it.
This is also a potential for deadlocks, as ASP.NET has a custom synchronization context which makes sure the context of the request is availiable when in a continuation of an async call.
I would recommend that instead, you'd use the synchronous API provided by entity-framework, as you won't actually be enjoying the scalability one can get from asynchronous calls.
Edit:
In the comments, you asked:
As i am currently doing with hangefire eliminate the async effect ? if yes then will it be better to use sync methods instead ? or using sync or async with hangeffire will be exactly the same
First, you have to understand what the benefits of async are. You don't do it because it's cool, you do it because it serves a purpose. What is that purpose? Being able to scale under load. How does it do that? When you await an async method, control is yielded back to the caller. For example, you have an incoming request, you query you database. You can either sit there and wait for the query to finish, or you can re-use that thread to serve more incomong requests. That's the true power.
If you don't actually plan on receiving a decent amount of requests (such that you'll starve the thread-pool), you won't actually see any benefit from async. Currently, the way you've implemented it, you won't see any of those benefits because you're blocking the async calls. All you'll see, possibly, are deadlocks.

This very much depends on the way HangFire is implemented. If it just queuing tasks to be invoked in ThreadPool the only effect will be, that one of your threads will be blocked until the request is ended. However if there is a custom SynchronizationContext this can lead to a serious deadlock.
Consider, if you really want to wait for your scheduled job to be done. Maybe all you want is just a fire and forget pattern. This way your method will be like:
public void Scan()
{
ScanAsync("test"); // smoothly ignore the task
}
If you do need to wait, you can instead try using async void method:
public async void Scan()
{
await ScanAsync("test");
DoSomeOtherJob();
}
There are many controversies about using async void as you cannot wait for this method to end, nor you can handle possible errors.
However, in event driven application this can be the only way. For more informations you can refer to: Async Void, ASP.Net, and Count of Outstanding Operations

Force a Task to continue on the current thread?

I'm making a port of the AKKA framework for .NET (don't take this too serious now, it is a weekend hack of the Actor part of it right now)
I'm having some problems with the "Future" support in it.
In Java/Scala Akka, Futures are to be awaited synchronously with an Await call.
Much like the .NET Task.Wait()
My goal is to support true async await for this.
It works right now, but the continuation is executed on the wrong thread in my current solution.
This is the result when passing a message to one of my actors that contain an await block for a future.
As you can see, the actor always executes on the same thread, while the await block executes on a random threadpool thread.
actor thread: 6
await thread 10
actor thread: 6
await thread 12
actor thread: 6
actor thread: 6
await thread 13
...
The actor gets a message using a DataFlow BufferBlock<Message>
Or rather, I use RX over the bufferblock to subscribe to messages.
It is configured like this:
var messages = new BufferBlock<Message>()
{
BoundedCapacity = 100,
TaskScheduler = TaskScheduler.Default,
};
messages.AsObservable().Subscribe(this);
So far so good.
However, when I await on a future result.
like so:
protected override void OnReceive(IMessage message)
{
....
var result = await Ask(logger, m);
// This is not executed on the same thread as the above code
result.Match()
.With<SomeMessage>(t => {
Console.WriteLine("await thread {0}",
System.Threading.Thread.CurrentThread.GetHashCode());
})
.Default(_ => Console.WriteLine("Unknown message"));
...
I know this is normal behavior of async await, but I really must ensure that only one thread has access to my actor.
I don't want the future to run synchronously, I want to to run async just like normal, but I want the continuation to run on the same thread as the message processor/actor does.
My code for the future support looks like this:
public Task<IMessage> Ask(ActorRef actor, IMessage message)
{
TaskCompletionSource<IMessage> result =
new TaskCompletionSource<IMessage>();
var future = Context.ActorOf<FutureActor>(name : Guid.NewGuid().ToString());
// once this object gets a response,
// we set the result for the task completion source
var futureActorRef = new FutureActorRef(result);
future.Tell(new SetRespondTo(), futureActorRef);
actor.Tell(message, future);
return result.Task;
}
Any ideas what I can do to force the continuation to run on the same thread that started the above code?

I'm making a port of the AKKA framework for .NET
Sweet. I went to an Akka talk at CodeMash '13 despite having never touched Java/Scala/Akka. I saw a lot of potential there for a .NET library/framework. Microsoft is working on something similar, which I hope will eventually be made generally available (it's currently in a limited preview).
I suspect that staying in the Dataflow/Rx world as much as possible is the easier approach; async is best when you have asynchronous operations (with a single start and single result for each operation), while Dataflow and Rx work better with streams and subscriptions (with a single start and multiple results). So my first gut reaction is to either link the buffer block to an ActionBlock with a specific scheduler, or use ObserveOn to move the Rx notifications to a specific scheduler, instead of trying to do it on the async side. Of course I'm not really familiar with the Akka API design, so take that with a grain of salt.
Anyway, my async intro describes the only two reliable options for scheduling await continuations: SynchronizationContext.Current and TaskScheduler.Current. If your Akka port is more of a framework (where your code does the hosting, and end-user code is always executed by your code), then a SynchronizationContext may make sense. If your port is more of a library (where end-user code does the hosting and calls your code as necessary), then a TaskScheduler would make more sense.
There aren't many examples of a custom SynchronizationContext, because that's pretty rare. I do have an AsyncContextThread type in my AsyncEx library which defines both a SynchronizationContext and a TaskScheduler for that thread. There are several examples of custom TaskSchedulers, such as the Parallel Extensions Extras which has an STA scheduler and a "current thread" scheduler.

Task scheduler decides whether to run a task on a new thread or on the current thread.
There is an option to force running it on a new thread, but none forcing it to run on the current thread.
But there is a method Task.RunSynchronously() which Runs the Task synchronously on the current TaskScheduler.
Also if you are using async/await there is already a similar question on that.