I've got an Uploaderclass with one method -Upload
public static int Upload(string endpoint,object objectToBeUploaded)
{
Source.Token.ThrowIfCancellationRequested();
var repos = new UploadRepository(endpoint);
return repos.Upload(objectToBeUploaded);
}
The Source is a static CancellationTokenSource available in the project.
I also have a list of endpoints I need to upload a certain object for.
The code in the Form (it's a very small project using WinForms) looks like this:
private async Task UploadObjectAsync(
string endpoint,
object objectToBeUploaded)
{
try
{
int elementId = await Task.Factory.StartNew(
() => Uploader.Upload(endpoint,objectToBeUploaded));
//do something with the returned value..
}
catch(OperationCanceledEception ex)
{
//handle the exception..
}
}
And then I set the btnUpload.Click handler like this so I can later use it:
this.btnUpload.Click += async (s, e) =>
{
foreach(var endpoint in endpoints)
{
await UploadObjectASsync(endpoint,someObject);
}
}
The problem is that whenever I start uploading to all the endpoints(how they are obtained is irrelevant) and I decide to cancel the uploading process using Source.Cancel(); the first UploadObjectAsyncwill always go through since
the Source.Token.ThrowIfCancellationRequested(); check in the Upload method has already been passed. The rest of tasks will be cancelled normally and handled gracefully.
How am I to restructure this code in order to make sure that the first UploadObjectAsync Task will also be cancelled?
It is worth mentioning that I also don't have access to the source code of the uploading process itself (service reference) - the repos.Upload(objectToBeUploaded) in my Upload method.
You need to make your UploadRepository.Upload take a CancellationToken.
Specially when that's the one doing the I/O operation.. That's when the async/await really pays-off.
That will also help you get rid of that: Task.Factory.StartNew since the Upload method will return Task already. There will be no need to spin off a task.
In your current setup, given enough time for the tasks to start (and go through your ThrowIfCancellationRequested) you won't be able to cancel any upload. Even if it takes 30 seconds.
Also, you might be interested in: Task.Run
There isn't anything practical you can do. The Upload method doesn't take a token. The first task has already passed the cancelation check by the time you hit the cancel button. You can prove to yourself the cancel is a timing issue by adding a 10 second sleep ahead of throw if cancelled call. All tasks would then cancel.
The problem is that you can't stop the process that happens inside the Upload function unless it checks for the state of the CancellationToken any terminates itself.
So what you could do is to abort the thread that is executing by doing something like this:
int elementId = await Task.Factory.StartNew(() =>
{
try
{
using (Source.Token.Register(Thread.CurrentThread.Interrupt))
{
return Uploader.Upload(endpoint, objectToBeUploaded));
}
}
catch (ThreadInterruptedException ex)
{
throw new OperationCanceledEception(ex)
}
}, Source.Token);
Using the Source.Token.Register(delegate) function you cause the token to call that function in case the token is cancelled. This way the thread that is currently executing the the uploaded and should throw a exception right away.
This method only works in case the thread enters the WaitSleepJoin-State from time to time, because the exception is only raised in case the thread is in that state. Have a look at the documentation of the Thread.Interrupt function.
The alternative is to use Thread.Abort and the ThreadAbortedException. This will kill your thread in any case, but it may corrupt the internal state of your service, because locks the thread holds won't be released properly. So be very careful using this method.
Related
We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings
Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.
Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.
Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc
For an async method that returns a Task<bool>, I need to take some actions when the method completes. The async method looks like this:
async Task<bool> EntryExists(string entry)
{
return await Task.Run(() => call_that_returns_bool());
}
I call it and attach a continuation task to it to perform follow-up actions:
EntryExists(my_entry).ContinueWith(t =>
{
if(t.Result) ...
});
However, I now need to conditionally wait for chained task to complete. Means depending upon a parameter, I either need to return immediately to the caller, or wait till the tasks finish. I change the above code to look like this:
var T = EntryExists(my_entry).ContinueWith(t =>
{
if(t.Result) ...
});
if(wait) T.Wait(); //wait is my parameter
Running this, the code gets stuck forever at T.Wait() when wait parameter is true, as if T never kicked off. I then tried the following:
var T = EntryExists(my_entry).ContinueWith(t =>
{
if(t.Result) ...
});
T.Start();
if(wait) T.Wait();
Upon which it tells me that
Start may not be called on a continuation task
I know I'm missing something basic, but after been coding for the last 15 hours, my brain isn't helping much. Can someone point out what I need to do?
You shouldn't block on async code. In short, you're very likely to be blocking the thread that the async method needs to return to (in this case you are, as it's the UI thread). The solution is to use async-await all the way through, rather than trying to run it synchronously. If you absolutely have to present a synchronous method, then at least provide as simpler wrapper as possible, rather than mixing ContinueWith and async-await.
Your outer call can be rewritten as:
async Task<{something}> MakeCallAndContinue()
{
try
{
await EntryExists(my_entry);
// additional continuation code
}
catch (Exception e)
{
// handle in some way
}
}
var task = MakeCallAndContinue();
if (wait) await task;
It's unusual to want to start a task and then not await it, which is what you're doing if wait is false. The error handler I've put in the above code is to ensure that you don't get an exception thrown on to an unawaited task. If you did that, then it would be thrown out somewhere else, and probably kill your process if you haven't declared a global handler.
You won't be able to use the above in your WPF command as-is, because it's async. However, you can have async WPF command handlers, as explained here. If you did want to do it synchronously then you would need to call .Wait(), but as Stephen Cleary explains in my first link, you have to use ConfigureAwait(false) on all awaited tasks all the way down in order to prevent one of them from trying to return to the occupied UI thread, and deadlocking.
I'm designing the software architecture for a product who can instantiate a series of "agents" doing some useful things.
Let's say each agent implement an interface having a function:
Task AsyncRun(CancellationToken token)
Because since these agents are doing a lot of I/O it could make some sense having as an async function. More over, the AsyncRun is supposed never complete, if no exception or explict cancellation occour.
Now the question is: main program has to run this on multiple agents, I would like to know the correct way of running that multiple task, signal each single completion ( that are due to cancellation/errors ):
for example I'm thinking on something like having an infinite loop like this
//.... all task cretaed are in the array tasks..
while(true)
{
await Task.WhenAny(tasks)
//.... check each single task for understand which one(s) exited
// re-run the task if requested replacing in the array tasks
}
but not sure if it is the correct ( or even best way )
And moreover I would like to know if this is the correct pattern, especially because the implementer can mismatch the RunAsync and do a blocking call, in which case the entire application will hang.
// re-run the task if requested replacing in the array tasks
This is the first thing I'd consider changing. It's far better to not let an application handle its own "restarting". If an operation failed, then there's no guarantee that an application can recover. This is true for any kind of operation in any language/runtime.
A better solution is to let another application restart this one. Allow the exception to propagate (logging it if possible), and allow it to terminate the application. Then have your "manager" process (literally a separate executable process) restart as necessary. This is the way all modern high-availability systems work, from the Win32 services manager, to ASP.NET, to the Kubernetes container manager, to the Azure Functions runtime.
Note that if you do want to take this route, it may make sense to split up the tasks to different processes, so they can be restarted independently. That way a restart in one won't cause a restart in others.
However, if you want to keep all your tasks in the same process, then the solution you have is fine. If you have a known number of tasks at the beginning of the process, and that number won't change (unless they fail), then you can simplify the code a bit by factoring out the restarting and using Task.WhenAll instead of Task.WhenAny:
async Task RunAsync(Func<CancellationToken, Task> work, CancellationToken token)
{
while (true)
{
try { await work(token); }
catch
{
// log...
}
if (we-should-not-restart)
break;
}
}
List<Func<CancellationToken, Task>> workToDo = ...;
var tasks = workToDo.Select(work => RunAsync(work, token));
await Task.WhenAll(tasks);
// Only gets here if they all complete/fail and were not restarted.
the implementer can mismatch the RunAsync and do a blocking call, in which case the entire application will hang.
The best way to prevent this is to wrap the call in Task.Run, so this:
await work(token);
becomes this:
await Task.Run(() => work(token));
In order to know whether the task completes successfully, or is cancelled or faulted, you could use a continuation. The continuation will be invoked as soon as the task finishes, whether that's because of failure, cancellation or completion. :
using (var tokenSource = new CancellationTokenSource())
{
IEnumerable<IAgent> agents; // TODO: initialize
var tasks = new List<Task>();
foreach (var agent in agents)
{
var task = agent.RunAsync(tokenSource.Token)
.ContinueWith(t =>
{
if (t.IsCanceled)
{
// Do something if cancelled.
}
else if (t.IsFaulted)
{
// Do something if faulted (with t.Exception)
}
else
{
// Do something if the task has completed.
}
});
tasks.Add(task);
}
await Task.WhenAll(tasks);
}
In the end you will wait for the continued tasks. Also see this answer.
If you are afraid that the IAgent implementations will create blocking calls and want to prevent the application from hanging, you can wrap the call to the async method in Task.Run. This way the call to the agent is executed on the threadpool and is therefore non-blocking:
var task = Task.Run(async () =>
await agent.RunAsync(tokenSource.Token)
.ContinueWith(t =>
{
// Same as above
}));
You may want to use Task.Factory.StartNew instead to mark the task as longrunning for example.
I am trying to get the proper 'structure' for monitoring the state of a game from external source(s) using (Tasks) async/await in order to run the tasks in an infinite loop, however the current way its written seems to just freeze up my UI.
What I have so far:
(in the "state machine" class)
// Start monitoring the game state for changes
public void Start()
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
IsRunning = true;
task = Task.Factory.StartNew(async () =>
{
while (true)
{
await Task.Run(()=>CheckForStateChange());
await Task.Delay(1000); // Pause 1 second before checking state again
}
}, token, TaskCreationOptions.LongRunning, TaskScheduler.FromCurrentSynchronizationContext());
}
Without the above "Task.Delay" line the UI completely freezes up. With the "Task.Delay" line it doesn't freeze up, but if I try to drag the window it skips back to where I began dragging it.
My assumption with the current code is that the 'await Task.Run()' executes and upon completion the 'await Task.Delay()' executes and then on completion returns to the beginning of the while(true) infinite loop. (ie. not running in parallel).
The CheckForStateChange() signature is as follows:
private void CheckForStateChange()
{
// ... A bunch of code to determine and update the current state value of the object
}
Nothing special there, simple non-async method. I have read through lots of examples / questions here on StackOverflow and I used to have CheckForStateChange as returning a Task (with awaitable actions inside the method) and many other iterations of code (with the same results).
Finally I call the Start() method from the main win32 form (button) as follows:
private void btnStartSW_Click(object sender, EventArgs e)
{
// Start the subscription of the event handler
if(!state.IsRunning)
{
state.StateChange += new SummonersWar.StateChangeHandler(OnGameStateChange);
state.Start();
}
}
I think the above code is the simplest form I have written the code structure in so far, but apparently its still not written 'properly'. Any help would be appreciated.
UPDATE:
The publisher side (state machine class):
// ------ Publisher of the event ---
public delegate void StateChangeHandler(string stateText);
public event StateChangeHandler StateChange;
protected void OnStateChange() // TODO pass text?
{
if (StateChange != null)
StateChange(StateText());
}
Where the StateText() method is just a temporary way of retrieving a 'text' representation of the current state (and is really a placeholder at this point until I organize it into a tidier struct)
IsRunning is purely a public bool.
And the handler in the UI thread:
private void OnGameStateChange(string stateText)
{
// Game State Changed (update the status bar)
labelGameState.Text = "State: " + stateText;
}
Why the UI freezes
In terms of the main question: you're already calling your CheckForStateChange via Task.Run, so there is no way that your CheckForStateChange will freeze the UI unless it includes calls which are marshalled back to the UI thread (i.e. Control.Invoke or SynchronizationContext.Post/Send used explicitly, or implicitly via a Task started on the UI TaskScheduler).
The best place to start looking is your StateChange handlers (i.e. StateChangeHandler). Also have a look at where the StateChange event is raised. You'll find thread marshalling code at one of these sites.
Other issues
You're passing the TaskScheduler pointing to the UI SynchronizationContext to the outer task. You're also passing in TaskCreationOptions.LongRunning. In simple terms you're telling the task factory to "start a task on a dedicated thread, and on the current thread". These two are mutually exclusive requirements and you can pretty safely drop them both.
If, as a result of the above, your outer task happens to execute on the UI thread, it won't really trip you up as the inner call is wrapped in Task.Run, but this probably isn't the behaviour you expect.
You are storing the result of Task.Factory.StartNew inside a task field or property. Note, however, that your Task.Factory.StartNew call returns a Task<Task>, so the saved Task instance will transition to completed state almost immediately unless you call Unwrap on it and get to the inner task. To avoid this entire mess, just use Task.Run to create the outer task (as it has Unwrap semantics built in). If you do that, you can ditch the inner Task.Run completely, like so:
public bool IsRunning
{
get
{
return task.Status == TaskStatus.Running;
}
}
public void Start()
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
task = Task.Run(async () =>
{
while (true)
{
CheckForStateChange(token);
token.ThrowIfCancellationRequested();
await Task.Delay(1000); // Pause 1 second before checking state again
}
}, token);
// Uncomment this and step through `CheckForStateChange`.
// When the execution hangs, you'll know what's causing the
// postbacks to the UI thread and *may* be able to take it out.
// task.Wait();
}
Since you have a CancellationToken you need to be passing it to CheckForStateChange, and checking it periodically - otherwise it only gets checked once, when the Task is started, and then never again.
Note that I have also provided a different IsRunning implementation. Volatile state is hard to get right. If the framework is giving it to you for free, you should use it.
Final word
Overall this entire solution feels like a bit of a crutch for something that should be done more reactively - but I can think of scenarios where this sort of design is valid. I'm just not convinced that yours is really one of them.
EDIT: how to find what's blocking the UI
I'll get downvoted to oblivion for this, but here goes:
The sure way to find what's causing postbacks to the UI thread is to deadlock with it. There's plenty of threads here on SO telling you how to avoid that, but in your case - we'll cause it on purpose and you'll know exactly what calls you need to avoid when you're polling for changes - although whether or not it will be possible to avoid these calls, remains to be seen.
I've put a task.Wait instruction at the end of my code snippet. Provided that you call Start on the UI thread, that should cause a deadlock with something inside your CheckForStateChange, and you will know what it is that you need to work around.
First of all, let me describe the flow of the program I am writing.
There is a list of objects in the GUI and when the user clicks on one, the object data is read from the disk and is loaded. This could take around 3-4 seconds.
Let's say the user is impatient and clicks on another object while the first one is still loading. The program will load the second object and at the same time, will cancel the loading for the first object.
As the user could spam load operations before any of them could complete successfully, I have implemented an action block to queue up all the load operations.
So I have an action block like this:
this.loadObjectActionBlock = new ActionBlock<CaseObject>(
c => this.LoadCaseData(CaseObject),
new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 10,
});
Every time the user has clicked on a Case Object, I will call:
this.loadObjectActionBlock.Post(CaseObject);
And the CaseObject will be processed by a function I have defined like this:
private void LoadCaseData(CaseObject caseObject)
{
caseObject.LoadCaseData();
}
What I need to do right now is, I need to wait until ALL CaseObjects are loaded so that I could continue my code after that.
I have tried detecting when all cases are processed by calling
if (this.loadObjectActionBlock.InputCount == 0)
{
this.loadObjectActionBlock.Complete();
}
after caseObject.LoadCaseData(); but this leads to strange results when load actions happen way too fast and the action block is told not to accept any more inputs. If I understand correctly, the InputCount property only looks at the number of jobs left in the queue.
So what I'd like to do is I'd like to await the ActionBlock like:
await this.loadObjectActionBlock.Completion;
when everything in the queue has been fully processed.
I may not be using the ActionBlock as it is intended to, so if there are any alternatives on this, please kindly suggest anything that I could read up on.
TLDR: I'd like to process multiple tasks (started by user) concurrently and wait for all to be complete then followed by a single task.
Thanks in advance :)
The program will load the second object and at the same time, will cancel the loading for the first object.
A queue is not the appropriate solution for this behavior, especially since a TPL block can only be completed once.
If you want to implement this behavior, simply ensure the cancellation token is observed before continuing with the next operation:
private static void ProcessCase(CaseObject caseObject, CancellationToken token)
{
caseObject.LoadCaseData();
token.ThrowIfCancellationRequested();
... // Further processing goes here
}
Called from the UI thread as:
static CancellationTokenSource _cts;
private static async Task ProcessCaseAsync(CaseObject caseObject)
{
if (_cts != null)
_cts.Cancel();
_cts = new CancellationTokenSource();
await Task.Run(() => ProcessCase(caseObject, _cts.Token));
}