After countless hours of testing,searching, and asking question i'm still stuck on 'always-running' tasks, or does that not exist in async io/ parallell programming?
currently I have a program that polls webapi service (soap) to get status of sensor (IOT based).
The hiearchy of our programming is just flatten down, step1 then step2 then step3 and so on and so forth.
Now the problem here is if step 3 fails with an exception step 4-6 will not be executed.
So i had in mind to split up in tasks each with there coresponding task.
For instance; step 1 start program connect to webapi, step 2 get a list of sensors in memory, step 3 start tasks.
There are 3 main tasks: 1 task for getting the statusses for the sensors, another to check if the webserver is online, and the last one to ask the server if there are new sensors of old one getting deleted.
so step 4 to step 6 are 3 tasks. -> these task will always be running since there are just polling the server, this happens on timer ticks
step 4 each 1 second, step 5 each 10 seconds, and step 6 every minute.
Oke so far so good, I probably call var task1 = Task.Run( () => somevoidtask);
Question: How to handle here if one of the tasks has failed and needs to be restarted?
I was testing around and came across autoresetevent.
How will I handle the exception (gets logged) and restart that task?
On microsoft docs i commonly see:
var task = Task.Run();
task.wait();
if(task.status == task.faulted)
{
get exception
}
just simple pseudo.
my current test project I have this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace taskTest
{
public class Program
{
public static void Main(string[] args)
{
var source = new CancellationTokenSource(20000);
var source2 = new CancellationTokenSource();
var source3 = new CancellationTokenSource();
var token1 = source.Token;
var token2 = source2.Token;
var token3 = source3.Token;
Task1(token1);
Task2(token2);
Task3(token3);
Console.ReadLine();
}
private static void Task1(CancellationToken token)
{
while (true)
{
if (token.IsCancellationRequested)
{
Console.WriteLine("Cancellation is request for this task.");
return;
}
Task.Delay(1000);
}
}
private static void Task2(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
try
{
throw new NullReferenceException();
}
catch (Exception ex)
{
}
}
Console.WriteLine("Task2 is cancelled");
}
private static void Task3(CancellationToken token)
{
Task.Run(async () =>
{
while (!token.IsCancellationRequested)
{
Console.WriteLine($"Task3: getting executed every 10 second");
await Task.Delay(10000);
}
Console.WriteLine("Task3 is cancelled");
});
}
}
}
Also maybe a best practice I assume not all my code is good.
Exceptions can be tricky since it can be difficult to know if they have put the program in an unrecoverable state or not. If that has happened the best thing to do is to let your program crash and rely on an external system to restart it.
For exceptions you are reasonably sure are safe to continue from, just catch them, log it, and continue. Or perform any other recovery process needed.
I do not like any of your examples. Task1/2 will just block until canceled and only task2 have any kind of exception handling.
For reoccurring tasks I would recommend a timer, using things like Task.Delay is just a async-wrapper around a timer anyway. Use one that triggers on a background thread if you are sure there are no thread safety issues. Handle all exceptions that you consider safe to catch in the event-handler of the timer, consider adding an unhandled exception handler to log any other exceptions before closing your application. Making your application a service is a decent way to let windows deal with restarting your application if it fails.
Related
I have a Windows service that reads data from the database and processes this data using multiple REST API calls.
Originally, this service ran on a timer where it would read unprocessed data from the database and process it using multiple threads limited using SemaphoreSlim. This worked well except that the database read had to wait for all processing to finish before reading again.
ServicePointManager.DefaultConnectionLimit = 10;
Original that works:
// Runs every 5 seconds on a timer
private void ProcessTimer_Elapsed(object sender, ElapsedEventArgs e)
{
var hasLock = false;
try
{
Monitor.TryEnter(timerLock, ref hasLock);
if (hasLock)
{
ProcessNewData();
}
else
{
log.Info("Failed to acquire lock for timer."); // This happens all of the time
}
}
finally
{
if (hasLock)
{
Monitor.Exit(timerLock);
}
}
}
public void ProcessNewData()
{
var unproceesedItems = GetDatabaseItems();
if (unproceesedItems.Count > 0)
{
var downloadTasks = new Task[unproceesedItems.Count];
var maxThreads = new SemaphoreSlim(semaphoreSlimMinMax, semaphoreSlimMinMax); // semaphoreSlimMinMax = 10 is max threads
for (var i = 0; i < unproceesedItems .Count; i++)
{
maxThreads.Wait();
var iClosure = i;
downloadTasks[i] =
Task.Run(async () =>
{
try
{
await ProcessItemsAsync(unproceesedItems[iClosure]);
}
catch (Exception ex)
{
// handle exception
}
finally
{
maxThreads.Release();
}
});
}
Task.WaitAll(downloadTasks);
}
}
To improve efficiency, I rewrite the service to run GetDatabaseItems in a separate thread from the rest so that there is a ConcurrentDictionary of unprocessed items between them that GetDatabaseItems fills and ProcessNewData empties.
The problem is that while 10 unprocessed items are send to ProcessItemsAsync, they are processed two at a time instead of all 10.
The code inside of ProcessItemsAsync calls var response = await client.SendAsync(request); where the delay occurs. All 10 threads make it to this code but come out of it two at a time. None of this code changed between the old version and the new.
Here is the code in the new version that did change:
public void Start()
{
ServicePointManager.DefaultConnectionLimit = maxSimultaneousThreads; // 10
// Start getting unprocessed data
getUnprocessedDataTimer.Interval = getUnprocessedDataInterval; // 5 seconds
getUnprocessedDataTimer.Elapsed += GetUnprocessedData; // writes data into a ConcurrentDictionary
getUnprocessedDataTimer.Start();
cancellationTokenSource = new CancellationTokenSource();
// Create a new thread to process data
Task.Factory.StartNew(() =>
{
try
{
ProcessNewData(cancellationTokenSource.Token);
}
catch (Exception ex)
{
// error handling
}
}, TaskCreationOptions.LongRunning
);
}
private void ProcessNewData(CancellationToken token)
{
// Check if task has been canceled.
while (!token.IsCancellationRequested)
{
if (unprocessedDictionary.Count > 0)
{
try
{
var throttler = new SemaphoreSlim(maxSimultaneousThreads, maxSimultaneousThreads); // maxSimultaneousThreads = 10
var tasks = unprocessedDictionary.Select(async item =>
{
await throttler.WaitAsync(token);
try
{
if (unprocessedDictionary.TryRemove(item.Key, out var item))
{
await ProcessItemsAsync(item);
}
}
catch (Exception ex)
{
// handle error
}
finally
{
throttler.Release();
}
});
Task.WhenAll(tasks);
}
catch (OperationCanceledException)
{
break;
}
}
Thread.Sleep(1000);
}
}
Environment
.NET Framework 4.7.1
Windows Server 2016
Visual Studio 2019
Attempts to fix:
I tried the following with the same bad result (two await client.SendAsync(request) completing at a time):
Set Max threads and ServicePointManager.DefaultConnectionLimit to 30
Manually create threads using Thread.Start()
Replace async/await pattern with sync HttpClient calls
Call data processing using Task.Run(async () => and Task.WaitAll(downloadTasks);
Replace the new long-running thread for ProcessNewData with a timer
What I want is to run GetUnprocessedData and ProcessNewData concurrently with an HttpClient connection limit of 10 (set in config) so that 10 requests are processed at the same time.
Note: the issue is similar to HttpClient.GetAsync executes only 2 requests at a time? but the DefaultConnectionLimit is increased and the service runs on a Windows Server. It also creates more than 2 connections when original code runs.
Update
I went back to the original project to make sure it still worked, it did. I added a new timer to perform some unrelated operations and the httpClient issue came back. I removed the timer, everything worked. I added a new thread to do parallel processing, the problem came back.
This is not a direct answer to your question, but a suggestion for simplifying your service that could make the debugging of any problem easier. My suggestion is to implement the producer-consumer pattern using an iterator for producing the unprocessed items, and a parallel loop for consuming them. Ideally the parallel loop would have async delegates, but since you are targeting the .NET Framework you don't have access to the .NET 6 method Parallel.ForEachAsync. So I will suggest the slightly wasteful approach of using a synchronous parallel loop that blocks threads. You could use either the Parallel.ForEach method, or the PLINQ like in the example below:
private IEnumerable<Item> Iterator(CancellationToken token)
{
while (true)
{
Task delayTask = Task.Delay(5000, token);
foreach (Item item in GetDatabaseItems()) yield return item;
delayTask.GetAwaiter().GetResult();
}
}
public void Start()
{
//...
ThreadPool.SetMinThreads(degreeOfParallelism, Environment.ProcessorCount);
new Thread(() =>
{
try
{
Partitioner
.Create(Iterator(token), EnumerablePartitionerOptions.NoBuffering)
.AsParallel()
.WithDegreeOfParallelism(degreeOfParallelism)
.WithCancellation(token)
.ForAll(item => ProcessItemAsync(item).GetAwaiter().GetResult());
}
catch (OperationCanceledException) { } // Ignore
}).Start();
}
Online demo.
The Iterator fetches unprocessed items from the database in batches, and yields them one by one. The database won't be hit more frequently than once every 5 seconds.
The PLINQ query is going to fetch a new item from the Iterator each time it has a worker available, according to the WithDegreeOfParallelism policy. The setting EnumerablePartitionerOptions.NoBuffering ensures that it won't try to fetch more items in advance.
The ThreadPool.SetMinThreads is used in order to boost the availability of ThreadPool threads, since the PLINQ is going to use lots of them. Without it the ThreadPool will not be able to satisfy the demand immediately, although it will gradually inject more threads and eventually will catch up. But since you already know how many threads you'll need, you can configure the ThreadPool from the start.
In case you dislike the idea of blocking threads, you can find a simple substitute of the Parallel.ForEachAsync here, based on the TPL Dataflow library. It requires installing a NuGet package.
The issue turned out to be the place where ServicePointManager.DefaultConnectionLimit is set.
In the version where HttpClient was only doing two requests at a time, ServicePointManager.DefaultConnectionLimit was being set before the threads were being created but after the HttpClient was initialized.
Once I moved it into the constructor before the HttpClient is initialized, everything started working.
Thank you very much to #Theodor Zoulias for the help.
TLDR; Set ServicePointManager.DefaultConnectionLimit before initializing the HttpClient.
I'm trying to find out how to use WhenAll to let two methods run at once, and once they both finish, collect the results without blocking by using .Result
I have this little console app test:
using System.Diagnostics;
using System.Threading.Tasks;
namespace ConsoleApplication2
{
class Program
{
public static void Main(string[] args)
{
var run = TaskRunner();
Debug.WriteLine(run);
if (run.IsCompleted)
{
Debug.WriteLine("this worked!");
} else
{
Debug.WriteLine("this failed!");
}
}
public static async Task<string> TaskRunner()
{
var taskOne = OneAsync();
var taskTwo = TwoAsync();
var tasks = await Task.WhenAll(taskOne, taskTwo);
var retval = tasks[0] + tasks[1];
return retval;
}
public static Task<string> OneAsync()
{
return Task.Run(() =>
{
return "test1";
});
}
public static Task<string> TwoAsync()
{
return Task.Run(() =>
{
return "test2";
});
}
}
}
This currently prints this worked! to my Output window... However, if I comment out Debug.WriteLine(run); it prints this failed!... Why does the Task complete simply by being logged to the output window?
I'm trying to understand a huge problem in a complex piece of code and this little test is my MCVE to hopefully shed some light on what is happening behind the scenes.
This happens just by pure chance. The way you are starting your task is with Task.Run. This essentially creates a new thread on which the (synchronous) action is executed. It returns a task for the completion of that thread.
So OneAsync and TwoAsync will each spawn a new thread that then immediately returns a string. This will happen very quickly but there’s still some overhead for creating those threads which means that it won’t be instantaneous.
TaskRunner then calls both those methods (spawning the threads), and then asynchronously waits for both threads to finish. Since the threads are not completely instantly, this TaskRunner method also won’t complete instantly.
Now, in your main, you are starting the asynchronous TaskRunner, which we figured will take “a very short moment”. You do not await the task, so the execution continues immediately. Debug.WriteLine is executed to print something (it probably doesn’t really matter that it’s the task in question that is being printed), and then you are checking the state of the task.
Since printing stuff is relatively slow (compared to other operations), this is probably the reason why the tasks ends up being completed. And when you remove the printing, the if is just reached too quickly for the task to finish.
As you likely noticed, working like that with asynchronous tasks does not appear to be a good idea. That’s why you should always await the task when you depend on its result.
// note the *async* here; async main methods are supported with C# 7.1
public static async void Main(string[] args)
{
var run = TaskRunner();
// await the task
await run;
if (run.IsCompleted)
{
Debug.WriteLine("this worked!");
}
else
{
Debug.WriteLine("this failed!");
}
}
I have a headless UWP application that uses an external library to connect to a serial device and send some commands. It runs an infinite loop (while true) with a 10 minute pause between loops. The measurement process takes around 4 minutes.
The external library needs to run 3 measurements and after each it signals by raising an event. When the event is raised the 4th time I know that I can return the results.
After 4 hours (+/- a few seconds) the library stops raising events (usually it raises the event one or 2 times and then it halts, no errors, nothing).
I implemented in DoMeasureAsync() below a CancellationTokenSource that was supposed to set the IsCancelled property on the TaskCompletionSource after 8 minutes so that the task returns and the loop continues.
Problem:
When the measurement does not complete (the NMeasureCompletionSource never gets its result set in class CMeasure), the task from nMeasureCompletionSource is never cancelled. The delegate defined in RespondToCancellationAsync() should run after the 8 minutes.
If the measurement runs ok, I can see in the logs that the code in the
taskAtHand.ContinueWith((x) =>
{
Logger.LogDebug("Disposing CancellationTokenSource...");
cancellationTokenSource.Dispose();
});
gets called.
Edit:
Is it possible that the GC comes in after the 4 hours and maybe deallocates some variables and doing so makes the app to not be able to send the commands to the sensor? - It is not the case
What am I missing here?
//this gets called in a while (true) loop
public Task<PMeasurement> DoMeasureAsync()
{
nMeasureCompletionSource = new TaskCompletionSource<PMeasurement>();
cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromMinutes(8));
var t = cMeasure.Run(nitrateMeasureCompletionSource, cancellationTokenSource.Token);
var taskAtHand = nitrateMeasureCompletionSource.Task;
taskAtHand.ContinueWith((x) =>
{
Logger.LogDebug("Disposing CancellationTokenSource...");
cancellationTokenSource.Dispose();
});
return taskAtHand;
}
public class CMeasure
{
public async Task Run(TaskCompletionSource<PMeasurement> tcs, CancellationToken cancellationToken)
{
try
{
NMeasureCompletionSource = tcs;
CancellationToken = cancellationToken;
CancellationToken.Register(async () => await RespondToCancellationAsync(), useSynchronizationContext: false);
CloseDevice(); //Closing device if for some reason is still open
await Task.Delay(2500);
TheDevice = await GetDevice();
measurementsdone = 0;
Process(); //start the first measurement
}
catch (Exception ex)
{
DisconnectCommManagerAndCloseDevice();
NMeasureCompletionSource.SetException(ex);
}
}
public async Task RespondToCancellationAsync()
{
if (!NitrateMeasureCompletionSource.Task.IsCompleted)
{
Logger.LogDebug("Measure Completion Source is not completed. Cancelling...");
NMeasureCompletionSource.SetCanceled();
}
DisconnectCommManagerAndCloseDevice();
await Task.Delay(2500);
}
private void Process()
{
if (measurementsdone < 3)
{
var message = Comm.Measure(m); //start a new measurement on the device
}
else
{
...
NMeasureCompletionSource.SetResult(result);
}
}
//the method called when the event is raised by the external library
private void Comm_EndMeasurement(object sender, EventArgs e)
{
measurementsdone++;
Process();
}
}
After more testing I have reached the conclusion that there is no memory leak and that all the objects are disposed. The cancellation works well also.
So far it appears that my problem comes from the execution of the headless app on the Raspberry Pi. Although I am using the deferral = taskInstance.GetDeferral(); it seems that the execution is stopped at some point...
I will test more and come back with the results (possibly in a new post, but I will put a link here as well).
Edit:
Here is the new post: UWP - Headless app stops after 3 or 4 hours
Edit 2:
The problem was from a 3rd party library that I had to use and it had to be called differently from a headless app. Internally it was creating its own TaskScheduler if SynchronizationContext.Current was null.
I have made an interesting observation which I would like to fully understand.
The easiest way to explain this is by capturing it with this little sample console application:
namespace AsyncAwaitTestApp
{
using System;
using System.Diagnostics;
using System.Threading.Tasks;
class Program
{
static void Main(string[] args)
{
var timer = new Stopwatch();
// First Run:
// -------------------------
Console.WriteLine("Running GiveMeATask in parallel...");
timer.Start();
Parallel.For(0, 500, (i) =>
{
SillyClass.GiveMeATask().Wait();
});
timer.Stop();
Console.WriteLine($"GiveMeATask run completed. Total time: {timer.Elapsed.TotalSeconds} seconds.");
// Second Run:
// -------------------------
Console.WriteLine("-------------------------------");
Console.WriteLine("Running AwaitATask in parallel...");
timer.Restart();
Parallel.For(0, 500, (i) =>
{
SillyClass.AwaitATask().Wait();
});
timer.Stop();
Console.WriteLine($"AwaitATask run completed. Total time: {timer.Elapsed.TotalSeconds} seconds.");
Console.ReadLine();
}
}
public class SillyClass
{
public static TimeSpan SomeTime = TimeSpan.FromSeconds(3);
public static Task GiveMeATask()
{
return Task.Delay(SomeTime);
}
public static async Task AwaitATask()
{
await Task.Delay(SomeTime);
}
}
}
It is also available as a Gist.
SillyClass has two methods which both return a Task. Inside both methods I have a Task.Delay of 3 seconds to simulate an expensive network or IO bound operation.
The first method GiveMeATask simply returns the task from Task.Delay. The second method AwaitATask awaits the Task.Delay with the async/await keywords.
Both methods get called from Parallel.For to simulate concurrent calls.
What I found interesting is that the first run always takes about double the time as the second run, no matter how many times I repeat it:
As far as I understand the async/await pattern has to capture a synchronisation context, which adds a little bit of overhead, but I would be surprised if that is the reason?
I also understand that in certain applications such as a WPF application or an ASP.NET application where only 1 thread can access the UI/Request context an synchronously blocking .Wait() can cause deadlocks, but this is not the case for console applications and evidently I don't have a deadlock here, because both runs complete just fine.
So I am clearly not understanding the full picture of the above and would appreciate someone who can shed more light on this?
Is there a way to set a value for how long a thread should (maximally) be alive when you start the thread?
Said in another way, with "pseudocode", is there anything like this:
Thread t = new Thread();
t.start();
t.abort_after_x_seconds(30);
which would make the thread abort if it lived more than 30 seconds.
Edit: I still can't get it to work, what I originally had is:
while(true)
{
if(...)
{
Thread t = new Thread(new ThreadStart(startMethod));
t.start();
}
Thread.sleep(...);
}
the problem is that sometimes the threads will hang (I'm not implementing what the threads do so I don't know exactly why (it's a school project, we're noobs at organizing)), so I want to kill those threads. I tried using Tasks and CancellationTokens as in the examples below, but when the Task hangs
it can't check if a cancellation request has occured.
Most of the time, you shouldn't be using Threads, use Tasks instead. They are more convenient and more efficient.
Aborting something is not safe, you should use cooperative cancellation instead. If you're calling a method that supports cancellation, then just pass it a cancellation token that will be cancelled after 30 seconds.
So your code could look like this (using .Net 4.5):
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30)));
var task = Task.Run(() => YourMethod(cts.Token), cts.Token);
[EDIT: My response was far too slow. But I'll leave this here for the sample code.]
You should use co-operative cancellation for this purpose. The thread itself will need to detect when it should exit, and respond appropriately.
There's a thing called a CancellationToken produced from a CancellationTokenSource that you can use for this purpose.
There's even a CancellationTokenSource constructor which lets you set a timeout.
Here's some sample code to demonstrate its use:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
class Program
{
private void run()
{
using (var tokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)))
{
var task = Task.Run(() => exampleOne(tokenSource.Token));
task.Wait();
}
using (var tokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)))
{
var task = Task.Run(() => exampleTwo(tokenSource.Token));
task.Wait();
}
Console.WriteLine("Done.");
}
static void exampleZero()
{
Console.WriteLine("Starting exampleZero()");
try
{
Thread.Sleep(10000); // Simulate work.
}
catch (OperationCanceledException)
{
Console.WriteLine("Operation cancelled.");
}
Console.WriteLine("Exiting exampleZero()");
}
static void exampleOne(CancellationToken cancellation)
{
Console.WriteLine("Starting exampleOne()");
// Busy loop processing.
while (!cancellation.IsCancellationRequested)
{
// Do some work.
}
Console.WriteLine("Exiting exampleOne()");
}
static void exampleTwo(CancellationToken cancellation)
{
Console.WriteLine("Starting exampleTwo()");
while (!cancellation.WaitHandle.WaitOne(100)) // Wait 100ms between work.
{
// Do some work.
}
Console.WriteLine("Exiting exampleTwo()");
}
static void Main()
{
new Program().run();
}
}
}
As commenters have said, using Abort is bad practice and not guaranteed to abort immediately.
Why would you want to keep the thread alive? The thread will be released back to the pool when the task assigned to it is completed. The next time the task is run on the thread will automatically be given from the pool of threads either by creating another new one or re-using one that is available in the threadpool.
Sounds like your logic/code is bad and needs to be fixed rather than waiting for something for x seconds then terminating it, which in itself will cause knock on problems.
Perhaps you need a timer instead which can tick after 30 seconds then you can disable the timer and kill the task at hand.