I'm developing an application using some multithreading.
I have a thread that produces some data each 200ms (vibration data from an acquisition device). And each time i reacieve the data I start several tasks to do stuff. On my development PC there is no error. But when I deploy the project on a less powerful tablet I have the following message appearing several times :
thread is dead. priority cannot be accessed.
Here is my code :
private void myCallback1Axis(IAsyncResult ar)
{
GC.Collect();
try
{
if (runningTask == ar.AsyncState)
{
data = reader.EndReadWaveform(ar); // GET THE DATA
// LAUNCH FFTs' THREADS
CancellationToken ct = cts.Token;
task1 = System.Threading.Tasks.Task.Factory.StartNew(() =>
{
try
{
if (ct.IsCancellationRequested)
{
Console.WriteLine("Task {0}: Cancelling", task1.Id);
return;
}
Console.WriteLine("Task {0}: {1}/2 In progress", task1.Id, 1);
ConsumeToFFT(new FFT_Parameters(data.GetRawData(), overlap1, (int)Fmax1,
NbLines1, HP, avg1, window1, switchFreqUnit1.Value, swVelo.Value));
}
catch (OperationCanceledException)
{
// Any clean up code goes here.
Console.WriteLine("Task {0}: Cancelling", task1.Id);
//throw; // To ensure that the calling code knows the task was cancelled.
}
catch (Exception)
{
// Clean up other stuff
//throw; // If the calling code also needs to know.
}
}, ct);
}
catch (DaqException ex)
{
MessageBox.Show(ex.Message);
runningTask = null;
myTask.Dispose();
}
}
I read that I have to :
Put threads into background before starting (here)
But nothing more. I'm stuck with those messages and I can not trace them. Any ideas how can I fix this?
Thank you very much.
(I'm using c#, visual studio, .net 4.0)
Actually I was not able to trace the error, and by a certain miracle it's not there any more. The only thing that I've changed is here :
BlockingCollection<double[]> flowData = new BlockingCollection<double[]>(1);
Declaring my data as BlockingCollection and then use it like that in my consumer :
double[] dataToProcess = flowData.Take();
and like that in my producer :
flowData.TryAdd(data.GetRawData());
I think that the fact that my data was not protected against thread race condition generated the error. Even if I am not using any Thread Priority in my code, having better protection of my shared data was the solution to get rid of my error.
Sorry guys, it's not the greatest explanation but at least I don't have the error anymore. Thank you for your help.
Related
After countless hours of testing,searching, and asking question i'm still stuck on 'always-running' tasks, or does that not exist in async io/ parallell programming?
currently I have a program that polls webapi service (soap) to get status of sensor (IOT based).
The hiearchy of our programming is just flatten down, step1 then step2 then step3 and so on and so forth.
Now the problem here is if step 3 fails with an exception step 4-6 will not be executed.
So i had in mind to split up in tasks each with there coresponding task.
For instance; step 1 start program connect to webapi, step 2 get a list of sensors in memory, step 3 start tasks.
There are 3 main tasks: 1 task for getting the statusses for the sensors, another to check if the webserver is online, and the last one to ask the server if there are new sensors of old one getting deleted.
so step 4 to step 6 are 3 tasks. -> these task will always be running since there are just polling the server, this happens on timer ticks
step 4 each 1 second, step 5 each 10 seconds, and step 6 every minute.
Oke so far so good, I probably call var task1 = Task.Run( () => somevoidtask);
Question: How to handle here if one of the tasks has failed and needs to be restarted?
I was testing around and came across autoresetevent.
How will I handle the exception (gets logged) and restart that task?
On microsoft docs i commonly see:
var task = Task.Run();
task.wait();
if(task.status == task.faulted)
{
get exception
}
just simple pseudo.
my current test project I have this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace taskTest
{
public class Program
{
public static void Main(string[] args)
{
var source = new CancellationTokenSource(20000);
var source2 = new CancellationTokenSource();
var source3 = new CancellationTokenSource();
var token1 = source.Token;
var token2 = source2.Token;
var token3 = source3.Token;
Task1(token1);
Task2(token2);
Task3(token3);
Console.ReadLine();
}
private static void Task1(CancellationToken token)
{
while (true)
{
if (token.IsCancellationRequested)
{
Console.WriteLine("Cancellation is request for this task.");
return;
}
Task.Delay(1000);
}
}
private static void Task2(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
try
{
throw new NullReferenceException();
}
catch (Exception ex)
{
}
}
Console.WriteLine("Task2 is cancelled");
}
private static void Task3(CancellationToken token)
{
Task.Run(async () =>
{
while (!token.IsCancellationRequested)
{
Console.WriteLine($"Task3: getting executed every 10 second");
await Task.Delay(10000);
}
Console.WriteLine("Task3 is cancelled");
});
}
}
}
Also maybe a best practice I assume not all my code is good.
Exceptions can be tricky since it can be difficult to know if they have put the program in an unrecoverable state or not. If that has happened the best thing to do is to let your program crash and rely on an external system to restart it.
For exceptions you are reasonably sure are safe to continue from, just catch them, log it, and continue. Or perform any other recovery process needed.
I do not like any of your examples. Task1/2 will just block until canceled and only task2 have any kind of exception handling.
For reoccurring tasks I would recommend a timer, using things like Task.Delay is just a async-wrapper around a timer anyway. Use one that triggers on a background thread if you are sure there are no thread safety issues. Handle all exceptions that you consider safe to catch in the event-handler of the timer, consider adding an unhandled exception handler to log any other exceptions before closing your application. Making your application a service is a decent way to let windows deal with restarting your application if it fails.
I try to start some action in background, I am not interested in its result, but in case of an error I'd like to log it, and - of course - prevent the application (here: a Windows service) from crashing.
public static void CreateAndStartTaskWithErrorLogging(Action _action, string _componentName, string _originalStacktrace = null)
{
DateTime started = HighPrecisionClock.Now;
Task task = new Task(_action);
task.ContinueWith(_continuation => _continuation.LogExceptions(_componentName, started, _originalStacktrace));
task.ConfigureAwait(false);
task.Start();
}
internal static void LogExceptions(this Task _t, string _componentName, DateTime _started, string _originalStacktrace = null)
{
try
{
_t.Wait(1000);
}
catch (Exception ex)
{
Logger.LogError(_componentName, $"An exception occurred in a fire-and-forget task which was started at {_started}.\r\n" +
$"The original stack trace is:\r\n{_originalStacktrace}");
Logger.LogException(_componentName, ex);
}
try
{
_t.Dispose();
}
catch (Exception dex)
{
Logger.LogException(_componentName, dex);
}
}
Without ConfigureAwait(false) and without _t.Dispose(), the catch works and logs the exception. But the application crashes several seconds later (i.e. on the Finalizer thread?). The entry in the Microsoft Event Viewer shows that exception.
With ConfigureAwait and _t.Dispose(), I do not see the exception in the logs, the application just crashes.
What's wrong with the idea shown above?
Edit:
Meanwhile I tested without ConfigureAwait but with _t.Dispose. I could catch about 10 such exceptions, and none made the application crash. That seems to solve the issue, but I do not understand the reason for that, so the situation is still bad.
What does ConfigureAwait(false) do to Exceptions in the task (or in tasks started within that task, e.g. by a Parallel.ForEach further down)?
Why does the Dispose - which is called on the continuation, not the task proper according to a comment - prevent the crash (the Finalizer does not call Dispose, but Dispose may set some flags influencing its behavior)?
Edit 2:
Also that does not work all the time, only most of the time. Suggested solution 1 below also fails sometimes.
In the crashing context, the function is called with Utilities.TaskExtensions.CreateAndStartTaskWithErrorLogging(() => DataStore.StoreSyncedData(data), Name);, where DataStore is set to a composite which in turn calls Parallel.ForEach(m_InnerDataStores, _store => { _store.StoreSyncedData(_syncedData); }); on its members. One of them writes a video with the Accord library, which sometimes causes an AccessViolation at <Module>.avcodec_encode_video2(libffmpeg.AVCodecContext*, libffmpeg.AVPacket*, libffmpeg.AVFrame*, Int32*), i.e. the exception may come from non-managed code.
Of course, I could try to catch it somewhere down there - but that's not the objective of this method. I expect it to be able to safely run any code in the background without crashing the application.
This is my suggestion for logging errors:
public static void OnExceptionLogError(this Task task, string message)
{
task.ContinueWith(t =>
{
// Log t.Exception
}, TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously);
}
Usage example:
var task = Task.Run(action);
task.OnExceptionLogError("Oops!");
try
{
await task;
}
catch
{
// No need to log exception here
}
I am working on an application that talks to a motion controller over ethernet.
To connect to the controller I use a library provided by the supplier, to connect you create an instance of the controller than then tell it to connect, this has the chance to block for a few seconds (with no controllable timeout) if there is no controller present. This cause freeze ups in the UI.
To avoid this I thought I would be able to use Tasks to run the connection in a different thread.
ConnectionTask = Task.Factory.StartNew(() =>
{
try
{
RMCLink rmc = RMCLink.CreateEthernetLink(DeviceType.RMC70, "192.168.0.55");
RMC.Connect();
}
catch
{
this.logger.Log("Failed to connect");
}
}, TaskCreationOptions.LongRunning);
This has no effect whatsoever and the UI still locks up.
I think I am using them properly as if I replace it with the below code the UI is fine even though the separate thread takes a few seconds before the message comes out.
ConnectionTask = Task.Factory.StartNew(() =>
{
int x = 1;
while (x != 0) x++;
this.logger.Log("Failed to connect");
}, TaskCreationOptions.LongRunning);
Is there any way I can identify what is going on and prevent calls that I do not know anything about their inner workings from locking the UI thread.
Use async/await, something along the lines of:
public async void MyButton_Click(object sender, EventArgs e)
{
await CreateEthernetLink();
this.logger.Log("Connected!");
}
private async Task CreateEthernetLink()
{
var task = Task.Run(() => {
try
{
RMCLink rmc = RMCLink.CreateEthernetLink(DeviceType.RMC70, "192.168.0.55");
rmc.Connect();
}
catch
{
this.logger.Log("Failed to connect");
}});
await task;
}
The await will capture the current thread (or SynchronizationContext - in this case the UI thread which is being blocked) and restore it after the async work has been completed.
So the threading is all handled for you behind the scenes and you should notice no difference in your application other than the fact that your application no longer freezes when performing connections.
EDIT: I also noticed in your code your initializing rmc but calling connect on RMC. I don't think this is correct.
We have implemented a queue for uploading files to box.net.
All files are uploaded successfully. but,I am getting the following exception 1 or 2 times in a week. I couldn't found any cause for this exception.
Exception-Message:
Thread was being aborted.
Exception-Source:
mscorlib
Exception-StackTrace:
at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
at System.Threading.Monitor.Wait(Object obj, Int32 millisecondsTimeout, Boolean exitContext)
at System.Threading.Monitor.Wait(Object obj)
at Box.netAPIWebApp.Service.BoxService.monitorOnUploadQueue() in C:\Project\BackupProjects\BoxNetFileUpload\Box.netAPIWebApp\Source\Service\BoxService.cs:line 90
Can any one help on this?
private static readonly BoxService instance = new BoxService();
private Queue<FileCabinetUploadHistory> uploadQueue = new Queue<FileCabinetUploadHistory>();
private BoxService()
{
Thread monitorThread = new Thread(new ThreadStart(monitorOnUploadQueue));
monitorThread.Start();
}
private FileCabinetUploadHistory RemoveFromUploadQueue()
{
lock (uploadQueue)
{
return uploadQueue.Dequeue();
}
}
private void monitorOnUploadQueue()
{
FileCabinetUploadHistory fileCabinetUploadHistory = null;
try
{
while (true)
{
if (uploadQueue.Count < 1)
{
lock (uploadQueue)
{
Monitor.Wait(uploadQueue);
}
}
fileCabinetUploadHistory = uploadQueue.Peek();
if (fileCabinetUploadHistory != null)
{
StartFileUpload(fileCabinetUploadHistory);
}
}
}
catch (Exception exception)
{
log.Error("Error:--> Class name: BoxService, Method name: monitorOnUploadQueue() \n", exception);
}
}
public void AddToUploadQueue(FileCabinetUploadHistory fileCabinetUploadHistory)
{
lock (uploadQueue)
{
if (!uploadQueue.Contains(fileCabinetUploadHistory))
{
uploadQueue.Enqueue(fileCabinetUploadHistory);
Monitor.Pulse(uploadQueue);
}
}
}
Basically a ThreadAbortException means exactly that: Your thread received an external signal to kill itself. Now ThreadAbortException is a bit special because it can not be handled. It just keeps on terminating your thread by rethrowing itself every time you catch it. See http://ericlippert.com/2009/03/06/locks-and-exceptions-do-not-mix/ for details.
So now you probably ask yourself who sent the external signal mentioned above. I don't know. The code you showed does not suffice to say. But there is a very good chance that someone still has a handle of the monitorThread and calls thread.Abort() on it. Does your codebase contain .Abort()? If so rest assured that it's a very bad idea. Again see the above link for details.
If you absolutely have to terminate a thread that is waiting on a monitor there are better ways. For example make the thread wait on multiple monitors at the same time: One for the queue and one to signal for termination. Then instead of killing the thread by abort you just pulse the termination monitor and let the thread shut itself down.
By the way, you are accessing your queue in an unsafe way. Write access seems to be under lock condition but read access (Count, Peek) is not. This is not the way locking is supposed to be used and bad things can (and eventually will) happen. Don't do it! See http://blog.coverity.com/2014/03/12/can-skip-lock-reading-integer/ to find out why.
I got it.
ThreadAbortException occurs when Application pool recycles after every 29 hours.
I'm running Service Bus on Azure, pumping about 10-100 messages per second.
Recently I've switched to .net 4.5 and all excited refactored all the code to have 'async' and 'await' at least twice in each line to make sure it's done 'properly' :)
Now I'm wondering whether it's actually for better or for worse. If you could have a look at the code snippets and let me know what your thoughts are. I especially worried if the thread context switching is not giving me more grief than benefit, from all the asynchrony... (looking at !dumpheap it's definitely a factor)
Just a bit of description - I will be posting 2 methods - one that does a while loop on a ConcurrentQueue, waiting for new messages and the other method that sends one message at a time. I'm also using the Transient Fault Handling block exactly as Dr. Azure prescribed.
Sending loop (started at the beginning, waiting for new messages):
private async void SendingLoop()
{
try
{
await this.RecreateMessageFactory();
this.loopSemaphore.Reset();
Buffer<SendMessage> message = null;
while (true)
{
if (this.cancel.Token.IsCancellationRequested)
{
break;
}
this.semaphore.WaitOne();
if (this.cancel.Token.IsCancellationRequested)
{
break;
}
while (this.queue.TryDequeue(out message))
{
try
{
using (message)
{
//only take send the latest message
if (!this.queue.IsEmpty)
{
this.Log.Debug("Skipping qeued message, Topic: " + message.Value.Topic);
continue;
}
else
{
if (this.Topic == null || this.Topic.Path != message.Value.Topic)
await this.EnsureTopicExists(message.Value.Topic, this.cancel.Token);
if (this.cancel.Token.IsCancellationRequested)
break;
await this.SendMessage(message, this.cancel.Token);
}
}
}
catch (OperationCanceledException)
{
break;
}
catch (Exception ex)
{
ex.LogError();
}
}
}
}
catch (OperationCanceledException)
{ }
catch (Exception ex)
{
ex.LogError();
}
finally
{
if (this.loopSemaphore != null)
this.loopSemaphore.Set();
}
}
Sending a message:
private async Task SendMessage(Buffer<SendMessage> message, CancellationToken cancellationToken)
{
//this.Log.Debug("MessageBroadcaster.SendMessage to " + this.GetTopic());
bool entityNotFound = false;
if (this.MessageSender.IsClosed)
{
//this.Log.Debug("MessageBroadcaster.SendMessage MessageSender closed, recreating " + this.GetTopic());
await this.EnsureMessageSender(cancellationToken);
}
try
{
await this.sendMessageRetryPolicy.ExecuteAsync(async () =>
{
message.Value.Body.Seek(0, SeekOrigin.Begin);
using (var msg = new BrokeredMessage(message.Value.Body, false))
{
await Task.Factory.FromAsync(this.MessageSender.BeginSend, this.MessageSender.EndSend, msg, null);
}
}, cancellationToken);
}
catch (MessagingEntityNotFoundException)
{
entityNotFound = true;
}
catch (OperationCanceledException)
{ }
catch (ObjectDisposedException)
{ }
catch (Exception ex)
{
ex.LogError();
}
if (entityNotFound)
{
if (!cancellationToken.IsCancellationRequested)
{
await this.EnsureTopicExists(message.Value.Topic, cancellationToken);
}
}
}
The code above is from a 'Sender' class that sends 1 message/second. I have about 50-100 instances running at any given time, so it could be quite a number of threads.
Btw do not worry about EnsureMessageSender, RecreateMessageFactory, EnsureTopicExists too much, they are not called that often.
Would I not be better of just having one background thread working through the message queue and sending messages synchronously, provided all I need is send one message at a time, not worry about the async stuff and avoid the overheads coming with it.
Note that usually it's a matter of milliseconds to send one Message to Azure Service Bus, it's not really expensive. (Except at times when it's slow, times out or there is a problem with Service Bus backend, it could be hanging for a while trying to send stuff).
Thanks and sorry for the long post,
Stevo
Proposed Solution
Would this example be a solution to my situation?
static void Main(string[] args)
{
var broadcaster = new BufferBlock<int>(); //queue
var cancel = new CancellationTokenSource();
var run = Task.Run(async () =>
{
try
{
while (true)
{
//check if we are not finished
if (cancel.IsCancellationRequested)
break;
//async wait until a value is available
var val = await broadcaster.ReceiveAsync(cancel.Token).ConfigureAwait(false);
int next = 0;
//greedy - eat up and ignore all the values but last
while (broadcaster.TryReceive(out next))
{
Console.WriteLine("Skipping " + val);
val = next;
}
//check if we are not finished
if (cancel.IsCancellationRequested)
break;
Console.WriteLine("Sending " + val);
//simulate sending delay
await Task.Delay(1000).ConfigureAwait(false);
Console.WriteLine("Value sent " + val);
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}, cancel.Token);
//simulate sending messages. One every 200mls
for (int i = 0; i < 20; i++)
{
Console.WriteLine("Broadcasting " + i);
broadcaster.Post(i);
Thread.Sleep(200);
}
cancel.Cancel();
run.Wait();
}
You say:
The code above is from a 'Sender' class that sends 1 message/second. I
have about 50-100 instances running at any given time, so it could be
quite a number of threads.
This is a good case for async. You save lots of threads here. Async reduces context switching because it is not thread-based. It does not context-switch in case of something requiring a wait. Instead, the next work item is being processed on the same thread (if there is one).
For that reason you async solution will definitely scale better than a synchronous one. Whether it actually uses less CPU at 50-100 instances of your workflow needs to be measured. The more instances there are the higher the probability of async being faster becomes.
Now, there is one problem with the implementation: You're using a ConcurrentQueue which is not async-ready. So you actually do use 50-100 threads even in your async version. They will either block (which you wanted to avoid) or busy-wait burning 100% CPU (which seems to be the case in your implementation!). You need to get rid of this problem and make the queuing async, too. Maybe a SemaphoreSlim is of help here as it can be waited on asynchronously.
First, keep in mind that Task != Thread. Tasks (and async method continuations) are scheduled to the thread pool, where Microsoft has put in tons of optimizations that work wonders as long as your tasks are fairly short.
Reviewing your code, one line raises a flag: semaphore.WaitOne. I assume you're using this as a kind of signal that there is data available in the queue. This is bad because it's a blocking wait inside an async method. By using a blocking wait, the code changes from a lightweight continuation into a much heavier thread pool thread.
So, I would follow #usr's recommendation and replace the queue (and the semaphore) with an async-ready queue. TPL Dataflow's BufferBlock<T> is an async-ready producer/consumer queue available via NuGet. I recommend this one first because it sounds like your project could benefit from using dataflow more extensively than just as a queue (but the queue is a fine place to start).
Other async-ready data structures exist; my AsyncEx library has a couple of them. It's also not hard to build a simple one yourself; I have a blog post on the subject. But I recommend TPL Dataflow in your situation.