We have a Windows service that calls a 3rd party method that can hang when improperly configured by the end user, which is difficult to test for beforehand. We're handling this risk by calling the method in a Task with a timeout:
private int? FooWithTimeout()
{
var timeout = TimeSpan.FromMinutes(1);
var task = Task.Run(Foo);
if (!task.Wait(timeout))
{
Log("Foo timed out...");
return null;
}
if (task.IsFaulted)
{
Log("Foo threw an exception...");
return null;
}
return task.Result;
}
The 3rd party method blocks forever waiting for input from a resource that cannot respond. It does not support cancellation in any way, shape, or form, nor does it have a built-in timeout. Our concern is that as the service runs these tasks will continue blocking and slowly accumulate, eventually consuming a large amount of resources.
Is that a valid concern? Do we need to abort/dispose the tasks in some way? If so, what is the correct way to do so?
Addendum: The 3rd party in question is Crystal Reports. It hangs when asked to print to a printer that requires some sort of additional input from the user (for example, Microsoft XPS Document Writer will prompt you for where to save the file if you print to it). And by hang, I mean it attempts to show a user prompt to get the additional input, but it's in a Windows Service so nobody ever sees the user prompt and it waits forever for a human being to tell it how to print. We allow end users to configure which printer the service attempts to print to, and there isn't really any way to tell if a given printer requires additional input short of attempting to print to it.
Our concern is that as the service runs these tasks will continue blocking and slowly accumulate
This is a valid concern. You can reduce the consequences of that by starting those tasks on a special thread-pool that uses small stacks. That way there is less memory usage. But it's not a complete fix. If you app must function for a long time (not a GUI app intended for a few hours of use) then this solution will prove unacceptable because eventually the app will suffer from resource exhaustion.
.NET has no way to terminate an uncooperative thread. You need to run these actions in their own processes. You can then terminate those processes safely.
Using AppDomains might be safe as well but it's less certain. It's possible that per-process state is being corrupted when an AppDomain and threads in it are aborted. Also, not all thread can be aborted. In particular IO operations.
Using separate processes is also not a guarantee that no state corruption will result from termination. But in practice most corruptible state live in memory. Process termination likely clears all inconsistent state.
One approach could be to create separate thread which would monitor for any newly created windows - if any new window is created - thread would try to forcefully to close them.
Enumerate windows:
How to enumerate all windows belonging to a particular process using .NET?
And close non wanted windows:
How to use WM_Close in C#?
May be will not work, just a proposal... :-)
Related
So, I basically have this:
public void DoThisThing()
{
Task.Run(() =>
{
while(true)
{
//Do things
}
}
}
The start of the application basically calls the DoThisThing() method and enters it's own loop.
So, if I just close the application, what happens to this task? Does it just end? does it continue forever? Does it go on for a little bit until garbage collection gets it? Does it have a way to know the application ended?
I googled, but I couldn't get a simple answer, and I feel like there definitely is one.
The first question is, how this task is even executed. According to the Documentation:
Queues the specified work to run on the ThreadPool and returns a task or Task handle for that work.
Each programm starts with one Thread, but can start further. This one is the Main Thread, the GUI Thread and a few other names. As a general rule, if that main thread is ending, all others threads it started are canceled too. This is a OS level rule, to avoid Zombie Threads with infinite loops running all over the place.
The ThreadPool and all it's data - including sheduled and running Threads - will be collected by the "End of Application" Garbage Colleciton. If not, there is propably some OS features to make sure they end as well. But I prefer not to mention those OS fallbacks, as we really should not be using them ever. There are for cases when even the GC can no longe run properly (like Programm being killed via Task Manager).
One of the big challenges of Multitasking and -threading is keeping the main Thread alive, but without blocking it so further I/O can happen. In a GUI you have that work done for you (with the EventQueue).
All which is said below is implementation details - FOR WINDOWS - and mostly undocumented behavior. Do not rely on any of the information.
As an implementation detail, this task will most likely be scheduled to execute on a thread pool thread.
If the task has not started by the time the process exit starts, it won't matter it was queued in the first place.
If the task is currently executing, then according to some of the implementation details of process shutdown on Windows eventually only one thread will be executing which will not be the one executing this task. So, it will be forcibly terminated in that case.
If the task has already finished execution, whether through completion or by throwing an exception then there's no thread occupied by it. However, if the exception was left unobserved then the finalizer - should it get a chance to execute - will throw that. Please note that finalizers are also not guaranteed to execute under any circumstances.
This page should have been visible, but Microsoft's latest screw up in revoking access to old MSDN blogs continues.
Similarly, if you can manage to track the first link on this page then do so and read it.
P.S.: Here's the link for Raymond's blog. What you'll find from both sources is that only one thread continues the process shutdown.
The answer depends on the content of the while loop.
If the loop is running some logic that runs entirely within the scope and control of the main program, then closing the application will terminate everything.
However, if the loop is calling some external routines or operating system functions (Example: write to a file, open a network connection, run a command, start a batch job, etc), then closing the application will not terminate everything.
Based on your sample, in brief: Yes
Tasks that are created by TPL (using Task.Run or Task.Factory.StartNew) by default are background threads. So closing application will immediately terminate them.
This post could be helpfull.
I need to execute an infinite while loop and want to initiate the execution in global.asax.
My question is how exactly should I do it? Should I start a new Thread or should I use Async and Task or anything else? Inside the while loop I need to do await TaskEx.Delay(5000);
How do I do this so it will not block any other processes and will not create memory leaks?
I use VS10,AsyncCTP3,MVC4
EDIT:
public void SignalRConnectionRecovery()
{
while (true)
{
Clients.SetConnectionTimeStamp(DateTime.UtcNow.ToString());
await TaskEx.Delay(5000);
}
}
All I need to do is to run this as a singleton instance globally as long as application is available.
EDIT:SOLVED
This is the final solution in Global.asax
protected void Application_Start()
{
Thread signalRConnectionRecovery = new Thread(SignalRConnectionRecovery);
signalRConnectionRecovery.IsBackground = true;
signalRConnectionRecovery.Start();
Application["SignalRConnectionRecovery"] = signalRConnectionRecovery;
}
protected void Application_End()
{
try
{
Thread signalRConnectionRecovery = (Thread)Application["SignalRConnectionRecovery"];
if (signalRConnectionRecovery != null && signalRConnectionRecovery.IsAlive)
{
signalRConnectionRecovery.Abort();
}
}
catch
{
///
}
}
I found this nice article about how to use async worker: http://www.dotnetfunda.com/articles/article613-background-processes-in-asp-net-web-applications.aspx
And this:
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6
But I think for my needs this one will be perfect:
http://forums.asp.net/t/1433665.aspx/1
ASP.NET is not designed to handle this kind of requirement. If you need something to run constantly, you would be better off creating a windows service.
Update
ASP.NET is not designed for long running tasks. It's designed to respond quickly to HTTP requests. See Cyborgx37's answer or Can I use threads to carry out long-running jobs on IIS? for a few reasons why.
Update
Now that you finally mentioned you are working with SignalR, I see that you are trying to host SignalR within ASP.NET, correct? I think you're going about this the wrong way, see the example NuGet package referenced on the project wiki. This example uses an IAsyncHttpHandler to manage tasks.
You can start a thread in your global.asax, however it will only run till your asp.net process get recycled. This will happen at least once a day, or when no one uses of your site. If the process get recycled, the only way the thread is restarted agian, is when you have a hit on your site. So the thread is not running continueuosly.
To get a continues process it is better to start a windows service.
If you do the 'In process' solution, it realy depends on what your are doing. The Thread itself will not cause you any problems in memory or deadlocks. You should add a meganism to stop your thread when the application stops. Otherwise restarting will take a long time, because it will wait for your thread to stop.
This is an old post, but as I was seraching for this, I would like to report that in .NET 4.5.2 there is a native way to do it with QueueBackgroundWorkItem.
Take a look at this post: https://blogs.msdn.microsoft.com/webdev/2014/06/04/queuebackgroundworkitem-to-reliably-schedule-and-run-background-processes-in-asp-net/
MarianoC
It depends what you are trying to accomplish in your while loop, but in general this is the kind of situation where a Windows Service is the best answer. Installing a Windows Service is going to require that you have admin privileges on the web server.
With an infinite loop you end up with a lot of issues regard the Windows message pump. This is the thing that keeps a Windows application alive even when the application isn't "doing" anything. Without it, a program simply ends.
The problem with an infinite loop is that the application is stuck "doing" something, which prevents other applications (or threads) from "doing" their thing. There have been a few workarounds, such as the DoEvents in Windows Forms, but they all have some serious drawbacks when it comes to responsiveness and resource management. (Acceptable on a small LOB application, maybe not on a web server.) Even if the while-loop is on a separate thread, it will use up all available processing power.
Asynchronus programming is really designed more for long-running processes, such as waiting for a database to return a result or waiting for a printer to come online. In these cases, it's the external process that is taking a long time, not a while-loop.
If a Window Service is not possible, then I think your best bet is going to be setting up a separate thread with its own message pump, but it's a bit complicated. I've never done it on a web server, but you might be able to start an Application. This will provide a message pump for you and allow you to respond to Windows events, etc. The only problem is that this is going to start a Windows application (either WPF or WinForms), which may not be desirable on a web server.
What are you trying to accomplish? Is there another way you might go about it?
I found this nice article about how to use async worker, will give it a try. http://www.dotnetfunda.com/articles/article613-background-processes-in-asp-net-web-applications.aspx
And this:
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6
But I think for my needs this one will be perfect:
http://forums.asp.net/t/1433665.aspx/1
I currently have an application which is basically a wrapper for ~10 "LongRunning" Tasks. Each thread should keep running indefinitely, but sometimes they lock up or crash, and sometimes the wrapper app spontaneously exits (I haven't been able to track that down yet). Additionally, the wrapper application can currently only be running for one user, and that user has to be the one to restart the threads or relaunch the whole app.
I currently have a monitor utility to let me know when the threads stop doing work so that they can be manually restarted, but I'd like to automatically restart them instead. I'd also like the wrapper to be available to everyone to check the status of the threads, and for the threads to be running even when the wrapper isn't.
Based on these goals, I think I want to separate the threads into a Windows Service, and convert the wrapper into something which can just connect to the service to check its status and manipulate it.
How would I go about doing this? Is this a reasonable architecture? Should I turn each thread into a separate service, or should I have a single multi-threaded service?
Edit: All the tasks log to the same set of output files (via a TextWriter.Synchronized(StreamWriter)), and I would want to maintain that behavior.
They also all currently share the same database connection, which means I need to get them all to agree to close the connection at the same time when it's necessary. However, if they were split up they could each use their own database connection, and I wouldn't need to worry about synchronizing that. I actually suspect that this step is one of the current failure points, so splitting it up would be a Good Thing.
I would suggest you to stay inside one multithreading service if possible. Just make sure that threads are handled correctly when Service Stop is triggered. Put brake flags inside blocks of code that will take a lot of time to execute. This way you will make your service responsive on Stop event. Log any exceptions and make sure to wait for all threads to exit until service is finally stopped. This will prevent you to run same "task" in multiple threads.
Maintaining one service is in the end easier then multiple services.
Splitting to multiple services would be reasonable if you require some separate functionalities that can run or not beside each other.
I don't think moving the threads to a Windows Service removes any of the problems. The service will still crash randomly and the threads will still exit randomly.
I assume that your long-running tasks implement a kind of worker loop. Wrap the body of that loop in a try-catch and log all exceptions. Don't rethrow them so that the task does not ever exit. Examine the logs to find the bugs.
I am implementing a protocol library. Here a simplified description.
The main thread within the main function will always check, whether some data is available on the the networkstream (within a tcpclient). Let us say response is the received message and thread is a running thread.
thread = new Thread(new ThreadStart(function));
thread.IsBackground = true;
thread.Start();
while(true){
response = receiveMessage();
if (response != null)
{
thread.Suspend();
//I am searching for an alternative for the line above and not thread.Abort().
thread2 = new Thread(new ThreadStart(function2));
thread2.IsBackground = true;
thread2.Start();
}
}
So far so good, there are actually more messages to come within the while loop and there is also a statemachine for handling different sort of incoming messages, but this should be enough.
(There are also more than just the functions "function" and "function2").
So anyways how the functions look inside is not clear in this application, since the protocol is hidden from the programmer and meant to be a library. This means the protocol will start some programmer-defined functions as a thread depending on at what state in the protocol the program is.
So if then a special response is received (e.g. a callAnotherFunction message), I want to terminate
a thread (here named "thread") abruptly, lets say within 100 ms. But I do not know whether it executes within a loop or without and how much processing is needed until it terminates.
How to stop these threads without deprecated Suspend or Exceptionthrowing Abort function?
(Note that I cannot force the programmer of the functions to catch the ThreadAbortException.)
Or do I need a different programme architecture?
(Btw I have decided to put the loop within receiveMessage for polling the network stream into the main function, since anytime a message can appear).
Starting a thread without having a reliable way to terminate it is a bad practice. Suspend/Abort are one of those unreliable ways to terminate a thread because you may terminate a thread in a state that corrupts your entire program and you have no way to avoid it from happening.
You can see how to kill a thread safely here: Killing a .NET thread
If the "user" is giving you a method to run in a thread, then the user should also give you a method to stop the code from running. Think of it as a contract: you promise the user that you will call the stop method and they promise that the stop method will actually stop the thread. If your user violates that contract then they will be responsible for the issues that arise, which is good because you don't want to be responsible for your user's errors :).
Note that I cannot force the programmer of the functions to catch the ThreadAbortException.
Since Suspend/Abort are bad practice, the programmer doesn't need to catch the ThreadAbortException, however they should catch the ThreadInterruptedException as part of their "contract."
Remember that there are two situations you need to worry about:
The thread is executing some code.
The thread is in a blocking state.
In the case that the thread is executing some code, all you can do is notify the thread that it can exit and wait until it processes the notification. You may also skip the waiting and assume that you've leaked a resource, in which case it's the user's fault again because they didn't design their stop method to terminate their thread in a timely fashion.
In the case where the thread is in a blocking state and it's not blocking on a notification construct (i.e. semaphore, manual reset event, etc) then you should call Thread.Interrupt() to get it out of the blocking state- the user must handle the ThreadInterruptedException.
Suspend is really evil especially in a way you are trying to use it - to stop thread execution forever. It will leave all locks that thread had and also will not release resources.
Thread Abort is slightly better since it will at least try to terminate thread cleaner and locks will have chance to be released.
To properly do that you really need your thread's code to cooperate in termination. Events, semaphores or even simple bool value checked by the thread may be enough.
It may be better to re-architect your solution to have queue of messages and process them on separate thread. Special message may simply empty the queue.
You need some sort of cancellation protocol between your application and wherever function comes from. Then you can share some sort of cancellation token between function and your message loop. If message loop recognizes that function needs to be stopped you signal that by setting that token which must be tested by function on proper occasions. The simplest way would be to share a condition variable which can be atomically set from within your message loop and atomically read from function.
I'd however consider using the proper Asynchronous IO patterns combined with Tasks provided by the .NET framework out-of-the box along with proper cancellation mechanisms.
So function refers to code which you have little control over? This is pretty typical of 3rd party libraries. Most of the time they do not have builtin abilities to gracefully terminate long running operations. Since you have no idea how these functions are implemented you have very few options. In fact, your only guaranteed safe option is to spin these operations up in their own process and communicate with them via WCF. That way if you need to terminate the operation abruptly you would just kill the process. Killing another process will not corrupt the state of the current process like what would happen if you called Thread.Abort on thread within the current process.
I am having problems closing an application that uses WaitForSingleObject() with an INFINITE timout.
The full picture is this. I am doing the following to allow my application to handle the device wakeup event:
Register the event with:
CeRunAppAtEvent("\\\\.\\Notifications\\NamedEvents\\WakeupEvent",
NOTIFICATION_EVENT_WAKEUP);
Start a new thread to wait on:
Thread waitForWakeThread = new Thread(new ThreadStart(WaitForWakeup));
waitForWakeThread.Start();
Then do the following in the target method:
private void WaitForWakeup()
{
IntPtr handle = CreateEvent(IntPtr.Zero, 0, 0, "WakeupEvent");
while (true)
{
WaitForSingleObject(handle, INFINITE);
MessageBox.Show("Wakey wakey");
}
}
This all works fine until I try to close the application when, predictably, WaitForSingleObject continues to wait and does not allow the app to close properly. We only allow one instance of our app to run at a time and we check for this on startup. It appears to continue running until the device is soft reset.
Is there a way to kill the handle that WaitForSingleObject is waiting for, to force it to return?
Many thanks.
Use WaitForMultipleObject instead, and pass 2 handles. The existing one, and one for an event called something like 'exit'. During app shutdown, SetEvent on the exit event, and the WaitForMultipleObject will return and you can get it to exit the thread gracefully.
You need to switch on the return value of WaitForMultipleObject to do the appropriate behaviour depending on which one of the handles was triggered.
Possibly, also, you can set the thread to be a background thread. This will prevent it from stopping your application from shutting down when the main thread terminates.
See:
http://msdn.microsoft.com/en-us/library/system.threading.thread.isbackground.aspx
This is what I would do...
Use the EventWaitHandle class instead of calling CreateEvent directly. There shouldn't be any need to use the Windows API other than CeRunAppAtEvent (and API calls make code ugly...). Get this working first.
Before creating the thread, create a ManualResetEvent variable that is not initially flagged. Call it "TerminateEvent".
Replace the WaitForSingleObject API call with WaitHandle.WaitAny(WaitHandle[]) and pass an array containing "TerminateEvent" and the EventWaitHandle class wrapping the CeRunAppAtEvent notification.
Your loop can use the return value of WaitAny to determine what to do. The return value is the array index of the wait handle that unblocked the thread, so you can determine whether to continue the loop or not.
To cleanly end the thread, you can call "Set" on your "TerminateEvent" and then "Join" the thread to wait for it to terminate.
'This all works fine until I try to close the application when, predictably, WaitForSingleObject continues to wait and does not allow the app to close properly.'
Any app can close, no matter what its threads are doing. If you call ExitProcess(0) from any thread in your app, the app will close, no matter if there are threads waiting INFINITE on some API/sychro, sleeping, running on another processor, whatever. The OS will change the state of all theads that are not running to 'never run again' and use its interprocessor driver to hard-interrupt any other processors that are actually running your thread code. Once all the threads are stopped, the OS frees handles, segments etc and your app no longer exists.
Problems arise when developers try to 'cleanly' shut down threads that are stuck - like yours, when the app is closing. So..
Do you have a TThread.WaitFor, or similar, in an OnClose/OnCloseQuery handler, FormDestroy or destructor? If you have, and have no vital reason to ensure that the thread is terminated, just comment it out!
This allows the main form to close and so your code will finally reach the ExitProcess() it has been trying to get at since you clicked on the red cross button
You could, of coure, just call ExitProcess() yourself, but this may leave you with resources leaked in other proceses - database connections, for example.
'216/217 errors on close if I don't stop the threads'. This often happens because developers have followed the er... 'unfortunate' Delphi thread examples and communicate with threads by directly exchanging data between secondary thread fields and main thread fields, (eg. TThread.synchronize). This just sucks and is hell-bent on causing problems, even in the app run, never mind at shutdown when a form has been destroyed and a thread is trying to write to it or a thread has been destroyed and a main-thread form is trying ot call methods on it. It is much safer to communicate asynchronously with threads by means of queueing/PostMessaging objects that outlive both of them, eg. objects created in the thread/form and freed in the form/thread, or by means of a (thread-safe), pool of objects created in an initialization section. Forms can then close/free safely while associated threads may continue to pointlessly fill up objects for handling until the main form closes, ExitProcess() is reached and the OS annihilates the threads.
'My Form handle is invalid because it has closed but my thread tries to post a message to it'. If the PostMessage excepts, exit your thread. A better way is similar to the approach above - only post messages to a window that outlives all forms. Create one in an initialization section with a trivial WndProc that only handles one const message number that all threads use for posting. You can use wParam to pass the TwinControl instance that the thread is trying to communicate with, (usually a form variable), while lParam passes the object being communicated. When it gets a message from a thread, WndProc calls 'Peform' on the TwinControl passed and the TwinControl will get the comms object in a message-handler. A simple global boolean, 'AppClosing', say, can stop the WndProc calling Peform() on TwinControls that are freeing themselves during shutdown. This approach also avoids problems arising when the OS recreates your form window with a different handle - the Delphi form handle is not used and Windows will not recreate/change the handle of the simple form created in initialization.
I have followed these approaches for decades and do not get any shutdown problems, even with apps with dozens of threads slinging objects around on queues.
Rgds,
Martin
Of course the preferable way to solve this is to use WaitForMultipleObjects, or any other suitable function that is able to wait for multiple criterias (such as WaitForMultipleObjects, MsgWaitForMultipleObjects, etc.).
However if you have no control over which function is used - there're some tricky methods to solve this.
You may hack the functions imported from system DLL, by altering in memory the import table of any module. Since WaitForMultipleObjects is exported from kernel32.dll - it's ok.
using this technics you may redirect the function caller into your hands, and there you will be able to use the WaitForMultipleObjects.