We Keep Coding

Design: Task-Asynchronous Pattern (TAP with await / async), vs threads with signalling vs other thread structures

Help with ideas for redesign of the below C# program would be greatly appreciated. I am trying to pick between implementing multithreading using 1) TAP, 2) course-grained threads that contain spinners that terminate when their bools are set to false, or 3) the same threads using signalling instead of these bools. I will explain the program below, to make the case clear.
The Program
The program is a game automation application in C# that I am developing as a fun way to learn the language and C# (5.0) features better. It has a UI that needs to remain responsive while the app runs.
When a particular tab in the UI is opened, the program fires up a new thread called "Scan" that, in a new method in another class, scans various memory locations and updates labels in the UI with these quickly changing values using the UI's SynchronizationContext. This goes on in a while(scanning) loop, for as long as scanning bool is true (usually the full life-duration of the program).
When the user clicks the Start button on this tab, the program fires up two new threads that does the following: Thread "Run" moves the character around following a particular path. Thread "Action" hits particular buttons and performs actions at the same time as the player runs the path. If a certain scenario occurs, the program should stop the running thread and the action thread temporarily, run a method, and when it finishes, go back to the running and action'ing.
When the user clicks the Stop button on this tab, the automation should halt and threads terminate.
The Challenge
I have already created a working version using continuous spinner loops in each thread that takes care of the various work. The spinners run using a while(myBool). For the three threads the bools are: scanning, running and actioning.
When I want to stop a thread I set the bool to false, and use a Thread.Join to wait for the thread to terminate gracefully before proceeding. The threads can, as mentioned, be stopped by the user clicking the Stop button, or automatically by the program as part of its functionality. In the latter case a thread is stopped, Joined, and then at a later stage restarted.
After having done a lot of reading and research on threading and the new async programming tools in C# 5.0, I have realized that the way I am currently doing it might be very clumsy and unprofessional. It creates lots of synchronization/thread-safety issues, and as the goal of all of this is to learn more about C# I wanted to get your take on whether I should change it to a fine-grained asynchrounous programming approach instead, using TAP with async and await as appropriate.
Does this sound like a case where Tasks with cancellation tokens could be useful? The threads are after all long-running operations, so I was concerned that using the thread pool (Task.Run) would cause bad hygiene in the thread pool (over-subscription). If async programming seems like a bad match here, what about using threads as I have done, but instead use signalling to start and stop the threads?
Any thoughts greatly appreciated.

No. TPL was designed to run shorter tasks where the allocation of new threads all time would hurt perfomance. It got quite nice features like job queues and work stealing (a TPL thread can take jobs from another thread). It can of course have longer running task, but you wont get so many benefits from that. On the contrarary, you force TPL to allocate new threads.
However, the question is a bit general in the sense that we need more information about your actual implementation to know what you should use. For the Scan thread it's quite obvious that it should run in a single thread.
But for the others it's hard to know. Do they do work all the time or periodically? If they do work all the time you should keep them in seperate threads.
As for the thread syncronization there is another alternative. You could use a ConcurrentQueue to queue up everything that has to be drawn. In that way you do not need any synchronization. Just let the UI thread check the queue and draw anything in it, while the producers can continue to do their work.
In fact, in that way you can move anything not related to UI drawing to other threads. That should also improve the responsiveness in your application.
public void ActionRunnerThreadFunc()
{
_drawQueue.Enqueue(new SpaceShipRenderer(x, y));
}
public void UIThreadFunc()
{
IItemRender item;
if (_drawQueue.TryDequeue(out item))
item.Draw(drawContext);
}

How to see how much prossesing time a C# Windows Forms application needs?

I have a C# Windows Forms application wicht does some camera control and computer vision. For all the parts which take longer for calculation I used seperate threads. But there are still some parts which are in the callback functions of the GUI. As I understand, all these callback functions are executed in the same thread. Is there a way to see how much time this thread is working or idle? What percentage of idle time is needed such that the GUI is still responsive?

It's recommended that you shouldn't block the UI thread for more than 50ms, otherwise it will affect the UI responsiveness. I.e., two UI callbacks queued with Form.BeginInvoke, each taking ~50ms to complete, may introduce some unpleasant UI experience to the user.
It doesn't make sense to update the UI more often than the user can react to it (i.e, ~24 frames per second). So, you should throttle the UI thread callbacks and give user input events a priority.
I recently posted an example of how it can possibly be done:
https://stackoverflow.com/a/21654436/1768303

For simple tasks you could use a stopwatch and measure the time manually. However I think you'll need to check what a performance profiler is.
Also - there is little situations in which your GUI needs that heavy processing. In most cases the problem comes from putting too much calculations in event handlers instead of implementing them somewhere outside and then update the form when finished. It's less of a single/multi-threading problem and more of using available events properly.

System.Threading Sleep function help? (C#)

I'm new in C# and I'm using System.Threading.
I have this code:
UISystem.SetScene(Scene_Menu);
Thread.Sleep (9000);
p.Text="HELLO";
Thread.Sleep(9000);
p.Text="WORLD";
It delays 18 seconds, but the p.Text="HELLO" doesn't show between the sleep functions. What's the problem with my code?
Thanks.
Timers don't work since I can't edit p from a separate thread.
Application.DoEvents() is a Windows Forms function, I'm building an application in PS Vita.

You have discovered why you should never use Thread.Sleep. It is useful for only two things. (1) Writing test cases that need to simulate a thread being busy for a certain number of seconds, and (2) Sleeping for zero milliseconds tells the operating system "I cede the rest of my time slice to another process if there exists one that wants it"; it's a politeness thing.
You should never use thread.Sleep to introduce a delay as you are doing for exactly the reason you have discovered. You are setting a property, but setting a property does not cause the operating system to repaint the screen. Consider if it did; you might have a thousand property sets in a method, and you would have to repaint the screen after all of them, which would look ugly and be very slow.
Instead what happens is the property is set and the object makes a note to the operating system that says when this thread is available to handle operating system messages again, please repaint me. Your program is, instead of telling the operating system "I'm done, go ahead and see if there are any message for me" that instead you want the thread to do nothing for nine seconds.
Now, you can tell the program to check for messages by calling DoEvents but using DoEvents is also a bad idea and you should not do it. Doing so essentially causes your program to exhibit symptoms of Attention Deficit Disorder; you have not finished the current job and you are looking to see if there are new jobs to do without removing the old jobs from the call stack! Suppose those new jobs in turn get interrupted, and so on, and so on. The stack grows without bound, which is very bad. DoEvents is a "worst practice", just like sleeping a thread. You can get away with it in small simple programs but it leads to big trouble when the program becomes complex.
Moreover: yes, DoEvents will paint your control, but that is all it will do. For the next nine seconds, the application will appear to the user to be completely hung. That is a very bad user experience.
The right thing to do if you want to introduce a delay is to asynchronously wait. In C# 4 and earlier the standard way to do that is to create a timer, and when the timer ticks, do the next thing.
Now, you say that you cannot use a timer because you need to access the control from the UI thread. That's fine. The timer's tick event handler will run on the UI thread, not on a separate thread. You can safely use a timer.
In C# 5, the right thing to do is to use the new await keyword to introduce an asynchronous wait. That is, a wait that does other stuff while it is waiting, instead of going to sleep while it is waiting. In C# 5 you would write your code as:
UISystem.SetScene(Scene_Menu);
await Task.Delay (9000);
p.Text="HELLO";
await Task.Delay(9000);
p.Text="WORLD";
C# 5 is at present in beta; for details on this new feature see:
http://msdn.microsoft.com/en-us/async
For a gentle introduction to async and an explanation of why DoEvents is bad news, see my MSDN magazine article:
http://msdn.microsoft.com/en-us/magazine/hh456401.aspx

How can I run UI on a thread other than the Main UI thread in .net?

I know there are other questions around this but most end up with the answer don't do what I am about to suggest. So I know you aren't supposed to. The reason for this question is I want to do it anyway, how can I do it...
Here is why I want to break the rules...
Let's say I have a complicated application, it's version 1 and we want our customers to submit errors to us in the event of crashes or hangs. Let's now say I have a button on the top of the main form they click to submit reports.
Let's now imagine that the application hung because of a deadlock...
It would be nice if that small piece of UI and a handler for that button could live on a thread other than the main ui thread so that it isn't caught up in the deadlock. When clicked it would gather all the call stacks for the other threads and submit them to our error reporting service.
Now, knowing the scenario, can this be done in .net?

Yes, there is no magic in creating UI on another thread than the "main thread". The important rule to always keep in mind is to interact with that UI on the thread that created it.
Still, I feel that you are attacking this from the wrong angle. You should probably instead make an effort to push all work off the main thread. That way you minimize the risk for that thread to freeze, and then you don't need to resort to unorthodox solutions for the error reporting.

I have various cases of creating forms on non-main thread, and it works fine every time.
Create a new Thread, and show a Form from it. New message loop will be created for that thread and everything will run fine.
What magic will you use to gather data from the crashed app and locked main thread, that's up to you :)

if application hung, your main message loop is dead, thus ui will not work. As workaround for your problem i'd consider usage of external application (another exe) which will be invoked in case of report
in any case, if you want to invoke UI from other thread you should perform context switch In case of winforms, follow this answer

It sounds like you'd like to keep the UI alive, even when some other operation is mired in a deadlock. If so, perhaps Asynchronous Programming would be of use. Using Async to manage a potentially hung up task would allow the remainder of the application to remain responsive.
we want our customers to submit errors to us in the event of crashes or hangs
You might also consider adding some degree if instrumentation/reporting, so that you'll have this data without requiring user input.

Multithreading on Windows Forms

I want to paralelize a 3D voxel editor built on top of Windows Forms, it uses a raycaster to render so dividing the screen and getting each thread on a pool to render a part of it should be trivial.
The problem arises in that Windows Forms' thread must run as STA - I can get other threads to start and do the work but blocking the main thread while waiting for them to finish causes strange random deadlocks as expected.
Keeping the main thread unblocked would also be a problem - if, for example, the user uses a floodfill tool the input would be processed during the rendering process which would cause "in-between" images (an object partially colored, for example). Copying the entire image before every frame isn't doable either because the volumes are big enough to offset any performance gain if it has to be copied every frame.
I want to know if there is any workaround to get the amin thread to appear blocked to the user in a way that it will not be actually blocked but will delay the processing of input till the next frame.
If it isn't possible, is there a better design for dealing with this?
EDIT: Reading the anwsers I think I wasn't clear that the raycaster runs in real time, so showing progress dialogs won't work at all. Unfortunately the FPS is low enough (5-40 depending on various factors) for the input between frames to produce unwanted results.
I have already tried to implement it blocking the UI thread and using some threads of a ThreadPool to process and it works fine except for this problem with STA.

This is a common problem. With windows forms you can have only one UI thread. Don't run your algorithm on the UI thread because then the UI will appear frozen.
I recommend running your algorithm and waiting for it to finish before updating the UI. A class called BackgroundWorker comes pre-built to do just this very thing.
Edit:
Another fact about the UI thread is that it handles all of the mouse and keyboard events, along with system messages that are sent to the window. (Winforms is really just Win32 surrounded by a nice API.) You cannot have a stable application if the UI thread is saturated.
On the other hand, if you start several other threads and try to draw directly on the screen with them, you may have two problems:
You're not supposed to draw on the UI with any thread but the UI thread. Windows controls are not thread safe.
If you have a lot of threads, context switching between them may kill your performance.
Note that you (and I) shouldn't claim a performance problem until it has been measured. You could try drawing a frame in memory and swapping it in at an appropriate time. Its called double-buffering and is very common in Win32 drawing code to avoid screen flicker.
I honestly don't know if this is feasible with your target frame rate, or if you should consider a more graphics-centered library like OpenGL.

Am I missing something or can you just set your render control (and any other controls that generate input events) to disabled while you're rendering a frame? That will prevent unwanted inputs.
If you still want to accept events while you're rendering but don't want to apply them until the next frame, you should leave your controls enabled and post the detail of the event to an input queue. That queue should then be processed at the start of every frame.
This has the affect that the user can still click buttons and interact with the UI (the GUI thread does not block) and those events are not visible to the renderer until the start of the next frame. At 5 FPS, the user should see their events are processed within 400ms worst case (2 frames), which isn't quite fast enough, but better than threading deadlocks.
Perhaps something like this:
Public InputQueue<InputEvent> = new Queue<InputEvent>();
// An input event handler.
private void btnDoSomething_Click(object sender, EventArgs e)
{
lock(InputQueue)
{
InputQueue.Enqueue(new DoSomethingInputEvent());
}
}
// Your render method (executing in a background thread).
private void RenderNextFrame()
{
Queue<InputEvent> inputEvents = new Queue<InputEvent>();
lock(InputQueue)
{
inputEvents.Enqueue(InputQueue.Dequeue());
}
// Process your input events from the local inputEvents queue.
....
// Now do your render based on those events.
....
}
Oh, and do your rendering on a background thread. Your UI thread is precious, it should only do the most trivial work. Matt Brundell's suggestion of BackgroundWorker has lots of merit. If it doesn't do what you want, the ThreadPool is also useful (and simpler). More powerful (and complex) alternatives are the CCR or the Task Parallel Library.

Show a modal "Please Wait" dialog using ShowDialog, then close it once your rendering is finished.
This will prevent the user from interacting with the form while still allowing you to Invoke to the UI thread (which is presumably your problem).

If you don't want all the features offered by the BackgroundWorker you can simply use the ThreadPool.QueueUserWorkItem to add something to the thread pool and use a background thread. It would be easy to show some kind of progress while the background thread was performing it's operations as you can provide a delegate callback to notify you whenever a particular background thread is done. Take a look at ThreadPool.QueueUserWorkItem Method (WaitCallback, Object) to see what I'm referring you to. If you need something more complex you could always use the APM async method to perform your operations as well.
Either way I hope this helps.
EDIT:
Notify user somehow that changes are being made to the UI.
On a(many) background threads using the ThreadPool perform the ops you need to perform to the UI.
For each operation keep a reference to the state for the operation so that you know when it completed in the WaitCallback. Maybe put them in some type of hash / collection to keep ref to them.
Whenever an operation completes remove it from the collection that contains a ref to the ops that were performed.
Once all operations have completed (hash / collection) has no more references in it render the UI with the changes applied. Or possibly incrementally update the UI
I'm thinking that if you are making so many updates to the UI while you are performing your operations that is what is causing your problems. That's also why I recommended the use of SuspendLayout, PerformLayout as you may have been performing so many updates to the UI the main thread was getting overwhelmed.
I am no expert on threading though, just trying to think it through myself. Hope this helps.

Copying the entire image before every frame isn't doable either because the volumes are big enough to offset any performance gain if it has to be copied every frame.
Then don't copy the off-screen buffer on every frame.