We Keep Coding

Using threadpool with thread completion updates inside a lock

Following several tutorials on threaded app development, I've implemented a small practice utility that uses both Task.Run, and Threadpool.QueueUserWorkItem to recursively index a folder and generate a checksum manifest of all contained files.
While the application works well, I'm unable to get the UI to update during the portion that specifically uses a threadpool. The UI does update correctly using await Application.Current.Dispatcher.BeginInvoke when using Task.Run.
The desired behavior is for the progress to continue to update within ProcessChecksums as each of the threadpool's threads finish their individual task.
int queuedThreads = 0;
object locker = new object();
CancellationTokenSource cancellationTokenSource;
List<string> lstFilePaths = new List<string>();
Manifest manifestData;
event Action<double> OnProgressChange;
event Action<string> OnStatusChange;
async void GenerateManifest(Object sender, RoutedEventArgs e) {
status = "Indexing Files";
progress = 1;
cancellationTokenSource = new CancellationTokenSource();
await Task.Run(() => Async_GenerateManifest(cancellationTokenSource.Token), cancellationTokenSource.Token);
if (!cancellationTokenSource.Token.IsCancellationRequested) {
Finished();
}else{
Cancelled();
}
}
async Task Async_GenerateManifest(CancellationToken cancellationToken) {
if (cancellationToken.IsCancellationRequested) { return; }
Async_IndexFiles(initialpath, cancellationToken);
//Works Perfectly
await Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Background, new Action(() => {
OnStatusChange("Generating Checksums");
OnProgressChange(10);
}));
ProcessChecksums(cancellationToken);
//Works Perfectly
await Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Background, new Action(() => {
OnStatusChange("Saving Manifest");
OnProgressChange(90);
}));
SaveManifestFile(cancellationToken);
}
void ProcessChecksums(CancellationToken cancellationToken) {
List<FileData> lstFileData = new List<FileData>();
for (int i = 0; i < lstFilePaths.Count; i++) {
if (cancellationToken.IsCancellationRequested) { return; }
string s = lstFilePaths[i];
lock (locker) queuedThreads++;
ThreadPool.QueueUserWorkItem( x => {
manifestData.AddFileData(new FileData(s, GenerateChecksum(s)));
});
}
lock (locker) {
while (queuedThreads > 0) {
if (cancellationToken.IsCancellationRequested) { return; }=
Monitor.Wait(locker);
//Can't use await Dispatcher.BeginInvoke as is inside a lock, doesn't update GUI while waiting.
OnProgressChange((((queuedThreads - lstFilePaths.Count) * -1) / lstFilePaths.Count) - 0.2);
}
}
}
string GenerateChecksum(string filePath) {
//Time-consuming checksum generation
//...
lock (locker) {
queuedThreads--;
Monitor.Pulse(locker);
}
return BitConverter.ToString(checksum);
}

The standard pattern for updating the UI with progress updates from a background thread is to use IProgress<T>/Progress<T>. The more modern approach has several benefits over direct use of Dispatcher.
// You'd want to set these while on the UI thread.
// E.g., in your constructor:
// _percentProgress = new Progress<double>(value => ...);
// _statusProgress = new Progress<string>(value => ...);
IProgress<double> _percentProgress;
IProgress<string> _statusProgress;
async void GenerateManifest(Object sender, RoutedEventArgs e) {
status = "Indexing Files";
progress = 1;
cancellationTokenSource = new CancellationTokenSource();
await Task.Run(() => GenerateManifest(cancellationTokenSource.Token));
if (!cancellationTokenSource.Token.IsCancellationRequested) {
Finished();
}else{
Cancelled();
}
}
void GenerateManifest(CancellationToken cancellationToken) {
if (cancellationToken.IsCancellationRequested) { return; }
Async_IndexFiles(initialpath, cancellationToken);
_statusProgress?.Report("Generating Checksums");
_percentProgress?.Report(10);
ProcessChecksums(cancellationToken);
_statusProgress?.Report("Saving Manifest");
_percentProgress?.Report(90);
SaveManifestFile(cancellationToken);
}
void ProcessChecksums(CancellationToken cancellationToken) {
List<FileData> lstFileData = new List<FileData>();
for (int i = 0; i < lstFilePaths.Count; i++) {
if (cancellationToken.IsCancellationRequested) { return; }
string s = lstFilePaths[i];
lock (locker) queuedThreads++;
ThreadPool.QueueUserWorkItem( x => {
manifestData.AddFileData(new FileData(s, GenerateChecksum(s)));
});
}
lock (locker) {
while (queuedThreads > 0) {
if (cancellationToken.IsCancellationRequested) { return; }
Monitor.Wait(locker);
_percentProgress?.Report((((queuedThreads - lstFilePaths.Count) * -1) / lstFilePaths.Count) - 0.2);
}
}
}
There are a few other problems with the code, too. QueueUserWorkItem should probably be replaced with Task.Run and an await Task.WhenAll to ensure exceptions don't tear down the process. The if (...) Finished(); else Cancelled(); is probably best represented by changing the return type to Task. Use ThrowIfCancellationRequested instead of IsCancellationRequested. And there's a kind of Monitor.Wait without a Pulse that is just used for progress updates, which is odd; it would be cleaner to allow that code to report its own progress. If you take a look at each of these individually, you should find your code ends up cleaner.

WPF Thread Delay

I have a method which I call in a new task with
// get the dispatcher for the UI thread
var uiDispatcher = Dispatcher.CurrentDispatcher;
Task.Factory.StartNew(() => BackgroundThreadProc(uiDispatcher));
In the method BackgroundThreadProc() I need a delay of few seconds. I tried it with the DispatcherTimer and the task.delay function but it didn't work. The only thing which worked was the System.Threading.Thread.Sleep(1) but I think the Thread.Sleep() function isn't the best solution.
This is my function:
public void BackgroundThreadProc(Dispatcher uiDispatcher)
{
for (var i = 0; i < 100; i++)
{
var task = Task.Delay(1000).ContinueWith(t =>
{
// create object
var animal = new Animal { Name = "test" + i };
uiDispatcher.Invoke(new Action(() => log(animal)));
});
}
}
As I found out it didn't work because the DispatcherTimer is running in the UI thread. How I can accomplish the delay in the function which is in a other thread than the UI thread?
Update:
Now I tried it with the timer:
public void BackgroundThreadProc(Dispatcher uiDispatcher)
{
for (var i = 0; i < 100; i++)
{
var _delayTimer = new System.Timers.Timer();
_delayTimer.Interval = 1000;
//_delayTimer.Enabled = true;
_delayTimer.Elapsed += delegate
{
var animal = new Animal { Name = "test" + i };
uiDispatcher.Invoke(new Action(() => log(animal)));
_delayTimer.Stop();
};
_delayTimer.Start();
}
}

Use Task.Delay to introduce a delay asynchrnoously:
var task = Task.Delay(1000)
.ContinueWith(t => BackgroundThreadProc());

Are you limited to C# 4.0? I assume you're not, because Task.Delay wouldn't be available.
So, make BackgroundThreadProc an async method and use await inside it:
// get the dispatcher for the UI thread
var uiDispatcher = Dispatcher.CurrentDispatcher;
var task = BackgroundThreadProc(uiDispatcher));
// ...
public async Task BackgroundThreadProc(Dispatcher uiDispatcher)
{
for (var i = 0; i < 100; i++)
{
await Task.Delay(1000).ConfigureAwait(false);
// create object
var animal = new Animal { Name = "test" + i };
uiDispatcher.Invoke(new Action(() => log(animal)));
}
}
You really don't need Task.Factory.StartNew here, the execution will continue on thread pool after await Task.Delay.
Apparently, you're only updating the UI from this BackgroundThreadProc. If that's the case, just remove ConfigureAwait(false) and don't use uiDispatcher.Invoke:
public async Task BackgroundThreadProc()
{
for (var i = 0; i < 100; i++)
{
await Task.Delay(1000);
// create object
var animal = new Animal { Name = "test" + i };
log(animal);
}
}
This loop will be executing asynchronously on the WPF UI thread.
Otherwise, if you do have any other CPU-bound work before Task.Delay, then you may need Task.Factory.StartNew to avoid freezing the UI (note Unwrap):
var task = Task.Factory.StartNew(() =>
BackgroundThreadProc(uiDispatcher)).Unwrap();
You can also use Task.Run, which unwraps the inner task automatically:
var task = Task.Run(() => BackgroundThreadProc(uiDispatcher));

Recursive Task Queue

I'm trying to iterate through a queue - taking 1 item from the queue, processing it in a background task, updating the UI, and then taking the next item, and so on. The problem is the first item is processed in a background task (thread) but then the subsequent items are processed in the UI thread - blocking the UI.
Does anyone know why this happens and how to get around this problem? My full test code is below. Note: this code is for my learning and future reference - not any real application.
public partial class MainWindow : Window
{
private Queue<int> testQueue = new Queue<int>();
private TaskScheduler uiScheduler;
public MainWindow()
{
InitializeComponent();
this.uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();
this.testQueue = new Queue<int>();
this.testQueue.Enqueue(3);
this.testQueue.Enqueue(6);
this.testQueue.Enqueue(7);
this.testQueue.Enqueue(11);
this.testQueue.Enqueue(13);
}
// just a method that takes about 1 second to run on a modern pc
private double SumRootN(int root)
{
double result = 0;
for (int i = 1; i < 10000000; i++)
{
result += Math.Exp(Math.Log(i) / root);
}
return result;
}
private void testQueueButton_Click(object sender, RoutedEventArgs e)
{
this.processQueue();
}
private void processQueue()
{
if (this.testQueue.Count > 0)
{
int root = this.testQueue.Dequeue();
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
this.processQueue();
}, uiScheduler);
}
else
{
this.statusText.Text += "Done\n";
}
}
}

Thank you for posting a repro which allowed me to debug.
Task.Factory.StartNew runs your task on the scheduler (factoryScheduler ?? currentTaskScheduler ?? threadPoolScheduler). You got into case 2: Your new task inherits the scheduler from its parent.
I noticed your curious use of recursive calls in order to simulate a loop. If you do it like this, the problem goes away:
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
}, uiScheduler).ContinueWith(t => { this.processQueue(); });

That's because you are using TaskScheduler.FromCurrentSynchronizationContext() - you do know what it does right? (makes it run on the same thread it is called, in your case the UI)
EDIT: usr answered you why is that happening, but you could also do this (for a quasi parallel processing):
int root = this.testQueue.Dequeue();
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
}, uiScheduler);
this.processQueue();

Create multiple threads and wait for all of them to complete

How can I create multiple threads and wait for all of them to complete?

It depends which version of the .NET Framework you are using. .NET 4.0 made thread management a whole lot easier using Tasks:
class Program
{
static void Main(string[] args)
{
Task task1 = Task.Factory.StartNew(() => doStuff());
Task task2 = Task.Factory.StartNew(() => doStuff());
Task task3 = Task.Factory.StartNew(() => doStuff());
Task.WaitAll(task1, task2, task3);
Console.WriteLine("All threads complete");
}
static void doStuff()
{
//do stuff here
}
}
In previous versions of .NET you could use the BackgroundWorker object, use ThreadPool.QueueUserWorkItem(), or create your threads manually and use Thread.Join() to wait for them to complete:
static void Main(string[] args)
{
Thread t1 = new Thread(doStuff);
t1.Start();
Thread t2 = new Thread(doStuff);
t2.Start();
Thread t3 = new Thread(doStuff);
t3.Start();
t1.Join();
t2.Join();
t3.Join();
Console.WriteLine("All threads complete");
}

I think you need WaitHandler.WaitAll. Here is an example:
public static void Main(string[] args)
{
int numOfThreads = 10;
WaitHandle[] waitHandles = new WaitHandle[numOfThreads];
for (int i = 0; i < numOfThreads; i++)
{
var j = i;
// Or you can use AutoResetEvent/ManualResetEvent
var handle = new EventWaitHandle(false, EventResetMode.ManualReset);
var thread = new Thread(() =>
{
Thread.Sleep(j * 1000);
Console.WriteLine("Thread{0} exits", j);
handle.Set();
});
waitHandles[j] = handle;
thread.Start();
}
WaitHandle.WaitAll(waitHandles);
Console.WriteLine("Main thread exits");
Console.Read();
}
FCL has a few more convenient functions.
(1) Task.WaitAll, as well as its overloads, when you want to do some tasks in parallel (and with no return values).
var tasks = new[]
{
Task.Factory.StartNew(() => DoSomething1()),
Task.Factory.StartNew(() => DoSomething2()),
Task.Factory.StartNew(() => DoSomething3())
};
Task.WaitAll(tasks);
(2) Task.WhenAll when you want to do some tasks with return values. It performs the operations and puts the results in an array. It's thread-safe, and you don't need to using a thread-safe container and implement the add operation yourself.
var tasks = new[]
{
Task.Factory.StartNew(() => GetSomething1()),
Task.Factory.StartNew(() => GetSomething2()),
Task.Factory.StartNew(() => GetSomething3())
};
var things = Task.WhenAll(tasks);

I've made a very simple extension method to wait for all threads of a collection:
using System.Collections.Generic;
using System.Threading;
namespace Extensions {
public static class ThreadExtension {
public static void WaitAll (this IEnumerable<Thread> threads) {
if (threads != null) {
foreach (Thread thread in threads) {
thread.Join();
}
}
}
}
}
Then you simply call:
List<Thread> threads = new List<Thread>();
// Add your threads to this collection
threads.WaitAll();

In .NET 4.0, you can use the Task Parallel Library.
In earlier versions, you can create a list of Thread objects in a loop, calling Start on each one, and then make another loop and call Join on each one.

If you don't want to use the Task class (for instance, in .NET 3.5), you can just start all your threads, and then add them to the list and join them in a foreach loop.
Example:
List<Thread> threads = new List<Thread>();
// Start threads
for (int i = 0; i < 10; i++) {
int tmp = i; // Copy value for closure
Thread t = new Thread(() => Console.WriteLine(tmp));
t.Start();
threads.Add(t);
}
// Join threads (wait threads)
foreach (Thread thread in threads) {
thread.Join();
}

I don't know if there is a better way, but the following describes how I did it with a counter and background worker thread.
private object _lock = new object();
private int _runningThreads = 0;
private int Counter{
get{
lock(_lock)
return _runningThreads;
}
set{
lock(_lock)
_runningThreads = value;
}
}
Now whenever you create a worker thread, increment the counter:
var t = new BackgroundWorker();
// Add RunWorkerCompleted handler
// Start thread
Counter++;
In work completed, decrement the counter:
private void RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
Counter--;
}
Now you can check for the counter anytime to see if any thread is running:
if(Couonter>0){
// Some thread is yet to finish.
}

Most proposed answers don't take into account a time-out interval, which is very important to prevent a possible deadlock. Next is my sample code. (Note that I'm primarily a Win32 developer, and that's how I'd do it there.)
//'arrRunningThreads' = List<Thread>
//Wait for all threads
const int knmsMaxWait = 3 * 1000; //3 sec timeout
int nmsBeginTicks = Environment.TickCount;
foreach(Thread thrd in arrRunningThreads)
{
//See time left
int nmsElapsed = Environment.TickCount - nmsBeginTicks;
int nmsRemain = knmsMaxWait - nmsElapsed;
if(nmsRemain < 0)
nmsRemain = 0;
//Then wait for thread to exit
if(!thrd.Join(nmsRemain))
{
//It didn't exit in time, terminate it
thrd.Abort();
//Issue a debugger warning
Debug.Assert(false, "Terminated thread");
}
}

In my case, I could not instantiate my objects on the the thread pool with Task.Run() or Task.Factory.StartNew(). They would not synchronize my long running delegates correctly.
I needed the delegates to run asynchronously, pausing my main thread for their collective completion. The Thread.Join() would not work since I wanted to wait for collective completion in the middle of the parent thread, not at the end.
With the Task.Run() or Task.Factory.StartNew(), either all the child threads blocked each other or the parent thread would not be blocked, ... I couldn't figure out how to go with async delegates because of the re-serialization of the await syntax.
Here is my solution using Threads instead of Tasks:
using (EventWaitHandle wh = new EventWaitHandle(false, EventResetMode.ManualReset))
{
int outdex = mediaServerMinConnections - 1;
for (int i = 0; i < mediaServerMinConnections; i++)
{
new Thread(() =>
{
sshPool.Enqueue(new SshHandler());
if (Interlocked.Decrement(ref outdex) < 1)
wh.Set();
}).Start();
}
wh.WaitOne();
}

Problem with Invoke to parallelize foreach

I have a problem with using System.Threading.Tasks.Parallel.ForEach. The body foreach progressBar want to update.
But Invoke method sometimes freeze.
I attach the code to the form which is prograssbar and Buton.
private void button1_Click(object sender, EventArgs e)
{
DateTime start = DateTime.Now;
pforeach();
Text = (DateTime.Now - start).ToString();
}
private void pforeach()
{
int[] intArray = new int[60];
int totalcount = intArray.Length;
object lck = new object();
System.Threading.Tasks.Parallel.ForEach<int, int>(intArray,
() => 0,
(x, loop, count) =>
{
int value = 0;
System.Threading.Thread.Sleep(100);
count++;
value = (int)(100f / (float)totalcount * (float)count);
Set(value);
return count;
},
(x) =>
{
});
}
private void Set(int i)
{
if (this.InvokeRequired)
{
var result = Invoke(new Action<int>(Set), i);
}
else
progressBar1.Value = i;
}
Sometimes it passes without a problem, but usually it freeze on
var result = Invoke (new Action <int> (Set), i).
Try to kick me in the problem.
Thank you.

Your problem is that Invoke (and queueing a Task to the UI TaskScheduler) both require the UI thread to be processing its message loop. However, it is not. It is still waiting for the Parallel.ForEach loop to complete. This is why you see a deadlock.
If you want the Parallel.ForEach to run without blocking the UI thread, wrap it into a Task, as such:
private TaskScheduler ui;
private void button1_Click(object sender, EventArgs e)
{
ui = TaskScheduler.FromCurrentSynchronizationContext();
DateTime start = DateTime.Now;
Task.Factory.StartNew(pforeach)
.ContinueWith(task =>
{
task.Wait(); // Ensure errors are propogated to the UI thread.
Text = (DateTime.Now - start).ToString();
}, ui);
}
private void pforeach()
{
int[] intArray = new int[60];
int totalcount = intArray.Length;
object lck = new object();
System.Threading.Tasks.Parallel.ForEach<int, int>(intArray,
() => 0,
(x, loop, count) =>
{
int value = 0;
System.Threading.Thread.Sleep(100);
count++;
value = (int)(100f / (float)totalcount * (float)count);
Task.Factory.StartNew(
() => Set(value),
CancellationToken.None,
TaskCreationOptions.None,
ui).Wait();
return count;
},
(x) =>
{
});
}
private void Set(int i)
{
progressBar1.Value = i;
}

I was looking at how I did this and this change may help you:
In my constructor I have this line:
TaskScheduler uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();
Then I do this:
private void changeProgressBar()
{
(new Task(() =>
{
mainProgressBar.Value++;
mainProgressTextField.Text = mainProgressBar.Value + " of " + mainProgressBar.Maximum;
})).Start(uiScheduler);
}
This gets rid of needing to use Invoke, and if you use the Task method then it may solve your problem.
I think these were all in System.Threading.Tasks;