We Keep Coding

Using threadpool with thread completion updates inside a lock

Following several tutorials on threaded app development, I've implemented a small practice utility that uses both Task.Run, and Threadpool.QueueUserWorkItem to recursively index a folder and generate a checksum manifest of all contained files.
While the application works well, I'm unable to get the UI to update during the portion that specifically uses a threadpool. The UI does update correctly using await Application.Current.Dispatcher.BeginInvoke when using Task.Run.
The desired behavior is for the progress to continue to update within ProcessChecksums as each of the threadpool's threads finish their individual task.
int queuedThreads = 0;
object locker = new object();
CancellationTokenSource cancellationTokenSource;
List<string> lstFilePaths = new List<string>();
Manifest manifestData;
event Action<double> OnProgressChange;
event Action<string> OnStatusChange;
async void GenerateManifest(Object sender, RoutedEventArgs e) {
status = "Indexing Files";
progress = 1;
cancellationTokenSource = new CancellationTokenSource();
await Task.Run(() => Async_GenerateManifest(cancellationTokenSource.Token), cancellationTokenSource.Token);
if (!cancellationTokenSource.Token.IsCancellationRequested) {
Finished();
}else{
Cancelled();
}
}
async Task Async_GenerateManifest(CancellationToken cancellationToken) {
if (cancellationToken.IsCancellationRequested) { return; }
Async_IndexFiles(initialpath, cancellationToken);
//Works Perfectly
await Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Background, new Action(() => {
OnStatusChange("Generating Checksums");
OnProgressChange(10);
}));
ProcessChecksums(cancellationToken);
//Works Perfectly
await Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Background, new Action(() => {
OnStatusChange("Saving Manifest");
OnProgressChange(90);
}));
SaveManifestFile(cancellationToken);
}
void ProcessChecksums(CancellationToken cancellationToken) {
List<FileData> lstFileData = new List<FileData>();
for (int i = 0; i < lstFilePaths.Count; i++) {
if (cancellationToken.IsCancellationRequested) { return; }
string s = lstFilePaths[i];
lock (locker) queuedThreads++;
ThreadPool.QueueUserWorkItem( x => {
manifestData.AddFileData(new FileData(s, GenerateChecksum(s)));
});
}
lock (locker) {
while (queuedThreads > 0) {
if (cancellationToken.IsCancellationRequested) { return; }=
Monitor.Wait(locker);
//Can't use await Dispatcher.BeginInvoke as is inside a lock, doesn't update GUI while waiting.
OnProgressChange((((queuedThreads - lstFilePaths.Count) * -1) / lstFilePaths.Count) - 0.2);
}
}
}
string GenerateChecksum(string filePath) {
//Time-consuming checksum generation
//...
lock (locker) {
queuedThreads--;
Monitor.Pulse(locker);
}
return BitConverter.ToString(checksum);
}

The standard pattern for updating the UI with progress updates from a background thread is to use IProgress<T>/Progress<T>. The more modern approach has several benefits over direct use of Dispatcher.
// You'd want to set these while on the UI thread.
// E.g., in your constructor:
// _percentProgress = new Progress<double>(value => ...);
// _statusProgress = new Progress<string>(value => ...);
IProgress<double> _percentProgress;
IProgress<string> _statusProgress;
async void GenerateManifest(Object sender, RoutedEventArgs e) {
status = "Indexing Files";
progress = 1;
cancellationTokenSource = new CancellationTokenSource();
await Task.Run(() => GenerateManifest(cancellationTokenSource.Token));
if (!cancellationTokenSource.Token.IsCancellationRequested) {
Finished();
}else{
Cancelled();
}
}
void GenerateManifest(CancellationToken cancellationToken) {
if (cancellationToken.IsCancellationRequested) { return; }
Async_IndexFiles(initialpath, cancellationToken);
_statusProgress?.Report("Generating Checksums");
_percentProgress?.Report(10);
ProcessChecksums(cancellationToken);
_statusProgress?.Report("Saving Manifest");
_percentProgress?.Report(90);
SaveManifestFile(cancellationToken);
}
void ProcessChecksums(CancellationToken cancellationToken) {
List<FileData> lstFileData = new List<FileData>();
for (int i = 0; i < lstFilePaths.Count; i++) {
if (cancellationToken.IsCancellationRequested) { return; }
string s = lstFilePaths[i];
lock (locker) queuedThreads++;
ThreadPool.QueueUserWorkItem( x => {
manifestData.AddFileData(new FileData(s, GenerateChecksum(s)));
});
}
lock (locker) {
while (queuedThreads > 0) {
if (cancellationToken.IsCancellationRequested) { return; }
Monitor.Wait(locker);
_percentProgress?.Report((((queuedThreads - lstFilePaths.Count) * -1) / lstFilePaths.Count) - 0.2);
}
}
}
There are a few other problems with the code, too. QueueUserWorkItem should probably be replaced with Task.Run and an await Task.WhenAll to ensure exceptions don't tear down the process. The if (...) Finished(); else Cancelled(); is probably best represented by changing the return type to Task. Use ThrowIfCancellationRequested instead of IsCancellationRequested. And there's a kind of Monitor.Wait without a Pulse that is just used for progress updates, which is odd; it would be cleaner to allow that code to report its own progress. If you take a look at each of these individually, you should find your code ends up cleaner.

How can I get the result from WhenAll(Array of tasks)

I have the following code.
var tasks = new[]
{
Task.Factory.StartNew(() => GetSomething1()),
Task.Factory.StartNew(() => GetSomething2()),
Task.Factory.StartNew(() => GetSomething3())
};
var things = Task.WhenAll(tasks);
How can I get the results from all three tasks separately and print them?

In order to get the results seperately you have multiple ways I would do something like this:
var task1 = GetSomething1();
var task2 = GetSomething2();
var task3 = GetSomething3();
// your method will continue when everything's completed, but you won't tie up a thread to just hang around until that time.
await Task.WhenAll(task1, task2, task3);
var result1 = await task1;
var result2 = await task2;
var result3 = await task3;

One way is to make each Task responsible for storing its own result, and then all you have to do is await the collection of Tasks. Note that you'll have to use await to make the WhenAll() execute the tasks that you pass into it.
var results = new int[3];
var tasks = new[] {
Task.Factory.StartNew(() => results[0] = GetSomething1()),
Task.Factory.StartNew(() => results[1] = GetSomething2()),
Task.Factory.StartNew(() => results[2] = GetSomething3())
};
await Task.WhenAll(tasks);
Console.WriteLine(results[0]);
Console.WriteLine(results[1]);
Console.WriteLine(results[2]);
Working demo: https://dotnetfiddle.net/HS32QA
Note that you may need to be careful with using e.g. a List<T> instead of an array, and then calling list.Add(result), as there is no guarantee of the order in which the tasks are executed or when they will be finished.

You should use async..await pattern with When, e.g.
private async Task MyExecution() {
var tasks = new[] {
//TODO: Task.Run is a better option than Task.Factory.StartNew
Task.Factory.StartNew(() => GetSomething1()),
Task.Factory.StartNew(() => GetSomething2()),
Task.Factory.StartNew(() => GetSomething3())
};
// Await until all tasks are completed
await Task.WhenAll(tasks);
// Since all tasks are completed, we can (safely) query for their `Result`s:
var things = tasks
.Select(task => task.Result) // or task => await task
.ToArray();
// Let's print the things
for (int i = 0; i < things.Length; ++i)
Console.WriteLine($"Task #{i + 1} returned {things[i]}");
...
}

Have you considered actually using async functions? Then you have an array of tasks with the results, and avoid the highly unpredictable behaviours of Task.Factory.StartNew.
private async Task MyExecution()
{
var tasks = new[] {
GetSomething1(),
GetSomething2(),
GetSomething3()
};
// Await until all tasks are completed
await Task.WhenAll(tasks);
foreach(var t in tasks){
//t.Result is available
}
}
public static async Task<int> GetSomething1() { return 1; }
public static async Task<int> GetSomething2() { return 2; }
public static async Task<int> GetSomething3() { return 3; }
Fiddle here: https://dotnetfiddle.net/3ffs9L

TaskFactory, Starting a new Task when one ends

I have found many methods of using the TaskFactory but I could not find anything about starting more tasks and watching when one ends and starting another one.
I always want to have 10 tasks working.
I want something like this
int nTotalTasks=10;
int nCurrentTask=0;
Task<bool>[] tasks=new Task<bool>[nThreadsNum];
for (int i=0; i<1000; i++)
{
string param1="test";
string param2="test";
if (nCurrentTask<10) // if there are less than 10 tasks then start another one
tasks[nCurrentThread++] = Task.Factory.StartNew<bool>(() =>
{
MyClass cls = new MyClass();
bool bRet = cls.Method1(param1, param2, i); // takes up to 2 minutes to finish
return bRet;
});
// How can I stop the for loop until a new task is finished and start a new one?
}

Check out the Task.WaitAny method:
Waits for any of the provided Task objects to complete execution.
Example from the documentation:
var t1 = Task.Factory.StartNew(() => DoOperation1());
var t2 = Task.Factory.StartNew(() => DoOperation2());
Task.WaitAny(t1, t2)

I would use a combination of Microsoft's Reactive Framework (NuGet "Rx-Main") and TPL for this. It becomes very simple.
Here's the code:
int nTotalTasks=10;
string param1="test";
string param2="test";
IDisposable subscription =
Observable
.Range(0, 1000)
.Select(i => Observable.FromAsync(() => Task.Factory.StartNew<bool>(() =>
{
MyClass cls = new MyClass();
bool bRet = cls.Method1(param1, param2, i); // takes up to 2 minutes to finish
return bRet;
})))
.Merge(nTotalTasks)
.ToArray()
.Subscribe((bool[] results) =>
{
/* Do something with the results. */
});
The key part here is the .Merge(nTotalTasks) which limits the number of concurrent tasks.
If you need to stop the processing part way thru just call subscription.Dispose() and everything gets cleaned up for you.
If you want to process each result as they are produced you can change the code from the .Merge(...) like this:
.Merge(nTotalTasks)
.Subscribe((bool result) =>
{
/* Do something with each result. */
});

This should be all you need, not complete, but all you need to do is wait on the first to complete and then run the second.
Task.WaitAny(task to wait on);
Task.Factory.StartNew()

Have you seen the BlockingCollection class? It allows you to have multiple threads running in parallel and you can wait from results from one task to execute another. See more information here.

The answer depends on whether the tasks to be scheduled are CPU or I/O bound.
For CPU-intensive work I would use Parallel.For() API setting the number of thread/tasks through MaxDegreeOfParallelism property of ParallelOptions
For I/O bound work the number of concurrently executing tasks can be significantly larger than the number of available CPUs, so the strategy is to rely on async methods as much as possible, which reduces the total number of threads waiting for completion.
How can I stop the for loop until a new task is finished and start a
new one?
The loop can be throttled by using await:
static void Main(string[] args)
{
var task = DoWorkAsync();
task.Wait();
// handle results
// task.Result;
Console.WriteLine("Done.");
}
async static Task<bool> DoWorkAsync()
{
const int NUMBER_OF_SLOTS = 10;
string param1="test";
string param2="test";
var results = new bool[NUMBER_OF_SLOTS];
AsyncWorkScheduler ws = new AsyncWorkScheduler(NUMBER_OF_SLOTS);
for (int i = 0; i < 1000; ++i)
{
await ws.ScheduleAsync((slotNumber) => DoWorkAsync(i, slotNumber, param1, param2, results));
}
ws.Complete();
await ws.Completion;
}
async static Task DoWorkAsync(int index, int slotNumber, string param1, string param2, bool[] results)
{
results[slotNumber] = results[slotNumber} && await Task.Factory.StartNew<bool>(() =>
{
MyClass cls = new MyClass();
bool bRet = cls.Method1(param1, param2, i); // takes up to 2 minutes to finish
return bRet;
}));
}
A helper class AsyncWorkScheduler uses TPL.DataFlow components as well as Task.WhenAll():
class AsyncWorkScheduler
{
public AsyncWorkScheduler(int numberOfSlots)
{
m_slots = new Task[numberOfSlots];
m_availableSlots = new BufferBlock<int>();
m_errors = new List<Exception>();
m_tcs = new TaskCompletionSource<bool>();
m_completionPending = 0;
// Initial state: all slots are available
for(int i = 0; i < m_slots.Length; ++i)
{
m_slots[i] = Task.FromResult(false);
m_availableSlots.Post(i);
}
}
public async Task ScheduleAsync(Func<int, Task> action)
{
if (Volatile.Read(ref m_completionPending) != 0)
{
throw new InvalidOperationException("Unable to schedule new items.");
}
// Acquire a slot
int slotNumber = await m_availableSlots.ReceiveAsync().ConfigureAwait(false);
// Schedule a new task for a given slot
var task = action(slotNumber);
// Store a continuation on the task to handle completion events
m_slots[slotNumber] = task.ContinueWith(t => HandleCompletedTask(t, slotNumber), TaskContinuationOptions.ExecuteSynchronously);
}
public async void Complete()
{
if (Interlocked.CompareExchange(ref m_completionPending, 1, 0) != 0)
{
return;
}
// Signal the queue's completion
m_availableSlots.Complete();
await Task.WhenAll(m_slots).ConfigureAwait(false);
// Set completion
if (m_errors.Count != 0)
{
m_tcs.TrySetException(m_errors);
}
else
{
m_tcs.TrySetResult(true);
}
}
public Task Completion
{
get
{
return m_tcs.Task;
}
}
void SetFailed(Exception error)
{
lock(m_errors)
{
m_errors.Add(error);
}
}
void HandleCompletedTask(Task task, int slotNumber)
{
if (task.IsFaulted || task.IsCanceled)
{
SetFailed(task.Exception);
return;
}
if (Volatile.Read(ref m_completionPending) == 1)
{
return;
}
// Release a slot
m_availableSlots.Post(slotNumber);
}
int m_completionPending;
List<Exception> m_errors;
BufferBlock<int> m_availableSlots;
TaskCompletionSource<bool> m_tcs;
Task[] m_slots;
}

Converting loop to tasks

I have the following synchronous code:
foreach ( var step in result ) {
step.Run();
}
I tried to convert it to tasks but I failed to do so. I tried to convert it using Task.WhenAll like this (and I did append async to the method signature):
var tasks = new List<Task>();
foreach ( var step in result ) {
tasks.Add( new Task( () => step.Run() ) );
}
await Task.WhenAll( tasks );
This returns immediately and doesn't execute the Run() method. Then I tried to convert it to the following code:
var tasks = new List<Task>();
foreach ( var step in result ) {
tasks.Add( new Task( () => step.Run() ) );
}
var task = Task.WhenAll( tasks );
task.Wait();
This blocks forever. However, when I create a within the loop it works:
foreach ( var step in result ) {
var t = Task.Run( () => step.Run() );
t.Wait();
}
If I use instead await Task.Run( () => step.Run() ); it awaits only the first one and resumes the main thread.
The run method looks like this:
public async void Run() {
var result = Work();
if ( null != result && result.Count > 0 ) {
var tasks = new List<Task>();
foreach ( var step in result ) {
await Task.Run( () => step.Run() );
}
}
}
All steps implement a Work() method (which is abstract in a base class). My first step looks like this:
class NoWorkStep : WorkerStep {
protected override IList<WorkerStep> Work() {
Console.WriteLine( "HERE" );
List<WorkerStep> newList = new List<WorkerStep>();
for ( int i = 0; i < 10; i++ ) {
newList.Add( new NoWorkStep2() );
}
return newList;
}
}
And my second step looks like this:
class NoWorkStep2 : WorkerStep {
protected override IList<WorkerStep> Work() {
Console.WriteLine( "HERE-2" );
return new List<WorkerStep>();
}
}
I simple create an instance of NoWorkStep and call instance.Run().
Where do I have a problem with executing the steps with Task.WhenAll?
Edit: Calling code after I changed the Run method to async Task RunAsync:
private static async void doIt() {
var step = new NoWorkStep();
await step.RunAsync();
}

Lets map out the problems with your code:
new Task(() => step.Run())
This returns a cold Task, meaning the Task isn't actually started. In order for it to start you would need to call:
new Task(() => step.Run()).Start)
But, you shouldn't use new Task anyway, you should use Task.Run.
If I use instead await Task.Run( () => step.Run() ); it awaits only
the first one and resumes the main thread.
That is because Run is async void which cannot be awaited. async void is ment to be used only in top level event handlers, where this clearly isn't the case here.
If you want to await on until all the tasks are completed, you can do that following:
public async Task RunAsync()
{
var result = Work();
var stepTasks = result.Select(step => Task.Run(() => step.Run()));
await Task.WhenAll(steps);
}
This will guarantee all tasks have completed execution once RunAsync finishes.

You don't seem to be starting the tasks.
Try:
var tasks = new List<Task>();
foreach (var step in result)
{
var t = new Task(() => step.Run());
t.Start();
tasks.Add(t);
}
Task.WhenAll(tasks);

You can use Parallel.ForEach.
Parallel.ForEach(result, step => step.Run());
This way you don't even muck around with the lower level parts of the Parallel Framework.

Is it possible to execute two async methods in parallel?

If I execute two tasks, I can execute the two tasks at the same time and wait until the two tasks are finished. With this code:
Task<bool> tsk01 = Task.Run(()=> my code; return true);
Task<bool> tsk02 = Task.Run(()=> my code; return true);
Task.WaitAll(tsk01, tsk02);
//next code
In this case the following code is only executed when all tasks are finished. But the application is not blocked.
However if I have this code:
private async void Button_Click_2(object sender, RoutedEventArgs e)
{
Task<bool> tsk01 = miMetodoAsync01();
Task<bool> tsk02 = miMetodoAsync02();
Task.WaitAll(tsk01, tsk02);
}
private async Task<bool> miMetodoAsync02()
{
return await TaskEx.Run<bool>(() =>
{
Int64 resultado = 0;
for (Int64 i = 1; i < 1000000000; i++)
{
resultado = resultado + 1;
}
return true;
}).ConfigureAwait(false);
}
private async Task<bool> miMetodoAsync01()
{
return await TaskEx.Run<bool>(() =>
{
Int64 resultado = 0;
for (Int64 i = 1; i < 1000000000; i++)
{
resultado = resultado + 1;
}
return true;
}).ConfigureAwait(false);
}
In this second option the application is blocked because the WaitAll seems to wait for a response from the tasks that never happens.
Why in the first case the application is not blocked and in the second one it is?

Both your examples will block the UI thread. That's what Task.WaitAll means.
However, you can use TaskEx.WhenAll:
await TaskEx.WhenAll(tsk01, tsk02);

The method Task.WaitAll will block the UI thread as it waits for all tasks to return before continuing.
The code examples that you gave for creating a Task are basically the same (albeit written slightly different ways). Both the return Task<bool>.
The difference is the function being run inside both of your lambda expressions. Your first example has a "my code" reference and returns. The second example you created two counters.
If your "my code" is defined differently than the the counters created in the second example, or if you are only returning true in your lambda expression, then you will get the appearance of one waiting over the other.
Simply returning true will end the threads immediately after they are created. Where-as the counter takes time to compute (also depending on your CPU speed).
If you add the same counter into your function of the first example, you will find that both take the same time, and Task.WaitAllblocks your UI. You can use the System.Diagnositics.StopWatch class to time it.
static void Main(string[] args)
{
string test = Console.ReadLine();
System.Diagnostics.Stopwatch t = new System.Diagnostics.Stopwatch();
t.Start();
Task<bool> task1 = Task.Run<bool>(() => { return true; });
Task<bool> task2 = Task.Run<bool>(() => { return true; });
Task.WaitAll(task1, task2);
t.Stop();
Console.WriteLine("Elapsed time: " + t.Elapsed);
System.Diagnostics.Stopwatch t2 = new System.Diagnostics.Stopwatch();
t2.Start();
Task<bool> task3 = asyncMethod1();
Task<bool> task4 = asyncMethod2();
Task.WaitAll(task3, task4);
t2.Stop();
Console.WriteLine("Elapsed time: " + t2.Elapsed);
Console.Read();
}