We Keep Coding

Proper way to implement my own async Worker Service(s) according to TAP pattern

I'm newbie in async. And making WPF app for scraping and API calls purposes. WPF's UI is needed only for service monitoring and settings control, so all services will run simultaneously in the background. And most of them is doing similar work.
For this one I need to implement strategy like this:
Start worker on threadpool
Worker must send request and process response from Website
2.1 If response processed and some new data appeared - raise an event
2.2 If request failed - handle an error
2.3 If there are many error percent for last x requests - stop worker
No matter the last request completed/failed/running we must send another request
Another request should be sent not earlier than the set delay but should not exceed the delay too much (as far as possible).
private _workTask;
private List<ScrapeParameters> _scrapeParams = new();
public event EventHandler<ScrapedEventArgs>? NewDataScraped;
//Can I run Worker like this?
public void RunWorker()
{
if(_workTask.IsCompleted)
_workTask = WorkAsync(_token)
}
private async Task WorkAsync(CancellationToken cancelToken)
{
List<Task> processTasks = new();
while(true)
{
if(cancelToken.IsCancellationRequested) return;
//Delay could be from 0.5 second to any value
var delayTask = Task.Delay(WorkerDelay);
var completedTasks = processTasks.Where(t => t.IsCompleted)
var setToHandle = new HashSet<Task>(completedTasks);
foreach(var task in setToHandle)
{
//Theoretical logic to handle errors and completion
if(task.IsFaulted)
HandleFaultedTask(task);
else
CountCompleted();
processTasks.Remove(task);
}
//Theoretical logic to obtain the desired parameters.
var currParameters = GetParameters();
processTasks.Add(ProcessAsync(currParameters, cancelToken));
await delayTask;
}
}
//This method usually takes around 2-4 seconds
private async Task ProcessAsync(ScrapeParameters parameters CancellationToken cancelToken)
{
//Some work with http requests
var response = await Client.GetAsync(parameters.ToUri());
...
//Processing response
...
if(newData != null)
NewDataScraped?.Invoke(new(newData));
}
Does my implementation matches the TAP pattern?
Especially I would like to focus on RunWorker() and setToHandle

Using IProgress when reporting progress for async await code vs progress bar control

private static async Task FuncAsync(DataTable dt, DataRow dr)
{
try
{
await Task.Delay(3000); //assume this is an async http post request that takes 3 seconds to respond
Thread.Sleep(1000) //assume this is some synchronous code that takes 2 second
}
catch (Exception e)
{
Thread.Sleep(1000); //assume this is synchronous code that takes 1 second
}
}
private async void Button1_Click(object sender, EventArgs e)
{
List<Task> lstTasks = new List<Task>();
DataTable dt = (DataTable)gridview1.DataSource;
foreach (DataRow dr in dt.Rows)
{
lstTasks.Add(FuncAsync(dr["colname"].ToString());
}
while (lstTasks.Any())
{
Task finishedTask = await Task.WhenAny(lstTasks);
lstTasks.Remove(finishedTask);
await finishedTask;
progressbar1.ReportProgress();
}
}
Assuming the datatable has got 10000 rows.
In the code, on button click, at the 1st iteration of the for loop, an async api request is made. While it takes 3 seconds, the control immediately goes to the caller. So the for loop can make the next iteration, and so on.
When the api response arrives, the code below the await runs as a callback. Thus blocking the UI thread and any incomplete for loop iterations will be delayed until the callback completes irrespective of whether I use await WhenAny or WhenAll.
All code runs on the UI thread due to the presence of synchronization context. I can do ConfigureAwait false on Task.Delay so the callbacks run on separate threads in order to unblock the ui thread.
Say 1000 iterations are made when the 1st await returns and when the 1st iterations await call back runs the following iterations will have completed completed awaits so their callbacks will run. Effectively callbacks will run one after the other if configure await is true. If false then they will run in parallel on separate threads.
So I think that the progress bar that I am updating in the while loop is incorrect because - by the time the code reaches the while block, most of the initial for loop iterations will have been already completed. I hope that I have understood correctly so far.
I have the following options to report progress from inside the task:
using IProgress (I think this is more suitable to report progress from another thread [for example when using Task.Run], or in usual async await if the configure await is false, resulting in code below the await to run in separate thread otherwise it will not show the progress bar moving as the ui thread will be blocked running the callbacks. In my current example code always runs on the same UI thread). So I was thinking the below point may be more appropriate solution.
making the Task non-static so that I can access the progress bar from within the Task and do porgressbar1.PerformStep().
Another thing I have noticed is that await WhenAll doesn't guarantee that IProgress is fully executed.

The IProgress<T> implementation offered natively by the .NET platform, the Progress<T> class, has the interesting characteristic of notifying the captured SynchronizationContext asynchronously, by invoking its Post method. This characteristic sometimes results to unexpected behavior. For example can you guess what effect has the code below to the Label1 control?
IProgress<string> progress = new Progress<string>(s => Label1.Text = s);
progress.Report("Hello");
Label1.Text = "World";
What text will be eventually written to the label, "Hello" or "World"? The correct answer is: "Hello". The delegate s => Label1.Text = s is invoked asynchronously, so it runs after the execution of the Label1.Text = "World" line, which is invoked synchronously.
Implementing a synchronous version of the Progress<T> class is quite trivial. All you have to do is copy-paste Microsoft's source code, rename the class from Progress<T> to SynchronousProgress<T>, and change the line m_synchronizationContext.Post(... to m_synchronizationContext.Send(.... This way every time you invoke the progress.Report method, the call will block until the invocation of the delegate on the UI thread is completed. The unfortunate implication of this is that if the UI thread is blocked for some reason, for example because you used the .Wait() or the .Result to wait synchronously for the task to complete, your application will deadlock.
The asynchronous nature of the Progress<T> class is rarely a problem in practice, but if you want to avoid thinking about it you can just manipulate the ProgressBar1 control directly. After all you are not writing a library, you are just writing code in the event handler of a button to make some HTTP requests. My suggestion is to forget about the .ConfigureAwait(false) hackery, and just let the main workflow of your asynchronous event handler to stay on the UI thread from start to end. If you have synchronous blocking code that needs to be offloaded to a ThreadPool thread, use the Task.Run method to offload it. To create your tasks, instead of manually adding tasks to a List<Task>, use the handly LINQ Select operator to project each DataRow to a Task. Also add a reference to the System.Data.DataSetExtensions assembly, so that the DataTable.AsEnumerable extension method becomes available. Finally add a throttler (a SemaphoreSlim), so that your application makes efficient use of the available network bandwidth, and it doesn't overburden the target machine:
private async void Button1_Click(object sender, EventArgs e)
{
Button1.Enabled = false;
const int maximumConcurrency = 10;
var throttler = new SemaphoreSlim(maximumConcurrency, maximumConcurrency);
DataTable dataTable = (DataTable)GridView1.DataSource;
ProgressBar1.Minimum = 0;
ProgressBar1.Maximum = dataTable.Rows.Count;
ProgressBar1.Step = 1;
ProgressBar1.Value = 0;
Task[] tasks = dataTable.AsEnumerable().Select(async row =>
{
await throttler.WaitAsync();
try
{
await Task.Delay(3000); // Simulate an asynchronous HTTP request
await Task.Run(() => Thread.Sleep(2000)); // Simulate synchronous code
}
catch
{
await Task.Run(() => Thread.Sleep(1000)); // Simulate synchronous code
}
finally
{
throttler.Release();
}
ProgressBar1.PerformStep();
}).ToArray();
await Task.WhenAll(tasks);
Button1.Enabled = true;
}

You can simply add a wrapper function:
private IProgress<double> _progress;
private int _jobsFinished = 0;
private int _totalJobs = 1000;
private static async Task FuncAsync()
{
try
{
await Task.Delay(3000); //assume this is an async http post request that takes 3 seconds to respond
Thread.Sleep(1000); //assume this is some synchronous code that takes 2 second
}
catch (Exception e)
{
Thread.Sleep(1000); //assume this is synchronous code that takes 1 second
}
}
private async Task AwaitAndUpdateProgress()
{
await FuncAsync(); // Can also do Task.Run(FuncAsync) to run on a worker thread
_jobsFinished++;
_progress.Report((double) _jobsFinished / _totalJobs);
}
And then just WhenAll after adding all the calls.

When to use Task.Run and not

I asked this question yesterday and still don't understand the difference in using
task = Task.Run(() => RunLongRunningMethod(cts.Token));
and
task = RunLongRunningMethod(cts.Token);
I've read through Task.Run Etiquette and Proper Usage and it appears to be mostly using Task.Run as long as it's used correctly (not in the implementation)
Is there any other reading material about this and or can someone explain the difference between the two?
My code is below which works okay using both methods.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Drawing;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApp1
{
public partial class Form1 : Form
{
private static HttpClient client { get; set; }
private Task task { get; set; }
private CancellationTokenSource cts { get; set; }
private bool buttonStartStopState { get; set; }
public Form1()
{
InitializeComponent();
}
private async Task RunLongRunningMethod(CancellationToken cancellationToken)
{
try
{
while (true)
{
if (cancellationToken.IsCancellationRequested)
{
cancellationToken.ThrowIfCancellationRequested();
}
// Some CPU bound work here
// Then call async
var success = await GetUrlAsync(#"https://www.bbc.co.uk/");
Thread.Sleep(2000); // simulate blocking only
}
}
catch (OperationCanceledException)
{
// Just exit without logging. Operation cancelled by user.
}
catch (Exception ex)
{
// Report Error
}
}
private async Task<bool> GetUrlAsync(string url)
{
if (client == null)
{
client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true, AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip });
client.BaseAddress = new Uri("https://www.bbc.co.uk/");
client.DefaultRequestHeaders.Add("Accept", "*/*");
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/62.0");
client.DefaultRequestHeaders.Connection.Add("Keep-Alive");
client.DefaultRequestHeaders.Add("DNT", "1");
}
var response = await client.GetAsync(url);
var contents = await response.Content.ReadAsStringAsync();
Debug.WriteLine($"{DateTime.Now} {response.StatusCode}");
return true;
}
private void buttonStartStop_Click(object sender, EventArgs e)
{
buttonStartStopState = !buttonStartStopState;
if(buttonStartStopState)
{
cts = new CancellationTokenSource();
// What is difference between this
//task = Task.Run(() => RunLongRunningMethod(cts.Token));
// And this?
task = RunLongRunningMethod(cts.Token);
// This always runs instantly
Debug.WriteLine($"{DateTime.Now} buttonStartStopState:{buttonStartStopState}");
}
else
{
cts.Cancel();
cts = null;
}
}
private async void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
if(cts != null)
{
cts.Cancel(); // CancellationTokenSource
cts = null;
}
if (!task.IsCompleted)
{
//this.Hide();
e.Cancel = true;
await task;
this.Close();
}
}
}
}

Basically, use async/await throughout your code base so that nothing is blocking threads. (e.g. your current use of Thread.Sleep is blocking, as I commented).
Once you've reached that point, you've already got hot Tasks returned by your async functions, and these functions will return as soon as they're not making further progress.
At this point, you have a decision to make. Do you have a long running task that is CPU bound? If so, that is when you might consider using Task.Run, because that explicitly asks for the work to be done elsewhere (the thread pool). It's generally ok to allow I/O dominated tasks to come back onto the UI thread briefly, which is what you'll get by default and means that you don't have to do anything special to access UI objects.
Hopefully, at this point, you'll not want to use Task.Run at all in your example.
But it is possible that your long running task is a combination of truly async I/O operations and some CPU intensive operations, and you still don't want those occupying the UI thread. At this point, you should normally be considering using ConfigureAwait(false) on your awaitables. But you may wish to also use Task.Run here.
In either case, if you're wanting to interact with UI objects again, you'll have to Invoke to get back onto the UI thread. Make sure you do this at the right "granularity". Don't Invoke 5 or 6 separate basic setting of properties on UI objects. But also don't Invoke back onto the UI thread before actually doing the CPU intensive operations - they're why you tried to move them to a different thread in the first place!

Just to add to the 2 previous answers
Answer by #Neil
Answer by #Thierry V
It looks like you are running inside WinForms which runs in STA (Single Threaded Apartment) model, meaning there is only thread processing UI queued messages.
Therefore when you run task = Task.Run(() => RunLongRunningMethod(cts.Token)); this does everything on a thread pool thread, where as by default the task = RunLongRunningMethod(cts.Token); will block the UI for the duration of Thread.Sleep(2000); // simulate blocking only as your await call will be queued onto the the UI Dispatcher as you do not have ConfigureAwait(false).
Overall not running it on a background thread means:
Your Thread.Sleep(x) or the actual work time taken will block the UI
You will put more pressure on the dispatcher as each await will be scheduled to be executed by the UI dispatcher. (in your instance if it is only a single await, it is not an issue, but if you started to do 100's or even 1000's awaits, this can start showing noticeable performance loss!)

task = Task.Run(() => RunLongRunningMethod(cts.Token));
Will queue RunLongRunningMethod to be executed by one of the threads in the task pool at some point in the future. The next line of code will be executed immediately.
Whereas
RunLongRunningMethod(cts.Token);
Will execute the code on the current thread, and will not run the next line of code until it has finished.

RunLongRunningMethod(cts.Token); execute your action immediately in the same thread. That means it can block your UI if your code is in the UI thread.
task = Task.Run(() => RunLongRunningMethod(cts.Token)); contrariwise means that you want to execute right away your action. This line queues the task to run on the ThreadPool and returns a task handle for that work.
Normally, we use:
Task.Run when you want to execute a long-running code and don't wait if the task finishes. a calculation in background i.e and
task = RunLongRunningMethod normally when you want to await task, i.e
await RunLongRunningMethod(token);
//this method DoSomeThing is not executed until your your long-running code finishs
DoSomeThing();

WebClient DownloadStringAsync blocked - never finished

I have specific problem with WebClient in my Windows Phone app (using MVVM)
private string _lastCurrencyRatesJson;
private bool _lastCurrencyRatesJsonLoaded = false;
private void GetLastCoursesFromApiAsync()
{
var uri = new Uri(string.Format(OperationGetLastCourses, AppSettings.ApiEndpoint, AppSettings.ApiKey));
var client = new WebClient { Encoding = Encoding.UTF8 };
client.DownloadStringCompleted += client_DownloadStringCompleted;
client.DownloadStringAsync(uri);
}
void client_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
_lastCurrencyRatesJson = e.Result;
_lastCurrencyRatesJsonLoaded = true;
}
public List<CurrencyRate> GetLastCourses()
{
var worker = new Thread(GetLastCoursesFromApiAsync);
worker.Start();
while (!_lastCurrencyRatesJsonLoaded)
{
}
.....
The problem is that client_DownloadStringCompleted is never fired BUT when I change GetLastCourses this way:
public List<CurrencyRate> GetLastCourses()
{
var worker = new Thread(GetLastCoursesFromApiAsync);
worker.Start();
// whetever here, but any while...
client_DownloadStringCompleted is fired and data are obtained. It means, connectivity is ok.
I had very similar problems with DownloadStringTaskAsyn. Example:
private async Task<string> GetCoursesForDayFromApiAsJson(DateTime date)
{
var uri = new Uri(string.Format(OperationGetCoursesForDay, AppSettings.ApiEndpoint, AppSettings.ApiKey, date.ToString(DateFormat)));
var client = new WebClient { Encoding = Encoding.UTF8 };
return await client.DownloadStringTaskAsync(uri);
}
Again, at the line with await is application waiting for the data but the DownloadStringTaskAsync is never finished and my UI is still loading.
Any ideas what could be wrong?
SITUATION ONE DAY AGO
So, it looks that WP application is working just with one thread. It means, current thread have to be "finished" and then is DownloadStringTaskAsync finished and the code under the await executed. When I want to work with Task.Result I can not. Never.
When I create another Thread and I am trying to wait for thread completetion (using Join()), created Thread is never finsihed and the code after Join() is never executed.
There is any example on the Internet and I absolutely don't know, why exists some DownloadStringTaskAsync when it is not applicable.

You're blocking the UI thread by your while loop and at the same time, the DownloadStringCompleted event wants to execute on the UI loop. This causes a deadlock, so nothing happens. What you need to do is to let GetLastCourses() return (and whatever method calls that), so that the event handler can execute. This means that the code that handles the results should be in that event handler (not in GetLastCourses()).
With async-await, you didn't provide all of your code, but it's likely that you're encountering pretty much the same issue by calling Wait() or Result on the returned Task. If replace that with await, you code will work. Though that requires you to make all your code from GetCoursesForDayFromApiAsJson() up async.

I'd recommend to use the HttpClient class from Microsoft NuGet package and use the async/await downloading pattern rather than using event-based WebClient class:
Uri uri = new Uri(string.Format(OperationGetLastCourses, AppSettings.ApiEndpoint, AppSettings.ApiKey));
using (HttpClient client = new HttpClient())
{
string result = await client.GetStringAsync(uri);
}

Waiting for all async WebClient calls to finish

I am using the WebClient class in C#, 4.0. I need to hit a REST service with 30,000 different IDs, and grab the status result (200 or 404). Here is the method that makes the calls (eventCounter is a CountdownEvent object):
private void doWork()
{
initDB();
List<string> _lines = new List<string>();
//pull all UpcIds into a List
using (StreamReader _rdr = new StreamReader(#"C:\Users\kkohut\Dropbox\ROVI\Application Support\BestBuy\upc_test2.txt"))
{
string _line;
while ((_line = _rdr.ReadLine()) != null)
{
_lines.Add(_line);
}
}
numIds = _lines.Count();
for (int i = 0; i < numIds; i++)
{
string _upcId = _lines[i];
WebClient c = new WebClient();
c.DownloadDataCompleted += new DownloadDataCompletedEventHandler(c_DownloadDataCompleted);
c.DownloadDataAsync(new Uri(BASE_URL + _upcId), _upcId);
}
//this is not working correctly. Code execution hits this line and waits, without processing any of the
//the DownloadDataCompleted eventhandlers
eventCounter.Wait();
}
Here is the DownloadDataCompleted event handler
void c_DownloadDataCompleted(object sender, DownloadDataCompletedEventArgs e)
{
DataSet _ds = new DataSet();
string _upcId = e.UserState.ToString();
string _status = "404";
try
{
if (!e.Cancelled && e.Error == null)
{
string _result = System.Text.Encoding.UTF8.GetString(e.Result);
if (_result.IndexOf("<code>200</code>") > 0)
{
_status = "200";
}
}
}
catch (Exception ex)
{
_status = "404";
}
finally
{
updateDB(_upcId, _status);
eventCounter.Signal(1);
txtLog.Text += string.Format("{0}\t{1}\t{2}\r\n",ctr, _upcId, _status);
}
}
If I comment out the eventCounter.Wait() statement, the calls work, but I have no way of knowing when they complete. This is a winforms app, so as long as I keep the form running, all the calls complete. But if I uncomment the eventCounter.Wait() statement, no calls get processed. It appears that the Wait() statement is blocking the async calls to start with. Every example I have found uses this approach, but none of them are signaling the CountdownEvent in the completed event handler. Thoughts?

The WebClient Class implements the Event-based Asynchronous Pattern (EAP).
In this pattern, the XXXAsync Method captures the current SynchronizationContext (i.e. the UI thread in a WPF or WinForms application). When the operation completes, the event handler is executed in this context.
(See also: On which thread(s) does WebClient raise its events?)
Problem: If you invoke a blocking method on the UI thread, the event handler will not run before the blocking method returns.
Solution: Wait asynchronously for the CountdownEvent to complete, not synchronously.
You can use the ThreadPool.RegisterWaitForSingleObject Method to register a callback for the WaitHandle of the CountdownEvent.