We Keep Coding

Queue of BackgroundWorkers raise event when complete [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I need to execute n BackgroundWorkers, when they're complete I want to raise an event and do something with the result of all their work. My use case it to make the queue, fill it, then run it only once. To accomplish this I made a class ParallelQueue. With my initial testing it seems to work, however I'm concerned that the condition _max == _iteration is not the best was to evaluate all work in the queue has been done. Or that my use of Queue is not thread safe, what should I use to accomplish this? (ConcurrentQueue?) If this question is too general I'll remove it, Thanks.
public class ParallelQueue
{
private Queue<BackgroundWorker> _queue;
private readonly object _key = new object();
private int _max = 0;
private int _iteration = 0;
private bool _ran = false;
public ParallelQueue()
{
_queue = new Queue<BackgroundWorker>();
}
public delegate void BackgroundQueueCompleted(object sender, RunWorkerCompletedEventArgs e);
public event BackgroundQueueCompleted QueueCompleted;
public void Add(BackgroundWorker worker)
{
worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(WorkerCompleted);
_queue.Enqueue(worker);
}
public void Run()
{
lock (_key)
{
if(!_queue.Any()) throw new ArgumentOutOfRangeException("ParallelQueue cannot be empty");
if (_ran) throw new InvalidOperationException("ParallelQueue can only be run once");
_ran = true;
_max = _queue.Count();
Parallel.For(0, _queue.Count, (i, state) =>
{
BackgroundWorker worker = _queue.Dequeue();
worker.RunWorkerAsync();
});
}
}
private void WorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
Interlocked.Increment(ref _iteration);
if (_max == _iteration)
{
QueueCompleted?.Invoke(this, e);
}
}
}
example using ParallelQueue
public class Program
{
static void Main(string[] args)
{
var queue = new ParallelQueue();
queue.QueueCompleted += MyQueueCompletedHandler;
for (int i = 0; i < 10; i++)
{
BackgroundWorker bw = new BackgroundWorker();
bw.DoWork += new DoWorkEventHandler((sender, e) =>
{
Thread.Sleep(500);
});
queue.Add(bw);
}
queue.Run();
Console.ReadLine();
}
private static void MyQueueCompletedHandler(object sender, RunWorkerCompletedEventArgs e)
{
Console.WriteLine("queue is complete");
}
}

The BackgroundWorker class has become practically obsolete after the introduction of the Task Parallel Library (TPL) in 2010. If the results of the work you have to do is homogeneous, you can just use the Parallel.ForEach method, or, if you are familiar with LINQ, use PLINQ. Here is a PLINQ example:
var input = new List<int>() { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
int[] results = input
.AsParallel()
.AsOrdered() // optional
.WithDegreeOfParallelism(2) // optional
.Select(x => { Thread.Sleep(500); return x * 2; }) // simulate some work
.ToArray();
If the results are heterogeneous, you can create a Task<TResult> for each piece of work, store it in a List<Task>, wait all the tasks, and get the results through the Result property of each. Example:
var task1 = Task.Run(() => { Thread.Sleep(500); return 1; });
var task2 = Task.Run(() => { Thread.Sleep(500); return "Helen"; });
var task3 = Task.Run(() => { Thread.Sleep(500); return DateTime.Now; });
var list = new List<Task>() { task1, task2, task3 };
Task.WaitAll(list.ToArray());
int result1 = task1.Result;
string result2 = task2.Result;
DateTime result3 = task3.Result;

how to wait c# webbrowser to complete document load in TPL

Hi I have this two blocks of code
when I navigate the browser to a url and try to wait for it I get a deadlock I think=) any help suggestions are much appreciated!
I'm just adding some more text here so I am able to post the question :/ sorry
foreach (var davaType in davaTypes)
{
Parallel.ForEach(years, year =>
{
ts.Add(Task.Run(() =>
{
var th = new Thread(async () =>
{
await doWorkAsync(year, davaType, tarafType);
Application.Run();
});
th.IsBackground = true;
th.SetApartmentState(ApartmentState.STA);
th.Start();
}));
});
}
and
public static async Task doWorkAsync(int year, string davaType, string tarafType)
{
using (var browser = new MyBrowser())
{
Console.WriteLine("Im Created Here");
browser.Navigate("some url");
while (!browser.MyLoaded)
{
Console.WriteLine("waiting");
await Task.Delay(10);
}
Console.WriteLine("complete");
browser.Document?.GetElementById("DropDownList1")?.SetAttribute("value", davaType);
browser.Document.GetElementById("DropDownList3")?.SetAttribute("value", tarafType);
browser.Document.GetElementById("DropDownList4")?.SetAttribute("value", year.ToString());
browser.Document.GetElementById("Button1").MyClick();
await browser.WaitPageLoad(10);
Console.WriteLine("Im DoneHere");
}
}

You can use TaskCompletionSource to create a task and await for it after the page is loaded.
See: UWP WebView await navigate.

I have experienced this problem before and the solution I have applied is as follows;
private void waitTillLoad()
{
WebBrowserReadyState loadStatus = default(WebBrowserReadyState);
int waittime = 100000;
int counter = 0;
while (true)
{
loadStatus = webBrowser1.ReadyState;
Application.DoEvents();
if ((counter > waittime) || (loadStatus == WebBrowserReadyState.Uninitialized) || (loadStatus == WebBrowserReadyState.Loading) || (loadStatus == WebBrowserReadyState.Interactive))
{
break; // TODO: might not be correct. Was : Exit While
}
counter += 1;
}
counter = 0;
while (true)
{
loadStatus = webBrowser1.ReadyState;
Application.DoEvents();
if (loadStatus == WebBrowserReadyState.Complete)
{
break; // TODO: might not be correct. Was : Exit While
}
counter += 1;
}
}

You need to create only one separate thread for WinForms. Other threads will be created by individual browsers themselves. What you do need is handling their DocumentCompleted event. The first time the document completes you enter and submit data, the second time it completes you get the result. When all browsers finish, you exit the winforms thread and you're done.
Note: To launch bots from winforms thread, I made use of Idle event handler, just because it was the first solution that came to my mind. You may find better way.
using System;
using System.Collections.Generic;
using System.Threading;
using System.Windows.Forms;
namespace ConsoleBrowsers
{
class Program
{
static void Main(string[] args)
{
string url = "https://your.url.com";
string[] years = { "2001", "2002", "2003"};
string[] davas = { "1", "2", "3" };
string taraf = "1";
int completed = 0, succeeded = 0;
var bots = new List<Bot>();
var started = false;
var startBots = (EventHandler)delegate (object sender, EventArgs b)
{
if (started)
return;
started = true;
foreach (var dava in davas)
foreach (var year in years)
bots.Add(new Bot { Year = year, Dava = dava, Taraf = taraf, Url = url }.Run((bot, result) =>
{
Console.WriteLine($"{result}: {bot.Year}, {bot.Dava}");
succeeded += result ? 1 : 0;
if (++completed == years.Length * davas.Length)
Application.ExitThread();
}));
};
var forms = new Thread((ThreadStart) delegate {
Application.EnableVisualStyles();
Application.Idle += startBots;
Application.Run();
});
forms.SetApartmentState(ApartmentState.STA);
forms.Start();
forms.Join();
Console.WriteLine($"All bots finished, succeeded: {succeeded}, failed:{bots.Count - succeeded}.");
}
}
class Bot
{
public string Url, Dava, Taraf, Year;
public delegate void CompletionCallback(Bot sender, bool result);
WebBrowser browser;
CompletionCallback callback;
bool submitted;
public Bot Run(CompletionCallback callback)
{
this.callback = callback;
browser = new WebBrowser();
browser.ScriptErrorsSuppressed = true;
browser.DocumentCompleted += Browser_DocumentCompleted;
browser.Navigate(Url);
return this;
}
void Browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (browser.ReadyState == WebBrowserReadyState.Complete)
{
if (submitted)
{
callback(this, true);
return;
}
var ok = SetValue("DropDownList1", Dava);
ok &= SetValue("DropDownList3", Taraf);
ok &= SetValue("DropDownList4", Year);
if (ok)
ok &= Click("Button1");
if (!(submitted = ok))
callback(this, false);
}
}
bool SetValue(string id, string value, string name = "value")
{
var e = browser?.Document?.GetElementById(id);
e?.SetAttribute(name, value);
return e != null;
}
bool Click(string id)
{
var element = browser?.Document?.GetElementById(id);
element?.InvokeMember("click");
return element != null;
}
}
}

How to properly detect when multiple threads within another thread are fully complete?

I'm running several BackgroundWorkerthreads that are being used to execute queries to retrieve DataSets all within another BackgroundWorker thread. Let's call the thread that is running these multiple threads the 'Host Thread' and the others 'Query Thread'. What I am trying to do is to tell when all of the query threads are finished populating their DataSets by utilizing the host thread's RunWorkerCompleted event. The first line in this event handler is
while (dataSets.Count < count) { Thread.Sleep(100); }
//dataSets is a Dictionary<string, DataSet>
where count is the total amount of DataSets that are expected to be returned. My issue seems to be that dataSets.Count seems to become == to count before all of the DataSets are populated.
Here is my full code (Unhelpful/Sensitive information removed)
var hostThread = new BackgroundWorker();
hostThread.RunWorkerCompleted += new RunWorkerCompletedEventHandler(queryWorker_RunWorkerCompleted);
hostThread.DoWork += (send, even) =>
{
foreach (var cs in _connectionStrings)
{
var queryThread = new BackgroundWorker();
queryThread.DoWork += (se, eve) =>
{
var set = DataHandlers.TryGetDataSet(_query, cs, domain, username, pass);
dataSets.Add(((DataRow)set.Tables[0].Rows[0]).ItemArray[0].ToString(), set);
};
queryThread.RunWorkerAsync();
}
};
hostThread.RunWorkerAsync();
The RunWorkerCompleted :
var bw = new BackgroundWorker();
bw.DoWork += (s, ev) =>
{
//Waiting for all DataSets to get populated
while (dataSets.Count < count) { Thread.Sleep(100); }
//Thread.Sleep(5000); If I add this, everything works fine, but when I start running more queries in each query thread this needs to be increased.
this.Invoke((MethodInvoker)delegate()
{
this.Cursor = Cursors.Default;
this.Hide();
foreach (var set in dataSets)
{
if (set == null)
break;
//THIS BLOCK IS NEVER HIT IF I LEAVE OUT THE FIVE SECOND SLEEP
var workflowList = new List<string>();
foreach (var row in set.Value.Tables[0].Rows)
{
workflowList.Add(((DataRow)row).ItemArray[_licensed ? 1 : 0].ToString());
}
((MainForm)this.OwnedForms[0]).ClientWorkflows = new KeyValuePair<string, List<string>>(set.Key, workflowList);
}
//This gets hit before setting properties on a child form because it still thinks there are no DataSets in the dataSets dictionary
((MainForm)this.OwnedForms[0]).ShowDialog();
this.Close();
});
};
bw.RunWorkerAsync();
So as I stated in the comments in the code - I know that at some point the DataSets will be valid as long as I add a long enough sleep after the while loop. So what would be the best way to tell when all of the query threads are actually completed within the host thread completed event handler?
EDIT: Per #ndd this is what I ended up using.
var queryTasks = new List<Task>();
var parentTask = Task.Factory.StartNew(() =>
{
foreach (var cs in appConfigStrings)
{
queryTasks.Add(Task.Factory.StartNew(() => GetDataSets(mainForm, cs.Key, cs.Value)));
}
var array = queryTasks.ToArray();
Task.WaitAll(array);
});
parentTask.ContinueWith((t) =>
{
this.Invoke((MethodInvoker)delegate()
{
this.Cursor = Cursors.Default;
this.Hide();
foreach (var set in dataSets)
{
var workflowList = new List<string>();
foreach (var row in set.Value.Tables[0].Rows)
{
workflowList.Add(((DataRow)row).ItemArray[_licensed ? 1 : 0].ToString());
}
((MainForm)this.OwnedForms[0]).ClientWorkflows = new KeyValuePair<string, List<string>>(set.Key, workflowList);
}
((MainForm)this.OwnedForms[0]).ShowDialog();
this.Close();
});
});

Personally I am in never favor of Sleep as it is not predictable. If I had to use BackgroundWorker then I would likely go with IsBusy property to determine whether the BackgroundThread is done or not.
Sample code with TPL, please note this is just an example, in real world you may want to handle exceptions, pass cancellation token and other things :)
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace BGToTPL
{
class Program
{
static void Main(string[] args)
{
Task[] tasks = new Task[20];
//Parent task is starting 20 child tasks
var parentTask = Task.Run(() =>
{
Console.WriteLine("Parent threadid: " + System.Threading.Thread.CurrentThread.ManagedThreadId);
for (int i = 0; i < 20; i++)
{
tasks[i] = Task.Factory.StartNew(() =>
{
Console.WriteLine("Child threadid: " + System.Threading.Thread.CurrentThread.ManagedThreadId);
Task.Delay(15000);
});
}
});
parentTask.Wait();
Console.WriteLine("Parent task has started creating and running all the child tasks, now waiting for child tasks to be over.");
//Now wait for all the tasks to be done
Task.WaitAll(tasks);
Console.WriteLine("All the tasks are done");
Console.ReadKey();
}
}
}
And the output

How about something like this. — of course just if TPL is no option:
private readonly IList<BackgroundWorker> workers = new List<BackgroundWorker>();
private void Run()
{
var worker1 = new BackgroundWorker();
worker1.DoWork += (sender, args) => Thread.Sleep(1000);
worker1.RunWorkerCompleted += (sender, args) => this.CheckThreads();
var worker2 = new BackgroundWorker();
worker2.DoWork += (sender, args) => Thread.Sleep(1000);
worker2.RunWorkerCompleted += (sender, args) => this.CheckThreads();
lock (this.workers)
{
this.workers.Add(worker1);
this.workers.Add(worker2);
}
worker1.RunWorkerAsync();
worker2.RunWorkerAsync();
}
private void CheckThreads()
{
lock (this.workers)
{
if (this.workers.All(w => !w.IsBusy))
{
Console.WriteLine("All workers completed");
}
}
}

Recursive Task Queue

I'm trying to iterate through a queue - taking 1 item from the queue, processing it in a background task, updating the UI, and then taking the next item, and so on. The problem is the first item is processed in a background task (thread) but then the subsequent items are processed in the UI thread - blocking the UI.
Does anyone know why this happens and how to get around this problem? My full test code is below. Note: this code is for my learning and future reference - not any real application.
public partial class MainWindow : Window
{
private Queue<int> testQueue = new Queue<int>();
private TaskScheduler uiScheduler;
public MainWindow()
{
InitializeComponent();
this.uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();
this.testQueue = new Queue<int>();
this.testQueue.Enqueue(3);
this.testQueue.Enqueue(6);
this.testQueue.Enqueue(7);
this.testQueue.Enqueue(11);
this.testQueue.Enqueue(13);
}
// just a method that takes about 1 second to run on a modern pc
private double SumRootN(int root)
{
double result = 0;
for (int i = 1; i < 10000000; i++)
{
result += Math.Exp(Math.Log(i) / root);
}
return result;
}
private void testQueueButton_Click(object sender, RoutedEventArgs e)
{
this.processQueue();
}
private void processQueue()
{
if (this.testQueue.Count > 0)
{
int root = this.testQueue.Dequeue();
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
this.processQueue();
}, uiScheduler);
}
else
{
this.statusText.Text += "Done\n";
}
}
}

Thank you for posting a repro which allowed me to debug.
Task.Factory.StartNew runs your task on the scheduler (factoryScheduler ?? currentTaskScheduler ?? threadPoolScheduler). You got into case 2: Your new task inherits the scheduler from its parent.
I noticed your curious use of recursive calls in order to simulate a loop. If you do it like this, the problem goes away:
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
}, uiScheduler).ContinueWith(t => { this.processQueue(); });

That's because you are using TaskScheduler.FromCurrentSynchronizationContext() - you do know what it does right? (makes it run on the same thread it is called, in your case the UI)
EDIT: usr answered you why is that happening, but you could also do this (for a quasi parallel processing):
int root = this.testQueue.Dequeue();
Task<double>.Factory.StartNew(() => SumRootN(root))
.ContinueWith(t =>
{
this.statusText.Text += String.Format("root {0} : {1}\n", root, t.Result);
}, uiScheduler);
this.processQueue();

Monitor multiple Threading.Timers after Disposal

I have a process, which creates a dynamic list of timers(System.Threading.Timer) and continues to run until a signal is received to terminate. Once a signal is received to terminate I want any existing timer callbacks to complete (See Below):
private IList<Timer> _timers = new List<Timer>();
...
...
private void WaitOnExecutingThreads()
{
var waiters = new List<ManualResetEvent>(_timers.Count);
foreach (var timer in _timers)
{
var onWait = new ManualResetEvent(false);
waiters.Add(onWait);
timer.Dispose(onWait);
}
WaitHandle.WaitAll(waiters.ToArray());
waiters.ForEach(x=> x.Dispose());
}
This code works right now, but I would like to monitor the ongoing thread callbacks once the timers are disposed. My intent is to write to a log at a given interval "Timer A is still running".
I started playing with:
ThreadPool.RegisterWaitForSingleObject(....)
add added the following:
(Note:I created a class ThreadContext which contains the timer and associated data)
private void WaitOnExecutingThreads()
{
var waiters = new List<ManualResetEvent>();
WaitOrTimerCallback IsRunning = (x, timeout) => { if (timeout) { Log(x + "is still running"); } };
foreach (var threadContext in _threadContexts)
{
var onWait = new ManualResetEvent(false);
threadContext.Timer.Dispose(onWait);
ThreadPool.RegisterWaitForSingleObject(onWait, IsRunning , threadContext.ThreadInfo.Id, new TimeSpan(0, 0, 30), false);
waiters.Add(onWait);
}
WaitHandle.WaitAll(waiters.ToArray());
waiters.ForEach(x=> x.Dispose());
}
I feel like this should be a straight forward task in C# .net 4.0. In my simple unit test, My IsRunning callback fires quite a bit after the wait. I do not perform any further execution after this call. but I am writing quite a bit of code that I am not too comfortable with and feel like this will fail.
Is there a simpler solution or I am misunderstanding something?
UPDATE
Based on Peter R. suggestion I came up with the following below. Granted its more lines of code but I don't have to register a single thread object. If all the threads are still executing after disposal I sleep for 10 seconds and check again for this example.
private void WaitOnExecutingThreads()
{
foreach (var threadContext in _threadContexts)
{
threadContext.DisposeWaiter = new ManualResetEvent(false);
threadContext.Timer.Dispose(threadContext.DisposeWaiter);
}
while(_threadContexts.Count > 0)
{
for(var i = 0; i < _threadContexts.Count; i++)
{
var threadContext = _threadContexts[i];
var isComplete = threadContext.DisposeWaiter.WaitOne(0);
if(isComplete)
{
Console.WriteLine(string.Format("{0}: {1} has completed", DateTime.Now, threadContext.Name));
_threadContexts.RemoveAt(i);
}
else
{
Console.WriteLine(string.Format("{0}: {1} is still running", DateTime.Now, threadContext.Name));
}
}
if (_threadContexts.Count > 0)
{
Thread.Sleep(new TimeSpan(0, 0, 10));
}
}
}
....
public class ThreadContext
{
public string Name { get; set; }
public Timer Timer { get; set; }
public WaitHandle DisposeWaiter { get; set; }
}
_

If your handlers haven't completed, your ManualResetEvents will not be signalled. So, you could simply test if the event is in a signaled state or not. i.e.
var isComplete = waiters[0].WaitOne(0);