We Keep Coding

Restart concurrent tasks as soon as they fail for x number of times

I have a console app that is making HTTP queries and adding/updating products in my database according to response. Some fail and need to be retried a few times.
The way I came up with was to use a dictionary to store the product ID and a Task. Then I can check all the task results and re-run.
This is working but it strikes me as inefficient. Tasks are not being re-created until all tasks have finished. It would be more efficient if they were immediately restarted but I can't figure out how to do this. Also every retry involves a query to the database as only the ID is stored.
I made small app that shows how I am currently retrying failed requests.
Can someone suggest a more efficient method for retrying?
class Program
{
private static void Main(string[] args)
{
HttpQuery m = new HttpQuery();
var task = Task.Run(() => m.Start());
Task.WaitAll(task);
Console.WriteLine("Finished");
Console.ReadLine();
}
}
class HttpQuery
{
public async Task Start()
{
// dictionary where key represent reference to something that needs to be processed and bool whether it has completed or not
ConcurrentDictionary<int, Task<bool>> monitor = new ConcurrentDictionary<int, Task<bool>>();
// start async tasks.
Console.WriteLine("starting first try");
for (int i = 0; i < 1000; i++)
{
Console.Write(i+",");
monitor[i] = this.Query(i);
}
// wait for completion
await Task.WhenAll(monitor.Values.ToArray());
Console.WriteLine();
// start retries
// number of retries per query
int retries = 10;
int count = 0;
// check if max retries exceeded or all completed
while (count < retries && monitor.Any(x => x.Value.Result == false))
{
// make list of numbers that failed
List<int> retryList = monitor.Where(x => x.Value.Result == false).Select(x => x.Key).ToList();
Console.WriteLine("starting try number: " + (count+1) + ", Processing: " + retryList.Count);
// create list of tasks to wait for
List<Task<bool>> toWait = new List<Task<bool>>();
foreach (var i in retryList)
{
Console.Write(i + ",");
monitor[i] = this.Query(i);
toWait.Add(monitor[i]);
}
// wait for completion
await Task.WhenAll(toWait.ToArray());
Console.WriteLine();
count++;
}
Console.WriteLine("ended");
Console.ReadLine();
}
public async Task<bool> Query(int i)
{
// simulate a http request that may or may not fail
Random r = new Random();
int delay = i * r.Next(1, 10);
await Task.Delay(delay);
if (r.Next(0,2) == 1)
{
return true;
}
else
{
return false;
}
}
}

You can create another method and wrap all these ugly retry logic. All of that ugly code goes away :)
public async Task Start()
{
const int MaxNumberOfTries = 10;
List<Task<bool>> tasks = new List<Task<bool>>();
for (int i = 0; i < 1000; i++)
{
tasks.Add(this.QueryWithRetry(i, MaxNumberOfTries));
}
await Task.WhenAll(tasks);
}
public async Task<bool> QueryWithRetry(int i, int numOfTries)
{
int tries = 0;
bool result;
do
{
result = await Query(i);
tries++;
} while (!result && tries < numOfTries);
return result;
}

Updating multiple progress bars from multiple tasks in task list using IProgress<T>

I have multiple tasks that will be running in parallel. They are gathered in a list of tasks. Each task has a progress bar associated with it. I've been trying to implement the IProgress API in order to update the progress bars while the task runs. However, the progress bar only updates when the task is completed.
I have a set of test code that simulates my problem, and no matter how I modify it, the task bars only update when the task is completed.
As each task is created, I pass Progress object that will update the right progress bar. This Progress object is passed to the inner task to be updated with each loop. The Progress.Report() method will check for an InvokeRequired, although I don't think this should be necessary if you're using the IProgress API with asynchronous methods.
I've included the code. The form simply has a button (button1) that starts all the processes, and there are 11 progress bars that should update with each task.
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private async void button1_Click(object sender, EventArgs e)
{
List<ProgressBar> listOfProgressBars = new List<ProgressBar>
{
progressBar1,
progressBar2,
progressBar3,
progressBar4,
progressBar5,
progressBar6,
progressBar7,
progressBar8,
progressBar9,
progressBar10,
progressBar11
};
List<Task<bool>> taskList = new List<Task<bool>>();
foreach (ProgressBar pBar in listOfProgressBars)
{
pBar.Value = 0;
pBar.Minimum = 0;
pBar.Maximum = 100;
}
int i = 11;
for (int j = 1; j <= i; j++)
{
ProgressBar thisProgressBar = listOfProgressBars[j - 1];
var progress = new Progress<int>((int value) =>
{
UpdateProgressBar(thisProgressBar, value);
});
taskList.Add(InnerProcess(j, progress));
}
await Task.WhenAll(taskList);
}
public void UpdateProgressBar(ProgressBar pBar, int value)
{
if (this.InvokeRequired)
{
this.EndInvoke(this.BeginInvoke(new MethodInvoker(delegate() { UpdateProgressBar(pBar, value); })));
}
else
{
pBar.Value = value;
}
}
public Task<bool> InnerProcess(int waitTime, IProgress<int> progress)
{
return Task<bool>.Run(() =>
{
var job = new LongJob();
job.Delay(waitTime, progress);
return true;
});
}
class LongJob
{
public void Delay(int i, IProgress<int> progress)
{
for (int j = 1; j <= i; j++)
{
Thread.Sleep(500);
if (progress != null)
{
progress.Report(j/i*100);
}
}
}
}
}

progress.Report(j/i*100);
j/i is integer division, since j is less that i until completion, j/i is 0.
cast j to a float and all should be good

You have a bug in your progress calculation due to integer division. Change it to this:
progress.Report((int)(((double)j) / i * 100));

How wait all thread

I have code, that create 5 threads. I need wait, until all threads finished their work, and after return value. How can I do this?
public static int num=-1;
public int GetValue()
{
Thread t=null;
for (int i = 0; i <=5; i++)
{
t = new Thread(() => PasswdThread(i));
t.Start();
}
//how wait all thread, and than return value?
return num;
}
public void PasswdThread(int i)
{
Thread.Sleep(1000);
Random r=new Random();
int n=r.Next(10);
if (n==5)
{
num=r.Next(1000);
}
}
Of course this is not a real code. The actual code is much more complicated, so I simplified it.
P.S. Look carefully. I am not use Task, so I can't use method Wait() or WaitAll(). Also I can't use Join(), because Join wait one thread. If they start wait thread, which already finished they work, the will wait infinity.

Make an array of thread like below and call WaitAll function
List<Thread> threads = new List<Thread>();
Thread thread = null;
for (int i = 0; i <=5; i++)
{
t = new Thread(() => PasswdThread(i));
t.Start();
threads.add(t);
}
Thread.WaitAll(thread);
//how wait all thread, and than return value?
return num;

create a ManualResetEvent handle for each your thread, and then call WaitHandle.WaitAll(handles) in your main thread.
static WaitHandle[] handles = new WaitHandle[5];
`
public void PasswdThread(int i)
{
handles[i] = new ManualResetEvent(false);
Thread.Sleep(1000);
Random r=new Random();
int n=r.Next(10);
if (n==5)
{
num=r.Next(1000);
}
handles[i].Set();
}
Get more information on http://msdn.microsoft.com/en-us/library/z6w25xa6.aspx

I think you can use Thread.WaitAll(thread_array) or in other case you can also use Thread.Sleep(100)
In Thread.sleep, 100 is number of milliseconds. So in this case thread would sleep for 100 milliseconds.
And in Thread.WaitAll - thread_Array is array of threads that you wanna wait.

As this question is effectively a duplicate, please see this answer, (code copied below, all credit to Reed Copsey.
class Program
{
static void Main(string[] args)
{
int numThreads = 10;
ManualResetEvent resetEvent = new ManualResetEvent(false);
int toProcess = numThreads;
// Start workers.
for (int i = 0; i < numThreads; i++)
{
new Thread(delegate()
{
Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
// If we're the last thread, signal
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}).Start();
}
// Wait for workers.
resetEvent.WaitOne();
Console.WriteLine("Finished.");
}
}
Aside
Also note that your PasswdThread code will not produce random numbers. The Random object should be declared statically, outside of your method, to produce random numbers.
Additionally you never use the int i parameter of that method.

I would use TPL for this, imo it's the most up to date technique for handling this sort of synchronization. Given the real life code is probably more complex, I'll rework the example slightly:
public int GetValue()
{
List<Task<int>> tasks = new List<Task<int>>();
for (int i = 0; i <=5; i++)
{
tasks.Add(PasswdThread(i));
}
Task.WaitAll(tasks);
// You can now query all the tasks:
foreach (int result in tasks.Select(t => t.Result))
{
if (result == 100) // Do something to pick the desired result...
{
return result;
}
}
return -1;
}
public Task<int> PasswdThread(int i)
{
return Task.Factory.StartNew(() => {
Thread.Sleep(1000);
Random r=new Random();
int n=r.Next(10);
if (n==5)
{
return r.Next(1000);
}
return 0;
});
}

Thread t=null;
List<Thread> lst = new List<Thread();
for (int i = 0; i <=5; i++)
{
t = new Thread(() => PasswdThread(i));
lst.Add(t);
t.Start();
}
//how wait all thread, and than return value?
foreach(var item in lst)
{
while(item.IsAlive)
{
Thread.Sleep(5);
}
}
return num;

Is using IEnumerator is the best way to implement asynchronous while multithreading is seems to be slow?

I am trying to create some kind of framework that simplifies process of writing object interaction algorithm. (One object -- many clients(algorithms))
For example I want to implement algorithm that do some very simple job and waits some condition meet in a loop:
public void MakeVerySimpleJob() { }
public void AsyncLoop()
{ // real algorithm can be more complex, but job is always very simple
while (true)
{
MakeVerySimpleJob();
WakeUpCondition = "As fast as u can!";
JobComplete.Set();
WakeUp.WaitOne();
}
}
void main()
{
Thread MyThread = new Thread(AsyncLoop);
MyThread.Start();
var w = new System.Diagnostics.Stopwatch(); w.Start();
for (int i = 0; i < 100000; i++)
{
// waitin for thread
JobComplete.WaitOne();
// ok we did it
WakeUpCondition = null;
WakeUp.Set();
}
w.Stop();
}
AutoResetEvent JobComplete = new AutoResetEvent(false);
AutoResetEvent WakeUp = new AutoResetEvent(false);
Unfortunately it consumes about 500ms to do 100000 simple jobs.
Ok multithreading is not acceptable in my case, but I dont want to force users to write algorithms in this manner:
// invoke it again and again
public void PseudoAsyncLoop()
{
if (CurrentState == 1)
{
MakeVerySimpleJob();
CurrentState = 2;
return;
}
else is (CurrentState == some_sate)
{
}
}
int CurrentState = 0;
So i look at Enumerators. With Enumerators user can implement their own algorithm in traditional style:
public IEnumerable<bool> PseudoAsyncLoop()
{
while (true)
{
MakeVerySimpleJob();
WakeUpCondition = "As fast as u can!";
yield return true;
}
}
public string WakeUpCondition { get; private set; }
void main()
{
var MyLoop = PseudoAsyncLoop();
var LoopEnumerator = MyLoop.GetEnumerator();
var w = new System.Diagnostics.Stopwatch(); w.Start();
for(int i = 0; i < 100000; i ++)
{
LoopEnumerator.MoveNext();
// ok we did it
WakeUpCondition = null;
}
w.Stop();
}
Now it takes about 3ms, great. But i think its something wrong with that all...
My questions is:
Am I in right direction?
How does professional programmers solves that types of problems?
May be there is some ways to optimize multithreaded version?

I don't completely understand what are you doing or why, but if this is actually representative of your code, then you can speed it up by using one of the -Slim synchronization primitives. There is no AutoResetEventSlim, but you can use SemaphoreSlim instead:
private readonly SemaphoreSlim JobComplete = new SemaphoreSlim(0, 1);
private readonly SemaphoreSlim WakeUp = new SemaphoreSlim(0, 1);
private void AsyncLoop()
{
while (true)
{
MakeVerySimpleJob();
WakeUpCondition = "As fast as u can!";
JobComplete.Release();
WakeUp.Wait();
}
}
private void main()
{
Thread MyThread = new Thread(AsyncLoop);
MyThread.Start();
for (int i = 0; i < 100000; i++)
{
JobComplete.Wait();
WakeUpCondition = null;
WakeUp.Release();
}
}
This results in about 5.5× faster execution on my machine.

For loop in many threads

How can I run each call for loop in another thread, but continuation of ExternalMethod should wait to ending of last working thread from for loop (and synchronize) ?
ExternalMethod()
{
//some calculations
for (int i = 0; i < 10; i++)
{
SomeMethod(i);
}
//continuation ExternalMethod
}

One approach would be to use a ManualResetEvent.
Consider the following code (note that this should not be taken as a working example, stuck on OSX so don't have VS nor a C# compiler to hand to check this over):
static ManualResetEvent mre = new ManualResetEvent(false);
static int DoneCount = 0;
static int DoneRequired = 9;
void ExternalMethod() {
mre.Reset();
for (int i = 0; i < 10; i++) {
new Thread(new ThreadStart(ThreadVoid)).Start();
}
mre.WaitOne();
}
void ThreadVoid() {
Interlocked.Increment(ref DoneCount);
if (DoneCount == DoneRequired) {
mre.Set();
}
}
IMPORTANT - This possibly isn't the best way to do it, just an example of using ManualResetEvent, and it will suit your needs perfectly fine.
If you're on .NET 4.0 you can use a Parallel.For loop - explained here.

System.Threading.Tasks.Parallel.For(0, 10, (i) => SomeMethod(i));

One approach is to use a CountdownEvent.
ExternalMethod()
{
//some calculations
var finished = new CountdownEvent(1);
for (int i = 0; i < 10; i++)
{
int capture = i; // This is needed to capture the loop variable correctly.
finished.AddCount();
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
SomeMethod(capture);
}
finally
{
finished.Signal();
}
}, null);
}
finished.Signal();
finished.Wait();
//continuation ExternalMethod
}
If CountdownEvent is not available then here is an alternate approach.
ExternalMethod()
{
//some calculations
var finished = new ManualResetEvent(false);
int pending = 1;
for (int i = 0; i < 10; i++)
{
int capture = i; // This is needed to capture the loop variable correctly.
Interlocked.Increment(ref pending);
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
SomeMethod(capture);
}
finally
{
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
}
}, null);
}
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
finished.WaitOne();
//continuation ExternalMethod
}
Note that in both examples the for loop itself is treating as a parallel work item (it is on a separate thread from the other work items afterall) to avoid a really subtle race condition that might occur if the first work item signals the event before the next work item is queued.

For .NET 3.5, maybe something like this:
Thread[] threads = new Thread[10];
for (int x = 0; x < 10; x++)
{
threads[x] = new Thread(new ParameterizedThreadStart(ThreadFun));
threads[x].Start(x);
}
foreach (Thread thread in threads) thread.Join();
It may seem counterintuitive to use the Join() method, but since you are effectively doing a WaitAll-type pattern, it doesn't matter what order the joins are executed.