We Keep Coding

What is a good way to parallelize many HTTP web requests?

I am building a generic URI retrieval system. Essentially there's a generic class Retriever<T> and it maintains a queue of URIs to be retrieved. It has a separate thread that handles that queue as fast as it can. An example of a type of URI, as indicated in the question title, is HTTP type URIs.
The problem is, when I get down to requesting that the resource be retrieved, via an abstract method T RetrieveResource(Uri location), it slow down due to a lack of asynchrony.
Changing the return type of RetrieveResource to Task<T> was my first thought. However, that seems to make tasks pile up and cause lots of problems when we have thousands of outstanding tasks. It appears to create many actual threads instead of utilizing the thread pool. I imagine this just slows everything down because there are too many things going on at once, so nothing individually is making significant progress.
It's expected that we will have a large number of queued items to retrieve and that they cannot be handled as fast as they are enqueued. There is an opportunity for the system to catch up, over time; but it's definitely not quick.
I've also thought about instead of maintaining a queue and a thread to handle it... to just queue a work item on the ThreadPool. However, I'm not sure that this is ideal if say I need to shut the system down before all work items are handled or later want to allow for prioritization or something.
We also know that retrieving a resource is a time consuming process (0.250 - 5 seconds), but not necessarily a resource intense process. We are fine parallelizing this out to hundreds of requests.
Our requirements are:
URIs can be enqueued from any thread, even when the system is working on the queue
Retrieval may need to later be capable of being prioritized
Retrieval should be able to be paused
Minimal spinning should occur when nothing is being retrieved (BlockingCollection is useful here).
Is there a good way to parallelize this without introducing unnecessary complexity?
Below is some existing code we have, as an example.
public abstract class Retriever<T> : IRetriever<T>, IDisposable
{
private readonly Thread worker;
private readonly BlockingCollection<Uri> pending;
private volatile int isStarted;
private volatile int isDisposing;
public event EventHandler<RetrievalEventArgs<T>> Retrieved;
protected Retriever()
{
this.worker = new Thread(this.RetrieveResources);
this.pending = new BlockingCollection<Uri>(new ConcurrentQueue<Uri>());
this.isStarted = 0;
this.isDisposing = 0;
}
~Retriever()
{
this.Dispose(false);
}
private void RetrieveResources()
{
while (this.isDisposing == 0)
{
while (this.isStarted == 0)
{
Monitor.Wait(this.pending);
}
Uri location = this.pending.Take();
// This is what needs to be concurrently done.
// In this example, it's synchronous, but just on a separate thread.
T result = this.RetrieveResource(location);
// At this point, we would fire our event with the retrieved data
}
}
protected abstract T RetrieveResource(Uri location);
protected void Dispose(bool disposing)
{
if (Interlocked.CompareExchange(ref this.isDisposing, 1, 0) == 1)
{
return;
}
if (disposing)
{
this.pending.CompleteAdding();
this.worker.Join();
}
}
public void Add(Uri uri)
{
try
{
this.pending.Add(uri);
}
catch (InvalidOperationException)
{
return;
}
}
public void AddRange(IEnumerable<Uri> uris)
{
foreach (Uri uri in uris)
{
try
{
this.pending.Add(uri);
}
catch (InvalidOperationException)
{
return;
}
}
}
public void Start()
{
if (Interlocked.CompareExchange(ref this.isStarted, 1, 0) == 1)
{
throw new InvalidOperationException("The retriever is already started.");
}
if (this.worker.ThreadState == ThreadState.Unstarted)
{
this.worker.Start();
}
Monitor.Pulse(this.pending);
}
public void Stop()
{
if (Interlocked.CompareExchange(ref this.isStarted, 0, 1) == 0)
{
throw new InvalidOperationException("The retriever is already stopped.");
}
}
public void Dispose()
{
this.Dispose(true);
GC.SuppressFinalize(this);
}
}
To build on the example above... a solution to this that I think adds too much complexity or rather, weird code... would be this.
private void RetrieveResources()
{
while (this.isDisposing == 0)
{
while (this.isStarted == 0)
{
Monitor.Wait(this.pending);
}
Uri location = this.pending.Take();
Task<T> task = new Task<T>((state) =>
{
return this.RetrieveResource(state as Uri);
}, location);
task.ContinueWith((t) =>
{
T result = t.Result;
RetrievalEventArgs<T> args = new RetrievalEventArgs<T>(location, result);
EventHandler<RetrievalEventArgs<T>> callback = this.Retrieved;
if (!Object.ReferenceEquals(callback, null))
{
callback(this, args);
}
});
task.Start();
}
}

I've come up with a pretty good solution I think. I abstracted both the method a resource is retrieved and the result's representation. This allows support for retrieval of arbitrary URIs with arbitrary results; kind of like some URI driven "ORM".
It supports variable concurrency levels. The other day when I posted the question, I was forgetting that asynchrony and concurrency are quite different and that all I was achieving with tasks was asynchrony and jamming up the task scheduler because what I really wanted was concurrency.
I added in cancellation because it seemed like a good idea to have start/stop capabilities.
public abstract class Retriever<T> : IRetriever<T>
{
private readonly object locker;
private readonly BlockingCollection<Uri> pending;
private readonly Thread[] threads;
private CancellationTokenSource cancellation;
private volatile int isStarted;
private volatile int isDisposing;
public event EventHandler<RetrieverEventArgs<T>> Retrieved;
protected Retriever(int concurrency)
{
if (concurrency <= 0)
{
throw new ArgumentOutOfRangeException("concurrency", "The specified concurrency level must be greater than zero.");
}
this.locker = new object();
this.pending = new BlockingCollection<Uri>(new ConcurrentQueue<Uri>());
this.threads = new Thread[concurrency];
this.cancellation = new CancellationTokenSource();
this.isStarted = 0;
this.isDisposing = 0;
this.InitializeThreads();
}
~Retriever()
{
this.Dispose(false);
}
private void InitializeThreads()
{
for (int i = 0; i < this.threads.Length; i++)
{
Thread thread = new Thread(this.ProcessQueue)
{
IsBackground = true
};
this.threads[i] = thread;
}
}
private void StartThreads()
{
foreach (Thread thread in this.threads)
{
if (thread.ThreadState == ThreadState.Unstarted)
{
thread.Start();
}
}
}
private void CancelOperations(bool reset)
{
this.cancellation.Cancel();
this.cancellation.Dispose();
if (reset)
{
this.cancellation = new CancellationTokenSource();
}
}
private void WaitForThreadsToExit()
{
foreach (Thread thread in this.threads)
{
thread.Join();
}
}
private void ProcessQueue()
{
while (this.isDisposing == 0)
{
while (this.isStarted == 0)
{
Monitor.Wait(this.locker);
}
Uri location;
try
{
location = this.pending.Take(this.cancellation.Token);
}
catch (OperationCanceledException)
{
continue;
}
T data;
try
{
data = this.Retrieve(location, this.cancellation.Token);
}
catch (OperationCanceledException)
{
continue;
}
RetrieverEventArgs<T> args = new RetrieverEventArgs<T>(location, data);
EventHandler<RetrieverEventArgs<T>> callback = this.Retrieved;
if (!Object.ReferenceEquals(callback, null))
{
callback(this, args);
}
}
}
private void ThowIfDisposed()
{
if (this.isDisposing == 1)
{
throw new ObjectDisposedException("Retriever");
}
}
protected abstract T Retrieve(Uri location, CancellationToken token);
protected virtual void Dispose(bool disposing)
{
if (Interlocked.CompareExchange(ref this.isDisposing, 1, 0) == 1)
{
return;
}
if (disposing)
{
this.CancelOperations(false);
this.WaitForThreadsToExit();
this.pending.Dispose();
}
}
public void Start()
{
this.ThowIfDisposed();
if (Interlocked.CompareExchange(ref this.isStarted, 1, 0) == 1)
{
throw new InvalidOperationException("The retriever is already started.");
}
Monitor.PulseAll(this.locker);
this.StartThreads();
}
public void Add(Uri location)
{
this.pending.Add(location);
}
public void Stop()
{
this.ThowIfDisposed();
if (Interlocked.CompareExchange(ref this.isStarted, 0, 1) == 0)
{
throw new InvalidOperationException("The retriever is already stopped.");
}
this.CancelOperations(true);
}
public void Dispose()
{
this.Dispose(true);
GC.SuppressFinalize(this);
}
}

How to use a mutex

I have one thread, that is sending data stored in a buffer of type List< string> via tcp. Another thread is writing into the buffer. As I am not very familiar with c# I'd like to know how I should use lock or Mutex correctly.
This is the code I'd like to use eventually:
while(buffer.isLocked())
{
buffer.wait();
}
buffer.lockBuffer();
buffer.add(tcpPacket);
buffer.unlockBuffer();
buffer.notify();
This is my current code. I hope someone can help me complete it.
public class Buffer
{
private Mutex mutex;
private List<string> buffer;
private bool locked = false;
public Buffer()
{
mutex = new Mutex(false);
buffer = new List<string>();
}
public bool isLocked()
{
return locked;
}
public void lockBuffer()
{
if (!locked)
{
//...
locked = true;
}
}
public void unlockBuffer()
{
if(locked)
{
mutex.ReleaseMutex();
locked = false;
}
}
public void wait()
{
mutex.WaitOne();
}
public void notify()
{
//...
}
}

It would be better if you use System.Collections.Concurrent.BlockingCollection. It doesn't require an external sync.
For those who don't use 4.0
using System;
using System.Collections.Generic;
using System.Threading;
namespace MyCollections
{
public class BlockingQueue<T> : IDisposable
{
Queue<T> _Queue = new Queue<T>();
SemaphoreSlim _ItemsInQueue = null;
SemaphoreSlim _FreeSlots = null;
int _MaxItems = -1;
public BlockingQueue(int maxItems=Int32.MaxValue)
{
_MaxItems = maxItems;
_ItemsInQueue = new SemaphoreSlim(0, maxItems);
_FreeSlots = new SemaphoreSlim(maxItems, maxItems);
}
public void Dispose()
{
if (_ItemsInQueue != null) _ItemsInQueue.Dispose();
if (_FreeSlots != null) _FreeSlots.Dispose();
}
public int Count
{
get { return _ItemsInQueue.CurrentCount; }
}
public void Add(T item)
{
if(_MaxItems != Int32.MaxValue) _FreeSlots.Wait();
lock (this)
{
_Queue.Enqueue(item);
_ItemsInQueue.Release();
}
}
public T Take()
{
T item = default(T);
_ItemsInQueue.Wait();
lock (this)
{
item = _Queue.Dequeue();
if (_MaxItems != Int32.MaxValue) _FreeSlots.Release();
}
return item;
}
}
}

The following code is not thread-safe. If two threads are entering this method at the same time, both might pass the if condition successfully.
public void lockBuffer()
{
if (!locked)
{
//...
locked = true;
}
}
You simply might want to do something like this:
lock (_sycnObject)
{
buffer.lockBuffer();
buffer.add(tcpPacket);
buffer.unlockBuffer();
buffer.notify();
}
I don't think you're doing something sophisticated that requires more than the simple to use lock-statement.

I wouldn't use Mutexes since I suppose you aren't dealing with multiple processes synchronization. Locks are pretty fine and simpler to implement:
class Buffer
{
private readonly object syncObject = new object();
private readonly List<string> buffer = new List<string>();
public void AddPacket(string packet)
{
lock (syncObject)
{
buffer.Add(packet);
}
}
public void Notify()
{
// Do something, if needed lock again here
// lock (syncObject)
// {
// Notify Implementation
// }
}
}
The usage is obviously (as you requested):
var myBuffer = new Buffer();
myBuffer.Add("Hello, World!");
myBuffer.Notify();

Frequent lag spikes in multi-threaded code

I'm attempting to make my simple C# graphics library multi-threaded. However, after the introduction of this code:
/* foreach (IAffector affector in affectorLookup.Values)
affector.Update(timestep); */
taskManager.Value = timestep; taskManager.Start();
foreach (IAffector affector in affectorLookup.Values)
taskManager.AddToQueue(affector.Update);
taskManager.StopWhenDone();
taskManager.Wait();
the simulation starts experiencing sharp lag-spikes, which seem to originate in TaskHandler.Run (I can't tell for sure, because adding the previous code makes my code profiler ignore anything outside TaskHandler.Run).
The task manager:
public class TaskManager
{
public delegate void MethodDel(float timestep);
private Queue<MethodDel> queue;
private List<TaskHandler> handlers;
private float value;
public float Value
{
get
{
return value;
}
set
{
this.value = value;
}
}
public TaskManager()
{
this.queue = new Queue<MethodDel>();
this.handlers = new List<TaskHandler>(System.Environment.ProcessorCount);
for (int t = 0; t < this.handlers.Capacity; ++t)
this.handlers.Add(new TaskHandler(this));
this.value = 0;
}
public void Start()
{
foreach (var handler in handlers)
handler.Start();
}
public void Stop()
{
lock (queue)
queue.Clear();
foreach (var handler in handlers)
handler.StopWhenDone();
}
public void StopWhenDone()
{
foreach (var handler in handlers)
handler.StopWhenDone();
}
public void AddToQueue(MethodDel method)
{
lock (queue)
queue.Enqueue(method);
}
public bool GetFromQueue(out MethodDel method)
{
lock (queue)
{
if (queue.Count == 0) { method = null; return false; }
method = queue.Dequeue();
return true;
}
}
public int GetQueueCount()
{
return queue.Count;
}
internal void Wait()
{
// Have to wait for them one at a time because the main thread is STA.
WaitHandle[] waitHandles = new WaitHandle[1];
// for (int t = 0; t < handlers.Count; ++t)
// waitHandles[t] = handlers[t].WaitHandle;
// WaitHandle.WaitAll(waitHandles);
for (int t = 0; t < handlers.Count; ++t)
{ waitHandles[0] = handlers[t].WaitHandle; WaitHandle.WaitAll(waitHandles); }
}
}
And the task handler:
public class TaskHandler
{
private TaskManager manager;
private Thread thread;
private bool stopWhenDone;
private ManualResetEvent waitHandle;
public ManualResetEvent WaitHandle
{
get
{
return waitHandle;
}
}
public TaskHandler(TaskManager manager)
{
this.manager = manager;
}
public void Start()
{
waitHandle = new ManualResetEvent(false);
stopWhenDone = false;
thread = new Thread(Run);
thread.IsBackground = true;
thread.SetApartmentState(ApartmentState.MTA);
thread.Start();
}
public void StopWhenDone()
{
this.stopWhenDone = true;
}
// Possible source of slowdown
private void Run()
{
TaskManager.MethodDel curMethod;
while (!stopWhenDone || manager.GetQueueCount() > 0)
{
if (manager.GetFromQueue(out curMethod))
{
curMethod(manager.Value);
}
}
waitHandle.Set();
}
}

Starting a thread is a heavy operation. Not sure if it's as heavy as you are experiencing, but that could be it. Also, having all your processing run parallel can be putting a big strain on your system with possibly little benefit...

I'm going to venture that the spikes have something to do with waitHandle.Set();
I like the overall design, but I have not used WaitHandle before, so I am unsure how this interacts with your design.

Synchronization Exception

I have two threads, one thread processes a queue and the other thread adds stuff into the queue.
I want to put the queue processing thread to sleep when its finished processing the queue
I want to have the 2nd thread tell it to wake up when it has added an item to the queue
However these functions call System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code on the Monitor.PulseAll(waiting); call, because I havent synchronized the function with the waiting object. [which I dont want to do, i want to be able to process while adding items to the queue]. How can I achieve this?
Queue<object> items = new Queue<object>();
object waiting = new object();
1st Thread
public void ProcessQueue()
{
while (true)
{
if (items.Count == 0)
Monitor.Wait(waiting);
object real = null;
lock(items) {
object item = items.Dequeue();
real = item;
}
if(real == null)
continue;
.. bla bla bla
}
}
2nd Thread involves
public void AddItem(object o)
{
... bla bla bla
lock(items)
{
items.Enqueue(o);
}
Monitor.PulseAll(waiting);
}

The answer is in the error message you posted:
"Object synchronization method was called from an unsynchronized block of code on the Monitor.PulseAll(waiting);"
You have to call Monitor.PulseAll(waiting) from inside the lock(waiting) block.
Also... you have to call Monitor.Wait from within a lock block as well.

If you have access to .NET 4.0, what you want to do can be achieved by BlockingCollection<T>.
If you want to do it yourself by means of the Monitor class and signaling with Pulse(), you are actually on the right track.
You get the exception because to call Wait(), Pulse() and PulseAll(), you have to own the lock on the specified object. You happen to miss this on waiting.
A sample basic thread-safe queue that can be used:
with foreach on the consumer,
with while or your favorite conditional construct on the producer side,
handles multiple producers/consumers and
uses lock(), Monitor.Pulse(), Monitor.PulseAll() and Monitor.Wait():
.
public class SignaledQueue<T>
{
Queue<T> queue = new Queue<T>();
volatile bool shutDown = false;
public bool Enqueue(T item)
{
if (!shutDown)
{
lock (queue)
{
queue.Enqueue(item);
//Pulse only if there can be waiters.
if (queue.Count == 1)
{
Monitor.PulseAll(queue);
}
}
return true;
}
//Indicate that processing should stop.
return false;
}
public IEnumerable<T> DequeueAll()
{
while (!shutDown)
{
do
{
T item;
lock (queue)
{
//If the queue is empty, wait.
if (queue.Count == 0)
{
if (shutDown) break;
Monitor.Wait(queue);
if (queue.Count == 0) break;
}
item = queue.Dequeue();
}
yield return item;
} while (!shutDown);
}
}
public void SignalShutDown()
{
shutDown = true;
lock (queue)
{
//Signal all waiting consumers with PulseAll().
Monitor.PulseAll(queue);
}
}
}
Sample usage:
class Program
{
static void Main(string[] args)
{
int numProducers = 4, numConsumers = 2;
SignaledQueue<int> queue = new SignaledQueue<int>();
ParameterizedThreadStart produce = delegate(object obj)
{
Random rng = new Random((int)obj);
int num = 0;
while (queue.Enqueue(++num))
{
Thread.Sleep(rng.Next(100));
}
};
ThreadStart consume = delegate
{
foreach (int num in queue.DequeueAll())
{
Console.Write(" {0}", num);
}
};
Random seedRng = new Random();
for (int i = 0; i < numProducers; i++)
{
new Thread(produce).Start(seedRng.Next());
}
for (int i = 0; i < numConsumers; i++)
{
new Thread(consume).Start();
}
Console.ReadKey(true);
queue.SignalShutDown();
}
}

Use Semaphore http://msdn.microsoft.com/library/system.threading.semaphore.aspx it was designed exactly for this

I prefer to use a callback that launches a processing thread that continues until it's caught up, with locks causing simultaneous readers and writers to wait in line:
public delegate void CallbackDelegate();
class Program
{
static void Main(string[] args)
{
Queue<object> items = new Queue<object>();
Processor processor = new Processor(items);
Adder adder = new Adder(items, new CallbackDelegate(processor.CallBack));
Thread addThread = new Thread(new ParameterizedThreadStart(adder.AddItem));
object objectToAdd = new object();
addThread.Start(objectToAdd);
}
}
class Processor
{
Queue<object> items;
public Processor(Queue<object> itemsArg)
{
items = itemsArg;
}
public void ProcessQueue()
{
lock (items)
{
while (items.Count > 0)
{
object real = items.Dequeue();
// process real
}
}
}
public void CallBack()
{
Thread processThread = new Thread(ProcessQueue);
processThread.IsBackground = true;
processThread.Start();
}
}
class Adder
{
Queue<object> items;
CallbackDelegate callback;
public Adder(Queue<object> itemsArg, CallbackDelegate callbackArg)
{
items = itemsArg;
callback = callbackArg;
}
public void AddItem(object o)
{
lock (items) { items.Enqueue(o); }
callback();
}
}

Thread Pooling help

Having some issue with Threadpooling here that I need some help with please. I am trying to write a Generator, and I need to allow users generate up to 10,000 lines with the code below. Problem with this is the line
WaitHandle.WaitAll(doneEvents);
Can only handle 64 WaitAll at a time, How can I best apply thread pooling to my code in this case?
public void GenerateInsertStatements(int iRequiredRows)
{
// One event is used for each row object
ManualResetEvent[] doneEvents = new ManualResetEvent[iRequiredRows];
Row[] rows = new Row[iRequiredRows];
for (int i = 0; i < iRequiredRows; i++)
{
doneEvents[i] = new ManualResetEvent(false);
Row row = new Row(this.Name, this.TableColumns, doneEvents[i]);
rows[i] = row;
ThreadPool.QueueUserWorkItem(row.ThreadPoolCallback, i);
}
WaitHandle.WaitAll(doneEvents);
using (sr = new StreamWriter(this.Name + ".sql"))
{
for(int i=0; i<rows.Length; i++)
{
WriteStatementToFile(i, rows[i].GeneratedInsertStatement);
}
}
}
Thanks in advance

I would use just one WaitHandle and one int. Like:
int done_when_zero; // This is a field of the class
ManualResetEvent evt = new ManualResetEvent (false); // Field
...
done_when_zero = iRequiredRows; // This goes before the loop
...
evt.WaitOne (); // this goes after the loop
evt.Reset (); // Prepare for next execution if needed
And then, at the end of ThreadPoolCallback:
if (Interlocked.Decrement (ref done_when_zero)) <= 0)
evt.Set ();

As it was already suggested using a counter and a single ManualResetEvent should work fine for you. Below is ThreadPoolWait class taken from .NET Matters: ThreadPoolWait and HandleLeakTracer (see Figure 3 Better Implementation of ThreadPoolWait for more info)
public class ThreadPoolWait : IDisposable
{
private int _remainingWorkItems = 1;
private ManualResetEvent _done = new ManualResetEvent(false);
public void QueueUserWorkItem(WaitCallback callback)
{
QueueUserWorkItem(callback, null);
}
public void QueueUserWorkItem(WaitCallback callback, object state)
{
ThrowIfDisposed();
QueuedCallback qc = new QueuedCallback();
qc.Callback = callback;
qc.State = state;
lock (_done) _remainingWorkItems++;
ThreadPool.QueueUserWorkItem(new WaitCallback(HandleWorkItem), qc);
}
public bool WaitOne() { return WaitOne(-1, false); }
public bool WaitOne(TimeSpan timeout, bool exitContext)
{
return WaitOne((int)timeout.TotalMilliseconds, exitContext);
}
public bool WaitOne(int millisecondsTimeout, bool exitContext)
{
ThrowIfDisposed();
DoneWorkItem();
bool rv = _done.WaitOne(millisecondsTimeout, exitContext);
lock (_done)
{
if (rv)
{
_remainingWorkItems = 1;
_done.Reset();
}
else _remainingWorkItems++;
}
return rv;
}
private void HandleWorkItem(object state)
{
QueuedCallback qc = (QueuedCallback)state;
try { qc.Callback(qc.State); }
finally { DoneWorkItem(); }
}
private void DoneWorkItem()
{
lock (_done)
{
--_remainingWorkItems;
if (_remainingWorkItems == 0) _done.Set();
}
}
private class QueuedCallback
{
public WaitCallback Callback;
public object State;
}
private void ThrowIfDisposed()
{
if (_done == null) throw new ObjectDisposedException(GetType().Name);
}
public void Dispose()
{
if (_done != null)
{
((IDisposable)_done).Dispose();
_done = null;
}
}
}

Probably not the most efficient solution, but it should work regardless of the 64 wait handles limit :
for(int i = 0; i < iRequiredRows; i++)
doneEvents[i].WaitOne();