We Keep Coding

How to buffer a burst of events into fewer resulting actions

I want to reduce multiple events into a single delayed action. After some trigger occurs I expect some more similar triggers to come, but I prefer not to repeat the resulting delayed action. The action waits, to give a chance of completion to the burst.
The question: How can I do it in an elegant reusable way?
Till now I used a property to flag the event and trigger a delayed action like below:
public void SomeMethod()
{
SomeFlag = true; //this will intentionally return to the caller before completing the resulting buffered actions.
}
private bool someFlag;
public bool SomeFlag
{
get { return someFlag; }
set
{
if (someFlag != value)
{
someFlag = value;
if (value)
SomeDelayedMethod(5000);
}
}
}
public async void SomeDelayedMethod(int delay)
{
//some bufferred work.
await Task.Delay(delay);
SomeFlag = false;
}
below is a shorter way, but still not generic or reusable... I want something concise that packages the actions and the flag, and keeps the functionality (returning to the caller before execution is complete (like today)). I also need to be able to pass an object reference to this action)
public void SerializeAccountsToConfig()
{
if (!alreadyFlagged)
{
alreadyFlagged = true;
SerializeDelayed(5000, Serialize);
}
}
public async void SerializeDelayed(int delay, Action whatToDo)
{
await Task.Delay(delay);
whatToDo();
}
private bool alreadyFlagged;
private void Serialize()
{
//some buferred work.
//string json = JsonConvert.SerializeObject(Accounts, Formatting.Indented);
//Settings1.Default.Accounts = json;
//Settings1.Default.Save();
alreadyFlagged = false;
}

Here's a thread-safe and reusable solution.
You can create an instance of DelayedSingleAction, and in the constructor you pass the action that you want to have performed. I believe this is thread safe, though there is a tiny risk that it will restart the timer just before commencing the action, but I think that risk would exist no matter what the solution is.
public class DelayedSingleAction
{
private readonly Action _action;
private readonly long _millisecondsDelay;
private long _syncValue = 1;
public DelayedSingleAction(Action action, long millisecondsDelay)
{
_action = action;
_millisecondsDelay = millisecondsDelay;
}
private Task _waitingTask = null;
private void DoActionAndClearTask(Task _)
{
Interlocked.Exchange(ref _syncValue, 1);
_action();
}
public void PerformAction()
{
if (Interlocked.Exchange(ref _syncValue, 0) == 1)
{
_waitingTask = Task.Delay(TimeSpan.FromMilliseconds(_millisecondsDelay))
.ContinueWith(DoActionAndClearTask);
}
}
public Task Complete()
{
return _waitingTask ?? Task.FromResult(0);
}
}
See this dotnetfiddle for an example which invokes one action continuously from multiple threads.
https://dotnetfiddle.net/el14wZ

Since you're interested in RX here simple console app sample:
static void Main(string[] args)
{
// event source
var burstEvents = Observable.Interval(TimeSpan.FromMilliseconds(50));
var subscription = burstEvents
.Buffer(TimeSpan.FromSeconds(3)) // collect events 3 seconds
//.Buffer(50) // or collect 50 events
.Subscribe(events =>
{
//Console.WriteLine(events.First()); // take only first event
// or process event collection
foreach (var e in events)
Console.Write(e + " ");
Console.WriteLine();
});
Console.ReadLine();
return;
}

Based on the solution proposed by Andrew, here is a more generic solution.
Declaration and instance creation of the delayed action:
public DelayedSingleAction<Account> SendMailD;
Create the instance inside a function or in the constructor (this can be a collection of such actions each working on a different object):
SendMailD = new DelayedSingleAction<Account>(SendMail, AccountRef, 5000);
repeatedly call this action
SendMailD.PerformAction();
Send mail is the action you will "burst control". Its signature matches :
public int SendMail(Account A)
{}
Here is the updated class
public class DelayedSingleAction<T>
{
private readonly Func<T, int> actionOnObj;
private T tInstance;
private readonly long millisecondsDelay;
private long _syncValue = 1;
public DelayedSingleAction(Func<T, int> ActionOnObj, T TInstance, long MillisecondsDelay)
{
actionOnObj = ActionOnObj;
tInstance = TInstance;
millisecondsDelay = MillisecondsDelay;
}
private Task _waitingTask = null;
private void DoActionAndClearTask(Task _)
{
Console.WriteLine(string.Format("{0:h:mm:ss.fff} DelayedSingleAction Resetting SyncObject: Thread {1} for {2}", DateTime.Now, System.Threading.Thread.CurrentThread.ManagedThreadId, tInstance));
Interlocked.Exchange(ref _syncValue, 1);
actionOnObj(tInstance);
}
public void PerformAction()
{
if (Interlocked.Exchange(ref _syncValue, 0) == 1)
{
Console.WriteLine(string.Format("{0:h:mm:ss.fff} DelayedSingleAction Starting the timer: Thread {1} for {2}", DateTime.Now, System.Threading.Thread.CurrentThread.ManagedThreadId, tInstance));
_waitingTask = Task.Delay(TimeSpan.FromMilliseconds(millisecondsDelay)).ContinueWith(DoActionAndClearTask);
}
}
public Task Complete()
{
return _waitingTask ?? Task.FromResult(0);
}
}

C# Parallel.For not executing every step

I have been working on a mock-up for an import service which currently runs in sequence. However my mock-up seems to exhibit a strange problem where by sometimes one or two items in the for loop aren't executed.
class Service
{
private Thread _worker;
private bool _stopping;
private CancellationTokenSource _cts;
private ParallelOptions _po;
private Repository _repository;
public void Start(Repository repository)
{
_repository = repository;
_cts = new CancellationTokenSource();
_po = new ParallelOptions {
CancellationToken = _cts.Token
};
_worker = new Thread(ProcessImport);
_worker.Start();
}
public void Stop()
{
_stopping = true;
_cts.Cancel();
if(_worker != null && _worker.IsAlive)
_worker.Join();
}
private void ProcessImport()
{
while (!_stopping)
{
var import = _repository.GetInProgressImport();
if (import == null)
{
Thread.Sleep(1000);
continue;
}
try
{
Parallel.For(0, 1000, _po, i => Work.DoWork(i, import, _cts.Token, _repository));
}
catch (OperationCanceledException)
{
// Unmark batch so it can be started again
batch = _repository.GetBatch(import.BatchId);
batch.Processing = false;
_repository.UpdateBatch(batch);
Console.WriteLine("Aborted import {0}", import.ImportId);
}
catch (Exception ex)
{
Console.WriteLine("Something went wrong: {0}", ex.Message);
}
}
}
}
class Work
{
public static void DoWork(int i, Import import, CancellationToken ct, Repository repository)
{
// Simulate doing some work
Thread.Sleep(100);
HandleAbort(ct);
Thread.Sleep(100);
HandleAbort(ct);
Thread.Sleep(100);
// Update the batch
var batch = repository.GetBatch(import.BatchId);
batch.Processed++;
if (batch.Processed == batch.Total)
{
batch.Finished = DateTime.Now;
batch.Processing = false;
}
repository.UpdateBatch(batch);
}
private static void HandleAbort(CancellationToken ct)
{
if (!ct.IsCancellationRequested)
return;
ct.ThrowIfCancellationRequested();
}
}
With this code, I often find that the batches are never complete and that batch.Processed = 999 or 998.
Can anyone shed any light on what I've done wrong.
Thanks in advance.
Edit:
To be clear about the repository/batch object - I believe in my current mock-up that it is threadsafe
class Repository
{
private ConcurrentBag<Batch> _batchData = new ConcurrentBag<Batch>();
private ConcurrentBag<Import> _importData = new ConcurrentBag<Import>();
public void CreateImport(Import import)
{
_importData.Add(import);
}
public Import GetInProgressImport()
{
var import = _importData
.Join(_batchData, i => i.BatchId, b => b.BatchId, (i, b) => new
{
Import = i,
Batch = b
})
.Where(j => j.Batch.Processed < j.Batch.Total && !j.Batch.Processing)
.OrderByDescending(j => j.Batch.Total - j.Batch.Processed)
.ThenBy(j => j.Batch.BatchId - j.Batch.BatchId)
.Select(j => j.Import)
.FirstOrDefault();
if (import == null)
return null;
// mark the batch as processing
var batch = GetBatch(import.BatchId);
batch.Processing = true;
UpdateBatch(batch);
return import;
}
public List<Import> ListImports()
{
return _importData.ToList();
}
public void CreateBatch(Batch batch)
{
_batchData.Add(batch);
}
public Batch GetBatch(Int64 batchId)
{
return _batchData.FirstOrDefault(b => b.BatchId == batchId);
}
public void UpdateBatch(Batch batch)
{
var batchData = _batchData.First(b => b.BatchId == batch.BatchId);
batchData.Total = batch.Total;
batchData.Processed = batch.Processed;
batchData.Started = batch.Started;
batchData.Finished = batch.Finished;
batchData.Processing = batch.Processing;
}
}
class Import
{
public Int64 ImportId { get; set; }
public Int64 BatchId { get; set; }
}
class Batch
{
public Int64 BatchId { get; set; }
public int Total { get; set; }
public int Processed { get; set; }
public DateTime Created { get; set; }
public DateTime Started { get; set; }
public DateTime Finished { get; set; }
public bool Processing { get; set; }
}
This is only a mock-up so there is no DB or other persistence behind my repository.
Also, I'm not competing my batch on the value of i, but rather the number of iterations of the loop (the work actually having been done) indicated by the Processed property of the batch object.
Thanks
Solution:
I had forgotten about the need synchronise the update of the batch. Should look like:
class Work
{
private static object _sync = new object();
public static void DoWork(int i, Import import, CancellationToken ct, Repository repository)
{
// Do work
Thread.Sleep(100);
HandleAbort(ct);
Thread.Sleep(100);
HandleAbort(ct);
Thread.Sleep(100);
lock (_sync)
{
// Update the batch
var batch = repository.GetBatch(import.BatchId);
batch.Processed++;
if (batch.Processed == batch.Total)
{
batch.Finished = DateTime.Now;
batch.Processing = false;
}
repository.UpdateBatch(batch);
}
}
private static void HandleAbort(CancellationToken ct)
{
if (!ct.IsCancellationRequested)
return;
ct.ThrowIfCancellationRequested();
}
}

Looks like lost updates on batch.Processed. Increments are not atomic. batch.Processed++; is racy. Use Interlocked.Increment.
It seems to me like you don't have a good understanding of threading right now. It's very dangerous to perform such elaborate threading without a good understanding. The mistakes you make are hard to test for but production will find them.

According to MSDN, the overloads of Parallel.For specify the second integer as toExclusive, meaning to goes up to but does not meet that value. In other words, 999 is the expected result, not 1000 - but note also that by starting at "0", your loop does execute 1,000 times.
From a glance, your code is parallel, so make sure you're not seeing the "999" call out of order from the "998" one - this is because by being executed in parallel, your code is inherently unordered, and can easily end up being very randomly rearranged. Also, read up on lock, as your code may be accessing values which it should be waiting for.

Pattern for 'routing' events through several subscribers?

I have a group of classes with the following interface:
public interface RoutedEventReceiver<T>
{
IDisposable Apply(IObservable<T> stream);
bool ShouldForwardEvent(T anEvent);
}
What I would like to do is to maintain a stack of these classes, with each event being filtered through the ShouldForwardEvent(T) predicate, and the resulting IObservable<T> passed to the next receiver. I also want to be able to push and pop new receivers while my program is running (at some point I may want to move from a stack to some other collection but for now a stack is sufficient).
What I have currently does work, but I don't feel like it is very "Rx". I am sure there must be a way to do what I want without all this imperative logic:
private void Refresh()
{
// _subscriptions is a list of previous subscriptions
foreach (var subscription in _subscriptions)
subscription.Dispose();
_subscriptions.Clear();
// _stream is my stream of incoming events
if (_stream != null)
{
var stream = _stream;
foreach (var eventReceiver in _eventReceivers)
{
// add the subscription so it can be disposed next Refresh()
_subscriptions.Add(eventReceiver.Apply(stream));
// filter the stream for the next event receiver
stream = stream.Where(eventReceiver.ShouldForwardEvent);
}
}
}
The above method is called whenever I Push or Pop on the stack.
Is there a cleaner, more functional way to express the above intent? I have tried .Publish() but with little success - perhaps I don't know it well enough.

I have managed to make the Publish approach work, but it doesn't afford me much other than getting rid of the need to keep a list of IDisposables:
private void Refresh()
{
_published.DisposeIfNotNull();
if (_stream != null)
{
var connectable = _stream.Publish();
_published = connectable.Connect();
var stream = connectable.AsObservable();
foreach (var eventReceiver in _eventReceivers)
{
eventReceiver.Apply(stream);
stream = stream.Where(eventReceiver.ShouldForwardEvent);
}
}
}

The class below (named CORStack for Chain Of Responsibility* Stack), tries to do what you're after. Internally it adds an ShouldHandle bool to the stream and uses this to determine whether to process. It exposes the standard Push, Pop, and Peek methods.
public sealed class CORStack<T>
{
Stack<StackFrame> _handlers;
public CORStack(IObservable<T> source)
{
_handlers = new Stack<StackFrame>();
_handlers.Push(new StackFrame(
source.Select(t => new ShouldHandleWrapper(t, true)),
new Handler<T>(new Action<T>(t => { }), true)));
}
public void Push(Handler<T> handler)
{
_handlers.Push(new StackFrame(_handlers.Peek().Observable, handler));
}
public Handler<T> Peek()
{
return _handlers.Peek().Handler;
}
public Handler<T> Pop()
{
var frame = _handlers.Pop();
frame.Dispose();
return frame.Handler;
}
class StackFrame : IDisposable
{
IDisposable _unsub;
public IObservable<ShouldHandleWrapper> Observable { get; private set; }
public Handler<T> Handler { get; private set; }
public StackFrame(IObservable<ShouldHandleWrapper> topOfStack, Handler<T> handler)
{
_unsub = topOfStack.Subscribe(shouldHandle =>
{
if (shouldHandle.ShouldHandle)
handler.Action.Invoke(shouldHandle.Value);
});
Observable = topOfStack.Select(shouldHandle =>
new ShouldHandleWrapper(shouldHandle.Value, shouldHandle.ShouldHandle && handler.Forward));
Handler = handler;
}
public void Dispose()
{
_unsub.Dispose();
}
}
class ShouldHandleWrapper
{
public readonly T Value;
public readonly bool ShouldHandle;
public ShouldHandleWrapper(T value, bool shouldHandle)
{
Value = value;
ShouldHandle = shouldHandle;
}
}
}
public class Handler<T>
{
public Action<T> Action { get; set; }
public bool Forward { get; set; }
public Handler(Action<T> action, bool forward)
{
Action = action;
Forward = forward;
}
}
*I realised that it's not a chain of responsibility, but can't think of a better name atm.

This is a case where I'd actually use Subjects. Create a subject for each handler, then subscribe to the stream and loop through the handlers passing the event as required. This avoids continually unsubscribing/resubscribing to the stream (and thus the Refresh method), which is not always appropriate. We use lock to guard against a new receiver being added or removed at the same moment as a new value is coming through the stream. If you can guarantee that cannot happen, then you can remove the lock statements.
public class YourClass<T> : IDisposable
{
private readonly Stack<Tuple<Subject<T>, RoutedEventReceiver<T>, IDisposable> _handlers;
private readonly IObservable<T> _stream;
private readonly IDisposable _streamSubscription;
public YourClass(IObservable<T> stream)
{
_handlers = new Stack<Tuple<Subject<T>, RoutedEventReceiver<T>, IDisposable>();
_stream = stream;
_streamSubscription = stream.Subscribe(OnNext, OnError, OnCompleted);
}
public void Dispose()
{
_streamSubscription.Dispose();
lock (_handlers)
{
foreach (var h in _handlers)
{
h.Item3.Dispose();
h.Item1.Dispose();
}
_handlers.Clear();
}
}
private void OnNext(T value)
{
lock (_handlers)
{
for (var h in _handlers)
{
h.Item1.OnNext(value);
if (!h.Item2.ShouldForwardEvent(value)) break;
}
}
}
private void OnError(Exception e)
{
lock (_handlers)
{
for (var h in _handlers) { h.Item1.OnError(e); }
}
}
private void OnCompleted()
{
lock (_handlers)
{
for (var h in _handlers) { h.Item1.OnCompleted(); }
}
}
public void Push(RoutedEventReceiver<T> handler)
{
lock (_handlers)
{
var subject = new Subject<T>;
_handlers.Push(Tuple.Create(subject, handler, handler.Apply(subject)));
}
}
public RoutedEventReceiver<T> Pop()
{
lock (_handlers)
{
var handler = _handlers.Pop();
handler.Item3.Dispose();
handler.Item1.Dispose();
return handler.Item2;
}
}
}

Synchronizing thread communication?

Just for the heck of it I'm trying to emulate how JRuby generators work using threads in C#.
Also, I'm fully aware that C# has built in support for yield return, I'm just toying around a bit.
I guess it's some sort of poor mans coroutines by keeping multiple callstacks alive using threads. (even though none of the callstacks should execute at the same time)
The idea is like this:
The consumer thread requests a value
The worker thread provides a value and yields back to the consumer thread
Repeat untill worker thread is done
So, what would be the correct way of doing the following?
//example
class Program
{
static void Main(string[] args)
{
ThreadedEnumerator<string> enumerator = new ThreadedEnumerator<string>();
enumerator.Init(() =>
{
for (int i = 1; i < 100; i++)
{
enumerator.Yield(i.ToString());
}
});
foreach (var item in enumerator)
{
Console.WriteLine(item);
};
Console.ReadLine();
}
}
//naive threaded enumerator
public class ThreadedEnumerator<T> : IEnumerator<T>, IEnumerable<T>
{
private Thread enumeratorThread;
private T current;
private bool hasMore = true;
private bool isStarted = false;
AutoResetEvent enumeratorEvent = new AutoResetEvent(false);
AutoResetEvent consumerEvent = new AutoResetEvent(false);
public void Yield(T item)
{
//wait for consumer to request a value
consumerEvent.WaitOne();
//assign the value
current = item;
//signal that we have yielded the requested
enumeratorEvent.Set();
}
public void Init(Action userAction)
{
Action WrappedAction = () =>
{
userAction();
consumerEvent.WaitOne();
enumeratorEvent.Set();
hasMore = false;
};
ThreadStart ts = new ThreadStart(WrappedAction);
enumeratorThread = new Thread(ts);
enumeratorThread.IsBackground = true;
isStarted = false;
}
public T Current
{
get { return current; }
}
public void Dispose()
{
enumeratorThread.Abort();
}
object System.Collections.IEnumerator.Current
{
get { return Current; }
}
public bool MoveNext()
{
if (!isStarted)
{
isStarted = true;
enumeratorThread.Start();
}
//signal that we are ready to receive a value
consumerEvent.Set();
//wait for the enumerator to yield
enumeratorEvent.WaitOne();
return hasMore;
}
public void Reset()
{
throw new NotImplementedException();
}
public IEnumerator<T> GetEnumerator()
{
return this;
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return this;
}
}
Ideas?

There are many ways to implement the producer/consumer pattern in C#.
The best way, I guess, is using TPL (Task, BlockingCollection). See an example here.

How can I write a conditional lock in C#?

The thing is I've been using the lock statement to protect a critical part of my code, but now, I realize I could allow concurrent execution of that critical code is some conditions are met.
Is there a way to condition the lock?

bool locked = false;
if (condition) {
Monitor.Enter(lockObject);
locked = true;
}
try {
// possibly critical section
}
finally {
if (locked) Monitor.Exit(lockObject);
}
EDIT: yes, there is a race condition unless you can assure that the condition is constant while threads are entering.

I'm no threading expert, but it sounds like you might be looking for something like this (double-checked locking). The idea is to check the condition both before and after acquiring the lock.
private static object lockHolder = new object();
if (ActionIsValid()) {
lock(lockHolder) {
if (ActionIsValid()) {
DoSomething();
}
}
}

Action doThatThing = someMethod;
if (condition)
{
lock(thatThing)
{
doThatThing();
}
}
else
{
doThatThing();
}

Actually, to avoid a race condition, I'd be tempted to use a ReaderWriterLockSlim here - treat concurrent access as a read lock, and exclusive access as a write lock. That way, if the conditions change you won't end up with some inappropriate code still executing blindly in the region (under the false assumption that it is safe); a bit verbose, but
(formatted for space):
if (someCondition) {
lockObj.EnterReadLock();
try { Foo(); }
finally { lockObj.ExitReadLock(); }
} else {
lockObj.EnterWriteLock();
try { Foo(); }
finally { lockObj.ExitWriteLock(); }
}

If you have many methods/properties that require conditional locking, you don't want to repeat the same pattern over and over again. I propose the following trick:
Non-repetitive conditional-lock pattern
With a private helper struct implementing IDisposable we can encapsulate the condition/lock without measurable overhead.
public void DoStuff()
{
using (ConditionalLock())
{
// Thread-safe code
}
}
It's quite easy to implement. Here's a sample class demonstrating this pattern:
public class Counter
{
private static readonly int MAX_COUNT = 100;
private readonly bool synchronized;
private int count;
private readonly object lockObject = new object();
private int lockCount;
public Counter(bool synchronized)
{
this.synchronized = synchronized;
}
public int Count
{
get
{
using (ConditionalLock())
{
return count;
}
}
}
public int LockCount
{
get
{
using (ConditionalLock())
{
return lockCount;
}
}
}
public void Increase()
{
using (ConditionalLock())
{
if (count < MAX_COUNT)
{
Thread.Sleep(10);
++count;
}
}
}
private LockHelper ConditionalLock() => new LockHelper(this);
// This is where the magic happens!
private readonly struct LockHelper : IDisposable
{
private readonly Counter counter;
private readonly bool lockTaken;
public LockHelper(Counter counter)
{
this.counter = counter;
lockTaken = false;
if (counter.synchronized)
{
Monitor.Enter(counter.lockObject, ref lockTaken);
counter.lockCount++;
}
}
private void Exit()
{
if (lockTaken)
{
Monitor.Exit(counter.lockObject);
}
}
void IDisposable.Dispose() => Exit();
}
}
Now, let's create a small sample program demonstrating its correctness.
class Program
{
static void Main(string[] args)
{
var onlyOnThisThread = new Counter(synchronized: false);
IncreaseToMax(c1);
var onManyThreads = new Counter(synchronized: true);
var t1 = Task.Factory.StartNew(() => IncreaseToMax(c2));
var t2 = Task.Factory.StartNew(() => IncreaseToMax(c2));
var t3 = Task.Factory.StartNew(() => IncreaseToMax(c2));
Task.WaitAll(t1, t2, t3);
Console.WriteLine($"Counter(false) => Count = {c1.Count}, LockCount = {c1.LockCount}");
Console.WriteLine($"Counter(true) => Count = {c2.Count}, LockCount = {c2.LockCount}");
}
private static void IncreaseToMax(Counter counter)
{
for (int i = 0; i < 1000; i++)
{
counter.Increase();
}
}
}
Output:
Counter(false) => Count = 100, LockCount = 0
Counter(true) => Count = 100, LockCount = 3002
Now you can let the caller decide whether locking (costly) is needed.

I'm guessing you've got some code that looks a little like this:
private Monkey GetScaryMonkey(int numberOfHeads){
Monkey ape = null;
lock(this) {
ape = new Monkey();
ape.AddHeads(numberOfHeads);
}
return ape;
}
To make this conditional couldn't you just do this:
private Monkey GetScaryMonkey(int numberOfHeads){
if ( numberOfHeads > 1 ) {
lock(this) {
return CreateNewMonkey( numberOfHeads );
}
}
return CreateNewMonkey( numberOfHeads );
}
Should work, no?

Use Double-checked locking pattern, as suggested above. that's the trick IMO :)
make sure you have your lock object as a static, as listed in not.that.dave.foley.myopenid.com's example.