We Keep Coding

Scheduling asynchronous jobs/tasks (and ignoring exceptions) using Observable.Timer

I have several asynchronous tasks/jobs that I need to run on a schedule and it seems that I could do this nicely using Observables. When a job fetches the data, an exception could occur (eg 404), and when the resultant data is processed, an error could also occur.
I have seen this answer by Enigmativity which seems like the perfect solution to wrap the IObservable<> so that if an error occurs (when I fetch the data) I can trap it and continue (ultimately skipping the processing for that particular fetch).
I understand that when an Observable errors it is meant to terminate, but given the answer I mentioned above, it seems that there are ways around this, which would make for a decent job scheduling system. Alternative approaches are welcome, but I would like to understand how to do this with Observables.
I would also like to provide some feedback/logging about the state of the job.
Currently, I have the below method, which won't compile!
job is the object that contains information about the job (eg a list of job runs and their outcomes/success/failure, run frequency, status, errors, boolean flag indicating if the job should proceed, etc)
interval(job) returns the frequency in milliseconds that the job should run at
runSelect(job) is a boolean method that signals if a job should proceed (I think this would be better replaced with an observable? And of course there is the option of using a CancellationToken, but again I'm not sure how to integrate that!)
select(job) is the method that fetches the data
subscribe(job) is the method that processes the data
public static IDisposable BuildObservable<TJob, TSelect>(TJob job, Func<TJob, int> interval, Func<TJob, bool> runSelect, Func<TJob, Task<TSelect>> select,
Func<TSelect, Task> subscribe)
where TJob : Job
where TSelect : class
{
return Observable.Timer(TimeSpan.Zero, TimeSpan.FromMilliseconds(interval(job)))
.SelectMany(x => Observable.FromAsync(async () =>
{
JobRunDetail jobRunDetail = job.StartNewRun();
if (runSelect(job))
{
jobRunDetail.SetRunningSelect();
return new { Result = await select(job), JobRunDetail = jobRunDetail };
}
else
{
jobRunDetail.SetAbandonedSelect();
return new { Result = (TSelect)null!, JobRunDetail = jobRunDetail };
}
}).ToExceptional())
.Subscribe(async (resultAndJobRunDetail) =>
{
//none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile
resultAndJobRunDetail.Value.JobRunDetail.SetRunningSubscribe();
try
{
if (resultAndJobRunDetail.Value.Result!= null)
await subscribe(resultAndJobRunDetail.Value.Result);
resultAndJobRunDetail.Value.JobRunDetail.SetCompleted();
}
catch (Exception ee)
{
resultAndJobRunDetail.Value.JobRunDetail.SetErrorSubscribe(ee);
}
});
}
As noted, none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile because resultAndJobRunDetail.Value is still an Observable<>, but when I remove the .ToExceptional() call, the value returned is no longer an Observable.
Clearly I'm missing something.
I have seen different answers on SO that use Do() rather than Subscribe() so I'm not sure which is appropriate. I have also seen answers that suggest using Retry() or one of the "observable error handling methods" but I'm not sure how these would work if I just want my job to keep repeating ad infinitum?
Ultimately, I'm still learning how the whole Observable infrastructure fits together, so I could well be completely off track!
It's worth nothing that searching Google for "Schedule Job using Observable" it pretty fruitless as Observables use schedulers!

I'm not sure if this helps or not, but your .ToExceptional() call was in the wrong place:
public static IDisposable BuildObservable<TJob, TSelect>(TJob job, Func<TJob, int> interval, Func<TJob, bool> runSelect, Func<TJob, Task<TSelect>> select,
Func<TSelect, Task> subscribe)
where TJob : Job
where TSelect : class
{
return Observable.Timer(TimeSpan.Zero, TimeSpan.FromMilliseconds(interval(job)))
.SelectMany(x => Observable.FromAsync(async () =>
{
JobRunDetail jobRunDetail = job.StartNewRun();
if (runSelect(job))
{
jobRunDetail.SetRunningSelect();
return new { Result = await select(job), JobRunDetail = jobRunDetail }.ToExceptional();
}
else
{
jobRunDetail.SetAbandonedSelect();
return new { Result = (TSelect)null!, JobRunDetail = jobRunDetail }.ToExceptional();
}
}))
.Subscribe(async (resultAndJobRunDetail) =>
{
//none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile
resultAndJobRunDetail.Value.JobRunDetail.SetRunningSubscribe();
try
{
if (resultAndJobRunDetail.Value.Result != null)
await subscribe(resultAndJobRunDetail.Value.Result);
resultAndJobRunDetail.Value.JobRunDetail.SetCompleted();
}
catch (Exception ee)
{
resultAndJobRunDetail.Value.JobRunDetail.SetErrorSubscribe(ee);
}
});
}

RX terminolgy: Async processing in RX operator when there are frequent observable notifications

The purpose is to do some async work on a scarce resource in a RX operator, Select for example. Issues arise when observable notifications came at a rate that is faster than the time it takes for the async operation to complete.
Now I actually solved the problem. My question would be what is the correct terminology for this particular kind of issue? Does it have a name? Is it backpressure? Research I did until now indicate that this is some kind of a pressure problem, but not necessarily backpressure from my understanding. The most relevant resources I found are these:
https://github.com/ReactiveX/RxJava/wiki/Backpressure-(2.0)
http://reactivex.io/documentation/operators/backpressure.html
Now to the actual code. Suppose there is a scarce resource and it's consumer. In this case exception is thrown when resource is in use. Please note that this code should not be changed.
public class ScarceResource
{
private static bool inUse = false;
public async Task<int> AccessResource()
{
if (inUse) throw new Exception("Resource is alredy in use");
var result = await Task.Run(() =>
{
inUse = true;
Random random = new Random();
Thread.Sleep(random.Next(1, 2) * 1000);
inUse = false;
return random.Next(1, 10);
});
return result;
}
}
public class ResourceConsumer
{
public IObservable<int> DoWork()
{
var resource = new ScarceResource();
return resource.AccessResource().ToObservable();
}
}
Now here is the problem with a naive implementation to consume the resource. Error is thrown because notifications came at a faster rate than the consumer takes to run.
private static void RunIntoIssue()
{
var numbers = Enumerable.Range(1, 10);
var observableSequence = numbers
.ToObservable()
.SelectMany(n =>
{
Console.WriteLine("In observable: {0}", n);
var resourceConsumer = new ResourceConsumer();
return resourceConsumer.DoWork();
});
observableSequence.Subscribe(n => Console.WriteLine("In observer: {0}", n));
}
With the following code the problem is solved. I slow down processing by using a completed BehaviorSubject in conjunction with the Zip operator. Essentially what this code does is to take a sequential approach instead of a parallel one.
private static void RunWithZip()
{
var completed = new BehaviorSubject<bool>(true);
var numbers = Enumerable.Range(1, 10);
var observableSequence = numbers
.ToObservable()
.Zip(completed, (n, c) =>
{
Console.WriteLine("In observable: {0}, completed: {1}", n, c);
var resourceConsumer = new ResourceConsumer();
return resourceConsumer.DoWork();
})
.Switch()
.Select(n =>
{
completed.OnNext(true);
return n;
});
observableSequence.Subscribe(n => Console.WriteLine("In observer: {0}", n));
Console.Read();
}
Question
Is this backpressure, and if not does it have another terminology associated?

You're basically implementing a form of locking, or a mutex. Your code an cause backpressure, it's not really handling it.
Imagine if your source wasn't a generator function, but rather a series of data pushes. The data pushes arrive at a constant rate of every millisecond. It takes you 10 Millis to process each one, and your code forces serial processing. This causes backpressure: Zip will queue up the unprocessed datapushes infinitely until you run out of memory.

Reactive programming normalize time series values

I have a remote program that sends an updated measurement every 10 milliseconds over a socket connection. In my client program I have wrapped this socket in an observable that generated these measurements. For my usecase it's important that the measurement arrive at 10 millisecond intervals. Of course this is not happening since network delays make it arrive a little earlier or later each message.
So basically what I have on my remote pc is program that sends this on a socket connection.
-- is 10 milliseconds
o--o--o--o--o--o--o--o--o--o--...
Which becomes this on my client due to network delays.
o-o---o-o--o---o--o-o--o-o-...
Now in my observable I want to "normalise" this so it will again emit a value each 10 millisecond.
--o--o--o--o--o--o--o--o--o--o...
Of course this will mean I will have to introduce a buffer time that it will store values and emit them on 10 millisecond interval. Is there a way I can accomplish this?
Here is some test code that will emit the event according to the way I described above.
using System;
using System.Collections.Generic;
using System.Reactive.Disposables;
using System.Reactive.Linq;
using System.Threading.Tasks;
using Microsoft.Reactive.Testing;
public class Program
{
protected static event EventHandler<EventArgs> CancelEvent;
private static Random random = new Random();
private static double GetRandomNumber(double minimum, double maximum)
{
return random.NextDouble() * (maximum - minimum) + minimum;
}
public static void Main()
{
var completed = false;
var scheduler = new TestScheduler();
var observable = Observable
.Interval(TimeSpan.FromMilliseconds(7.0), scheduler)
.SelectMany(e => Observable
.Return(e, scheduler)
.Delay(TimeSpan.FromMilliseconds(GetRandomNumber(0.0, 6.0)), scheduler)
)
.TimeInterval(scheduler)
.Select(t => t.Interval.Milliseconds);
var fromEvent = Observable.FromEventPattern<EventArgs>(
p => CancelEvent += p,
p => CancelEvent -= p,
scheduler
);
var cancellable = observable.TakeUntil(fromEvent);
var results = new List<int>();
using (cancellable.Subscribe(
results.Add,
e => { throw new Exception("No exception is planned! {0}", e); },
() => { completed = true; })
)
{
scheduler.AdvanceBy(TimeSpan.FromSeconds(3.5).Ticks);
CancelEvent(null, new EventArgs());
scheduler.AdvanceBy(TimeSpan.FromSeconds(3).Ticks);
}
Console.WriteLine("Have I completed indeed? {0}", completed);
Console.WriteLine("What emit time deltas been registered before cancellation?\n\t{0}", string.Join("ms\n\t", results));
}
}

This is theoretically similar to A way to push buffered events in even intervals.
That solution would look like this:
var source = new Subject<double>();
var bufferTime = TimeSpan.FromMilliseconds(100);
var normalizedSource = source
.Delay(bufferTime)
.Drain(x => Observable.Empty<int>().Delay(TimeSpan.FromMilliseconds(10)));
...with Drain defined as follows:
public static class ObservableDrainExtensions
{
public static IObservable<TOut> Drain<TSource, TOut>(this IObservable<TSource> source,
Func<TSource, IObservable<TOut>> selector)
{
return Observable.Defer(() =>
{
BehaviorSubject<Unit> queue = new BehaviorSubject<Unit>(new Unit());
return source
.Zip(queue, (v, q) => v)
.SelectMany(v => selector(v)
.Do(_ => { }, () => queue.OnNext(new Unit()))
);
});
}
}
However, I think you're going to run into problems with the 10 millisecond qualifier. That's too small a time to schedule. If I remember correctly, any delay less than 15ms is ignored by the schedulers and fired immediately. Given that, even if you used a larger interval (I tried with 100 ms), you're going to get some variance thanks to OS context switching, etc..

Custom Rx operator for throttling only when there's a been a recent value

I'm trying to create an Rx operator that seems pretty useful, but I've suprisingly not found any questions on Stackoverflow that match precisely. I'd like to create a variation on Throttle that lets values through immediately if there's been a period of inactivity. My imagined use case is something like this:
I have a dropdown that kicks off a web request when the value is changed. If the user holds down the arrow key and cycles rapidly through the values, I don't want to kick off a request for each value. But if I throttle the stream then the user has to wait out the throttle duration every time they just select a value from the dropdown in the normal manner.
So whereas a normal Throttle looks like this:
I want to create ThrottleSubsequent that look like this:
Note that marbles 1, 2, and 6 are passed through without delay because they each follow a period of inactivity.
My attempt at this looks like the following:
public static IObservable<TSource> ThrottleSubsequent<TSource>(this IObservable<TSource> source, TimeSpan dueTime, IScheduler scheduler)
{
// Create a timer that resets with each new source value
var cooldownTimer = source
.Select(x => Observable.Interval(dueTime, scheduler)) // Each source value becomes a new timer
.Switch(); // Switch to the most recent timer
var cooldownWindow = source.Window(() => cooldownTimer);
// Pass along the first value of each cooldown window immediately
var firstAfterCooldown = cooldownWindow.SelectMany(o => o.Take(1));
// Throttle the rest of the values
var throttledRest = cooldownWindow
.SelectMany(o => o.Skip(1))
.Throttle(dueTime, scheduler);
return Observable.Merge(firstAfterCooldown, throttledRest);
}
This seems to work, but I'm having a difficult time reasoning about this, and I get the feeling there are some edge cases here where things might get screwy with duplicate values or something. I'd like to get some feedback from more experienced Rx-ers as to whether or not this code is correct, and/or whether there is a more idiomatic way of doing this.

Well, here's a test suite (using nuget Microsoft.Reactive.Testing):
var ts = new TestScheduler();
var source = ts.CreateHotObservable<char>(
new Recorded<Notification<char>>(200.MsTicks(), Notification.CreateOnNext('A')),
new Recorded<Notification<char>>(300.MsTicks(), Notification.CreateOnNext('B')),
new Recorded<Notification<char>>(500.MsTicks(), Notification.CreateOnNext('C')),
new Recorded<Notification<char>>(510.MsTicks(), Notification.CreateOnNext('D')),
new Recorded<Notification<char>>(550.MsTicks(), Notification.CreateOnNext('E')),
new Recorded<Notification<char>>(610.MsTicks(), Notification.CreateOnNext('F')),
new Recorded<Notification<char>>(760.MsTicks(), Notification.CreateOnNext('G'))
);
var target = source.ThrottleSubsequent(TimeSpan.FromMilliseconds(150), ts);
var expectedResults = ts.CreateHotObservable<char>(
new Recorded<Notification<char>>(200.MsTicks(), Notification.CreateOnNext('A')),
new Recorded<Notification<char>>(450.MsTicks(), Notification.CreateOnNext('B')),
new Recorded<Notification<char>>(500.MsTicks(), Notification.CreateOnNext('C')),
new Recorded<Notification<char>>(910.MsTicks(), Notification.CreateOnNext('G'))
);
var observer = ts.CreateObserver<char>();
target.Subscribe(observer);
ts.Start();
ReactiveAssert.AreElementsEqual(expectedResults.Messages, observer.Messages);
and using
public static class TestingHelpers
{
public static long MsTicks(this int i)
{
return TimeSpan.FromMilliseconds(i).Ticks;
}
}
Seems to pass. If you wanted to reduce it, you could turn it into this:
public static IObservable<TSource> ThrottleSubsequent2<TSource>(this IObservable<TSource> source, TimeSpan dueTime, IScheduler scheduler)
{
return source.Publish(_source => _source
.Window(() => _source
.Select(x => Observable.Interval(dueTime, scheduler))
.Switch()
))
.Publish(cooldownWindow =>
Observable.Merge(
cooldownWindow
.SelectMany(o => o.Take(1)),
cooldownWindow
.SelectMany(o => o.Skip(1))
.Throttle(dueTime, scheduler)
)
);
}
EDIT:
Publish forces sharing of a subscription. If you have a bad (or expensive) source observable with subscription side-effects, Publish makes sure you only subscribe once. Here's an example where Publish helps:
void Main()
{
var source = UglyRange(10);
var target = source
.SelectMany(i => Observable.Return(i).Delay(TimeSpan.FromMilliseconds(10 * i)))
.ThrottleSubsequent2(TimeSpan.FromMilliseconds(70), Scheduler.Default) //Works with ThrottleSubsequent2, fails with ThrottleSubsequent
.Subscribe(i => Console.WriteLine(i));
}
static int counter = 0;
public IObservable<int> UglyRange(int limit)
{
var uglySource = Observable.Create<int>(o =>
{
if (counter++ == 0)
{
Console.WriteLine("Ugly observable should only be created once.");
Enumerable.Range(1, limit).ToList().ForEach(i => o.OnNext(i));
}
else
{
Console.WriteLine($"Ugly observable should only be created once. This is the {counter}th time created.");
o.OnError(new Exception($"observable invoked {counter} times."));
}
return Disposable.Empty;
});
return uglySource;
}

How to re-subscribe to sequence in particular point?

I'm trying to solve the following:
a) subscriber receives events from IObservable for some time. Then it unsubscribes, do some stuff and then subscribe again. Here it should start receiving events from exactly the same point where unsubscription was performed.
b) Such behavior is desirable for multiple subscribers model. E.g. when one has unsubscribed, others should continue receiving events.
Are there any suggestions from the RX side?
Thanks in advance!

Here's a reasonably simple Rx way to do what you want copied from my answer to this other question. I've created an extension method called Pausable that takes a source observable and a second observable of boolean that pauses or resumes the observable.
public static IObservable<T> Pausable<T>(
this IObservable<T> source,
IObservable<bool> pauser)
{
return Observable.Create<T>(o =>
{
var paused = new SerialDisposable();
var subscription = Observable.Publish(source, ps =>
{
var values = new ReplaySubject<T>();
Func<bool, IObservable<T>> switcher = b =>
{
if (b)
{
values.Dispose();
values = new ReplaySubject<T>();
paused.Disposable = ps.Subscribe(values);
return Observable.Empty<T>();
}
else
{
return values.Concat(ps);
}
};
return pauser.StartWith(false).DistinctUntilChanged()
.Select(p => switcher(p))
.Switch();
}).Subscribe(o);
return new CompositeDisposable(subscription, paused);
});
}
It can be used like this:
var xs = Observable.Generate(
0,
x => x < 100,
x => x + 1,
x => x,
x => TimeSpan.FromSeconds(0.1));
var bs = new Subject<bool>();
var pxs = xs.Pausable(bs);
pxs.Subscribe(x => { /* Do stuff */ });
Thread.Sleep(500);
bs.OnNext(true);
Thread.Sleep(5000);
bs.OnNext(false);
Thread.Sleep(500);
bs.OnNext(true);
Thread.Sleep(5000);
bs.OnNext(false);

It sounds like you need a "pausable" stream. Assuming that only 1 subscriber will handle the values at a time (while the other subscribers just wait), this solution is probably what you need.