We Keep Coding

Scheduling asynchronous jobs/tasks (and ignoring exceptions) using Observable.Timer

I have several asynchronous tasks/jobs that I need to run on a schedule and it seems that I could do this nicely using Observables. When a job fetches the data, an exception could occur (eg 404), and when the resultant data is processed, an error could also occur.
I have seen this answer by Enigmativity which seems like the perfect solution to wrap the IObservable<> so that if an error occurs (when I fetch the data) I can trap it and continue (ultimately skipping the processing for that particular fetch).
I understand that when an Observable errors it is meant to terminate, but given the answer I mentioned above, it seems that there are ways around this, which would make for a decent job scheduling system. Alternative approaches are welcome, but I would like to understand how to do this with Observables.
I would also like to provide some feedback/logging about the state of the job.
Currently, I have the below method, which won't compile!
job is the object that contains information about the job (eg a list of job runs and their outcomes/success/failure, run frequency, status, errors, boolean flag indicating if the job should proceed, etc)
interval(job) returns the frequency in milliseconds that the job should run at
runSelect(job) is a boolean method that signals if a job should proceed (I think this would be better replaced with an observable? And of course there is the option of using a CancellationToken, but again I'm not sure how to integrate that!)
select(job) is the method that fetches the data
subscribe(job) is the method that processes the data
public static IDisposable BuildObservable<TJob, TSelect>(TJob job, Func<TJob, int> interval, Func<TJob, bool> runSelect, Func<TJob, Task<TSelect>> select,
Func<TSelect, Task> subscribe)
where TJob : Job
where TSelect : class
{
return Observable.Timer(TimeSpan.Zero, TimeSpan.FromMilliseconds(interval(job)))
.SelectMany(x => Observable.FromAsync(async () =>
{
JobRunDetail jobRunDetail = job.StartNewRun();
if (runSelect(job))
{
jobRunDetail.SetRunningSelect();
return new { Result = await select(job), JobRunDetail = jobRunDetail };
}
else
{
jobRunDetail.SetAbandonedSelect();
return new { Result = (TSelect)null!, JobRunDetail = jobRunDetail };
}
}).ToExceptional())
.Subscribe(async (resultAndJobRunDetail) =>
{
//none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile
resultAndJobRunDetail.Value.JobRunDetail.SetRunningSubscribe();
try
{
if (resultAndJobRunDetail.Value.Result!= null)
await subscribe(resultAndJobRunDetail.Value.Result);
resultAndJobRunDetail.Value.JobRunDetail.SetCompleted();
}
catch (Exception ee)
{
resultAndJobRunDetail.Value.JobRunDetail.SetErrorSubscribe(ee);
}
});
}
As noted, none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile because resultAndJobRunDetail.Value is still an Observable<>, but when I remove the .ToExceptional() call, the value returned is no longer an Observable.
Clearly I'm missing something.
I have seen different answers on SO that use Do() rather than Subscribe() so I'm not sure which is appropriate. I have also seen answers that suggest using Retry() or one of the "observable error handling methods" but I'm not sure how these would work if I just want my job to keep repeating ad infinitum?
Ultimately, I'm still learning how the whole Observable infrastructure fits together, so I could well be completely off track!
It's worth nothing that searching Google for "Schedule Job using Observable" it pretty fruitless as Observables use schedulers!

I'm not sure if this helps or not, but your .ToExceptional() call was in the wrong place:
public static IDisposable BuildObservable<TJob, TSelect>(TJob job, Func<TJob, int> interval, Func<TJob, bool> runSelect, Func<TJob, Task<TSelect>> select,
Func<TSelect, Task> subscribe)
where TJob : Job
where TSelect : class
{
return Observable.Timer(TimeSpan.Zero, TimeSpan.FromMilliseconds(interval(job)))
.SelectMany(x => Observable.FromAsync(async () =>
{
JobRunDetail jobRunDetail = job.StartNewRun();
if (runSelect(job))
{
jobRunDetail.SetRunningSelect();
return new { Result = await select(job), JobRunDetail = jobRunDetail }.ToExceptional();
}
else
{
jobRunDetail.SetAbandonedSelect();
return new { Result = (TSelect)null!, JobRunDetail = jobRunDetail }.ToExceptional();
}
}))
.Subscribe(async (resultAndJobRunDetail) =>
{
//none of the resultAndJobRunDetail.Value.JobDetail or resultAndJobRunDetail.Value.Result statements will compile
resultAndJobRunDetail.Value.JobRunDetail.SetRunningSubscribe();
try
{
if (resultAndJobRunDetail.Value.Result != null)
await subscribe(resultAndJobRunDetail.Value.Result);
resultAndJobRunDetail.Value.JobRunDetail.SetCompleted();
}
catch (Exception ee)
{
resultAndJobRunDetail.Value.JobRunDetail.SetErrorSubscribe(ee);
}
});
}

Proper approach to dealing with intermittent errors and Observables

I am using the GraphQL.NET client to subscribe to data on a remote service. The client returns an Observable so when the subscription is created you, as expected, receive new messages in onNext and get errors (both initial connection errors, reconnection errors, and anything else) in onError. The GraphQL client has the ability to automatically reconnect if the initial connection fails or when an established connection drops.
I know that by convention, any messages coming in on onError is supposed to terminate the sequence of messages. However, somehow they are able to continue sending to onNext and onError after that first onError. I have tried reading through the code but it is confusing. There seems to be multiple nesting of Observable and I suspect they are creating a new sequence when they encounter an error.
To clarify my issue, suppose I had the following pseudo Event based wrapper class.
public class PubSubSubscription() {
...
public void CreateSubscription<TResponse>(string topic) {
// GraphQL client
var stream = client
.CreateSubscriptionStream<FixConnectionChangedSubscriptionResult>(...);
stream
.Subscribe(
response => {
// Do stuff with incoming data (validation, mapping, logging, etc.)
// send it on the UI
DataReceived?.Invoke(this, new DataReceivedEventArgs { Message = response });
},
ex => {
// ******************************
// Note that the Observable created by CreateSubscriptionStream()
// will call `onError` over-and-over since it _seems_ like it is
// creating (and re-creating) nested Observables in its own
// classes. In the event of an initial connection failure or
// re-connect it will raise an error and then automatically
// try again.
// ******************************
// send it on to UI
ErrorDetected?.Invoke(this, new ErrorDetectedEventArgs { Exception = ex });
});
}
...
}
I would then call it as follows (or close enough)...
...
var orders = ordersPubSub.CreateSubscription("/orders");
orders.DataReceived += OnDataReceived;
orders.ErrorDetected += OnErrorDetected;
void OnErrorDetected(object sender, ErrorDetectedEventArgs e) {
// Can be called multiple times
// Display message in UI
}
...
I am having trouble converting that event-based wrapper approach to an Observable wrapper approach.
public class PubSubSubscription() {
...
public IObservable<TResponse> CreateSubscription<TResponse>(string topic) {
// Observable that I give back to my UI
var eventSubject = new Subject<TResponse>();
// GraphQL client
var stream = client
.CreateSubscriptionStream<FixConnectionChangedSubscriptionResult>(...);
stream
.Subscribe(
response => {
// Do stuff with incoming data (validation, mapping, logging, etc.)
// send it on the UI
eventSubject.onNext(response);
},
ex => {
// ******************************
// Note that the Observable created by CreateSubscriptionStream()
// will call `onError` over-and-over since it _seems_ like it is
// creating (and re-creating) nested Observables in its own
// classes. In the event of an initial connection failure or
// re-connect it will raise an error and then automatically
// try again.
// ******************************
// send it on to UI
eventSubject.onError(ex);
});
return eventSubject.AsObservable();
}
...
}
This I would then call it as follows (or close enough)...
...
var orders = ordersPubSub.CreateSubscription("/orders");
orders
// Things I have tried...
// Do() by itself does not stop the exception from hitting onError (which makes sense)
.Do(
_ => { },
ex => // display in UI)
// Retry() seems to cause the GraphQL subscription to "go away" because I no longer see connection attempts
.Retry()
// Stops the exception from hitting onError but the sequence still stops since I need to return _something_ from this method
.Catch(() => {
// display in UI
return Observable.Empty<T>();
})
.Subscribe(
msg => // do something with data,
ex => // display in UI);
}
...
Bottom line is what is the proper approach to dealing with sequences that can be "temporarily interrupted"?
I am also unsure of the idea of pushing the responsibility of retries onto the observer. This means that I would need to duplicate the logic each time CreateSubscription() is called. Yet, if I move it into the CreateSubscription() method, I am still unsure how to let the observer know the interruption happened so the UI can be updated.
One approach I am playing with (after reading about it as a possible solution) is to wrap my TResponse in a "fake" SubscriptionResponse<TResponse> which has T Value and Exception Error properties so the outer Observable only has onNext called. Then in my Subscribe I add if/else logic to check if Error is non-null and react accordingly. But this just feels ugly... I would almost want to go back to using events...

If you have an unruly observable - one that produces multiple errors without ended - you can make it workable by doing this:
IObservable<int> unruly = ...;
IObservable<Notification<int>> workable =
unruly
.Materialize();
The Materialize operator turns the IObservable<int> into an IObservable<Notification<int>> where the OnCompleted, OnError, and OnNext messages all get converted to OnNext messages that you can inspect like this:
Now you can deal with the errors without the sequence ending. When you've cleared them you can restore the sequence with Dematerialize like so:
IObservable<int> ruly =
workable
.Where(x => x.Kind != NotificationKind.OnError)
.Dematerialize();

Can I check in observer if a new item arrived to the observable?

I'm accessing a memory area that belongs to a different computational process.
There are relatively infrequent changes in the area and I need to run a calculation when there are changes. I get notifications on change, but I need to wait a bit to make sure that no more changes are being made. I model it like this:
var readyToBeProcessed = changed
.Select(x => DateTime.Now)
.Throttle(TimeSpan.FromSeconds(5));
However my calculations take quite some time, and it is possible that memory changes while I'm doing them. In this case I need to mark this particular round of calculations as invalid.
But how do I know in my observer, when I finished the calculation, if another event arrived or not, while processing current event? If there are no events arrive since I started the calculation, then it's valid and I can store the result.
In practice, it's very rarely, that the event arrive in the pattern (fast enough) that allow the calculation to become invalid, still I'd like to cater for this case.
Note: I realize that I cannot guarantee to have always valid calculations. There is a small time between a change in memory is made and the time I receive the event. It is entirely possible, that the sequence is like this 1) I'm doing the calculation 2) memory changes 3) I finish the calculation and check the event, and decide the calculation is valid 4) memory change event arrives. I'm happy to live with this for now
readyToBeProcessed.Subscribe(x =>
{
Log.Info("Start work...");
// Do calculation here
...
// When finished
if (Is there a new item)
{
Log.Info("There were changes while we worked... Invalidating");
Invalidate();
}
else
{
Log.Info("Succeeded");
}
}, cancellationToken);
Is Reactive bad fit for this task?

Rx is actually a great choice here, I think, though you may need to model it a bit more explicitly.
Think of there being really five type of events: Item changes, do-Work-begins, and do-Work-ends, Invalidates, and Succeededs (I wish I could use better names, but I'm working off what you wrote).
Here's a marble diagram of how they would work:
t(sec) : 0--1--2--3--4--5--6--7--8--9--10-11-12-13-14-15-16...
item-change : *-*--**-----------------*-------------------------...
do-Work-begins: ---------------------*-----------------*----------...
do-Work-ends : -------------------------*------------------*-----...
invalidate : -------------------------*------------------------...
succeeded : --------------------------------------------*-----...
We begin work once there has been a 5 second lull in item changes. If there has been any changes during the work time, we want to invalidate upon work-completion. If not, we want to observe success.
var doWorkBegins = changed
.Select(x => DateTime.Now)
.Throttle(TimeSpan.FromSeconds(5));
var doWorkEnds = doWorkBegins
.SelectMany(x =>
{
Log.Info("Start work...");
// DoWork();
//
// should return an observable that returns a single value when complete.
// If DoWork is just a void, then can use
// return Observable.Return(Unit.Default);
});
var lists = changed
.Buffer(() => doWorkEnds)
.Publish().RefCount();
var succeeded = lists
.Where(l => l.Count == 0);
var invalidate = lists
.Where(l => l.Count > 0);
invalidate.Subscribe(x =>
{
Log.Info("There were changes while we worked... Invalidating");
Invalidate();
}, cancellationToken);
succeeded.Subscribe(x =>
{
Log.Info("Succeeded");
}, cancellationToken);

Ideally, I would recommend you use a Task for keeping track of your work, then you can use:
readyToBeProcessed
.Select(evt => Observable.StartAsync<Unit>(async (cancellationToken) =>
{
//pass cancellationToken to work
var result = await DoWork(cancellationToken);
//test token if needed
return result;
}))
.Switch()
.Subscribe();
When the next item arrives, the current token will be canceled.

How to cancel a Select in RX if it is not finished before the next event arrives

I have the following setup
IObservable<Data> source = ...;
source
.Select(data=>VeryExpensiveOperation(data))
.Subscribe(data=>Console.WriteLine(data));
Normally the events come seperated by a reasonable time frame.
Imagine a user updating a text box in a form. Our VeryExpensiveOperation
might take 5 seconds to complete and whilst it does an hour glass
is displayed on the screen.
However if during the 5 seconds the user updates the textbox again
I would want to send a cancelation to the current VeryExpensiveOperation
before the new one starts.
I would imagine a scenario like
source
.SelectWithCancel((data, cancelToken)=>VeryExpensiveOperation(data, token))
.Subscribe(data=>Console.WriteLine(data));
So every time the lambda is called is is called with a cancelToken which can be
used to manage canceling a Task. However now we are mixing Task, CancelationToken and RX.
Not quite sure how to fit it all together. Any suggestions.
Bonus Points for figuring out how to test the operator using XUnit :)
FIRST ATTEMPT
public static IObservable<U> SelectWithCancelation<T, U>( this IObservable<T> This, Func<CancellationToken, T, Task<U>> fn )
{
CancellationTokenSource tokenSource = new CancellationTokenSource();
return This
.ObserveOn(Scheduler.Default)
.Select(v=>{
tokenSource.Cancel();
tokenSource=new CancellationTokenSource();
return new {tokenSource.Token, v};
})
.SelectMany(o=>Observable.FromAsync(()=>fn(o.Token, o.v)));
}
Not tested yet. I'm hoping that a task that does not complete generates an IObservable that completes without firing any OnNext events.

You have to model VeryExpensiveOperation as an cancellable asynchronous thing. Either a Task or an IObservable. I'll assume it is a task with a CancellationToken:
Task<TResult> VeryExpensiveOperationAsync<TSource, TResult>(TSource item, CancellationToken token);
Then you do it like so:
source
.Select(item => Observable.DeferAsync(async token =>
{
// do not yield the observable until after the operation is completed
// (ie do not just do VeryExpensiveOperation(...).ToObservable())
// because DeferAsync() will dispose of the token source as soon
// as you provide the observable (instead of when the observable completes)
var result = await VeryExpensiveOperationAsync(item, token);
return Observable.Return(result);
})
.Switch();
The Select just creates a deferred observable that, when subscribed, will create a token and kick off the operation. If the observable is unsubscribed before the operation finishes, the token will be cancelled.
The Switch subscribes to each new observable that comes out of Select, unsubscribing from the previous observable it was subscribed to.
This has the effect you want.
P.S. this is easily testable. Just provide a mock source and a mock VeryExpensiveOperation that uses a TaskCompletetionSource provided by the unit test so the unit test can control exactly when new source items are produced and when tasks are completed. Something like this:
void SomeTest()
{
// create a test source where the values are how long
// the mock operation should wait to do its work.
var source = _testScheduler.CreateColdObservable<int>(...);
// records the actions (whether they completed or canceled)
List<bool> mockActionsCompleted = new List<bool>();
var resultStream = source.SelectWithCancellation((token, delay) =>
{
var tcs = new TaskCompletionSource<string>();
var tokenRegistration = new SingleAssignmentDisposable();
// schedule an action to complete the task
var d = _testScheduler.ScheduleRelative(delay, () =>
{
mockActionsCompleted.Add(true);
tcs.SetResult("done " + delay);
// stop listening to the token
tokenRegistration.Dispose();
});
// listen to the token and cancel the task if the token signals
tokenRegistration.Disposable = token.Register(() =>
{
mockActionsCompleted.Add(false);
tcs.TrySetCancelled();
// cancel the scheduled task
d.Dispose();
});
return tcs.Task;
});
// subscribe to resultStream
// start the scheduler
// assert the mockActionsCompleted has the correct sequence
// assert the results observed were what you expected.
}
You might run into trouble using testScheduler.Start() due to the new actions scheduled dynamically. a while loop with testScheduler.AdvanceBy(1) might work better.

Why not just use a Throttle?
http://rxwiki.wikidot.com/101samples#toc30
Throttle stops the flow of events until no more events are produced for a specified period of time. For example, if you throttle a TextChanged event of a textbox to .5 seconds, no events will be passed until the user has stopped typing for .5 seconds. This is useful in search boxes where you do not want to start a new search after every keystroke, but want to wait until the user pauses.
SearchTextChangedObservable = Observable.FromEventPattern<TextChangedEventArgs>(this.textBox, "TextChanged");
_currentSubscription = SearchTextChangedObservable.Throttle(TimeSpan.FromSeconds(.5)).ObserveOnDispatcher

Task Parallel Library - how to return immediately but have a parallel step

I have the following workflow that needs to happen in a non-blocking parallel processing manner. I would like the method DoStuff() to return immediately so I am using Task Parallel Libary
DoStuff():
Do some setup
Parse an Excel file
then for each row
Fill Template with parsed values
Convert filled template to Pdf
Convert pdf to Tiff
when all row processing has completed Create Summary Text File
when summary text file has completed, Finalize
I'm stumbling a bit with the "when all row processing has been completed" step since I want to return immediately. Is the following roughly what I should be doing?
public Task<ProcessingResult> DoStuff() {
return new Task<SetupResult>(SetUp)
.ContinueWith(ParseExcel, TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith(excelProcessing => {
var templateProcessing = excelProcessing.Result.RowParsing
.Select(template =>
new Task<TemplateFillingResult>(()=>FillTemplate)
.ContinueWith(ConvertToPdf, TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith(ConvertToTiff, TaskContinuationOptions.OnlyOnRanToCompletion)
).ToArray()
//-------------------------------------------------------------
// This is the part that seems wierd
//-------------------------------------------------------------
Task.Factory.ContinueWhenAll(templateTasks, t=> { }).Wait();
return new TemplatesProcessingResult(templateProcessing);
}, TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith(CreateSummaryFile, TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith(FinalizeProcessing, TaskContinuationOptions.OnlyOnRanToCompletion);

I think you're getting confused because you are trying to wire up all those components as continuations of the original event. If there is no compelling reason to make all of those calls continuations, then this can all be simply done with a single background thread (task).
var task = Task.Factory.StartNew(() =>
{
// setup
// var stuff = ParseFile()
// Executes elements in parallel and blocks this thread until all have completed, else bubbles the exception up
var transformations = excelProcessing.Result.RowParsing.AsParallel().Select(x =>
{
FillTemplate(x);
}).ToArray();
// create summary text file
// Finalize
return processingResult;
});
Basically, you can do all of that in a single thread and not have to worry about it. Marking up all those steps as continuations is pretty convoluted for what you need to do.
Then your calling code can simply block on the Result property of that guy to get the result of the asynchronous call:
try
{
var result = task.Result;
}
catch(AggregateException e)
{
e.Flatten().Handle(ex =>
{
// Do Stuff, return true to indicate handled
});
}
However, the one thing you will need to be cognizant of is exceptions. If this is going to be a fire and forget task, then if you have an exception, it's going to bubble all the way up and potentially kill your process.