We Keep Coding

Map errors to observable using C# ReactiveX

I have an observable MyObservable<Object> which can throw CustomExceptions where
private class CustomException : Exception
What I want to do is convert the CustomExceptions into objects and emit those in a new observable.
This is my solution so far but I was wondering if this could be done without having to directly call the Subject's onNext, onCompleted or onError methods.
var MySubject = new Subject<NewObject>();
MyObservable.Catch<Object, CustomException>(
ex =>
{
NewObject o = new NewObject(ex.Message);
MySubject.OnNext(o);
return Observable.Empty<Object>();
});
IObservable<IList<NewObject>> listObservable = MySubject.ToList();
Edit: Thanks ibebbs! Worked like a charm!

You can catch and map exceptions without a subject by using the Materialize() function as shown here:
var errorObservable = source
.Select(projection)
.Materialize()
.Where(notification => notification.Kind == NotificationKind.OnError)
.Select(notification => notification.Exception)
.OfType<CustomException>()
.Select(exception => new NewObject(exception.Message));
The Materialize function takes an IObservable<T> and maps it to a IObservable<Notification<T>> where each Notification has a Kind of OnNext, OnError or OnComplete. The above observable simply looks for Notifications with a Kind`` of OnError and with the Exception being an instance of CustomException then projects these exceptions into anIObservable```.
Here is a unit test showing this working:
[Fact]
public void ShouldEmitErrorsToObservable()
{
Subject<int> source = new Subject<int>();
List<int> values = new List<int>();
List<NewObject> errors = new List<NewObject>();
Func<int, int> projection =
value =>
{
if (value % 2 == 1) throw new CustomException("Item is odd");
return value;
};
Func<CustomException, IObservable<int>> catcher = null;
catcher = ex => source.Select(projection).Catch(catcher);
var errorObservable = source
.Select(projection)
.Materialize()
.Where(notification => notification.Kind == NotificationKind.OnError)
.Select(notification => notification.Exception)
.OfType<CustomException>()
.Select(exception => new NewObject(exception.Message));
var normalSubscription = source.Select(projection).Catch(catcher).Subscribe(values.Add);
var errorSubscription = errorObservable.Subscribe(errors.Add);
source.OnNext(0);
source.OnNext(1);
source.OnNext(2);
Assert.Equal(2, values.Count);
Assert.Equal(1, errors.Count);
}
However, as you can see with the construed catch mechanisms employed above, exception handling in Rx can be tricky to get right and even more difficult to do elegantly. Instead, consider that Exceptions should be Exceptional and, if you expect an class of error such that you've written a custom exception for it, then the error is not really exceptional but part of a process flow that must handle these errors.
In this instance, I would recommend projecting the observable into a class which embodies the "try this operation and record the result, be it a value or an exception" and using this further along the execution chain.
In the example below, I use a "Fallible" class to capture the result or exception of an operation and then subscribe to a stream of "Fallible" instances, separating the errors from values. As you will see, the code is both neater and better performing as both the errors and values share a single subscription to the underlying source:
internal class Fallible
{
public static Fallible<TResult> Try<TResult, TException>(Func<TResult> action) where TException : Exception
{
try
{
return Success(action());
}
catch (TException exception)
{
return Error<TResult>(exception);
}
}
public static Fallible<T> Success<T>(T value)
{
return new Fallible<T>(value);
}
public static Fallible<T> Error<T>(Exception exception)
{
return new Fallible<T>(exception);
}
}
internal class Fallible<T>
{
public Fallible(T value)
{
Value = value;
IsSuccess = true;
}
public Fallible(Exception exception)
{
Exception = exception;
IsError = true;
}
public T Value { get; private set; }
public Exception Exception { get; private set; }
public bool IsSuccess { get; private set; }
public bool IsError { get; private set; }
}
[Fact]
public void ShouldMapErrorsToFallible()
{
Subject<int> source = new Subject<int>();
List<int> values = new List<int>();
List<NewObject> errors = new List<NewObject>();
Func<int, int> projection =
value =>
{
if (value % 2 == 1) throw new CustomException("Item is odd");
return value;
};
var observable = source
.Select(value => Fallible.Try<int, CustomException>(() => projection(value)))
.Publish()
.RefCount();
var errorSubscription = observable
.Where(fallible => fallible.IsError)
.Select(fallible => new NewObject(fallible.Exception.Message))
.Subscribe(errors.Add);
var normalSubscription = observable
.Where(fallible => fallible.IsSuccess)
.Select(fallible => fallible.Value)
.Subscribe(values.Add);
source.OnNext(0);
source.OnNext(1);
source.OnNext(2);
Assert.Equal(2, values.Count);
Assert.Equal(1, errors.Count);
}
Hope it helps.

Observing changes only within a window that commits successfully

Suppose I have the following class:
public class Person : ReactiveObject, IEditableObject
{
private string name;
private string nameCopy;
public string Name
{
get { return this.name; }
set { this.RaiseAndSetIfChanged(ref this.name, value); }
}
public void BeginEdit()
{
this.nameCopy = this.name;
}
public void CancelEdit()
{
this.name = this.nameCopy;
}
public void EndEdit()
{
}
}
Now suppose I want to create an observable sequence (of Unit) that "ticks" whenever a change is committed to Name. That is, I only care about changes to Name that occur between a call to BeginEdit and and a subsequent call to EndEdit. Any changes prior to a call to CancelEdit should be ignored and the sequence should not tick.
I'm struggling to get my head around how I would do this with Rx. It seems I would need state in the pipeline somewhere in order to know whether the change occurred during the window of BeginEdit/EndEdit calls. I suppose I could timestamp everything and compare timestamps, but that seems a nasty hack.
I came pretty close using a dedicated Subject for edit actions along with Observable.Merge:
public class Person : ReactiveObject, IEditableObject
{
private readonly Subject<EditAction> editActions;
private readonly IObservable<Unit> changedDuringEdit;
private string name;
private string nameCopy;
public Person()
{
this.editActions = new Subject<EditAction>();
var nameChanged = this.ObservableForProperty(x => x.Name).Select(x => x.Value);
var editBeginning = this.editActions.Where(x => x == EditAction.Begin);
var editCommitted = this.editActions.Where(x => x == EditAction.End);
this.changedDuringEdit = nameChanged
.Buffer(editBeginning, _ => editCommitted)
.Where(x => x.Count > 0)
.Select(_ => Unit.Default);
}
public IObservable<Unit> ChangedDuringEdit
{
get { return this.changedDuringEdit; }
}
public string Name
{
get { return this.name; }
set { this.RaiseAndSetIfChanged(ref this.name, value); }
}
public void BeginEdit()
{
this.editActions.OnNext(EditAction.Begin);
this.nameCopy = this.name;
}
public void CancelEdit()
{
this.editActions.OnNext(EditAction.Cancel);
this.Name = this.nameCopy;
}
public void EndEdit()
{
this.editActions.OnNext(EditAction.End);
}
private enum EditAction
{
Begin,
Cancel,
End
}
}
However, if several changes are cancelled, and then one is committed, the observable ticks several times on commit (once for each prior cancellation, and once again for the commit). Not to mention the fact that I get a List<Unit> which I don't actually need. In a way, this would still satisfy my use case, but not my curiosity or sense of code aesthetic.
I feel like Join should solve this fairly elegantly:
var nameChanged = this.ObservableForProperty(x => x.Name).Select(_ => Unit.Default);
var editBeginning = this.editActions.Where(x => x == EditAction.Begin);
var editCommitted = this.editActions.Where(x => x == EditAction.End);
var editCancelled = this.editActions.Where(x => x == EditAction.Cancel);
var editCancelledOrCommitted = editCancelled.Merge(editCommitted);
this.changedDuringEdit = editBeginning
.Join(nameChanged, _ => editCancelledOrCommitted, _ => editCancelledOrCommitted, (editAction, _) => editAction == EditAction.End)
.Where(x => x)
.Select(_ => Unit.Default);
But this doesn't work either. It seems Join is not subscribing to editCancelledOrCommitted, for reasons I don't understand.
Anyone have any ideas how to go about this cleanly?

Here's how I'd do it:
IObservable<Unit> beginEditSignal = ...;
IObservable<Unit> commitSignal = ...;
IObservable<Unit> cancelEditSignal = ...;
IObservable<T> propertyChanges = ...;
// this will yield an array after each commit
// that has all of the changes for that commit.
// nothing will be yielded if the commit is canceled
// or if the changes occur before BeginEdit.
IObservable<T[]> commitedChanges = beginEditSignal
.Take(1)
.SelectMany(_ => propertyChanges
.TakeUntil(commitSignal)
.ToArray()
.Where(changeList => changeList.Length > 0)
.TakeUntil(cancelEditSignal))
.Repeat();
// if you really only want a `Unit` when something happens
IObservable<Unit> changeCommittedSignal = beginEditSignal
.Take(1)
.SelectMany(_ => propertyChanges
.TakeUntil(commitSignal)
.Count()
.Where(c => c > 0)
.Select(c => Unit.Default)
.TakeUntil(cancelEditSignal))
.Repeat();

You have a timing problem that I don't think you have articulated yet; when are you hoping for the changes to tick?
either as they occur
once the commit happens
The clear and obvious problem with 1) is that you don't know if the changes will be committed, so why would you raise them. IMO, this only leaves option 2). If the change is cancelled, then no event is raised.
Next question I have is, do you want each change raised? ie. for the process
[Begin]-->[Name="fred"]-->[Name="bob"]-->[Commit]
Should this raise 1 or 2 events when the Commit is made? As you are only pushing the token type Unit, it seems redundant to push two values. This now leads me to think that you just want to push a Unit value when EndEdit() is executed and the values have changed.
This leaves us with a painfully simple implementation:
public class Person : ReactiveObject, IEditableObject
{
private readonly ISubject<Unit> changedDuringEdit = new Subject<Unit>();
private string name;
private string nameCopy;
public string Name
{
get { return this.name; }
set { this.RaiseAndSetIfChanged(ref this.name, value); }
}
public void BeginEdit()
{
this.nameCopy = this.name;
}
public void CancelEdit()
{
this.name = this.nameCopy;
}
public void EndEdit()
{
if(!string.Equals(this.nameCopy, this.name))
{
changedDuringEdit.OnNext(Unit.Default);
}
}
public IObservable<Unit> ChangedDuringEdit
{
get { return this.changedDuringEdit.AsObservable(); }
}
}
Is this what you are looking for? If not can you help me understand the complexities I am missing? If it is then I would be keen to flesh this out so that I wasn't recommending using Subjects :-)

Saving subscription state for resuming it later

Update - solved
The final solution differs a bit from Brandon's suggestion but his answer brought me on the right track.
class State
{
public int Offset { get; set; }
public HashSet<string> UniqueImageUrls = new HashSet<string>();
}
public IObservable<TPicture> GetPictures(ref object _state)
{
var localState = (State) _state ?? new State();
_state = localState;
return Observable.Defer(()=>
{
return Observable.Defer(() => Observable.Return(GetPage(localState.Offset)))
.SubscribeOn(TaskPoolScheduler.Default)
.Do(x=> localState.Offset += 20)
.Repeat()
.TakeWhile(x=> x.Count > 0)
.SelectMany(x=> x)
.Where(x=> !localState.UniqueImageUrls.Contains(x.ImageUrl))
.Do(x=> localState.UniqueImageUrls.Add(x.ImageUrl));
});
}
IList<TPicture> GetPage(int offset)
{
...
return result;
}
Original Question
I'm currently struggling with the following problem. The PictureProvider implementation shown below is working with an offset variable used for paging results of a backend service providing the actual data. What I would like to implement is an elegant solution making the current offset available to the consumer of the observable to allow for resuming the observable sequence at a later time at the correct offset. Resuming is already accounted for by the intialState argument to GetPictures().
Recommendations for improving the code in a more RX like fashion would be welcome as well. I'm actually not so sure if the Task.Run() stuff is appropriate here.
public class PictureProvider :
IPictureProvider<Picture>
{
#region IPictureProvider implementation
public IObservable<Picture> GetPictures(object initialState)
{
return Observable.Create<Picture>((IObserver<Picture> observer) =>
{
var state = new ProducerState(initialState);
ProducePictures(observer, state);
return state;
});
}
#endregion
void ProducePictures(IObserver<Picture> observer, ProducerState state)
{
Task.Run(() =>
{
try
{
while(!state.Terminate.WaitOne(0))
{
var page = GetPage(state.Offset);
if(page.Count == 0)
{
observer.OnCompleted();
break;
}
else
{
foreach(var picture in page)
observer.OnNext(picture);
state.Offset += page.Count;
}
}
}
catch (Exception ex)
{
observer.OnError(ex);
}
state.TerminateAck.Set();
});
}
IList<Picture> GetPage(int offset)
{
var result = new List<Picture>();
... boring web service call here
return result;
}
public class ProducerState :
IDisposable
{
public ProducerState(object initialState)
{
Terminate = new ManualResetEvent(false);
TerminateAck = new ManualResetEvent(false);
if(initialState != null)
Offset = (int) initialState;
}
public ManualResetEvent Terminate { get; private set; }
public ManualResetEvent TerminateAck { get; private set; }
public int Offset { get; set; }
#region IDisposable implementation
public void Dispose()
{
Terminate.Set();
TerminateAck.WaitOne();
Terminate.Dispose();
TerminateAck.Dispose();
}
#endregion
}
}

I suggest refactoring your interface to yield the state as part of the data. Now the client has what they need to resubscribe where they left off.
Also, once you start using Rx, you should find that using synchronization primitives like ManualResetEvent are rarely necessary. If you refactor your code so that retrieving each page is its own Task, then you can eliminate all of that synchronization code.
Also, if you are calling a "boring web service" in GetPage, then just make it async. This gets rid of the need to call Task.Run among other benefits.
Here is a refactored version, using .NET 4.5 async/await syntax. It could also be done without async/await. I also added a GetPageAsync method that uses Observable.Run just in case you really cannot convert your webservice call to be asynchronous
/// <summary>A set of pictures</summary>
public struct PictureSet
{
public int Offset { get; private set; }
public IList<Picture> Pictures { get; private set; }
/// <summary>Clients will use this property if they want to pick up where they left off</summary>
public int NextOffset { get { return Offset + Pictures.Count; } }
public PictureSet(int offset, IList<Picture> pictures)
:this() { Offset = offset; Pictures = pictures; }
}
public class PictureProvider : IPictureProvider<PictureSet>
{
public IObservable<PictureSet> GetPictures(int offset = 0)
{
// use Defer() so we can capture a copy of offset
// for each observer that subscribes (so multiple
// observers do not update each other's offset
return Observable.Defer<PictureSet>(() =>
{
var localOffset = offset;
// Use Defer so we re-execute GetPageAsync()
// each time through the loop.
// Update localOffset after each GetPageAsync()
// completes so that the next call to GetPageAsync()
// uses the next offset
return Observable.Defer(() => GetPageAsync(localOffset))
.Select(pictures =>
{
var s = new PictureSet(localOffset, pictures);
localOffset += pictures.Count;
})
.Repeat()
.TakeWhile(pictureSet => pictureSet.Pictures.Count > 0);
});
}
private async Task<IList<Picture>> GetPageAsync(int offset)
{
var data = await BoringWebServiceCallAsync(offset);
result = data.Pictures.ToList();
}
// this version uses Observable.Run() (which just uses Task.Run under the hood)
// in case you cannot convert your
// web service call to be asynchronous
private IObservable<IList<Picture>> GetPageAsync(int offset)
{
return Observable.Run(() =>
{
var result = new List<Picture>();
... boring web service call here
return result;
});
}
}
Clients just need to add a SelectMany call to get their IObservable<Picture>. They can choose to store the pictureSet.NextOffset if they wish.
pictureProvider
.GetPictures()
.SelectMany(pictureSet => pictureSet.Pictures)
.Subscribe(picture => whatever);

Instead of thinking about how to save the subscription state, I would think about how to replay the state of the inputs (i.e. I'd try to create a serializable ReplaySubject that, on resume, would just resubscribe and catch back up to the current state).

Moq is slow to verify dependency after a large number calls

I've hit a snag when using Moq to simulate an dependency which is called a large number of times. When I call Verify, Moq takes a long time (several minutes) to respond, and sometimes crashes with a NullReferenceException (I guess this is understandable, given the amount of data that Moq would have to accumulate to do the Verify from a "cold start").
So my question is, is there another strategy that I can use to do this using Moq, or should I revert to a hand-crafted stub for this rather unusual case. Specifically, is there a way to tell Moq up front that I'm only interested in verifying specific filters on the parameters, and to ignore all other values?
Neither of the approaches below is satisfactory.
Given CUT and Dep:
public interface ISomeInterface
{
void SomeMethod(int someValue);
}
public class ClassUnderTest
{
private readonly ISomeInterface _dep;
public ClassUnderTest(ISomeInterface dep)
{
_dep = dep;
}
public void DoWork()
{
for (var i = 0; i < 1000000; i++) // Large number of calls to dep
{
_dep.SomeMethod(i);
}
}
}
Moq Strategy 1 - Verify
var mockSF = new Mock<ISomeInterface>();
var cut = new ClassUnderTest(mockSF.Object);
cut.DoWork();
mockSF.Verify(mockInt => mockInt.SomeMethod(It.Is<int>(i => i == 12345)),
Times.Once());
mockSF.Verify(mockInt => mockInt.SomeMethod(It.Is<int>(i => i == -1)),
Times.Never());
Moq Strategy 2 - Callback
var mockSF = new Mock<ISomeInterface>();
var cut = new ClassUnderTest(mockSF.Object);
bool isGoodValueAlreadyUsed = false;
mockSF.Setup(mockInt => mockInt.SomeMethod(It.Is<int>(i => i == 12345)))
.Callback(() =>
{
if (isGoodValueAlreadyUsed)
{
throw new InvalidOperationException();
}
isGoodValueAlreadyUsed = true;
});
mockSF.Setup(mockInt => mockInt.SomeMethod(It.Is<int>(i => i == -1)))
.Callback(() =>
{ throw new InvalidOperationException(); });
cut.DoWork();
Assert.IsTrue(isGoodValueAlreadyUsed);

Usually when such a limitation is reached, I would reconsider my design (no offense, I see your rep). Looks like the method under test does too much work, which is violation of the single responsibility principle. It first generates a large list of items, and then verifies a worker is called for each one of them, while also verifying that the sequence contains the right elements.
I'd split the functionality into a sequence generator, and verify that the sequence has the right elements, and another method which acts on the sequence, and verify that it executes the worker for each element:
namespace StackOverflowExample.Moq
{
public interface ISequenceGenerator
{
IEnumerable<int> GetSequence();
}
public class SequenceGenrator : ISequenceGenerator
{
public IEnumerable<int> GetSequence()
{
var list = new List<int>();
for (var i = 0; i < 1000000; i++) // Large number of calls to dep
{
list.Add(i);
}
return list;
}
}
public interface ISomeInterface
{
void SomeMethod(int someValue);
}
public class ClassUnderTest
{
private readonly ISequenceGenerator _generator;
private readonly ISomeInterface _dep;
public ClassUnderTest(ISomeInterface dep, ISequenceGenerator generator)
{
_dep = dep;
_generator = generator;
}
public void DoWork()
{
foreach (var i in _generator.GetSequence())
{
_dep.SomeMethod(i);
}
}
}
[TestFixture]
public class LargeSequence
{
[Test]
public void SequenceGenerator_should_()
{
//arrange
var generator = new SequenceGenrator();
//act
var list = generator.GetSequence();
//assert
list.Should().Not.Contain(-1);
Executing.This(() => list.Single(i => i == 12345)).Should().NotThrow();
//any other assertions
}
[Test]
public void DoWork_should_perform_action_on_each_element_from_generator()
{
//arrange
var items = new List<int> {1, 2, 3}; //can use autofixture to generate random lists
var generator = Mock.Of<ISequenceGenerator>(g => g.GetSequence() == items);
var mockSF = new Mock<ISomeInterface>();
var classUnderTest = new ClassUnderTest(mockSF.Object, generator);
//act
classUnderTest.DoWork();
//assert
foreach (var item in items)
{
mockSF.Verify(c=>c.SomeMethod(item), Times.Once());
}
}
}
}
EDIT:
Different approaches can be mixed to define a specific expectations, incl. When(), the obsoleted AtMost(), MockBehavior.Strict, Callback, etc.
Again, Moq is not designed to work on large sets, so there is performance penalty. You are still better off using another measures to verify what data will be passed to the mock.
For the example in the OP, here is a simplified setup:
var mockSF = new Mock<ISomeInterface>(MockBehavior.Strict);
var cnt = 0;
mockSF.Setup(m => m.SomeMethod(It.Is<int>(i => i != -1)));
mockSF.Setup(m => m.SomeMethod(It.Is<int>(i => i == 12345))).Callback(() =>cnt++).AtMostOnce();
This will throw for -1, for more than one invocation with 12, and assertion can be made on cnt != 0.

How to await an object inside IObservable with a specific property value?

To clarify, i have a method:
public static IObservable<Node> GetNodes()
{
var computers = GetComputersInLan();
return computers.Select(computerAddress => GetNode(computerAddress));
}
GetComputersInLan method returns IObservable of IPAddress
private static IObservable<IPAddress> GetComputersInLan()
{
var tasks = new List<Task<PingReply>>();
for (int i = 1; i < 255; i++)
{
Ping p = new Ping();
ipBytes[3] = (byte)(++ipBytes[3]);
IPAddress address = new IPAddress(ipBytes);
tasks.Add(p.SendPingAsync(address, 2000));
}
return tasks.ToObservable().Where(x => x.Result.Status == IPStatus.Success).Select(y => y.Result.Address);
}
GetNode method constructs a Node.
private static Node GetNode(IPAddress ipAddress)
{
return new Node(ipAddress, (IHandler)Activator.GetObject(typeof(Handler), "tcp://" + ipAddress + ":1337/handler"));
}
public class Node
{
private IHandler Handler { get; set; }
public IPAddress Address { get; set; }
public int AvailableCores { get; set; }
public async Task<TResult> Invoke<TResult>(Func<TResult> method)
{
AvailableCores--;
var result = await Task.Run<TResult>(() => Handler.Invoke(method));
AvailableCores++;
return result;
}
}
Handler is a remote computer, and AvailableCores represents its cpu cores.
What I want is to await method GetNodes to return the first Node that has more than 0 AvailableCores.
await GetNodes().FirstAsync(node => node.AvailableCore > 0)
But what happens, is that after enough calls to method Invoke, instead of waiting for cores to become available, it fires an exception "sequence contains no elements".

That is expected behavior for this method. FirstAsync will only check the current state of the items you pass to it, either returning the first match or throwing the exception you are encountering if there is no match.
You will have to manage the case of waiting until a core becomes available yourself. You could try FirstOrDefaultAsync to return null instead of throwing an exception when all cores are busy. From there, you will need some scheme to detect when a core becomes available for the next unit of work, be that an event or polling.