We Keep Coding

How can I route Observable values to different Subscribers?

This is all just pseudo code...
Ok here is my scenario, I have an incoming data stream that gets parsed into packets.
I have an IObservable<Packets> Packets
Each packet has a Packet ID, i.e. 1, 2, 3, 4
I want to create observables that only receive a specific ID.
so I do:
Packets.Where(p=>p.Id == 1)
for example... that gives me an IObservable<Packets> that only gives me packets of Id 1.
I may have several of these:
Packets.Where(p=>p.Id == 2)
Packets.Where(p=>p.Id == 3)
Packets.Where(p=>p.Id == 4)
Packets.Where(p=>p.Id == 5)
This essentially works, but the more Ids I want to select the more processing is required, i.e. the p=>p.Id will be run for every single Id, even after a destination Observable has been found.
How can I do the routing so that it is more efficient, something analogous:
Dictionary listeners;
listeners.GetValue(packet.Id).OnDataReceived(packet)
so that as soon as an id is picked up by one of my IObservables, then none of the others get to see it?
Updates
Added an extension based on Lee Campbell's groupby suggestion:
public static class IObservableExtensions
{
class RouteTable<TKey, TSource>
{
public static readonly ConditionalWeakTable<IObservable<TSource>, IObservable<IGroupedObservable<TKey, TSource>>> s_routes = new ConditionalWeakTable<IObservable<TSource>, IObservable<IGroupedObservable<TKey, TSource>>>();
}
public static IObservable<TSource> Route<TKey, TSource>(this IObservable<TSource> source, Func<TSource, TKey> selector, TKey id)
{
var grouped = RouteTable<TKey, TSource>.s_routes.GetValue(source, s => s.GroupBy(p => selector(p)).Replay().RefCount());
return grouped.Where(e => e.Key.Equals(id)).SelectMany(e => e);
}
}
It would be used like this:
Subject<Packet> packetSubject = new Subject<Packet>();
var packets = packetSubject.AsObservable();
packets.Route((p) => p.Id, 5).Subscribe((p) =>
{
Console.WriteLine("5");
});
packets.Route((p) => p.Id, 4).Subscribe((p) =>
{
Console.WriteLine("4");
});
packets.Route((p) => p.Id, 3).Subscribe((p) =>
{
Console.WriteLine("3");
});
packetSubject.OnNext(new Packet() { Id = 1 });
packetSubject.OnNext(new Packet() { Id = 2 });
packetSubject.OnNext(new Packet() { Id = 3 });
packetSubject.OnNext(new Packet() { Id = 4 });
packetSubject.OnNext(new Packet() { Id = 5 });
packetSubject.OnNext(new Packet() { Id = 4 });
packetSubject.OnNext(new Packet() { Id = 3 });
output is:
3, 4, 5, 4, 3
It only checks the Id for every group when it sees a new packet id.

Here's an operator that I wrote quite some time ago, but I think it does what you're after. I still think that a simple .Where is probably better - even with multiple subscribers.
Nevertheless, I wanted a .ToLookup for observables that operates like the same operator for enumerables.
It isn't memory efficient, but it implements IDisposable so that it can be cleaned up afterwards. It also isn't thread-safe so a little hardening might be required.
Here it is:
public static class ObservableEx
{
public static IObservableLookup<K, V> ToLookup<T, K, V>(this IObservable<T> source, Func<T, K> keySelector, Func<T, V> valueSelector, IScheduler scheduler)
{
return new ObservableLookup<T, K, V>(source, keySelector, valueSelector, scheduler);
}
internal class ObservableLookup<T, K, V> : IDisposable, IObservableLookup<K, V>
{
private IDisposable _subscription = null;
private readonly Dictionary<K, ReplaySubject<V>> _lookups = new Dictionary<K, ReplaySubject<V>>();
internal ObservableLookup(IObservable<T> source, Func<T, K> keySelector, Func<T, V> valueSelector, IScheduler scheduler)
{
_subscription = source.ObserveOn(scheduler).Subscribe(
t => this.GetReplaySubject(keySelector(t)).OnNext(valueSelector(t)),
ex => _lookups.Values.ForEach(rs => rs.OnError(ex)),
() => _lookups.Values.ForEach(rs => rs.OnCompleted()));
}
public void Dispose()
{
if (_subscription != null)
{
_subscription.Dispose();
_subscription = null;
_lookups.Values.ForEach(rs => rs.Dispose());
_lookups.Clear();
}
}
private ReplaySubject<V> GetReplaySubject(K key)
{
if (!_lookups.ContainsKey(key))
{
_lookups.Add(key, new ReplaySubject<V>());
}
return _lookups[key];
}
public IObservable<V> this[K key]
{
get
{
if (_subscription == null) throw new ObjectDisposedException("ObservableLookup");
return this.GetReplaySubject(key).AsObservable();
}
}
}
}
public interface IObservableLookup<K, V> : IDisposable
{
IObservable<V> this[K key] { get; }
}
You would use it like this:
IObservable<Packets> Packets = ...
IObservableLookup<int, Packets> lookup = Packets.ToLookup(p => p.Id, p => p, Scheduler.Default);
lookup[1].Subscribe(p => { });
lookup[2].Subscribe(p => { });
// etc
The nice thing with this is that you can subscribe to values by key before a value with that key has been produced by the source observable.
Don't forget to call lookup.Dispose() when done to clean up the resources.

I would suggest looking at GroupBy and then checking if there is a performance pay off. I assume there is, but is it significant?
Packets.GroupBy(p=>p.Id)
Example code with tests on how to use GroupBy as a type of router
var scheduler = new TestScheduler();
var source = scheduler.CreateColdObservable(
ReactiveTest.OnNext(100, 1),
ReactiveTest.OnNext(200, 2),
ReactiveTest.OnNext(300, 3),
ReactiveTest.OnNext(400, 4),
ReactiveTest.OnNext(500, 5),
ReactiveTest.OnNext(600, 6),
ReactiveTest.OnNext(700, 7),
ReactiveTest.OnNext(800, 8),
ReactiveTest.OnNext(900, 9),
ReactiveTest.OnNext(1000, 10),
ReactiveTest.OnNext(1100, 11)
);
var router = source.GroupBy(i=>i%4)
.Publish()
.RefCount();
var zerosObserver = scheduler.CreateObserver<int>();
router.Where(grp=>grp.Key == 0)
.Take(1)
.SelectMany(grp=>grp)
.Subscribe(zerosObserver);
var onesObserver = scheduler.CreateObserver<int>();
router.Where(grp => grp.Key == 1)
.Take(1)
.SelectMany(grp => grp)
.Subscribe(onesObserver);
var twosObserver = scheduler.CreateObserver<int>();
router.Where(grp => grp.Key == 2)
.Take(1)
.SelectMany(grp => grp)
.Subscribe(twosObserver);
var threesObserver = scheduler.CreateObserver<int>();
router.Where(grp => grp.Key == 3)
.Take(1)
.SelectMany(grp => grp)
.Subscribe(threesObserver);
scheduler.Start();
ReactiveAssert.AreElementsEqual(new[] { ReactiveTest.OnNext(400, 4), ReactiveTest.OnNext(800, 8)}, zerosObserver.Messages);
ReactiveAssert.AreElementsEqual(new[] { ReactiveTest.OnNext(100, 1), ReactiveTest.OnNext(500, 5), ReactiveTest.OnNext(900, 9)}, onesObserver.Messages);
ReactiveAssert.AreElementsEqual(new[] { ReactiveTest.OnNext(200, 2), ReactiveTest.OnNext(600, 6), ReactiveTest.OnNext(1000, 10) }, twosObserver.Messages);
ReactiveAssert.AreElementsEqual(new[] { ReactiveTest.OnNext(300, 3), ReactiveTest.OnNext(700, 7), ReactiveTest.OnNext(1100, 11)}, threesObserver.Messages);

You can use GroupBy to split the data. I would suggest you set up all subscriptions first and then activate your source. Doing so would result in one huge nested GroupBy query, but it is also possible to multi-cast your groups and subscribe to them individually. I wrote a small helper utility to do so below.
Because you still might want to add new routes after the source has been activated (done trough Connect), we use Replay to replay the groups. Replay is also a multi-cast operator so we wont need Publish to multi-cast.
public sealed class RouteData<TKey, TSource>
{
private IConnectableObservable<IGroupedObservable<TKey, TSource>> myRoutes;
public RouteData(IObservable<TSource> source, Func<TSource, TKey> keySelector)
{
this.myRoutes = source.GroupBy(keySelector).Replay();
}
public IDisposable Connect()
{
return this.myRoutes.Connect();
}
public IObservable<TSource> Get(TKey id)
{
return myRoutes.FirstAsync(e => e.Key.Equals(id)).Merge();
}
}
public static class myExtension
{
public static RouteData<TKey, TSource> RouteData<TKey, TSource>(this IObservable<TSource> source, Func<TSource, TKey> keySelector)
{
return new RouteData<TKey, TSource>(source, keySelector);
}
}
Example usage:
public class myPackage
{
public int Id;
public myPackage(int id)
{
this.Id = id;
}
}
class program
{
static void Main()
{
var source = new[] { 0, 1, 2, 3, 4, 5, 4, 3 }.ToObservable().Select(i => new myPackage(i));
var routes = source.RouteData(e => e.Id);
var subscription = new CompositeDisposable(
routes.Get(5).Subscribe(Console.WriteLine),
routes.Get(4).Subscribe(Console.WriteLine),
routes.Get(3).Subscribe(Console.WriteLine),
routes.Connect());
Console.ReadLine();
}
}

You may want to consider writing a custom IObserver that does your bidding. I've included an example below.
void Main()
{
var source = Observable.Range(1, 10);
var switcher = new Switch<int, int>(i => i % 3);
switcher[0] = Observer.Create<int>(val => Console.WriteLine($"{val} Divisible by three"));
source.Subscribe(switcher);
}
class Switch<TKey,TValue> : IObserver<TValue>
{
private readonly IDictionary<TKey, IObserver<TValue>> cases;
private readonly Func<TValue,TKey> idExtractor;
public IObserver<TValue> this[TKey decision]
{
get
{
return cases[decision];
}
set
{
cases[decision] = value;
}
}
public Switch(Func<TValue,TKey> idExtractor)
{
this.cases = new Dictionary<TKey, IObserver<TValue>>();
this.idExtractor = idExtractor;
}
public void OnNext(TValue next)
{
IObserver<TValue> nextCase;
if (cases.TryGetValue(idExtractor(next), out nextCase))
{
nextCase.OnNext(next);
}
}
public void OnError(Exception e)
{
foreach (var successor in cases.Values)
{
successor.OnError(e);
}
}
public void OnCompleted()
{
foreach (var successor in cases.Values)
{
successor.OnCompleted();
}
}
}
You would obviously need to implement idExtractor to extract the ids from your packet.

Remove an item from ienumerable<string>

I have a method that expect IEnumerable<string> as you can see here :
public static string FromDecimalAscii(IEnumerable<string> input)
{
return new string(input.Select(s => (char)int.Parse(s)).ToArray());
}
But every time the last record of my IEnumerable is empty so i got an error in this line because of that :
return new string(input.Select(s => (char)int.Parse(s)).ToArray());
So i have to remove that item from my IEnumerable .
The error:Input string was not in a correct format
Any ideas will be appreciated.
Best regards

You need just to filter collection with Where:
return new string(input.Where(s => !string.IsNullOrEmpty(s))
.Select(s => (char)int.Parse(s)).ToArray());
You can also use extension method to use TryParse:
static class Extensions
{
public delegate bool TryParseDelegate<TSource>(string s, out TSource source);
public static IEnumerable<TResult> WhereParsed<TSource, TResult>(
this IEnumerable<TSource> source,
TryParseDelegate<TResult> tryParse)
{
// TODO: check arguments against null first
foreach (var item in source)
{
TResult result;
if (tryParse(item != null ? item.ToString() : null, out result))
{
yield return result;
}
}
}
}
Usage:
var collection = input.WhereParsed<string, int>(int.TryParse)
.Cast<char>()
.ToArray();
return new string(collection);

LINQ query to perform a projection, skipping or wrapping exceptions where source throws on IEnumerable.GetNext()

I'd like a general solution but as an example, assume i have an IEnumerable<string>, where some can be parsed as integers, and some cannot.
var strings = new string[] { "1", "2", "notint", "3" };
Obviously if i did Select(s => int.Parse(s, temp)) it'd throw an exception when enumerated.
In this case i could do .All(s => int.TryParse(s, out temp)) first, however i want a general solution where i don't have to enumerate the IEnumerable twice.
Ideally i'd like to be able to do the following, which calls my magic exception skipping method:
// e.g. parsing strings
var strings = new string[] { "1", "2", "notint", "3" };
var numbers = strings.Select(s => int.Parse(s)).SkipExceptions();
// e.g. encountering null object
var objects = new object[] { new object(), new object(), null, new object() }
var objecttostrings = objects.Select(o => o.ToString()).SkipExceptions();
// e.g. calling a method that could throw
var myClassInstances = new MyClass[] { new MyClass(), new MyClass(CauseMethodToThrow:true) };
var myClassResultOfMethod = myClassInstances.Select(mci => mci.MethodThatCouldThrow()).SkipExceptions();
How can i write the SkipExceptions() extension method?
Some great answers for a SelectSkipExceptions() method, however i wonder if a SkipExceptions() method could be created, along the same lines as AsParallel().

How about this (you might want to give this special Select Extension a better name)
public static IEnumerable<TOutput> SelectIgnoringExceptions<TInput, TOutput>(
this IEnumerable<TInput> values, Func<TInput, TOutput> selector)
{
foreach (var item in values)
{
TOutput output = default(TOutput);
try
{
output = selector(item);
}
catch
{
continue;
}
yield return output;
}
}
Edit5
Added a using statement, thanks for the suggestion in comments
public static IEnumerable<T> SkipExceptions<T>(
this IEnumerable<T> values)
{
using(var enumerator = values.GetEnumerator())
{
bool next = true;
while (next)
{
try
{
next = enumerator.MoveNext();
}
catch
{
continue;
}
if(next) yield return enumerator.Current;
}
}
}
However this relies on the incoming IEnumerable not already being created (and therefore already having thrown Exceptions) as a list by the preceding Function. E.g. this would probably not work if you call it like this: Select(..).ToList().SkipExceptions()

Create a TryParseInt method that returns a Nullable<int>:
int? TryParseInt(string s)
{
int i;
if (int.TryParse(s, out i))
return i;
return null;
}
And use it in your query like that:
var numbers = strings.Select(s => TryParseInt(s))
.Where(i => i.HasValue)
.Select(i => i.Value);
See also this article by Bill Wagner, which presents a very similar case.
Now, i don't think you can write something like a generic SkipExceptions method, because you would catch the exception too late, and it would end the Select loop... But you could probably write a SelectSkipException method:
public static IEnumerable<TResult> SelectSkipExceptions<TSource, TResult>(
this IEnumerable<TSource> source,
Func<TSource, TResult> selector)
{
if (source == null)
throw new ArgumentNullException("source");
if (selector == null)
throw new ArgumentNullException("selector");
return source.SelectSkipExceptionsIterator(selector);
}
private static IEnumerable<TResult> SelectSkipExceptionsIterator<TSource, TResult>(
this IEnumerable<TSource> source,
Func<TSource, TResult> selector)
{
foreach(var item in source)
{
TResult value = default(TResult);
try
{
value = selector(item);
}
catch
{
continue;
}
yield return value;
}
}

Even the accepted answer may not be "general" enough. What if some day you find that you need to know what exceptions occurred?
The following extension
static class EnumeratorHelper {
//Don't forget that GetEnumerator() call can throw exceptions as well.
//Since it is not easy to wrap this within a using + try catch block with yield,
//I have to create a helper function for the using block.
private static IEnumerable<T> RunEnumerator<T>(Func<IEnumerator<T>> generator,
Func<Exception, bool> onException)
{
using (var enumerator = generator())
{
if (enumerator == null)
yield break;
for (; ; )
{
//You don't know how to create a value of T,
//and you don't know weather it can be null,
//but you can always have a T[] with null value.
T[] value = null;
try
{
if (enumerator.MoveNext())
value = new T[] { enumerator.Current };
}
catch (Exception e)
{
if (onException(e))
continue;
}
if (value != null)
yield return value[0];
else
yield break;
}
}
}
public static IEnumerable<T> WithExceptionHandler<T>(this IEnumerable<T> orig,
Func<Exception, bool> onException)
{
return RunEnumerator(() =>
{
try
{
return orig.GetEnumerator();
}
catch (Exception e)
{
onException(e);
return null;
}
}, onException);
}
}
will help. Now you can add SkipExceptions:
public static IEnumerable<T> SkipExceptions<T>(this IEnumerable<T> orig){
return orig.WithExceptionHandler(orig, e => true);
}
By using different onException callback, you can do different things
Break the iteration but ignore the exception: e => false
Try to continue iteration: e => true
Log the exception, etc

Here's a small complete program to demonstrate an answer inspired by the maybe monad. You might want to change the name of the 'Maybe' class, as it is inspired by rather than actually being a 'Maybe' as defined in other languages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace TestMaybe
{
class Program
{
static void Main(string[] args)
{
var strings = new string[] { "1", "2", "notint", "3" };
var ints = strings.Select(s => new Maybe<string, int>(s, str => int.Parse(str))).Where(m => !m.nothing).Select(m => m.value);
foreach (var i in ints)
{
Console.WriteLine(i);
}
Console.ReadLine();
}
}
public class Maybe<T1, T2>
{
public readonly bool nothing;
public readonly T2 value;
public Maybe(T1 input, Func<T1, T2> map)
{
try
{
value = map(input);
}
catch (Exception)
{
nothing = true;
}
}
}
}
Edit: depending on the needs of your code, you might also want nothing set to true if the result of map(input) is null.

This is the same answer as Thomas's, but with a lambda & LINQ expression. +1 for Thomas.
Func<string, int?> tryParse = s =>
{
int? r = null;
int i;
if (int.TryParse(s, out i))
{
r = i;
}
return r;
};
var ints =
from s in strings
let i = tryParse(s)
where i != null
select i.Value;

You could just chain the Where and Select method together.
var numbers = strings.Where(s =>
{
int i;
return int.TryParse(s, out i);
}).Select(int.Parse);
The use of the Where method effectively removes the need for you to write your own SkipExceptions method, because this is basically what you are doing.

.net Rx: in-order batch-processing of messages

I am attempting to implement an asynchronous workflow using Rx and I seem to be doing it completely wrong.
What I would like to do is this:
From an undefined asynchronous stream of un-parsed message strings (i.e. an IObservable<string>)
parse the message strings asynchronously, but preserve their order. (IObservable<Message>)
Batch up parsed Messages in groups of 100 or so (IObservable<IEnumerable<Message>>)
Send each batch, when complete, to the UI thread to be processed. Batches must arrive in the same order they were started.
I can't seem to get the order-preservation, and also Rx doesn't appear to be doing things asynchronously when I expected them to.
I made an attempt at order preservation by using an IEnumerable instead of an IObservable, and then calling the .AsParallel().AsOrdered() operators on it. Here is the code. See notes below for the issues I'm having:
private IObservable<IEnumerable<Message>> messageSource;
public IObservable<IEnumerable<Message>> MessageSource { get { return messageSource; } }
/// <summary>
/// Sub-classes of MessageProviderBase provide this IEnumerable to
/// generate unparsed message strings synchronously
/// </summary>
protected abstract IEnumerable<string> UnparsedMessages { get; }
public MessageProviderBase()
{
// individual parsed messages as a PLINQ query
var parsedMessages = from unparsedMessage in UnparsedMessages.AsParallel().AsOrdered()
select ParseMessage(unparsedMessage);
// convert the above PLINQ query to an observable, buffering up to 100 messages at a time
var batchedMessages
= parsedMessages.ToObservable().BufferWithTimeOrCount(TimeSpan.FromMilliseconds(200), 100);
// ISSUE #1:
// batchedMessages seems to call OnNext before all of the messages in its buffer are parsed.
// If you convert the IObservable<Message> it generates to an enumerable, it blocks
// when you try to enumerate it.
// Convert each batch to an IEnumerable
// ISSUE #2: Even if the following Rx query were to run asynchronously (it doesn't now, see the above comment),
// it could still deliver messages out of order. Only, instead of delivering individual
// messages out of order, the message batches themselves could arrive out of order.
messageSource = from messageBatch in batchedMessages
select messageBatch.ToEnumerable().ToList();
}

My answer below is somewhat based on Enigmativity's code, but fixes a number of race conditions related to completion and also adds support for cancellation and custom schedulers (which would make unit testing it significantly easier).
public static IObservable<U> Fork<T, U>(this IObservable<T> source,
Func<T, U> selector)
{
return source.Fork<T, U>(selector, Scheduler.TaskPool);
}
public static IObservable<U> Fork<T, U>(this IObservable<T> source,
Func<T, U> selector, IScheduler scheduler)
{
return Observable.CreateWithDisposable<U>(observer =>
{
var runningTasks = new CompositeDisposable();
var lockGate = new object();
var queue = new Queue<ForkTask<U>>();
var completing = false;
var subscription = new MutableDisposable();
Action<Exception> onError = ex =>
{
lock(lockGate)
{
queue.Clear();
observer.OnError(ex);
}
};
Action dequeue = () =>
{
lock (lockGate)
{
var error = false;
while (queue.Count > 0 && queue.Peek().Completed)
{
var task = queue.Dequeue();
observer.OnNext(task.Value);
}
if (completing && queue.Count == 0)
{
observer.OnCompleted();
}
}
};
Action onCompleted = () =>
{
lock (lockGate)
{
completing = true;
dequeue();
}
};
Action<T> enqueue = t =>
{
var cancellation = new MutableDisposable();
var task = new ForkTask<U>();
lock(lockGate)
{
runningTasks.Add(cancellation);
queue.Enqueue(task);
}
cancellation.Disposable = scheduler.Schedule(() =>
{
try
{
task.Value = selector(t);
lock(lockGate)
{
task.Completed = true;
runningTasks.Remove(cancellation);
dequeue();
}
}
catch(Exception ex)
{
onError(ex);
}
});
};
return new CompositeDisposable(runningTasks,
source.AsObservable().Subscribe(
t => { enqueue(t); },
x => { onError(x); },
() => { onCompleted(); }
));
});
}
private class ForkTask<T>
{
public T Value = default(T);
public bool Completed = false;
}
Here is a sample that randomizes the task execution time to test it:
AutoResetEvent are = new AutoResetEvent(false);
Random rand = new Random();
Observable.Range(0, 5)
.Fork(i =>
{
int delay = rand.Next(50, 500);
Thread.Sleep(delay);
return i + 1;
})
.Subscribe(
i => Console.WriteLine(i),
() => are.Set()
);
are.WaitOne();
Console.ReadLine();

Given you have:
IObservable<string> UnparsedMessages = ...;
Func<string, Message> ParseMessage = ...;
Then you could use a SelectAsync extension method like so:
IObservable<Message> ParsedMessages = UnparsedMessages.SelectAsync(ParseMessage);
The SelectAsync extension method processes each unparsed message asynchronously and ensures that the results come back in the order they arrived.
Let me know if this does what you need.
Here's the code:
public static IObservable<U> SelectAsync<T, U>(this IObservable<T> source,
Func<T, U> selector)
{
var subject = new Subject<U>();
var queue = new Queue<System.Threading.Tasks.Task<U>>();
var completing = false;
var subscription = (IDisposable)null;
Action<Exception> onError = ex =>
{
queue.Clear();
subject.OnError(ex);
subscription.Dispose();
};
Action dequeue = () =>
{
lock (queue)
{
var error = false;
while (queue.Count > 0 && queue.Peek().IsCompleted)
{
var task = queue.Dequeue();
if (task.Exception != null)
{
error = true;
onError(task.Exception);
break;
}
else
{
subject.OnNext(task.Result);
}
}
if (!error && completing && queue.Count == 0)
{
subject.OnCompleted();
subscription.Dispose();
}
}
};
Action<T> enqueue = t =>
{
if (!completing)
{
var task = new System.Threading.Tasks.Task<U>(() => selector(t));
queue.Enqueue(task);
task.ContinueWith(tu => dequeue());
task.Start();
}
};
subscription = source.Subscribe(
t => { lock(queue) enqueue(t); },
x => { lock(queue) onError(x); },
() => { lock(queue) completing = true; });
return subject.AsObservable();
}
I ended up needing to revisit this for work and wrote a more robust version of this code (based also on Richard's answer.)
The key advantage to this code is the absence of any explicit queue. I'm purely using task continuations to put the results back in order. Works like a treat!
public static IObservable<U> ForkSelect<T, U>(this IObservable<T> source, Func<T, U> selector)
{
return source.ForkSelect<T, U>(t => Task<U>.Factory.StartNew(() => selector(t)));
}
public static IObservable<U> ForkSelect<T, U>(this IObservable<T> source, Func<T, Task<U>> selector)
{
if (source == null) throw new ArgumentNullException("source");
if (selector == null) throw new ArgumentNullException("selector");
return Observable.CreateWithDisposable<U>(observer =>
{
var gate = new object();
var onNextTask = Task.Factory.StartNew(() => { });
var sourceCompleted = false;
var taskErrored = false;
Action<Exception> onError = ex =>
{
sourceCompleted = true;
onNextTask = onNextTask.ContinueWith(t => observer.OnError(ex));
};
Action onCompleted = () =>
{
sourceCompleted = true;
onNextTask = onNextTask.ContinueWith(t => observer.OnCompleted());
};
Action<T> onNext = t =>
{
var task = selector(t);
onNextTask = Task.Factory.ContinueWhenAll(new[] { onNextTask, task }, ts =>
{
if (!taskErrored)
{
if (task.IsFaulted)
{
taskErrored = true;
observer.OnError(task.Exception);
}
else
{
observer.OnNext(task.Result);
}
}
});
};
var subscription = source
.AsObservable()
.Subscribe(
t => { if (!sourceCompleted) lock (gate) onNext(t); },
ex => { if (!sourceCompleted) lock (gate) onError(ex); },
() => { if (!sourceCompleted) lock (gate) onCompleted(); });
var #return = new CompositeDisposable(subscription);
return #return;
});
}
And the SelectMany overloads to allow LINQ to be used are:
public static IObservable<U> SelectMany<T, U>(this IObservable<T> source, Func<T, Task<U>> selector)
{
return source.ForkSelect<T, U>(selector);
}
public static IObservable<V> SelectMany<T, U, V>(this IObservable<T> source, Func<T, Task<U>> taskSelector, Func<T, U, V> resultSelector)
{
if (source == null) throw new ArgumentNullException("source");
if (taskSelector == null) throw new ArgumentNullException("taskSelector");
if (resultSelector == null) throw new ArgumentNullException("resultSelector");
return source.Zip(source.ForkSelect<T, U>(taskSelector), (t, u) => resultSelector(t, u));
}
So these methods can now be used like this:
var observableOfU = observableOfT.ForkSelect(funcOfT2U);
Or:
var observableOfU = observableOfT.ForkSelect(funcOfT2TaskOfU);
Or:
var observableOfU =
from t in observableOfT
from u in funcOfT2TaskOfU(t)
select u;
Enjoy!

Get previous element in IObservable without re-evaluating the sequence

In an IObservable sequence (in Reactive Extensions for .NET), I'd like to get the value of the previous and current elements so that I can compare them. I found an example online similar to below which accomplishes the task:
sequence.Zip(sequence.Skip(1), (prev, cur) => new { Previous = prev, Current = cur })
It works fine except that it evaluates the sequence twice, which I would like to avoid. You can see that it is being evaluated twice with this code:
var debugSequence = sequence.Do(item => Debug.WriteLine("Retrieved an element from sequence"));
debugSequence.Zip(debugSequence.Skip(1), (prev, cur) => new { Previous = prev, Current = cur }).Subscribe();
The output shows twice as many of the debug lines as there are elements in the sequence.
I understand why this happens, but so far I haven't found an alternative that doesn't evaluate the sequence twice. How can I combine the previous and current with only one sequence evaluation?

There's a better solution to this I think, that uses Observable.Scan and avoids the double subscription:
public static IObservable<Tuple<TSource, TSource>>
PairWithPrevious<TSource>(this IObservable<TSource> source)
{
return source.Scan(
Tuple.Create(default(TSource), default(TSource)),
(acc, current) => Tuple.Create(acc.Item2, current));
}
I've written this up on my blog here: http://www.zerobugbuild.com/?p=213
Addendum
A further modification allows you to work with arbitrary types more cleanly by using a result selector:
public static IObservable<TResult> CombineWithPrevious<TSource,TResult>(
this IObservable<TSource> source,
Func<TSource, TSource, TResult> resultSelector)
{
return source.Scan(
Tuple.Create(default(TSource), default(TSource)),
(previous, current) => Tuple.Create(previous.Item2, current))
.Select(t => resultSelector(t.Item1, t.Item2));
}

#James World addendum looks great to me, if not for Tuple<>, which I almost always dislike: "Was .Item1 the previous? Or was it the current one? I can't remember. And what's the first argument to the selector, was it the previous item?".
For that part I liked #dcstraw definition of a dedicated ItemWithPrevious<T>. So there you go, putting the two together (hopefully I did not mix up previous with current) with some renaming and facilities:
public static class ObservableExtensions
{
public static IObservable<SortedPair<TSource>> CombineWithPrevious<TSource>(
this IObservable<TSource> source,
TSource initialValue = default(TSource))
{
var seed = SortedPair.Create(initialValue, initialValue);
return source.Scan(seed,
(acc, current) => SortedPair.Create(current, acc.Current));
}
public static IObservable<TResult> CombineWithPrevious<TSource, TResult>(
this IObservable<TSource> source,
Func<SortedPair<TSource>, TResult> resultSelector,
TSource initialValue = default(TSource))
{
var seed = SortedPair.Create(initialValue, initialValue);
return source
.Scan(seed,
(acc, current) => SortedPair.Create(current, acc.Current))
.Select(p => resultSelector(p));
}
}
public class SortedPair<T>
{
public SortedPair(T current, T previous)
{
Current = current;
Previous = previous;
}
public SortedPair(T current) : this(current, default(T)) { }
public SortedPair() : this(default(T), default(T)) { }
public T Current;
public T Previous;
}
public class SortedPair
{
public static SortedPair<T> Create<T>(T current, T previous)
{
return new SortedPair<T>(current, previous);
}
public static SortedPair<T> Create<T>(T current)
{
return new SortedPair<T>(current);
}
public static SortedPair<T> Create<T>()
{
return new SortedPair<T>();
}
}

Evaluating twice is an indicator of a Cold observable. You can turn it to a Hot one by using .Publish():
var pub = sequence.Publish();
pub.Zip(pub.Skip(1), (...
pub.Connect();

If you only need to access the previous element during subscription, this is probably the simplest thing that will work. (I'm sure there's a better way, maybe a buffer operator on IObservable? The documentation is pretty sparse at the moment, so I can't really tell you.)
EventArgs prev = null;
sequence.Subscribe(curr =>
{
if (prev != null)
{
// Previous and current element available here
}
prev = curr;
});
EventArgs is just a stand-in for the type of your event's argument.

It turns out you can use a variable to hold the previous value and refer to it and reassign it within the chain of IObservable extensions. This even works within a helper method. With the code below I can now call CombineWithPrevious() on my IObservable to get a reference to the previous value, without re-evaluating the sequence.
public class ItemWithPrevious<T>
{
public T Previous;
public T Current;
}
public static class MyExtensions
{
public static IObservable<ItemWithPrevious<T>> CombineWithPrevious<T>(this IObservable<T> source)
{
var previous = default(T);
return source
.Select(t => new ItemWithPrevious<T> { Previous = previous, Current = t })
.Do(items => previous = items.Current);
}
}