I just started messing around with reactive programming, and I know just enough to write code but not enough to figure out what's happening when I don't get what I expect. I don't really have a mentor available other than blog posts. I haven't found a very good solution to a situation I'm having, and I'm curious about the right approach.
The problem:
I need to get a Foo, which is partially composed of an array of Bar objects. I fetch the Bar objects from web services. So I represented each web service call as an IObservable from which I expect 0 or 1 elements before completion. I want to make an IObservable that will:
Subscribe to each of the IObservable instances.
Wait for up to a 2 second Timeout.
When either both sequences complete or the timeout happens:
Create an array with any of the Bar objects that were generated (there may be 0.)
Produce the Foo object using that Bar[].
I sort of accomplished this with this bit of code:
public Foo CreateFoo() {
var producer1 = webService.BarGenerator()
.Timeout(TimeSpan.FromSeconds(2), Observable.Empty<Bar>());
var producer2 = // similar to above
var pipe = producer1.Concat(producer2);
Bar[] result = pipe.ToEnumerable().ToArray();
...
}
That doesn't seem right, for a lot of reasons. The most obvious is Concat() will start the sequences serially rather than in parallel, so that's a 4-second timeout. I don't really care that it blocks, it's actually convenient for the architecture I'm working with that it does. I'm fine with this method becoming a generator of IObservable, but there's a few extra caveats here that seem to make that challenging when I try:
I need the final array to put producer1 and producer2's result in that order, if they both produce a result.
I'd like to use a TestScheduler to verify the timeout but haven't succeeded at that yet, I apparently don't understand schedulers at all.
This is, ultimately, a pull model, whatever gets the Foo needs it at a distinct point and there's no value to receiving it 'on the fly'. Maybe this tilts the answer to "Don't use Rx". To be honest, I got stuck enough I switched to a Task-based API. But I want to see how one might approach this with Rx, because I want to learn.
var pipe = producer1
.Merge(producer2)
.Buffer(Observable.Timer(TimeSpan.FromSeconds(2), testScheduler))
.Take(1);
var subscription = pipe
.Select(list => new Foo(list.ToArray())
.Subscribe(foo => {} /* Do whatever you want with your foo here.*/);
Buffer takes all elements emitted during a window (in our case in two seconds), and outputs a list.
If you want to stick with your pull model, instead of a subscription you could do:
var list = await pipe;
var foo = new Foo(list.ToArray());
//....
Related
I've watched the chat on LINQ with IAsyncEnumerable which has given me some insight on dealing with extension methods for IAsyncEnumerables, but wasn't detailed enough frankly for a real-world application, especially for my experience level, and I understand that samples/documentation don't really exist as of yet for IAsyncEnumerables
I'm trying to read from a file, do some transformation on the stream, returning a IAsyncEnumerable, and then send those objects downstream after an arbitrary number of objects have been obtained, like:
await foreach (var data in ProcessBlob(downloadedFile))
{
//todo add data to List<T> called listWithPreConfiguredNumberOfElements
if (listWithPreConfiguredNumberOfElements.Count == preConfiguredNumber)
await _messageHandler.Handle(listWithPreConfiguredNumberOfElements);
//repeat the behaviour till all the elements in the IAsyncEnumerable returned by ProcessBlob are sent downstream to the _messageHandler.
}
My understanding from reading on the matter so far is that the await foreach line is working on data that employs the use of Tasks (or ValueTasks), so we don't have a count up front. I'm also hesitant to use a List variable and just do a length-check on that as sharing that data across threads doesn't seem very thread-safe.
I'm using the System.Linq.Async package in the hopes that I could use a relevant extensions method. I can see some promise in the form of TakeWhile, but my understanding on how thread-safe the task I intend to do is not all there, causing me to lose confidence.
Any help or push in the right direction would be massively appreciated, thank you.
There is an operator Buffer that does what you want, in the package System.Interactive.Async.
// Projects each element of an async-enumerable sequence into consecutive
// non-overlapping buffers which are produced based on element count information.
public static IAsyncEnumerable<IList<TSource>> Buffer<TSource>(
this IAsyncEnumerable<TSource> source, int count);
This package contains operators like Amb, Throw, Catch, Defer, Finally etc that do not have a direct equivalent in Linq, but they do have an equivalent in System.Reactive. This is because IAsyncEnumerables are conceptually closer to IObservables than to IEnumerables (because both have a time dimension, while IEnumerables are timeless).
I'm also hesitant to use a List variable and just do a length-check on that as sharing that data across threads doesn't seem very thread-safe.
You need to think in terms of execution flows, not threads, when dealing with async; since you are await-ing the processing step, there isn't actually a concurrency problem accessing the list, because regardless of which threads are used: the list is only accessed once at a time.
If you are still concerned, you could new a list per batch, but that is probably overkill. What you do need, however, is two additions - a reset between batches, and a final processing step:
var listWithPreConfiguredNumberOfElements = new List<YourType>(preConfiguredNumber);
await foreach (var data in ProcessBlob(downloadedFile)) // CAF?
{
listWithPreConfiguredNumberOfElements.Add(data);
if (listWithPreConfiguredNumberOfElements.Count == preConfiguredNumber)
{
await _messageHandler.Handle(listWithPreConfiguredNumberOfElements); // CAF?
listWithPreConfiguredNumberOfElements.Clear(); // reset for a new batch
// (replace this with a "new" if you're still concerned about concurrency)
}
}
if (listWithPreConfiguredNumberOfElements.Any())
{ // process any stragglers
await _messageHandler.Handle(listWithPreConfiguredNumberOfElements); // CAF?
}
You might also choose to use ConfigureAwait(false) in the three spots marked // CAF?
I'm starting with reactive extensions and I'm having a problem where I'm not sure if I'm on the right track.
I'm using an Observable to create and consume a listener for an event broker with .NET. I created a "IncomingMessage" class which contains the messages from the eventbroker as they come in and I start creating the listener in the Observerable.Create function. That works very well.
Now I also want to get status notification from the listener as in "Connecting...", "Connected.", "Closing..." which are not an IncomingMessage so I created a class "BrokerEvent" with a "Message" property and an interface for "IncomingMessage" and "BrokerEvent". Now I send both via observer.OnNext(...) as they occur. That also works well so far.
However on the Subscriber side I'm now having a bit of a problem to filter the events I need.
I do:
GetObservable().Where(x => x is BrokerEvent ||
(x is IncomingMessage msg &&
msg.User == "test")).Subscribe(...)
That works however I then need to figure out the type in Subscribe again which seems a bit ugly.
After trying a bit I ended up doing this now...
var observable = GetObservable().Publish();
observable.OfType<BrokerEvent>().Subscribe(...);
observable.OfType<IncomingMessage>().Where(x=>x.User == "test").Subscribe(...);
var disposable = observable.Connect();
This also seems to work but as I'm new to reactive extensions I'm not quite sure if that has any unwanted sideeffects. I'm also not sure if it's the "right" way to include status messages into the stream at all. Is there any better way to handle that (possible without using Publish) or is that the way to go?
And to stop listening is it enough to just dispose the disposable I get from .Connect() or do I have to also dispose both disposables I get from .Subscribe()?
Thanks!
I'm assuming GetObservable returns IObservable<object>, which isn't ideal.
The best way to do the code you have above is as follows:
var observable = GetObservable().Publish();
var sub1 = observable.OfType<BrokerEvent>().Subscribe(_ => { });
var sub2 = observable.OfType<IncomingMessage>().Where(x => x.User == "test").Subscribe(_ => { });
var connectSub = observable.Connect();
var disposable = new CompositeDisposable(sub1, sub2, connectSub);
The composite disposable will then dispose of all the children when it is disposed.
If the two message streams have nothing to do with each other, that approach will work. However, since you basically have a control-message stream, and a data-message stream, I'm assuming the messages from one may be important when handling the messages in the other. In this case you may want to treat it like one stream.
In that case, you may want to create a Discriminated-Union type for your observable, which may make handling easier.
What you can do is create an 'event handler' class with three overloads of a 'process message'. One for object (default) that does nothing, one for the status type, one for incoming message. In .Subscribe use this syntax
m=>processor.Process((dynamic)m)
This will call the correct implementation or do nothing, as required.
If you want to filter before calling the Process, you can introduce a common class (ProcessableMesage or some such), or you can call .Merge on your OfType streams, or you can take the same approach as above by having a dynamic MessageFilter class.
Is there maybe something like a "when" statement in C#?
The reason I want this is because an if statement only checks once if a certain property is true at a particular time, but I want it to wait until the property is true.
Anybody know something I can do in C# that would be similar to a "when" statement?
What you want is SpinWait
e.g. SpinWait.SpinUntil(() => condition);
It will sit there until it either times out (with your specified timeout) or the condition is met.
There is no when control statement, but there are two options which might meet your needs:
You can use a while(predicate){} loop to keep looping until a condition is met. The predicate can be any expression which returns true/false - as long as the condition is true, it will loop. If you just want to wait without consuming too much CPU, you can Sleep() the thread within the loop:
while(name == "Sebastian")
{
// Code to execute
System.Threading.Thread.Sleep(1000);
}
If you property is a numeric range, you could use a for loop, but that doesn't sound like what you want.
If you want to deal with an asynchronous world than you may be should look at the library Rx.NET. Let's look at the simple example: suppose you want to read strings from the console and when user inputs word "hello" you need to print "world" in the response. This simple example can be implemented as follows:
var inputLines = new Subject<string>();
inputLines.Subscribe(info =>
{
if (info == "hello")
Console.Out.WriteLine("world");
});
while (true)
{
var line = Console.In.ReadLine();
inputLines.OnNext(line);
}
So, there are explicit when action, that we pass in the Subscribe(...) function.
In this simple example usage of Rx.NET is obviously unnecessarily and you shouldn't do it. But in more complex scenarios this is a very helpful library. You can see, that with Reactive Extensions you split the logic of your application from the main event-pool, where you can want to do some other work, not related to the application logic. Also, there is high flexibility that you can get with this library, because it's very dynamic - you can subscribe and unsubscribe different events in run-time at any time.
You can notice, that there is another way to solve my example in the event-based paradigm. We can simply use built-in events like this:
public static event EventHandler<string> InputEvent;
public void Run()
{
InputEvent += (sender, line) => {
if (line == "hello")
Console.WriteLine("world");
};
while (true) {
var line = Console.In.ReadLine();
InputEvent?.Invoke(this, line);
}
}
And this is a right point, sometimes you can replace Reactive Extensions with simple events because they are connected. But when you need to build a complex pipeline from many event sources and using many different tightly coupled actions, then Reactive Extensions allow you to nicely build this pipeline in the very declarative way.
You can use 'async' and 'await' to wait unit a certain 'Task' is complete. 'await' is somewhat similar to the 'when' statement you need. Only it pauses the current 'Task' until the awaited 'Task' finished with any result not just when an expression becomes 'true'. See also TaskCompletionSource.
What are you trying to achieve here? Are you running a synchronous process or are you waiting for something asynchronous to happen?
If you're synchronous then while is probably the correct solution:
var result = 0;
while(result != 6)
{
result = RollADie();
Console.WriteLine($"I rolled a {result}");
}
Console.WriteLine("At last, a six!");
But - if you're waiting for something asynchronous to happen then a different solution is called for. An asynchronous scenario is where you want your code to 'hang around and wait, doing nothing' until the condition is fulfilled.
In that case the modern C# solution is asynchronous programming using the async and await keywords, along with the Task class (and it's generic cousin Task<TResult>). That's probably a bit deep to go into here, but here's a pretty good primer.
What's important is that you don't use a solution based on while in order to deal with asynchronous processes. You'll just send the CPU spinning in circles, chasing it's own tail so to speak, when really you want to say "now stop working on this until X happens". Also, avoid any solution based on while combined with Thread.Sleep for related reasons.
I'm currently building a wizard system for an application, and we're using ReactiveUI and as a result Rx.
Each step in the wizard implements IWizardStep<T> where T is just the data type that the wizard ultimately produces.
Each step has the capability of surfacing which step should be available for the user to move to next, so as to enable branching based on data entry.
The step can be thought of having a similar structure to:
public interface IWizardStep<T>
{
IObservable<IStepChoice<T>> NextStepChoice {get;}
}
With IStepChoice<T> simply being:
public interface IStepChoice<T>
{
IWizardStep<T> Step {get;}
string Reason {get;}
}
In order to calculate the current path from the start to the end, for display to the user, I need to be able to go from the starting step, and walk along the NextStepChoice chain recursively until it hits a null (it's valid behaviour for the NextStepChoice observable to emit a null to indicate the end of the wizard).
I've taken a look at Observable.Scan but can't for the life of me work out how to get this working properly recursively.
I've also taken a look at Observable.Generate which looks promising, as this is a classic unfold-esque problem; the only issue is that Generate takes a function to determine when to break the loop, but I need to evaluate the inner observable to work this out.
Observable.Generate(
new WizardStepChoice<T>(start, null),
choice => choice != null,
choice => choice.ChosenStep.NextStepChoice,
choice => choice);
This would be ideal, and produce the output I'm after, but the NextStepChoice selector there obviously doesn't compile because it's an IObservable<IWizardStepChoice<T>> rather than an IWizardStepChoice<T>.
I've looked at using Aggregate and Scan but as these are more fold-driven operations, and I've only got the starting element, it's an unfold I'm looking for ala Generate, but I need it to be able to evaluate the nested observable.
Would Observable.Create perhaps be something I could utilise? I've tried it and come up with:
Path = CurrentStep.Select(_ => Observable.Create<IWizardStep<T>>(async observer =>
{
IWizardStepChoice<T> next = new WizardStepChoice<T>(start, null);
observer.OnNext(next.ChosenStep);
while (next != null)
{
next = await next.ChosenStep.NextStepChoice;
observer.OnNext(next.ChosenStep);
}
observer.OnCompleted();
return Disposable.Empty;
}).Aggregate(new List<IWizardStep<T>>(),
(l, s) =>
{
l.Add(s);
return l;
})).Switch().Publish().RefCount();
Which has all the right signature I want IWizardStep<T>->IReadOnlyList<IWizardStep<T>>, so at first glance it looks right, but it doesn't work; it fires, and I can step through, but it hangs once it hits the await and doesn't come back.
I've got a feeling I'm close, and this is a scheduling issue, so my question really is this:
What is the best approach to solve this, am I close?
If this is right, why is there an issue with the await, and how might I solve it?
Update
After a little bit of tinkering I noticed that the await was likely hanging as that observable hadn't yet (and wasn't going to) emit a value (duh), which I've now resolved by initialising each step with a value at the beginning of the wizard.
I've even sanity-checked this by adding a property to IWizardStep<T> - IWizardStepChoice<T> LatestStepChoice {get;} which is just hooked up with:
NextStepChoice.Subscribe(c => _latestStepChoice = c);
This is done on the step class itself, and I can confirm it's working just fine.
Yet still the await hangs, so I tried:
Making it Replay(1) so the await calling .Subscribe() would get the value - this didn't work
Making it Repeat() so even if something is subscribed it'll see the new value - this just made the whole thing hang.
Clearly I'm missing something here, I want it so that when the await queries the observable, it will be given the latest value seen, which is what I thought Replay(1) would achieve; I've also tried PublishLast() but then future updates don't get honoured because of the AsyncSubject<T> behaviour.
For now I've switched to using the self-subscribed property, but it's not ideal, I'd rather not have to break out of querying the observables if I can help it, it feels "hacky".
A recursive walk can transform the tree of observables into a single observable:
static IObservable<IWizardStep<T>> Walk<T>(IWizardStep<T> step)
{
if (step?.NextStepChoice == null)
return Observable.Return(step);
return step.NextStepChoice.SelectMany(choice => Walk(choice.Step)).StartWith(step);
}
Usage:
var steps = await Walk(step).ToArray();
Problem:
IEnumerable<Signal> feed = GetFeed();
var average1 = feed.MovingAverage(10);
var average2 = feed.MovingAverage(20);
var zipped = average1.Zip(average2, (x,y) => Tuple.Create(x,y));
When I iterate through "zipped", GetFeed().GetEnumerator() gets called twice and creates all sorts of synchronization issues. Is there a LINQ operator that can be used to broadcast values from single producer to multiple consumers? I know about Memoize, but in my case I can't predict buffer size to keep slow and fast consumers "happy".
I am thinking about writing my own operator that would keep separate queues for each consumer, but wanted to check if there is an existing solution.
What you want is Reactive Extensions. It's like LINQ to Objects, but in reverse: you don't pull values, they're pushed through observers.
It takes a little while to get used to it, but judging by what you've posted, it's exactly the right model for you.