I need a Buffer method that doesn't buffer on time or on a certain condition.
It should behave similar to this snapshot method:
Taking a snapshot of ReplaySubject<T> buffer
However it should not take a single snapshot, it should buffer when synchronous changes occur ond provide them as IObservable<IList<T>>.
I think there should be an almost simple solution as this Snapshot method, but I can't get my head around how to really solve this. (Note: The snapshot mehtod also works good for queries over multiple subjects)
Here is a test Method:
[TestMethod]
public async Task SyncBufferTest()
{
var i1 = new BehaviorSubject<int>(1);
var i2 = new BehaviorSubject<int>(4);
var sum = i1.CombineLatest(i2, (i1Value, i2Value) => i1Value + i2Value);
var listAsync = sum.SynchronousBuffer().Select(buf => buf.Last()).ToList().RunAsync(new CancellationToken());
Action syncChange1 = () =>
{
i1.OnNext(2);
i2.OnNext(5);
i1.OnNext(7);
};
Action syncChange2 = () =>
{
i1.OnNext(1);
i2.OnNext(1);
};
Action syncChange3 = () =>
{
i1.OnNext(3);
i1.OnCompleted();
i2.OnCompleted();
};
Task.Run(syncChange1)
.ContinueWith(t => syncChange2())
.ContinueWith(t => syncChange3());
var list = await listAsync;
CollectionAssert.AreEqual(new List<int> { 5, 12, 2, 4 }, list.ToList());
}
Background:
I am working on an architecture concept with a reactive data layer as the base of the application. The whole data layer consists of Subjects (as a "talking" data layer). In a single transaction multiple of these Subjects are changed. I have many Observables in a higher layer of my application that are queries to multiple of these Subjects. So I need this SynchronousBuffer to handle synchronous changes to all of these subjects in all of these queries to not get notified multiple times.
If you're looking for a reactive solution, it's always easier if you model your inputs as observables. In this case:
var i1 = new BehaviorSubject<int>(1);
var i2 = new BehaviorSubject<int>(4);
var sum = i1.CombineLatest(i2, (i1Value, i2Value) => i1Value + i2Value);
Action syncChange1 = () =>
{
i1.OnNext(2);
i2.OnNext(5);
i1.OnNext(7);
};
Action syncChange2 = () =>
{
i1.OnNext(1);
i2.OnNext(1);
};
Action syncChange3 = () =>
{
i1.OnNext(3);
i1.OnCompleted();
i2.OnCompleted();
};
IObservable<Action> actions = new Action[] { syncChange1, syncChange2, syncChange3 }.ToObservable();
Same as the question, just we're structuring our Actions as an observable series of changes. Now, magic can happen:
var openWindow = new Subject<int>();
var closeWindow = new Subject<int>();
var gatedActions = actions
.Select((a, i) => new Action(() => {
openWindow.OnNext(i);
a();
closeWindow.OnNext(i);
}));
Now we have windows defined, which can easily be passed into .Buffer() or .Window().
// alternative to window. Not used.
var buffer = sum.Buffer(openWindow, i => closeWindow.Where(cwi => cwi == i));
var listAsync = sum
.Window(openWindow, i => closeWindow.Where(cwi => cwi == i))
.SelectMany(w => w.TakeLast(1))
.ToList()
.RunAsync(new CancellationToken());
gatedActions.Subscribe(a => a(), () => { openWindow.OnCompleted(); closeWindow.OnCompleted(); });
var list = await listAsync; //output is {12, 2, 4}. The starting 5 can be worked in with a .Merge() or something.
Another approach is to try to define a time window within which you consider changes to be synchronous:
var synchronounsWindow = TimeSpan.FromMilliseconds(100);
var actions = new Action[] {syncChange1, syncChange2, syncChange3};
IObservable<Unit> allChanges = Observable.Merge(
i1.Select(_ => Unit.Default),
i2.Select(_ => Unit.Default)
);
Once we have a time window, you can apply the same windowing/buffering techniques as the other answer.
var buffer = sum.Buffer(allChanges.Throttle(synchronounsWindow)); //alternative to window if you like
IList<int> list = null;
var listAsync = sum
.Window(allChanges.Throttle(synchronounsWindow))
.SelectMany(w => w.TakeLast(1))
.ToList()
.Subscribe(l => { list = l;});
foreach (var a in actions)
{
a();
await Task.Delay(synchronounsWindow);
}
CollectionAssert.AreEqual(new List<int> { 12, 2, 4 }, list.ToList()); // again, skipping 5
Related
I have code which would need to have GroupBy and would need a unique BehaviorSubject per group of Switch().
We have a stream of stock market values that we group by Symbol and perform level crossing across a number of levels (defined by a BehaviorSubject and a switch to always use the latest values).
So I need to go from this:
var feed = new Subject<double>();
var levels = new BehaviorSubject<double[]>(new[] { 400.0, 500.0, 600.0, 700.0 });
levels
.Select(thresholds => feed
.Buffer(2, 1)
.Where(x => x.Count == 2)
.Select(x => new { LevelsCrossed = thresholds.GetCrossovers(x[0], x[1]), Previous = x[0], Current = x[1] })
.Where(x => x.LevelsCrossed.Any())
.SelectMany(x => x.LevelsCrossed.Select(level => new ThresholdCrossedEvent(level, x.Previous, x.Current))))
.Switch()
.Subscribe(x => Console.WriteLine(JsonConvert.SerializeObject(x)));
And adapt the above to take a stream of Tick below and group by Symbol, each with its own level threshold detection on each grouped Value.
class Tick
{
public string Symbol { get; set; } // The name.
public decimal Value { get; set; } // The value.
}
Outline:
Take Market data
Group by Symbol
Alert on levels (depending on group name, using a dictionary of BehaviorSubject)
Output
Use Switch() to always use latest values from the dictionary
With a naive implementation I have a wrapper class (ReactiveSymbolFeed below), however blurring non-reactive and reactive code can introduce potential concurrency issues that reactive extensions otherwise deals neatly with.
Questions please:
Am I introducing any side effects, or will this cause issue at scale (say 100,000 messages per second across 2,000 groups)?
Since we have many groups each with their own BehaviorSubject that needs Switch() - can we rewrite our Reactive Extensions statement block to include the thresholds levels per symbol group, or is the above wrapper class the right way to do this?
Further context and the wrapper class solution
Instead I create a ReactiveSymbolFeed wrapper that will form the value part of a dictionary per symbol key.
class ReactiveSymbolFeed
{
readonly BehaviorSubject<double[]> levels;
readonly Subject<double> feed;
public ReactiveSymbolFeed(double[] levels)
{
this.feed = new Subject<double>();
this.levels = new BehaviorSubject<double[]>(levels);
this.levels
.Select(thresholds => this.feed
.Buffer(2, 1)
.Where(x => x.Count == 2)
.Select(x => new { LevelsCrossed = thresholds.GetCrossovers(x[0], x[1]), Previous = x[0], Current = x[1] })
.Where(x => x.LevelsCrossed.Any())
.SelectMany(x => x.LevelsCrossed.Select(level => new ThresholdCrossedEvent(level, x.Previous, x.Current))))
.Switch()
.DistinctUntilChanged(x => x.Threshold)
.Subscribe(x => Console.WriteLine(JsonConvert.SerializeObject(x)));
}
public void OnNext(double value) => this.feed.OnNext(value);
public void UpdateThresholds(double[] levels) => this.levels.OnNext(levels);
}
And then use with the below:
// Setup the detection thresholds per Symbol - each Symbol has 1 set of thresholds
var dictionary = new Dictionary<string, ReactiveSymbolFeed>();
dictionary.Add("AAPL", new ReactiveSymbolFeed(new[] { 120.0, 125.0, 130.0 }));
dictionary.Add("VXX", new ReactiveSymbolFeed(new[] { 10.5, 15, 18.5, 20 }));
// Create some test tick data.
var ticks = new[]
{
new Tick { Symbol = "AAPL", Value = 119.0 },
new Tick { Symbol = "VXX", Value = 10.3 },
new Tick { Symbol = "VXX", Value = 10.8 },
new Tick { Symbol = "AAPL", Value = 121.0 },
new Tick { Symbol = "AAPL", Value = 121.0 }
// Followed by many other differnet Symbols and Values
};
// Loop through test data and dispatch it.
foreach(var tick in ticks)
{
if(dictionary.TryGetValue(tick.Symbol, out var value))
value.OnNext(tick.Value);
}
I'm using Tasks to perform a computation intensive operation. PerformCalculation method. The main parent task uses Task factory to create three child tasks and starts execution. Each child task shall return
a List<Dictionary<int,double>>
List<double> paramList = new List<double>{ 2, 2.5, 3};
CancellationTokenSource cts = new CancellationTokenSource();
Task parent = new Task(() =>
{
var tf = new TaskFactory<List<Dictionary<int, double>>>(cts.Token, TaskCreationOptions.AttachedToParent,
TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
var childTasks = new[] {
tf.StartNew(() => PerformCalculation(cts.Token, paramList[0],Task.CurrentId)),
tf.StartNew(() => PerformCalculation(cts.Token, paramList[1],Task.CurrentId)),
tf.StartNew(() => PerformCalculation(cts.Token, paramList[2],Task.CurrentId)) //3rd entry
};
The results of child tasks upon successful execution shall be in the form of a List<Dictionary<int, double>>.
Now my requirement is to write a lambda expression that would query the results from all of the child tasks once they are finished executing and populate in another list
i.e. list of list (List<List<Dictionary<int, double>>>)
// When all children are done, get the value returned from the
// non-faulting/canceled tasks.
tf.ContinueWhenAll(childTasks, completedTasks =>
completedTasks
.Where(t => t.Status == TaskStatus.RanToCompletion)**??Need HELP HERE ???**),CancellationToken.None)
,TaskContinuationOptions.ExecuteSynchronously);
});
I'll side step the question a bit because all these APIs are essentially obsolete (or at least not suitable for "mainstream" scenarios). This is actually fairly easy:
var childTasks = new[] {
Task.Run(() => PerformCalculation(cts.Token, paramList[0],Task.CurrentId)),
Task.Run(() => PerformCalculation(cts.Token, paramList[1],Task.CurrentId)),
Task.Run(() => PerformCalculation(cts.Token, paramList[2],Task.CurrentId)),
};
var results = Task.WhenAll(childTasks).Result;
TaskFactory is a fairly esoteric type that I have never seen used in the wild. Attached child tasks should be avoided since they add non-obvious dependencies. None of these are mistakes but they are smells.
Assumed that I have existed code like:
public IEnumerable<DataType> GetAllData(string[] ids) {
foreach(var id in ids) {
//this is a time-consuming operation, like query from database
var data = this.repo.Find(id);
yield return data;
}
}
I tried to apply Rx to the front-end code:
var observable = GetAllData(new[] { "1", "2", "3" }).ToObservable();
var subs = observable
.SubscribeOn(Scheduler.Default)
.Subscribe(
data => Console.WriteLine(data.Id),
() => Console.WriteLine("All Data Fetched Completed"));
And it's working properly.
But once I bind a subscription to the IObservable, is there any way I can stop it continue fetching data half-way? Dispose the subscription won't stop the enumeration.
Well, a simple approach is:
var cts = new CancellationTokenSource();
var observable = GetAllData(new[] { "1", "2", "3" }).ToObservable().TakeWhile(x => !cts.IsCancellationRequested);
var subs = observable
.SubscribeOn(Scheduler.Default)
.Subscribe(
data => Console.WriteLine(data.Id),
() => Console.WriteLine("All Data Fetched Completed"));
//...
cts.Cancel();
https://stackoverflow.com/a/31529841/2130786
I have to poll a database until it contains valid data.
To do it, I have a repository that should queried every n seconds in order to get a my very own entity, called DestinationResponse.
class DestinationResponse
{
bool HasDestination { get; set; }
bool Destination { get; set; }
}
When the DestinationResponse has its property HasDestination to true, the Destination is returned.
So, my observable sequence should get all the responses waiting for one to have HasDestination=true. It basically awaits for a response that HasDestination set to true. When this happens, it returns it and the sequence completes. It will only push one element at most!
My current approach is this:
var pollingPeriod = TimeSpan.FromSeconds(n);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var observable = Observable.Interval(pollingPeriod, scheduler)
.SelectMany(_ => destinationRepository.GetDestination().ToObservable())
.TakeWhile(response => !response.HasDestination)
.TakeLast(1)
.Select(response => response.Destination);
I know I's wrong, mainly because the Interval call will keep generating even if the last call to GetDestination hasn't finished.
NOTE:
repository.GetDestination() returns a Task<DestinationResponse> and it actually queries the database.
Merging the answer from Database polling with Reactive Extensions with your example code, I think gives you what you want.
var pollingPeriod = TimeSpan.FromSeconds(n);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var query = Observable.Timer(pollingPeriod , scheduler)
.SelectMany(_ => destinationRepository.GetDestination().ToObservable())
.TakeWhile(response => response.HasDestination)
.Retry() //Loop on errors
.Repeat() //Loop on success
.Select(response => response.Destination)
.Take(1);
This code may be the query I want. What do you think?
private IObservable<Destination> CreateOrderDestinationObservable(string boxId, int orderId)
{
var pollingPeriod = TimeSpan.FromSeconds(DestinationPollingDelay);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var observable = Observable.Timer(pollingPeriod, scheduler)
.SelectMany(_ => externalBridgeRepository.GetDestination(boxId, orderId).ToObservable())
.Where(response => response.HasDestination)
.Retry()
.Repeat()
.Take(1)
.Select(response => response.Destination);
return observable;
}
I have a simple use case where:
Receive a notification of events
Perform some action on the event
Print the content after x interval
How can I do the above step in a single Rx pipeline?
Something like below:
void Main()
{
var observable = Observable.Interval(TimeSpan.FromSeconds(1));
// Receive event and call Foo()
observable.Subscribe(x=>Foo());
// After 1 minute, I want to print the result of count
// How do I do this using above observable?
}
int count = 0;
void Foo()
{
Console.Write(".");
count ++;
}
I think this does what you want:
var observable =
Observable
.Interval(TimeSpan.FromSeconds(1))
.Do(x => Foo())
.Window(() => Observable.Timer(TimeSpan.FromMinutes(1.0)));
var subscription =
observable
.Subscribe(xs => Console.WriteLine(count));
However, it's a bad idea to mix state with observables. If you had two subscriptions you'd increment count twice as fast. It's better to encapsulate your state within the observable so that each subscription would get a new instance of count.
Try this instead:
var observable =
Observable
.Defer(() =>
{
var count = 0;
return
Observable
.Interval(TimeSpan.FromSeconds(1))
.Select(x =>
{
Console.Write(".");
return ++count;
});
})
.Window(() => Observable.Timer(TimeSpan.FromMinutes(0.1)))
.SelectMany(xs => xs.LastAsync());
var subscription =
observable
.Subscribe(x => Console.WriteLine(x));
I get this kind of output:
...........................................................59
............................................................119
............................................................179
............................................................239
Remembering that it starts with 0 then this is timing pretty well.
After seeing paulpdaniels answer I realized that I could replace my Window/SelectMany/LastAsync with the simpler Sample operator.
Also, if we don't really need the side-effect of incrementing a counter then this whole observable shrinks down to this:
var observable =
Observable
.Interval(TimeSpan.FromSeconds(1.0))
.Do(x => Console.Write("."))
.Sample(TimeSpan.FromMinutes(1.0));
observable.Subscribe(x => Console.WriteLine(x));
Much simpler!
I would use Select + Sample:
var observable = Observable.Interval(TimeSpan.FromSeconds(1))
.Select((x, i) => {
Foo(x);
return i;
})
.Do(_ => Console.Write("."))
.Sample(TimeSpan.FromMinutes(1));
observable.Subscribe(x => Console.WriteLine(x));
Select has an overload that returns the index of the current value, by returning that and then sampling at 1 minute intervals, you can get the last value emitted during that interval.