I am constructing a reactive pipeline that needs to expand (SelectMany) and then flatten (in this case, ToArray) whilst maintaining access to a piece of state obtained at the beginning of the pipeline.
Here is pseudo-code for what I am attempting:
return Observable
.Start(() => this.GetSearchResults(query))
.SelectMany(results => results.Hits) // results.Hits is a list of IDs. But there is also has a bool property that I want to keep through to the end of my pipeline
.SelectMany(hit => GetById(hit.Id)) // asynchronously load each result
.ToArray() // now need to pull all the results together into a containing data structure, and also include the bool flag from above in it
.Select(resolvedResults => new ...); // need access to both resolvedResults and the bool mentioned in the first comment above
So I'm trying to find a way to cleanly access some state determined at the beginning of the pipeline from the code at the end of the pipeline.
The first thing I tried was using anonymous types to bundle the bool with each result. This quickly got out of hand and was wasteful from a performance perspective.
The second thing I tried was using a subject as follows:
var state = new AsyncSubject<bool>();
return Observable
.Start(() => this.GetSearchResults(query))
.Do(results =>
{
state.OnNext(results.Flag);
state.OnCompleted();
}
.SelectMany(results => results.Hits)
.SelectMany(hit => GetById(hit.Id))
.ToArray()
.Zip(
state,
(results, state) => new ResultContainer(state, results));
This seems to work fine, but feels a little icky to me.
So what I'm wondering is whether there is a cleaner way to manage state in a reactive pipeline.
For reference, here is the actual code (rather than just pseudo-code):
public IObservable<ISearchResults<IContact>> Search(string query, int maximumResultCount = 100, float minimumScore = 0.1F)
{
Ensure.ArgumentNotNull(query, nameof(query));
var moreHitsAvailable = new AsyncSubject<bool>();
return Observable
.Start(
() => this.searchIndexService.Search<IContact>(query, maximumResultCount, minimumScore),
this.schedulerService.DataStoreScheduler)
.Do(
results =>
{
moreHitsAvailable.OnNext(results.MoreHitsAreAvailable);
moreHitsAvailable.OnCompleted();
})
.SelectMany(
results => results
.Hits
.Select(
hit => new
{
Id = hit.Id,
ParsedId = ContactId.Parse(hit.Id)
}))
.SelectMany(
result => this
.GetById(result.ParsedId)
.Select(
contact => new
{
Id = result.Id,
Contact = contact
}))
.Do(
result =>
{
if (result.Contact == null)
{
this.logger.Warn("Failed to find contact with ID '{0}' provided by the search index. Index may be out of date.", result.Id);
}
})
.Select(result => result.Contact)
.Where(contact => contact != null)
.ToArray()
.Zip(
moreHitsAvailable,
(results, more) => new SearchResults<IContact>(more, results.ToImmutableList()))
.PublishLast()
.ConnectUntilCompleted();
}
You could pop out to Query Comprehension Syntax and do something like this
var x = from result in Observable.Start(() => this.GetSearchResults())
let hasMore = result.MoreHitsAreAvailable
from hit in result.Hits
from contact in GetById(hit.Id)
select new { hasMore , contact};
Over to you how to deal with the duplicate hasMore values. As we know it will be just the single distinct value (all true or all false) you could group by.
Related
Basically I have an object with 2 different properties, both int and I want to get one list with all values from both properties. As of now I have a couple of linq queries to do this for me, but I am wondering if this could be simplified somehow -
var componentsWithDynamicApis = result
.Components
.Where(c => c.DynamicApiChoicesId.HasValue ||
c.DynamicApiSubmissionsId.HasValue);
var choiceApis = componentsWithDynamicApis
.Select(c => c.DynamicApiChoicesId.Value);
var submissionApis = componentsWithDynamicApis
.Select(c => c.DynamicApiSubmissionsId.Value);
var dynamicApiIds = choiceApis
.Union(submissionApis)
.Distinct();
Not every component will have both Choices and Submissions.
By simplify, I assume you want to combine into fewer statements. You can also simplify in terms of execution by reducing the number of times you iterate the collection (the current code does it 3 times).
One way is to use a generator function (assuming the type of items in your result.Components collection is Component):
IEnumerable<int> GetIds(IEnumerable<Component> components)
{
foreach (var component in components)
{
if (component.DynamicApiChoicesId.HasValue) yield return component.DynamicApiChoicesId.Value;
if (component.DynamicApiSubmissionsId.HasValue) yield return component.DynamicApiSubmissionsId.Value;
}
}
Another option is to use SelectMany. The trick there is to create a temporary enumerable holding the appropriate values of DynamicApiChoicesId and DynamicApiSubmissionsId. I can't think of a one-liner for this, but here is one option:
var dynamicApiIds = result
.Components
.SelectMany(c => {
var temp = new List<int>();
if (c.DynamicApiChoicesId.HasValue) temp.Add(c.DynamicApiChoicesId.Value);
if (c.DynamicApiSubmissionsId.HasValue) temp.Add(c.DynamicApiSubmissionsId.Value);
return temp;
})
.Distinct();
#Eldar's answer gave me an idea for an improvement on option #2:
var dynamicApiIds = result
.Components
.SelectMany(c => new[] { c.DynamicApiChoicesId, c.DynamicApiSubmissionsId })
.Where(c => c.HasValue)
.Select(c => c.Value)
.Distinct();
Similar to some of the other answers, but I think this covers all your bases with a very minimal amount of code.
var dynamicApiIds = result.Components
.SelectMany(c => new[] { c.DynamicApiChoicesId, c.DynamicApiSubmissionsId}) // combine
.OfType<int>() // remove nulls
.Distinct();
To map each element in the source list onto more than one element on the destination list, you can use SelectMany.
var combined = componentsWithDynamicApis
.SelectMany(x => new[] { x.DynamicApiChoicesId.Value, x.DynamicApiSubmissionsId.Value })
.Distinct();
I have not tested it but you can use SelectMany with filtering out the null values like below :
var componentsWithDynamicApis = result
.Components
.Select(r=> new [] {r.DynamicApiChoicesId,r.DynamicApiSubmissionsId})
.SelectMany(r=> r.Where(p=> p!=null).Cast<int>()).Distinct();
I have to poll a database until it contains valid data.
To do it, I have a repository that should queried every n seconds in order to get a my very own entity, called DestinationResponse.
class DestinationResponse
{
bool HasDestination { get; set; }
bool Destination { get; set; }
}
When the DestinationResponse has its property HasDestination to true, the Destination is returned.
So, my observable sequence should get all the responses waiting for one to have HasDestination=true. It basically awaits for a response that HasDestination set to true. When this happens, it returns it and the sequence completes. It will only push one element at most!
My current approach is this:
var pollingPeriod = TimeSpan.FromSeconds(n);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var observable = Observable.Interval(pollingPeriod, scheduler)
.SelectMany(_ => destinationRepository.GetDestination().ToObservable())
.TakeWhile(response => !response.HasDestination)
.TakeLast(1)
.Select(response => response.Destination);
I know I's wrong, mainly because the Interval call will keep generating even if the last call to GetDestination hasn't finished.
NOTE:
repository.GetDestination() returns a Task<DestinationResponse> and it actually queries the database.
Merging the answer from Database polling with Reactive Extensions with your example code, I think gives you what you want.
var pollingPeriod = TimeSpan.FromSeconds(n);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var query = Observable.Timer(pollingPeriod , scheduler)
.SelectMany(_ => destinationRepository.GetDestination().ToObservable())
.TakeWhile(response => response.HasDestination)
.Retry() //Loop on errors
.Repeat() //Loop on success
.Select(response => response.Destination)
.Take(1);
This code may be the query I want. What do you think?
private IObservable<Destination> CreateOrderDestinationObservable(string boxId, int orderId)
{
var pollingPeriod = TimeSpan.FromSeconds(DestinationPollingDelay);
var scheduler = new EventLoopScheduler(ts => new Thread(ts) {Name = "DestinationPoller"});
var observable = Observable.Timer(pollingPeriod, scheduler)
.SelectMany(_ => externalBridgeRepository.GetDestination(boxId, orderId).ToObservable())
.Where(response => response.HasDestination)
.Retry()
.Repeat()
.Take(1)
.Select(response => response.Destination);
return observable;
}
I want to build filters on IQueryable depending on user input and execute the query only at the end. I'm trying to understand concept behind and if this is going to work as expected.
Will the query below hit database on return fetchedWorkflowLogs.ToList() in the example below?
// Partition latest record for Instance
IQueryable<WorkflowLog> fetchedWorkflowLogs
= this._workflowLog_repo
.GetAll()
.GroupBy(log => log.ID)
.Select(
grp => new
{
grp = grp,
MaxID = grp.Max(log => log.ID)
}
)
.SelectMany(
temp0 => temp0.grp,
(temp0, log) => new
{
temp0 = temp0,
log = log
}
)
.Where(temp1 => (temp1.log.ID == temp1.temp0.MaxID))
.Select(temp1 => temp1.log);
// .. some more filters
// Last filter
// Filter by Project
if (model.ProjectID != null)
{
fetchedWorkflowLogs.Where(record => record.Project.ID == model.ProjectID);
}
return fetchedWorkflowLogs.ToList();
Agree with David. if GetAll() returns IQueryable, and you return fetchedWorkflowLogs.ToList() will result in the linq query being evaluated, and the query will hit database.
It's the best if you can debug and step through, and writing a test could help you if execuing program directly is difficult.
How do you coordinate observable sequences so one starts only when the other is finished?
I have 3 observable of different types:
var obs1 = ...
var obs2 = ...
var obs2 = ...
I want to do:
obs1.Subscribe( () => obs2.Subscribe( ()=> obs3.Subscribe( () => /* Hide Progress */ )));
But this code is really ugly. Is there some operator to do it? I tried using the And() extension method but I'm not sure that this is the correct way.
Well, if you don't mind introducing TPL, you can just use await:
await obs1;
await obs2;
await obs3;
If you want to observe the values of each while still using await, just add Do:
await obs1.Do(t1 => ...);
await obs2.Do(t2 => ...);
await obs3.Do(t3 => ...);
Does this do what you want?
obs1
.Concat(obs2)
.Concat(obs3)
.Subscribe(x => /* ... */ );
Obviously this only works with cold observables. If your obs2 & obs3 are hot you might miss values.
Enigmativity is correct, though you just have to use Select also.
obs1.Select(t => new { t, (U)null, (V)null })
.Concat(
obs2.Select(u => new { (T)null, u, (V)null }))
.Concat(
obs3.Select(v => new { (T)null, (U)null, v }))
.Subscribe(either =>
{
if (either.t != null) Observe(either.t);
else if (either.u != null) Observe(either.u);
else if (either.v != null) Observe(either.v);
else { throw new Exception("Oops."); }
})
See also a related blog post of mine: The Power of T
If you are only interested in observing obs3, you may want to write like this:
obs1.TakeLast(1)
.SelectMany(x => obs2)
.TakeLast(1)
.SelectMany(y => obs3)
.Subscribe(z => ... ); // z is the same type of obs3's data type
We take the last item from obs1 and when it arrives, we use SelectMany to subscribe to and output obs2. Then we repeatedly taking the last item from the returned Observable, and when the last item arrives, we again use SelectMany to subscribe to and output obs3. Afterwards you can subscribe to the returned Observable and do the handling of obs3 as you like.
This is related to my other question here. James World presented a solution as follows:
// idStream is an IObservable<int> of the input stream of IDs
// alarmInterval is a Func<int, TimeSpan> that gets the interval given the ID
var idAlarmStream = idStream
.GroupByUntil(key => key, grp => grp.Throttle(alarmInterval(grp.Key)))
.SelectMany(grp => grp.IgnoreElements().Concat(Observable.Return(grp.Key)));
<edit 2:
Question: How do I start the timers immediately without waiting for the first events to arrive? That's the root problem in my question, I guess. For that end, I planned on sending off dummy objects with the IDs I know should be there. But as I write in following, I ended up with some other problems. Nevertheless, I'd think solving that too would be interesting.
Forwards with the other interesting parts then! Now, if I'd like to group a complex object like the following and group by the key as follows (won't compile)
var idAlarmStream = idStream
.Select(i => new { Id = i, IsTest = true })
.GroupByUntil(key => key.Id, grp => grp.Throttle(alarmInterval(grp.Key)))
.SelectMany(grp => grp.IgnoreElements().Concat(Observable.Return(grp.Key)));
then I get into trouble. I'm unable to modify the part about SelectMany, Concat and Observable.Return so that the query would work as before. For instance, if I make query as
var idAlarmStream = idStream
.Select(i => new { Id = i, IsTest = true })
.GroupByUntil(key => key.Id, grp => grp.Throttle(alarmInterval(grp.Key)))
.SelectMany(grp => grp.IgnoreElements().Concat(Observable.Return(grp.Key.First())))
.Subscribe(i => Console.WriteLine(i.Id + "-" + i.IsTest);
Then two events are needed before an output can be observed in the Subscribe. It's the effect of the call to First, I gather. Furthermore, I woul like to use the complex object attributes in the call to alarmInterval too.
Can someone offer an explanation what's going on, perhaps even a solution? The problem in going with unmodified solution is that the grouping doesn't look Ids alone for the key value, but also the IsTest field.
<edit: As a note, the problem probably could be solved firsly by creating an explicit class or struct and then that implements a custom IEquatable and secondly then using James' code as-is so that grouping would happen by IDs alone. It feels like hack though.
Also, if you want to count the number of times you've seen an item before the alarm goes off you can do it like this, taking advantage of the counter overload in Select.
var idAlarmStream = idStream
.Select(i => new { Id = i, IsTest = true })
.GroupByUntil(key => key.Id, grp => grp.Throttle(alarmInterval(grp.Key))
.SelectMany(grp => grp.Select((count, alarm) => new { count, alarm }).TakeLast(1));
Note, this will be 0 for the first (seed) item - which is probably what you want anyway.
You are creating an anonymous type in your Select. Lets call it A1. I will assume your idStream is an IObservable. Since this is the Key in the GroupByUntil you do not need to worry about key comparison - int equality is fine.
The GroupByUntil is an IObservable<IGroupedObservable<int, A1>>.
The SelectMany as written is trying to be an IObservable<A1>. You need to just Concat(Observable.Return(grp.Key)) here - but the the type of the Key and the type of the Group elements must match or the SelectMany won't work. So the key would have to be an A1 too. Anonymous types use structural equality and the return type would be stream of A1 - but you can't declare that as a public return type.
If you just want the Id, you should add a .Select(x => x.Id) after the Throttle:
var idAlarmStream = idStream
.Select(i => new { Id = i, IsTest = true })
.GroupByUntil(key => key.Id, grp => grp.Throttle(alarmInterval(grp.Key)
.Select(x => x.Id))
.SelectMany(grp => grp.IgnoreElements().Concat(Observable.Return(grp.Key)));
If you want A1 instead - you'll need to create a concrete type that implements Equality.
EDIT
I've not tested it, but you could also flatten it more simply like this, I think this is easier! It is outputing A1 though, so you'll have to deal with that if you need to return the stream somewhere.
var idAlarmStream = idStream
.Select(i => new { Id = i, IsTest = true })
.GroupByUntil(key => key.Id, grp => grp.Throttle(alarmInterval(grp.Key))
.SelectMany(grp => grp.TakeLast(1));