I want to write a library that would communicate with a web server and expose data from it to the rest of the world. The web server is nothing special, it exposes several REST methods, mostly GET and POST.
Since I am relatively new to Reactive Extensions (but I love it already) I ask for advice. I decided that the interfaces of the library would expose IObservables. But I dont know how exactly to implement this. I'm thinking I have several options:
1) Expose IObservable<IEnumerable<T>>. Makes sense, REST service returns all requested data at once. User calls Subscribe(), only one IEnumerable is pushed, OnDone is called. So Subscribe() would need to be called multiple times.
2) Expose IObservable<T>. Could be a good choice in some cases I guess. Subscribe() would only be called once, to get other data, there would be methods Refresh() or NextPage() (...) to get more data to the stream. (then instead of IObservable<T> GetResource... it could be a property, IObservable<T> Resource { get; }
3) Forget Rx, do it old fashioned way via events (worst possible thing IMO)
4) Some other way?
Anyone with experience in this area? What I am concerned about is Refreshing (asking for new data), Paging, combining the results and generally having a good maintainable design.
Thx for any advice
I would suggest using the following interface:
public interface IRestService
{
IObservable<T> GetResources<T>();
}
There are a number of reasons behind this choice.
To expose an IObservable<IEnumerable<T>> mixes reactives with interactives (or observables with enumerables) and would force your queries to call .ToObservable() or, even worse, .ToEnumerable() to construct a basic SelectMany query. It's better for you to better to keep your queries and subscribing code nice an clean.
Now, you suggested that with an IObservable<T> you would only subscribe once and you would require a Refresh or NextPage call to get more data to the stream. That's not a good idea. You should instead think of a single subscription would return all the results of a single REST call and then call OnComplete. If you want to invoke a new REST call then just subscribe again.
Further, the semantics of a single subscription call are not clearly expressed in the code. So you need to think about maintaining your code. When you look at the code in the future what are you likely to think that the semantics are? I would suggest that the semantics of a a single subscription would map to a single REST call would be more likely. Your code would have the potential of being more confusing otherwise.
Even further, you should avoid a single subscription model because if any exception is thrown then your observable is done and, of course, calling web services can be very error prone. If you have an error with a multiple subscription model you can recover more easily.
I would also avoid IObservable<T> Resources { get; } because it suggests some sort of "fixed" value where instead it is more dynamic - in other words each call may give you different values. It is better to call a GetResources method rather than a Resources property.
Some, bottom-line, I'd have a IObservable<T> that abstracts a single call to your underlying REST service.
You're combining two concerns here which really should be addressed separately.
The first is your code going and getting data from other sources. The second is publishing that data to interested parties when there is new data available.
In regards to the first, the Reactive Extensions aren't going to help. Your concern here is to get the data on a timed interval; it has to be a timed interval because when making calls to REST services in your code, there's no callback, there's nothing for you to hook into that the service can call.
If there was some sort of callback into your code from the external service, then that could be wrapped in some IObservable<T> implementation which you could then subscribe to and perform your operations on (just forwarding the subscription, really).
The only way you could use the Reactive Extensions for the first concern is to set off a timer by using the static Timer method on the Observable class.
For the second concern (after you have your data and you want to notify subscribers), you absolutely can and should use an IObservable<T> implementation to notify subscribers.
In this case, I strongly recommend that you don't try and deviate from the intent of the Subscribe method on the IObservable<T> interface. You should expose a method that will give you the IObservable<T> that anyone can subscribe to (whether or not the IObservable<T> is hot or cold before Subscribe is called is up to you) and unsubscribe from by calling the Dispose method on the IDisposable interface implementation returned from the call to Subscribe.
That way, your clients can get the IObservable<T>, subscribe for the notifications they want, and then unsubscribe when they're done.
Related
I have a .NET library which, as part of an Object Model will emit notifications of certain occurrences.
It would seem to me that the main pros of events are approachability for beginners (and simplicity in certain consumption contexts) with the main negative being that they are not composable and hence are immediately forced into an Observable.FromEvent* if you want to do anything interesting without writing a code thicket.
The nature of the problem being solved is such that the event traffic won't be particularly frequent or voluminous (it's definitely not screaming RX), but there is definitely no requirement to support .NET versions prior to 4.0 [and hence I can use the built-in IObservable interface in System.Reactive without forcing any significant dependencies on consumers]. I'm interested in some general guidelines some specific concrete reasons to prefer IObservables over events from an API design perspective though - regardless of where my specific case might sit on the event - IObservable spectrum.
So, the question:
Is there anything concrete I'm making dramatically more difficult or problematic for API consumers if I go with the simplest thing and expose an event instead of an IObservable
Or, restated: Aside from the consumer having to do an Observable.FromEvent* to be able to compose events, is there really not a single reason to prefer an IObservable over an event when exposing a notification in an API?
Citations of projects that are using IObservable for not-screaming-RX stuff or coding guidelines would be ideal but are not critical.
NB as touched on in the comments with #Adam Houldsworth, I'm interested in concrete things wrt the API surface of a .NET 4+ library, not a survey of opinions as to which represents a better 'default architecture' for our times :)
NB this question has been touched on in IObserver and IObservable in C# for Observer vs Delegates, Events and IObservable vs Plain Events or Why Should I use IObservable?. The aspect of the question I'm asking has not been addressed in any of the responses due to SRP violations. Another slightly overlapping question is Advantages of .NET Rx over classic events?. Use of IObservable instead of events falls into that same category.
In the comments of this answer, OP refined his question as:
[Is it] indeed definitely the case that each and every event can
always be Adapted to be an IObservable?
To that question, the answer is basically yes - but with some caveats. Also note that the reverse is not true - see the section on reverse transformation and the conclusion for reasons why observables might be preferred to classic events because of the additional meaning they can convey.
For strict translation, all we need to do is map the event - which should include the sender as well as arguments - on to an OnNext invocation. The Observable.FromEventPattern helper method does a good job of this, with the overloads returning IObservable<EventPattern<T>> providing both the sender object and the EventArgs.
Caveats
Recall the Rx grammar. This can be stated in EBNF form as:
Observable Stream = { OnNext }, [ OnError | OnCompleted ] - or 0 or more OnNext events optionally followed by either an OnCompleted or an OnError.
Implicit in this is the idea that from the view of an individual subscriber events do not overlap. To be clear, this means that a subscriber will not be called concurrently. Additionally, it is quite possible that other subscribers can be called not only concurrently but also at different times. Often it is subscribers themselves that control pace of event flow (create back-pressure) by handling events slower than the pace at which they arrive. In this situation typical Rx operators queue against individual subscribers rather than holding up the whole subscriber pool. In contrast, classic .NET event sources will more typically broadcast to subscribers in lock-step, waiting for an event to be fully processed by all subscribers before proceeding. This is the long-standing assumed behaviour for classic events, but it is not actually anywhere decreed.
The C# 5.0 Language Specification (this is a word document, see section 1.6.7.4) and the .NET Framework Design Guidelines : Event Design have surprisingly little to say on the event behaviour. The spec observes that:
The notion of raising an event is precisely equivalent to invoking the
delegate represented by the event—thus, there are no special language
constructs for raising events.
The C# Programming Guide : Events section says that:
When an event has multiple subscribers, the event handlers are invoked
synchronously when an event is raised. To invoke events
asynchronously, see Calling Synchronous Methods Asynchronously.
So classic events are traditionally issued serially by invoking a delegate chain on a single thread, but there is no such restriction in the guidelines - and occasionally we see parallel invocation of delegates - but even here two instances of an event will usually be raised serially even if each one is broadcast in parallel.
There is nothing anywhere I can find in the official specifications that explicitly states that event instances themselves must be raised or received serially. To put it another way, there is nothing that says multiple instances of an event can't be raised concurrently.
This is in contrast to observables where it is explicitly stated in the Rx Design Guidelines that we should:
4.2. Assume observer instances are called in a serialized fashion
Note how this statement only addresses the viewpoint of an individual subscriber instance and says nothing about events being sent concurrently across instances (which in fact is very common in Rx).
So two takeways:
Whilst OnNext captures the idea of an event it is possible that the classic .NET event may violate the Rx Grammar by invoking events concurrently.
It is common for pure Rx Observables to have different semantics around the delivery of events under load because back-pressure is typically handled per subscriber rather than per-event.
As long as you don't raise events concurrently in your API, and you don't care about the back-pressure semantics, then translation to an observable interface via a mechanism like Rx's Observable.FromEvent is going to be just fine.
Reverse Transformation
On the reverse transformation, note that OnError and OnCompleted have no analogue in classic .NET events, so it is not possible to make the reverse mapping without some additional machinery and agreed usage.
For example, one could translate OnError and OnCompleted to additional events - but this is definitely stepping outside of classic event territory. Also, some very awkward synchronization mechanism would be required across the different handlers; in Rx, it is quite possible for one subscriber to receive an OnCompleted whilst another is still receiving OnNext events - it's much harder to pull this off in a classic .NET events transformation.
Errors
Considering behaviour in error cases: It's important to distinguish an error in the event source from one in the handlers/subscribers. OnError is there to deal with the former, in the latter case both classic events and Rx simply (and correctly) blow up. This aspect of errors does translate well in either direction.
Conclusion
.NET classic events and Observables are not isomorphic. You can translate from events to observables reasonably easily as long as you stick to normal usage patterns. It might be the case that your API requires the additional semantics of observables not so easily modelled with .NET events and therefore it makes more sense to go Observable only - but this is a specific consideration rather than a general one, and more of a design issue than a technical one.
As general guidance, I suggest a preference for classic events if possible as these are broadly understood and well supported and can be transformed - but don't hesitate to use observables if you need the extra semantics of source error or completion represented in the elegant form of OnError and OnCompleted events.
I've been reading a lot about Reactive extensions before finally dipping my toe, and after some rough starts I found them really interesting and useful.
Observables extension have this lovely optional parameter where you can pass your own time manager, effectively letting you manipulate time. In my case it helped a lot since I was doing some time related work (check this webservice every ten minutes, send one email per minute, etc) and it made testing a breeze. I would plug the TestScheduler in the component under test and simulate one day of events in a snap.
So if you have some workflows in your library where time plays a role in orchestration, I would really recommend using Observables as your output.
However if you are just raising events in direct responses to user inputs, I don't think that it's worth the added complexity for your users. As you noted they can wrap the event into their own Observable if needed.
You could have your cake and eat it too, although it would mean more work; offer a facade that turns your event library into a Observable-fueled one, by creating said Observable from your events. Or do it the other way: have an optional facade that suscribes to your observable and raise a classic event when triggered.
In my opinion there is a non-trivial technical step to take when dealing with reactive extensions and in this case it may come down to what your API consumers would be the most comfortable using
The IObservable is the IEnumerable of events, so the only question here is do you think IObservable will become the standard as the IEnumerable is now so yes its preferable, if you think its just a passing by thing that will fall in future use event instead.
The IObservable is better than event in most cases but I personally think I'll forget to use it as its not very common to be used and when the time arrives I'll have forgotten about it.
I know my answer is not of great help but only time will tell if RX will become the standard, I think it has good chances.
[EDIT] To make it more Concrete.
To the end user the only difference is that one is an interface an the other is not, making the interface more testable and expandable, because different sources can implement the same interface.
As Adam Houldsworth said one can be changed to the other easily to make no other difference.
To address your headline question:
No they should not be preferred on the basis that they exist in .NET 4 and are available to use. Preference depends on intended use, so a blanket preference is unwarranted.
That said, I would tend towards them as an alternative model to traditional C# events.
As I have commented on throughout this question, there are many ancillary benefits to approaching the API with IObservable, not least of which is external support and the range of choice available to the end consumer.
To address your inner question:
I believe there would be little difficulty between exposing events or IObserable in your API as there is a route to one from the other in both cases. This would put a layer over your API, but in actuality this is a layer you could also release.
It is my opinion that choosing one over the other isn't going to be part of the deciding reason why someone choose to use or not use your API.
To address your re-stated question:
The reason might be found in why there is an Observable.FromEvent in the first place :-) IObservable is gaining support in many places in .NET for reactive programming and forms part of many popular libraries (Rx, Ix, ReactiveUI), and also interoperates well with LINQ and IEnumerable and further into the likes of TPL and TPL DataFlow.
A non-Rx example of the observable pattern, so not specifically IObservable would be ObservableCollection for XAML apps.
WCF contains Server and Client side In my case Server is the program that produces data (logs). There are several subscribers that should be notified when a new bunch of data is ready to be posted.
So I think WCF Server should callback some method for all interested clients....
Should I use "CallbackContract" keyword to define one-way callback operation?
I need is void Log(string) method which will be called by server for all clients every time something should be possted, how to define such method in terms of WCF?
probably I should avoid "callback" but instead use string[] getLogs() method which will return new logs? then client may call getLogs method every one or two or three seconds to get logs for the last interval?
What you described looks like Publisher/Subscriber pattern. Check this out: http://blogs.msdn.com/b/tomholl/archive/2008/05/17/building-a-pub-sub-message-bus-with-wcf-and-msmq.aspx
There are also specialized frameworks, like NServiceBus that solve broader problem of asynchronous messaging.
what you are asking sounds reasonable, have a look at this one: WCF Callbacks; a beginners guide
I'm designing a desktop application with multiple layers: the GUI layer (WinForms MVP) holds references to interfaces of adapter classes, and these adapters call BL classes that do the actual work.
Apart from executing requests from the GUI, the BL also fires some events that the GUI can subscribe to through the interfaces. For example, there's a CurrentTime object in the BL that changes periodically and the GUI should reflect the changes.
There are two issues that involve multithreading:
I need to make some of the logic
calls asynchronous so that they don't block the GUI.
Some of the events the GUI recevies are fired from non-GUI threads.
At what level is it best to handle the multithreading? My intuition says that the Presenter is the most suitable for that, am I right? Can you give me some example application that does what I need? And does it make sense for the presenter to hold a reference to the form so it can invoke delegates on it?
EDIT: The bounty will probably go to Henrik, unless someone gives an even better answer.
I would look at using a Task-based BLL for those parts that can be described as "background operations" (that is, they're started by the UI and have a definite completion point). The Visual Studio Async CTP includes a document describing the Task-based Asynchronous Pattern (TAP); I recommend designing your BLL API in this way (even though the async/await language extensions haven't been released yet).
For parts of your BLL that are "subscriptions" (that is, they're started by the UI and continue indefinitely), there are a few options (in order of my personal preference):
Use a Task-based API but with a TaskCompletionSource that never completes (or only completes by being cancelled as part of application shutdown). In this case, I recommend writing your own IProgress<T> and EventProgress<T> (in the Async CTP): the IProgress<T> gives your BLL an interface for reporting progress (replacing progress events) and EventProgress<T> handles capturing the SynchronizationContext for marshalling the "report progress" delegate to the UI thread.
Use Rx's IObservable framework; this is a good match design-wise but has a fairly steep learning curve and is less stable than I personally like (it's a pre-release library).
Use the old-fashioned Event-based Asynchronous Pattern (EAP), where you capture the SynchronizationContext in your BLL and raise events by queuing them to that context.
EDIT 2011-05-17: Since writing the above, the Async CTP team has stated that approach (1) is not recommended (since it somewhat abuses the "progress reporting" system), and the Rx team has released documentation that clarifies their semantics. I now recommend Rx for subscriptions.
It depends on what type of application you are writing - for example - do you accept bugs? What are your data requirements - soft realtime? acid? eventually consistent and/or partially connected/sometimes disconnected clients?
Beware that there's a distinction between concurrency and asynchronocity. You can have asynchronocity and hence call method call interleaving without actually having a concurrently executing program.
One idea could be to have a read and write side of your application, where the write-side publishes events when it's been changed. This could lead to an event driven system -- the read side would be built from the published events, and could be rebuilt. The UI could be task-driven - in that a task to perform would produce a command that your BL would take (or domain layer if you so wish).
A logical next step, if you have the above, is to also go event-sourced. Then you would recreate internal state of the write-model through what has been previously committed. There's a google group about CQRS/DDD that could help you with this.
With regards to updating the UI, I've found that the IObservable interfaces in System.Reactive, System.Interactive, System.CoreEx are well suited. It allows you to skip around different concurrent invocation contexts - dispatcher - thread pool, etc, and it interops well with the Task Parallel Library.
You'd also have to consider where you put your business logic -- if you go domain driven I'd say you could put it in your application as you'd have an updating procedure in place for the binaries you distribute anyway, when time comes to upgrade, but there's also the choice of putting it on the server. Commands could be a nice way to perform the updates to the write-side and a convenient unit of work when connection-oriented code fails (they are small and serializable and the UI can be designed around them).
To give you an example, have a look at this thread, with this code, that adds a priority to the IObservable.ObserveOnDispatcher(...)-call:
public static IObservable<T> ObserveOnDispatcher<T>(this IObservable<T> observable, DispatcherPriority priority)
{
if (observable == null)
throw new NullReferenceException();
return observable.ObserveOn(Dispatcher.CurrentDispatcher, priority);
}
public static IObservable<T> ObserveOn<T>(this IObservable<T> observable, Dispatcher dispatcher, DispatcherPriority priority)
{
if (observable == null)
throw new NullReferenceException();
if (dispatcher == null)
throw new ArgumentNullException("dispatcher");
return Observable.CreateWithDisposable<T>(o =>
{
return observable.Subscribe(
obj => dispatcher.Invoke((Action)(() => o.OnNext(obj)), priority),
ex => dispatcher.Invoke((Action)(() => o.OnError(ex)), priority),
() => dispatcher.Invoke((Action)(() => o.OnCompleted()), priority));
});
}
The example above could be used like this blog entry discusses
public void LoadCustomers()
{
_customerService.GetCustomers()
.SubscribeOn(Scheduler.NewThread)
.ObserveOn(Scheduler.Dispatcher, DispatcherPriority.SystemIdle)
.Subscribe(Customers.Add);
}
... So for example with a virtual starbucks shop, you'd have a domain entity that has something like a 'Barista' class, which produces events 'CustomerBoughtCappuccino' : { cost : '$3', timestamp : '2011-01-03 12:00:03.334556 GMT+0100', ... etc }. Your read-side would subscribe to these events. The read side could be some data model -- for each of your screens that present data -- the view would have a unique ViewModel-class -- which would be synchronized with the view in an observable dictionary like this. The repository would be (:IObservable), and your presenters would subscribe to all of that, or just a part of it. That way your GUI could be:
Task driven -> command driven BL, with focus on user operations
Async
Read-write-segregated
Given that your BL only takes commands and doesn't on top of that display a 'good enough for all pages'-type of read-model, you can make most things in it internal, internal protected and private, meaning you can use System.Contracts to prove that you don't have any bugs in it (!). It would produce events that your read-model would read. You could take the main principles from Caliburn Micro about the orchestration of workflows of yielded asynchronous tasks (IAsyncResults).
There are some Rx design guidelines you could read. And cqrsinfo.com about event sourcing and cqrs. If you are indeed interested in going beyond the async programming sphere into the concurrent programming sphere, Microsoft has released a well written book for free, on how to program such code.
Hope it helps.
I would consider the "Thread Proxy Mediator Pattern". Example here on CodeProject
Basically all method calls on your Adaptors run on a worker thread and all results are returned on the UI thread.
The recommended way is using threads on the GUI, and then update your controls with Control.Invoke().
If you don't want to use threads in your GUI application, you can use the BackgroundWorker class.
The best practice is having some logic in your Forms to update your controls from outside, normally a public method. When this call is made from a thread that is not the MainThread, you must protect illegal thread accesses using control.InvokeRequired/control.Invoke() (where control is the target control to update).
Take a look to this AsynCalculatePi example, maybe it's a good starting point.
I have a multithreaded application that spawns threads for several hardware instruments. Each thread is basically an infinite loop (for the lifetime of the application) that polls the hardware for new data, and activates an event (which passes the data) each time it collects something new. There is a single listener class that consolidates all these instruments, performs some calculations, and fires a new event with this calculation.
However, I'm wondering if, since there is a single listener, it would be better to expose an IEnumerable<> method off these instruments, and use a yield return to return the data, instead of firing events.
I'd like to see if anybody knows of differences in these two methods. In particular, I'm looking for the best reliability, best ability to pause/cancel operation, best for threading purposes, general safety, etc.
Also, with the second method is it possible to still run the IEnumerable loop on a separate thread? Many of these instruments are somewhat CPU-bound, so ensuring each one is on a different thread is vital.
This sounds like a very good use case for the Reactive Extensions. There's a little bit of a learning curve to it but in a nutshell, IObservable is the dual of IEnumerable. Where IEnumerable requires you to pull from it, IObservable pushes its values to the observer. Pretty much any time you need to block in your enumerator, it's a good sign you should reverse the pattern and use a push model. Events are one way to go but IObservable has much more flexibility since it's composable and thread-aware.
instrument.DataEvents
.Where(x => x.SomeProperty == something)
.BufferWithTime( TimeSpan.FromSeconds(1) )
.Subscribe( x => DoSomethingWith(x) );
In the above example, DoSomethingWith(x) will be called whenever the subject (instrument) produces a DataEvent that has a matching SomeProperty and it buffers the events into batches of 1 second duration.
There's plenty more you could do such as merging in the events produced by other subjects or directing the notifications onto the UI thread, etc. Unfortunately documentation is currently pretty weak but there's some good information on Matthew Podwysocki's blog. (Although his posts almost exclusively mention Reactive Extensions for JavaScript, it's pretty much all applicable to Reactive Extensions for .NET as well.)
It's a close call, but I think I'd stick to the event model in this case, with the main decider behing that future maintenance programmers are less likely to understand the yield concept. Also, yield means the code processing each hardware request is in the same thread as the code generating the requests for processing. That's bad, because it could mean your hardware has to wait on the consumer code.
And speaking of consumers, another option is a producer/consumer queue. Your instruments can all push into the same queue and your single listener can then pop from it do whatever from there.
There's a pretty fundamental difference, push vs pull. The pull model (yield) being the harder one to implement from the instrument interface view. Because you'll have to store data until the client code is ready to pull. When you push, the client may or may not store, as it deems necessary.
But most practical implementations in multi-threading scenarios need to deal with the overhead in the inevitable thread context switch that's required to present data. And that's often done with pull, using a thread-safe bounded queue.
Stephen Toub blogs about a blocking queue which implements IEnumerable as an infinite loop using the yield keyword. Your worker threads could enqueue new data points as they appear and the calculation thread could dequeue them using a foreach loop with blocking semantics.
I don't think there's much difference performance-wise between the event and yield approach. Yield is lazy evaluated, so it leaves an opportunity to signal the producing threads to stop. If your code is thoughtfully documented then maintenance ought to be a wash, too.
My preference is a third option, to use a callback method instead of an event (even though both involve delegates). Your producers invoke the callback each time they have data. Callbacks can return values, so your consumer can signal producers to stop or continue each time they check in with data.
This approach can give you places to optimize performance if you have a high volume of data. In your callback you lock on a neutral object and append incoming data to a collection. The runtime internally uses an ready queue on the lock object, so this can serve as your queuing point.
This lets you choose a collection, such as a List<T> with predefined capacity, that is O(1) for appending. You can also double-buffer your consumer, with your callback appending to the "left" buffer while you consolidate from the "right" one, and so forth. This minimizes the amount of producer blocking and associated missed data, which is handy for bursty data. You can also readily measure high-water marks and processing rates as you vary the number of threads.
When is it appropriate to raise an event in C#?
As an example, in our system we have data objects being sent to us, say 50 per minute, from an external system. Upon receiving a data packet, we need to have it processed by another object. Would an event or a simple method call be better to use in this situation?
An event seems like a natural fit intuitively, but I'm not clear as to what advantage this may offer as compared to simply using a regular method call.
Events should be used when it is inappropriate for the code which originates the action to have direct knowledge of the code(s) which react to that action.
On one hand, an event does sound appropriate here, because the code which handles data reception should not be dependent on the implementation of the code which does something with said data. Otherwise the data reception code is now responsible for two things - receiving the data and delegating the processing of it.
On the other hand, if the particular processing of the data is directly tied to act of it being sent by the external caller, it may make more sense to make it a function call. We don't really have enough information in your question to say for sure.
IMO using a Queue would be an appropriate first step. Code processing the Queue can in turn either raise events or accept a delegate which performs different tasks based on the kind of data object. Action or Func should work well here.
Remember that when you use events, you have to ensure that handlers get unregistered in a timely manner or else you could have leaks.
In this case an event doesn't make sense. Events tend to be to inform about what is going on, not to replace function calls. They are more informative.
So, you may want to pass in an Action<> function, so that you can then call the function that was passed in, to do the processing.
This would be better than a function call, IMO.
You can look at this page for an example:
http://www.claassen.net/geek/blog/2007/12/action-func-never-write-another.html
Update: If you are not using C#3 then you may want to use a delegate instead, which, an event handler is a specialized delegate.