We Keep Coding

Rx.Net Using Observable.Create instead of Subject

I have code that uses a subject to create an observable. I'm able to pass in a value to the observable stream using subject.onNext(value). The problem is if I subscribe to that observable after a value was passed in, I would like to still get that value. From what I can tell, subjects don't hold onto value, they just pass them along like an event. So how do I change this code from using subject, to using an Observable.
private readonly Subject<int> _valueSubject = new Subject<int>();
public IObservable<int> ValueObservable => _valueSubject ;
public void SetValue(int valuePassedIn)
{
_valueSubject.OnNext(valuePassedIn);
}
Edit: Found out you can get latest values using BehaviorSubject, but would still like to know how to use Observable.Create instead of Subject in a scenario like this.

Assuming that you don't need to call the OnNext directly, a cold observable might be what you want:
IObservable<int> coldObservable = Observable.Create<int>(obs =>
{
obs.OnNext(1);
obs.OnNext(2);
obs.OnNext(3);
obs.OnCompleted();
return () => { };
});
Otherwise a ReplaySubject will allow you to keep a sized buffer of values that will be emitted as each observer subscribes. Not exactly the same as remembering all values I realize but this would not be a good idea anyway due to memory usage. Might be closer than the single value BehaviourSubject provides.
The following will allow 2 values to be available to new subscribers:
ISubject<int> replaySubject = new ReplaySubject<int>(2);
IObservable<int> observable;
[TestMethod]
public void TestMethod()
{
observable = replaySubject;
replaySubject.OnNext(1);
replaySubject.OnNext(2);
replaySubject.OnNext(3);
replaySubject.OnNext(4);
replaySubject.OnNext(5);
observable.Subscribe(OnValue);
}
Output:
Observed value:4
Observed value:5

Timeseries/temporal data in DDD on write side in CQRS

I am having trouble getting my head around how I would support timeseries/temporal data in DDD and how it would be handled on the write side using CQRS. Ultimately I would like to find a solution that also plays nice with event sourcing.
Using temperature forecasts as an example, a change in temperature could also affect the forecast energy demand for a region/location. Assuming temperature forecasts can go far in to the future (based on historic data), loading all the forecasts in to a Location aggregate I think would be impractical without applying some limit to the amount of data loaded.
What is a good/recommended approach for synchronising/storing this kind of data to be used on the write side in CQRS when keeping event sourcing in mind?
Are any of my attempts below (Option A or B) considered as suitable DDD/CQRS solutions?
Option A:
Allow temperature to be updated independently and subscribe to events using a process manager/saga to then recalculate the demand. This solution would help keep aggregate size small, however it feels like the aggregate boundary could be wrong as demand is dependent on temperature and now spread across commands/events.
// OverrideTemperatureForecastCommandHandler.cs
public void Handle(OverrideTemperatureForecast cmd)
{
var from = cmd.TemperatureOverrides.Min(t => t.DateTime);
var to = cmd.TemperatureOverrides.Max(t => t.DateTime);
TemperatureForecasts forecasts = temperatureForecastRepository.GetByLocation(cmd.LocationId, from, to);
forecasts.Override(cmd.TemperatureOverrides);
temperatureForecastRepository.Save(forecasts);
// raises
// TemperatureForecastsOverridden(locationId, overrides)
}
// TemperatureForecastsOverriddenProcessManager.cs
public void Handle(TemperatureForecastsOverridden #event)
{
var from = cmd.TemperatureOverrides.Min(t => t.DateTime);
var to = cmd.TemperatureOverrides.Max(t => t.DateTime);
// issue a command to recalculate the energy demand now temperature has changed...
commandBus.Send(new RecalculateEnergyDemand
{
LocationId = #event.LocationId,
From = from,
To = to
}));
}
// RecalculateEnergyDemandCommandHandler.cs
public void Handle(RecalculateEnergyDemand cmd)
{
EnergyDemand demandForecasts = energyDemandForecastRepository.GetByLocation(cmd.LocationId, cmd.From, cmd.To);
// have to fetch temperature forecasts again...
TemperatureForecasts temperatureForecasts = temperatureForecastRepository.GetByLocation(cmd.LocationId, cmd.From, cmd.To);
demandForecasts.AdjustForTemperature(temperatureForecasts);
energyDemandForecastRepository.Save(demandForecasts);
// raises
// ForecastDemandChanged(locationId, demandforecasts)
}
Option B:
Create a single aggregate 'Location' and pre-load forecast data internally based on a given date range. This feels cleaner from a DDD behaviour perspective however loading an aggregate constrained to time range feels a bit awkward to me (or is it just me?). Without limiting the size of the forecasts values the 'Location' aggregate could get huge.
// OverrideTemperatureForecastCommandHandler.cs
public void Handle(OverrideTemperatureForecast cmd)
{
var from = cmd.TemperatureOverrides.Min(t => t.DateTime);
var to = cmd.TemperatureOverrides.Max(t => t.DateTime);
// use from/to to limit internally the range of temperature and demand forecasts that get loaded in to the aggregate.
Location location = locationRepository.Get(cmd.LocationId, from, to);
location.OverrideTemperatureForecasts(cmd.TemperatureOverrides);
locationRepository.Save(forecasts);
// raises
// TemperatureForecastsOverridden(locationId, overrides)
// ForecastDemandChanged(locationId, demandforecasts)
}
For either option A or B, denormalisers on the read side could look something like:
// TemperatureDenormaliser.cs
public void Handle(TemperatureForecastsOverridden #event)
{
var from = #event.Overrides.Min(t => t.DateTime);
var to = #event.Overrides.Max(t => t.DateTime);
var temperatureDTOs = storage.GetByLocation(#event.LocationId, from, to);
// TODO ... (Add or update)
storage.Save(temperatureDTOs);
}
// EnergyDemandDenormalizer.cs
public void Handle(ForecastDemandChanged #event)
{
var from = #event.Overrides.Min(t => t.DateTime);
var to = #event.Overrides.Max(t => t.DateTime);
var demandDTOs = storage.GetByLocation(#event.LocationId, from, to);
// TODO ... (Add or update)
storage.Save(demandDTOs);
}

Event sourcing would not be an option with either of your examples.
As new events come in, the older ones become irrelevant. These do not necessarily need to be in one aggregate; There are no invariants to protect for the whole history of readings.
Series of events could be managed in a saga instead, only keeping a limited amount of knowledge, and cascading into result events.

How to seed an observable from a database

I'm trying to expose an observable sequence that gives observers all existing records in a database table plus any future items. For the sake of argument, lets say it's log entries. Therefore, I'd have something like this:
public class LogService
{
private readonly Subject<LogEntry> entries;
public LogService()
{
this.entries = new Subject<LogEntry>();
this.entries
.Buffer(...)
.Subscribe(async x => WriteLogEntriesToDatabaseAsync(x));
}
public IObservable<LogEntry> Entries
{
get { return this.entries; }
}
public IObservable<LogEntry> AllLogEntries
{
get
{
// how the heck?
}
}
public void Log(string message)
{
this.entries.OnNext(new LogEntry(message));
}
private async Task<IEnumerable<LogEntry>> GetLogEntriesAsync()
{
// reads existing entries from DB table and returns them
}
private async Task WriteLogEntriesToDatabaseAsync(IList<LogEntry> entries)
{
// writes entries to the database
}
}
My initial thought for the implementation of AllLogEntries was something like this:
return Observable.Create<LogEntry>(
async observer =>
{
var existingEntries = await this.GetLogEntriesAsync();
foreach (var existingEntry in existingEntries)
{
observer.OnNext(existingEntry);
}
return this.entries.Subscribe(observer);
});
But the problem with this is that there could log entries that have been buffered and not yet written to the database. Hence, those entries will be missed because they are not in the database and have already passed through the entries observable.
My next thought was to separate the buffered entries from the non-buffered and use the buffered when implementing AllLogEntries:
return Observable.Create<LogEntry>(
async observer =>
{
var existingEntries = await this.GetLogEntriesAsync();
foreach (var existingEntry in existingEntries)
{
observer.OnNext(existingEntry);
}
return this.bufferedEntries
.SelectMany(x => x)
.Subscribe(observer);
});
There are two problems with this:
It means clients of AllLogEntries also have to wait for the buffer timespan to pass before they receive their log entries. I want them to see log entries instantaneously.
There is still a race condition in that log entries could be written to the database between the point at which I finish reading the existing ones and the point at which I return the future entries.
So my question is: how would I actually go about achieving my requirements here with no possibility of race conditions, and avoiding any major performance penalties?

To do this via the client code, you will probably have to implement a solution using polling and then look for differences between calls. Possibly combining a solution with
Observable.Interval() : http://rxwiki.wikidot.com/101samples#toc28 , and
Observable.DistinctUntilChanged()
will give you sufficient solution.
Alternatively, I'd suggest you try to find a solution where the clients are notified when the DB/table is updated. In a web application, you could use something like SignalR to do this.
For example: http://techbrij.com/database-change-notifications-asp-net-signalr-sqldependency
If its not a web-application, a similar update mechanism via sockets may work.
See these links (these came from the accepted answer of SignalR polling database for updates):
http://xsockets.net/api/net-c#snippet61
https://github.com/codeplanner/XSocketsPollingLegacyDB

How to expose IObservable<T> properties without using Subject<T> backing field

In this answer to a question about Subject<T> Enigmativity mentioned:
as an aside, you should try to avoid using subjects at all. The
general rule is that if you're using a subject then you're doing
something wrong.
I often use subjects as backing fields for IObservable properties, which would have probably been .NET events in the days before Rx. e.g. instead of something like
public class Thing
{
public event EventHandler SomethingHappened;
private void DoSomething()
{
Blah();
SomethingHappened(this, EventArgs.Empty);
}
}
I might do
public class Thing
{
private readonly Subject<Unit> somethingHappened = new Subject<Unit>();
public IObservable<Unit> SomethingHappened
{
get { return somethingHappened; }
}
private void DoSomething()
{
Blah();
somethingHappened.OnNext(Unit.Default);
}
}
So, if I want to avoid using Subject what would be the correct way of doing this kind of thing? Or I should I stick to using .NET events in my interfaces, even when they'll be consumed by Rx code (so probably FromEventPattern)?
Also, a bit more details on why using Subject like this is a bad idea would be helpful.
Update: To make this question a bit more concrete, I'm talking about using Subject<T> as a way to get from non-Rx code (maybe you're working with some other legacy code) into the Rx world. So, something like:
class MyVolumeCallback : LegacyApiForSomeHardware
{
private readonly Subject<int> volumeChanged = new Subject<int>();
public IObservable<int> VolumeChanged
{
get
{
return volumeChanged.AsObservable();
}
}
protected override void UserChangedVolume(int newVolume)
{
volumeChanged.OnNext(newVolume);
}
}
Where, instead of using events, the LegacyApiForSomeHardware type makes you override virtual methods as a way of getting "this just happened" notifications.

For one thing, someone can cast the SomethingHappened back to an ISubject and feed things into it from the outside. At the very least, apply AsObservable to it in order to hide the subject-ness of the underlying object.
Also, subject broadcasting of callbacks doesn't do strictly more than a .NET event. For example, if one observer throws, the ones that are next in the chain won't be called.
static void D()
{
Action<int> a = null;
a += x =>
{
Console.WriteLine("1> " + x);
};
a += x =>
{
Console.WriteLine("2> " + x);
if (x == 42)
throw new Exception();
};
a += x =>
{
Console.WriteLine("3> " + x);
};
a(41);
try
{
a(42); // 2> throwing will prevent 3> from observing 42
}
catch { }
a(43);
}
static void S()
{
Subject<int> s = new Subject<int>();
s.Subscribe(x =>
{
Console.WriteLine("1> " + x);
});
s.Subscribe(x =>
{
Console.WriteLine("2> " + x);
if (x == 42)
throw new Exception();
});
s.Subscribe(x =>
{
Console.WriteLine("3> " + x);
});
s.OnNext(41);
try
{
s.OnNext(42); // 2> throwing will prevent 3> from observing 42
}
catch { }
s.OnNext(43);
}
In general, the caller is dead once an observer throws, unless you protect every On* call (but don't swallow exceptions arbitrarily, as shown above). This is the same for multicast delegates; exceptions will swing back at you.
Most of the time, you can achieve what you want to do without a subject, e.g. by using Observable.Create to construct a new sequence. Such sequences don't have an "observer list" that results from multiple subscriptions; each observer has its own "session" (the cold observable model), so an exception from an observer is nothing more than a suicide command in a confined area rather than blowing yourself up in the middle of a square.
Essentially, subjects are best used at the edges of the reactive query graph (for ingress streams that need to be addressable by another party that feeds in the data, though you could use regular .NET events for this and bridge them to Rx using FromEvent* methods) and for sharing subscriptions within a reactive query graph (using Publish, Replay, etc. which are Multicast calls in disguise, using a subject). One of the dangers of using subjects - which are very stateful due to their observer list and potential recording of messages - is to use them when trying to write a query operator using subjects. 99.999% of the time, such stories have a sad ending.

In an answer on the Rx forum, Dave Sexton (of Rxx), said as part an answer to something:
Subjects are the stateful components of Rx. They are useful for when
you need to create an event-like observable as a field or a local
variable.
Which is exactly what's happening with this question, he also wrote an in-depth follow up blog post on To Use Subject Or Not To Use Subject? which concludes with:
When should I use a subject?
When all of the following are true:
you don't have an observable or anything that can be converted into one.
you require a hot observable.
the scope of your observable is a type.
you don't need to define a similar event and no similar event already exists.
Why should I use a subject in that case?
Because you've got no choice!
So, answering the inner question of "details on why using Subject like this is a bad idea" - it's not a bad idea, this is one of the few places were using a Subject is the correct way to do things.

While I can't speak for Enigmativity directly, I imagine it's because it's very low-level, something you don't really need to use directly; everything that's offered by the Subject<T> class can be achieved by using the classes in the System.Reactive.Linq namespace.
Taking the example from the Subject<T> documentation:
Subject<string> mySubject = new Subject<string>();
//*** Create news feed #1 and subscribe mySubject to it ***//
NewsHeadlineFeed NewsFeed1 = new NewsHeadlineFeed("Headline News Feed #1");
NewsFeed1.HeadlineFeed.Subscribe(mySubject);
//*** Create news feed #2 and subscribe mySubject to it ***//
NewsHeadlineFeed NewsFeed2 = new NewsHeadlineFeed("Headline News Feed #2");
NewsFeed2.HeadlineFeed.Subscribe(mySubject);
This is easily achieved with the Merge extension method on the Observable class:
IObservable<string> feeds =
new NewsHeadlineFeed("Headline News Feed #1").HeadlineFeed.Merge(
new NewsHeadlineFeed("Headline News Feed #2").HeadlineFeed);
Which you can then subscribe to normally. Using Subject<T> just makes the code more complex. If you're going to use Subject<T> then you should be doing some very low-level processing of observables where the extension methods fail you.

One approach for classes which have simple one-off events, is to provide a ToObservable method which creates a meaningful cold observable based on an event.
This is more readable than using the Observable factory methods, and allows developers who don't use Rx to make use of the API.
public IObservable<T> ToObservable()
{
return Observable.Create<T>(observer =>
{
Action notifier = () =>
{
switch (Status)
{
case FutureStatus.Completed:
observer.OnNext(Value);
observer.OnCompleted();
break;
case FutureStatus.Cancelled:
observer.OnCompleted();
break;
case FutureStatus.Faulted:
observer.OnError(Exception);
break;
}
};
Resolve += notifier;
return () => Resolve -= notifier;
});
}

Best data structure for thread-safe list of subscriptions?

I am trying to build a subscription list. Let's take the example:
list of Publishers, each having a list of Magazines, each having a list of subscribers
Publishers --> Magazines --> Subscribers
Makes sense to use of a Dictionary within a Dictionary within a Dictionary in C#. Is it possible to do this without locking the entire structure when adding/removing a subscriber without race conditions?
Also the code gets messy very quickly in C# which makes me think I am not going down the right path. Is there an easier way to do this? Here are the constructor and subscribe method:
Note: The code uses Source, Type, Subscriber instead of the names above
Source ---> Type ---> Subscriber
public class SubscriptionCollection<SourceT, TypeT, SubscriberT>
{
// Race conditions here I'm sure! Not locking anything yet but should revisit at some point
ConcurrentDictionary<SourceT, ConcurrentDictionary<TypeT, ConcurrentDictionary<SubscriberT, SubscriptionInfo>>> SourceTypeSubs;
public SubscriptionCollection()
{
SourceTypeSubs = new ConcurrentDictionary<SourceT, ConcurrentDictionary<TypeT, ConcurrentDictionary<SubscriberT, SubscriptionInfo>>>();
}
public void Subscribe(SourceT sourceT, TypeT typeT, SubscriberT subT) {
ConcurrentDictionary<TypeT, ConcurrentDictionary<SubscriberT, SubscriptionInfo>> typesANDsubs;
if (SourceTypeSubs.TryGetValue(sourceT, out typesANDsubs))
{
ConcurrentDictionary<SubscriberT, SubscriptionInfo> subs;
if (typesANDsubs.TryGetValue(typeT, out subs))
{
SubscriptionInfo subInfo;
if (subs.TryGetValue(subT, out subInfo))
{
// Subscription already exists - do nothing
}
else
{
subs.TryAdd(subT, new SubscriptionInfo());
}
}
else
{
// This type does not exist - first add type, then subscription
var newType = new ConcurrentDictionary<SubscriberT, SubscriptionInfo>();
newType.TryAdd(subT, new SubscriptionInfo());
typesANDsubs.TryAdd(typeT, newType);
}
}
else
{
// this source does not exist - first add source, then type, then subscriptions
var newSource = new ConcurrentDictionary<TypeT, ConcurrentDictionary<SubscriberT, SubscriptionInfo>>();
var newType = new ConcurrentDictionary<SubscriberT, SubscriptionInfo>();
newType.TryAdd(subT, new SubscriptionInfo());
newSource.TryAdd(typeT, newType);
SourceTypeSubs.TryAdd(sourceT, newSource);
};
}

If you use ConcurrentDictionary, like you already do, you don't need locking, that's already taken care of.
But you still have to think about race conditions and how to deal with them. Fortunately, ConcurrentDictionary gives you exactly what you need. For example, if you have two threads, that both try to subscribe to source that doesn't exist yet at the same time, only one of them will succeed. But that's why TryAdd() returns whether the addition was successful. You can't just ignore its return value. If it returns false, you know some other thread already added that source, so you can retrieve the dictionary now.
Another option is to use the GetOrAdd() method. It retrieves already existing value, and creates it if it doesn't exist yet.
I would rewrite your code like this (and make it much simpler along the way):
public void Subscribe(SourceT sourceT, TypeT typeT, SubscriberT subT)
{
var typesAndSubs = SourceTypeSubs.GetOrAdd(sourceT,
_ => new ConcurrentDictionary<TypeT, ConcurrentDictionary<SubscriberT, SubscriptionInfo>>());
var subs = typesAndSubs.GetOrAdd(typeT,
_ => new ConcurrentDictionary<SubscriberT, SubscriptionInfo>());
subs.GetOrAdd(subT, _ => new SubscriptionInfo());
}