We Keep Coding

Getting the callstack from a method 'A' (MethodBase/MethodInfo) being called from 'B' or 'C' without the stacktrace

Well, I would like to do my own benchmarking system like spark in Minecraft (https://github.com/lucko/spark):
I'm using Harmony lib (https://github.com/pardeike/Harmony) which allows me to interact/modify methods and allows me to add a Prefix/Postfix on each call that will help me out with this stack.
The basic structure has something similar to (https://github.com/pardeike/Harmony/issues/355):
[HarmonyPatch]
class MyPatches
{
static IEnumerable<MethodBase> TargetMethods()
{
return AccessTools.GetTypesFromAssembly(Assembly.GetExecutingAssembly())
.SelectMany(type => type.GetMethods())
.Where(method => method.ReturnType != typeof(void) && method.Name.StartsWith("Do"));
}
static void Prefix(out Stopwatch __state, MethodBase __originalMethod)
{
__state = Stopwatch.StartNew();
// ...
}
static void Postfix(Stopwatch __state, MethodBase __originalMethod)
{
__state.Stop();
// ....
}
}
The problem here is that the __originalMethod doesn't take care if it was called from A or B.
So for example, we had patched string.Join method. And the we call from A or B, where A or B, is the full callstack of this method.
So first, we need to assign a ID to this call, and we need to create a Tree-based structure (which is hard to serialize later), from here (https://stackoverflow.com/a/36649069/3286975):
public class TreeModel : Tree<TreeModel>
{
public int ID { get; set; }
public TreeModel() { }
public TreeModel(TreeModel parent) : base(parent) { }
}
public class Tree<T> where T : Tree<T>
{
protected Tree() : this(null) { }
protected Tree(T parent)
{
Parent=parent;
Children=new List<T>();
if(parent!=null)
{
parent.Children.Add(this as T);
}
}
public T Parent { get; set; }
public List<T> Children { get; set; }
public bool IsRoot { get { return Parent==null; } }
public T Root { get { return IsRoot?this as T:Parent.Root; } }
public T RecursiveFind(Predicate<T> check)
{
if(check(this as T)) return this as T;
foreach(var item in Children)
{
var result=item.RecursiveFind(check);
if(result!=null)
{
return result;
}
}
return null;
}
}
Now, the thing is that we need to fill the Tree as long as we iterate all the method and instructions got from Harmony. Forget about Harmony for a second, I will explain only two facts about it.
The lib allows you first to get all patched methods through IEnumerable<MethodBase> TargetMethods() so, you have the Assembly X passed through reflection and filtered all methods that are allowed to be patched (some of them broke Unity, so I decided to skip methods from UnityEngine., UnityEditor. and System.* namespaces).
And we have also the ReadMethodBody method (https://harmony.pardeike.net/api/HarmonyLib.PatchProcessor.html#HarmonyLib_PatchProcessor_ReadMethodBody_System_Reflection_MethodBase_) from a given MethodBase it returns all IL stack instructions.
So we can start to iterate over and over in order to get all instructions and fill the entire tree. This is what I wrote last night:
internal static class BenchmarkEnumerator
{
internal static Dictionary<MethodBase, int> Mappings { get; } = new Dictionary<MethodBase, int>();
internal static Dictionary<int, TreeModel> TreeIDs { get; } = new Dictionary<int, TreeModel>();
internal static Dictionary<MethodBase, BenchmarkTreeModel> TreeMappings { get; } = new Dictionary<MethodBase, BenchmarkTreeModel>();
private static HashSet<int> IDUsed { get; } = new HashSet<int>();
public static int GetID(this MethodBase method)
{
return GetID(method, out _);
}
public static int GetID(this MethodBase method, out bool contains)
{
// A > X = X1
// B > X = X2
if (!Mappings.ContainsKey(method))
{
var id = Mappings.Count;
Mappings.Add(method, Mappings.Count);
IDUsed.Add(id);
contains = false;
return id;
}
contains = true;
return Mappings[method];
}
public static int GetFreeID()
{
int id;
Random rnd = new Random();
do
{
id = rnd.Next();
} while (IDUsed.Contains(id));
IDUsed.Add(id);
return id;
}
public static BenchmarkCall GetCall(int id)
{
return TreeIDs[id]?.Call;
}
public static BenchmarkCall GetCall(this MethodBase method)
{
return TreeIDs[Mappings[method]]?.Call;
}
}
The BenchmarkEnumerator class allow us to differentiate between A or B, but it doesn't care about the full hierarchy, only from the parent MethodBase itself, so I need to write something complex to take in care of the full call stack, which I said I have a problem to understand.
Then we have the TargetMethods:
private static IEnumerable<MethodBase> TargetMethods()
{
Model = new BenchmarkTreeModel();
var sw = Stopwatch.StartNew();
//int i = 0;
return Filter.GetTargetMethods(method =>
{
try
{
var instructions = PatchProcessor.ReadMethodBody(method);
var i = method.GetID(out var contains);
var tree = new TreeModel
{
ID = i
};
if (contains)
{
//var lastId = i;
i = GetFreeID();
tree.ID = i;
tree.FillMethodName($"{method.GetMethodSignature()}_{i}"); // TODO: Check this
tree.Parent = null;
tree.Children = TreeMappings[method].Forest.First().Children; // ??
//DictionaryHelper.AddOrAppend(TreeMappings, method, tree);
TreeMappings[method].Forest.Add(tree);
TreeIDs.Add(i, tree);
Model.Forest.Add(tree);
// UNIT TESTING: All contained methods at this point will have a parent.
// string.Join is being added as a method by a instruction, so when we try to patch it, it will have already a reference on the dictionary
// Here, we check if the method was already added by a instruction CALL
// Logic: If the method is already contained by the mapping dictionary
// then, we will exit adding a new that will have the same childs but a new ID
return false;
}
TreeIDs.Add(i, tree);
tree.FillMethodName($"{method.GetMethodSignature()}_{i}"); // TODO: Check this
foreach (var pair in instructions)
{
var opcode = pair.Key;
if (opcode != OpCodes.Call || opcode != OpCodes.Callvirt) continue;
var childMethod = (MethodBase)pair.Value;
var id = childMethod.GetID(out var _contains);
var subTree = new TreeModel(tree)
{
ID = id
};
if (_contains)
{
id = GetFreeID();
subTree.ID = id;
subTree.FillMethodName($"{childMethod.GetMethodSignature()}_{id}"); // TODO: Check this
subTree.Parent = TreeIDs[i];
subTree.Children = TreeMappings[childMethod].Forest.First().Children;
TreeIDs.Add(id, subTree);
continue;
}
TreeIDs.Add(id, subTree);
subTree.FillMethodName($"{childMethod.GetMethodSignature()}_{id}");
tree.Children.Add(subTree);
TreeMappings.Add(childMethod, new BenchmarkTreeModel());
TreeMappings[childMethod].Forest.Add(subTree);
}
TreeMappings.Add(method, new BenchmarkTreeModel());
TreeMappings[method].Forest.Add(tree);
Model.Forest.Add(tree);
return true;
//var treeModel = new TreeModel();
}
catch (Exception ex)
{
//Debug.LogException(new Exception(method.GetMethodSignature(), ex));
return false;
}
}, sw);
//return methods;
}
The GetMethodSignature is something like:
public static string GetMethodSignature(this MethodBase method)
{
if (method == null) return null;
return method.DeclaringType == null ? method.Name : $"{method.DeclaringType.FullName}.{method.Name}";
}
I think I'll replace it with the MethodBase.ToString instead (what do you think?)
Also, we have the BenchmarkCall class which allow us to take in care how many times the call was done and how many time it has spent at all:
[Serializable]
public class BenchmarkCall
{
public string Method { get; set; }
public double SpentMilliseconds { get; set; }
public long SpentTicks { get; set; }
public double MinSpentMs { get; set; } = double.MaxValue;
public double MaxSpentMs { get; set; } = double.MinValue;
public long MinSpentTicks { get; set; } = long.MaxValue;
public long MaxSpentTicks { get; set; } = long.MinValue;
public double AvgMs => SpentMilliseconds / TimesCalled;
public double AvgTicks => SpentTicks / (double)TimesCalled;
public BenchmarkCall()
{
}
public BenchmarkCall(MethodBase method)
{
Method = method.GetMethodSignature();
}
public override string ToString()
{
if (TimesCalled > 0)
return "BenchmarkCall{\n" +
$"Ticks[SpentTicks={SpentTicks},MinTicks={MinSpentTicks},MaxTicks={MaxSpentTicks},AvgTicks={AvgTicks:F2}]\n" +
$"Ms[SpentMs={SpentMilliseconds:F2},MinMs={MinSpentMs:F2},MaxMs={MaxSpentMs:F2},AvgMs={AvgMs:F2}]\n" +
"}";
return "BenchmarkCall{}";
}
}
}
So I think that my next movement will be to differentiate between X method being called from A or B (Xa or Xb) taking care of the full hierarchy (which I'm not sure how to do) instead of the parent method that calls it, maybe the code I wrote has some to do it with it, but I'm not sure (last night I was so tired, so I didn't code it taking care those facts), build up a list of method signatures with different IDs, and then fill up the tree, ID 1 is Xa and ID 2 is Xb (where I have problems also filling up the tree).
Also I'll need to use the Transpiler in order to alter all code instructions, so if a method has:
void method() {
X1();
X2();
}
We will need to add 2 methods (like prefix/postfix) to measure each instruction call:
void method() {
Start(1);
X1();
End(1);
Start(2);
X2();
End(2);
}
This will be a hard task, but I hope somebody could guide me with this out.

Map errors to observable using C# ReactiveX

I have an observable MyObservable<Object> which can throw CustomExceptions where
private class CustomException : Exception
What I want to do is convert the CustomExceptions into objects and emit those in a new observable.
This is my solution so far but I was wondering if this could be done without having to directly call the Subject's onNext, onCompleted or onError methods.
var MySubject = new Subject<NewObject>();
MyObservable.Catch<Object, CustomException>(
ex =>
{
NewObject o = new NewObject(ex.Message);
MySubject.OnNext(o);
return Observable.Empty<Object>();
});
IObservable<IList<NewObject>> listObservable = MySubject.ToList();
Edit: Thanks ibebbs! Worked like a charm!

You can catch and map exceptions without a subject by using the Materialize() function as shown here:
var errorObservable = source
.Select(projection)
.Materialize()
.Where(notification => notification.Kind == NotificationKind.OnError)
.Select(notification => notification.Exception)
.OfType<CustomException>()
.Select(exception => new NewObject(exception.Message));
The Materialize function takes an IObservable<T> and maps it to a IObservable<Notification<T>> where each Notification has a Kind of OnNext, OnError or OnComplete. The above observable simply looks for Notifications with a Kind`` of OnError and with the Exception being an instance of CustomException then projects these exceptions into anIObservable```.
Here is a unit test showing this working:
[Fact]
public void ShouldEmitErrorsToObservable()
{
Subject<int> source = new Subject<int>();
List<int> values = new List<int>();
List<NewObject> errors = new List<NewObject>();
Func<int, int> projection =
value =>
{
if (value % 2 == 1) throw new CustomException("Item is odd");
return value;
};
Func<CustomException, IObservable<int>> catcher = null;
catcher = ex => source.Select(projection).Catch(catcher);
var errorObservable = source
.Select(projection)
.Materialize()
.Where(notification => notification.Kind == NotificationKind.OnError)
.Select(notification => notification.Exception)
.OfType<CustomException>()
.Select(exception => new NewObject(exception.Message));
var normalSubscription = source.Select(projection).Catch(catcher).Subscribe(values.Add);
var errorSubscription = errorObservable.Subscribe(errors.Add);
source.OnNext(0);
source.OnNext(1);
source.OnNext(2);
Assert.Equal(2, values.Count);
Assert.Equal(1, errors.Count);
}
However, as you can see with the construed catch mechanisms employed above, exception handling in Rx can be tricky to get right and even more difficult to do elegantly. Instead, consider that Exceptions should be Exceptional and, if you expect an class of error such that you've written a custom exception for it, then the error is not really exceptional but part of a process flow that must handle these errors.
In this instance, I would recommend projecting the observable into a class which embodies the "try this operation and record the result, be it a value or an exception" and using this further along the execution chain.
In the example below, I use a "Fallible" class to capture the result or exception of an operation and then subscribe to a stream of "Fallible" instances, separating the errors from values. As you will see, the code is both neater and better performing as both the errors and values share a single subscription to the underlying source:
internal class Fallible
{
public static Fallible<TResult> Try<TResult, TException>(Func<TResult> action) where TException : Exception
{
try
{
return Success(action());
}
catch (TException exception)
{
return Error<TResult>(exception);
}
}
public static Fallible<T> Success<T>(T value)
{
return new Fallible<T>(value);
}
public static Fallible<T> Error<T>(Exception exception)
{
return new Fallible<T>(exception);
}
}
internal class Fallible<T>
{
public Fallible(T value)
{
Value = value;
IsSuccess = true;
}
public Fallible(Exception exception)
{
Exception = exception;
IsError = true;
}
public T Value { get; private set; }
public Exception Exception { get; private set; }
public bool IsSuccess { get; private set; }
public bool IsError { get; private set; }
}
[Fact]
public void ShouldMapErrorsToFallible()
{
Subject<int> source = new Subject<int>();
List<int> values = new List<int>();
List<NewObject> errors = new List<NewObject>();
Func<int, int> projection =
value =>
{
if (value % 2 == 1) throw new CustomException("Item is odd");
return value;
};
var observable = source
.Select(value => Fallible.Try<int, CustomException>(() => projection(value)))
.Publish()
.RefCount();
var errorSubscription = observable
.Where(fallible => fallible.IsError)
.Select(fallible => new NewObject(fallible.Exception.Message))
.Subscribe(errors.Add);
var normalSubscription = observable
.Where(fallible => fallible.IsSuccess)
.Select(fallible => fallible.Value)
.Subscribe(values.Add);
source.OnNext(0);
source.OnNext(1);
source.OnNext(2);
Assert.Equal(2, values.Count);
Assert.Equal(1, errors.Count);
}
Hope it helps.

return instance with modifying some attributes in C#

I'd like to know an idiom in C#. I write code related Remote Proxy.
public Task SetIndexAsync(int index)
{
var state = this.StateManager.TryGetStateAsync<ActorState>("MyState").GetAwaiter().GetResult().Value;
if (state == null)
{
state = new ActorState() { Temperature = 0 };
}
state.Index = index;
this.StateManager.SetStateAsync<ActorState>("MyState", state);
return Task.FromResult(true);
}
This is not cool. :(
I try to write like this. However, it is still not cool.
private ActorState StateProxy
{
get
{
return this.StateManager.TryGetStateAsync<ActorState>("MyState").GetAwaiter().GetResult().Value;
}
set
{
this.StateManager.AddOrUpdateStateAsync<ActorState>("MyState", value, (k, v) => value)
}
}
public Task SetIndexAsync(int index)
{
Func<ActorState, int, ActorState> addIndex = (state, idx) => { state.Index = idx; return state; };
addIndex(StateProxy, index);
return Task.FromResult(true);
}
If I could write one-liner which returns instance with modifying the attribute,
it might be better. Or any ideas for a cool solution for this?

Finally, I decide to go like this. I recall the design of ActiveRecord.
internal sealed class ActorState
{
[DataMember]
public List<string> PartitionNames { get; set; }
[DataMember]
public Dictionary<string, DateTime> PartitionLeases { get; set; }
private IActorStateManager stateManager;
internal static ActorState GetState(IActorStateManager stateManager)
{
var state = stateManager.GetStateAsync<ActorState>("MyState").GetAwaiter().GetResult();
state.stateManager = stateManager;
return state;
}
internal void Save()
{
stateManager.AddOrUpdateStateAsync<ActorState>("MySate", this, (k,v) => this ).GetAwaiter();
}
}
Usage:
var stateProxy = ActorState.GetState(this.StateManager);
List<string> keys = stateProxy.PartitionLeases.Keys.ToList();
foreach(string key in keys)
{
if (DateTime.Now - stateProxy.PartitionLeases[key] >= TimeSpan.FromSeconds(60))
stateProxy.PartitionLeases.Remove(key);
}
stateProxy.Save();

Rx: Count of Grouped Events in Moving Window

I have started looking at using Reactive Extensions with EventStore. As a proof of concept, I'd like to see if I can get Rx to consume an event stream and output the count of events grouped by type for a window of one second.
So, say that I am consuming a stream with the name "orders", I'd like to see something like the following appear in the console:
OrderCreated 201
OrderUpdated 111
(a second passes..)
OrderCreated 123
OrderUpdated 132
And so on.
So far, I have been able to get an output of the count of all events per second. But can't seem to be able to group them by event type.
The code I am using is based on a gist by James Nugent:
internal class EventStoreRxSubscription
{
public Subject<ResolvedEvent> ResolvedEvents { get; }
public Subject<SubscriptionDropReason> DroppedReasons { get; }
public EventStoreSubscription Subscription { get; }
public EventStoreRxSubscription(EventStoreSubscription subscription, Subject<ResolvedEvent> resolvedEvent, Subject<SubscriptionDropReason> droppedReasons)
{
Subscription = subscription;
ResolvedEvents = resolvedEvent;
DroppedReasons = droppedReasons;
}
}
static class EventStoreConnectionExtensions
{
public static Task<EventStoreRxSubscription> SubscribeTo(this IEventStoreConnection connection, string streamName, bool resolveLinkTos)
{
return Task<EventStoreRxSubscription>.Factory.StartNew(() => {
var resolvedEvents = new Subject<ResolvedEvent>();
var droppedReasons = new Subject<SubscriptionDropReason>();
var subscriptionTask = connection.SubscribeToStreamAsync(streamName, resolveLinkTos,
(subscription, #event) => resolvedEvents.OnNext(#event),
(subscription, dropReason, arg3) => droppedReasons.OnNext(dropReason));
subscriptionTask.Wait();
return new EventStoreRxSubscription(subscriptionTask.Result, resolvedEvents, droppedReasons);
});
}
}
class Program
{
static void Main(string[] args)
{
var connection = EventStoreConnection.Create(new IPEndPoint(IPAddress.Loopback, 1113));
connection.ConnectAsync();
var subscriptionTask = connection.SubscribeTo("orders", true);
subscriptionTask.Wait();
var events = subscriptionTask.Result.ResolvedEvents;
var query = events.Timestamp()
.Buffer(TimeSpan.FromSeconds(1))
.Select(e => e.Count);
query.Subscribe(Console.WriteLine);
Console.ReadLine();
}
}

I have done something similar to this before, I used Throttle to group all events within a set frequency, however you could use Buffer to get the count/collection for every period.
The example below provides an abstract example of how I achieved this, where AggregateType and AggregateFunction would be replaced by your own type and aggregation.
GroupByUntil allows you to group by a type within a set period.
subscription = observable
.GroupByUntil(e => e.Key, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e.Aggregate(new AggregateType(), (a, e) => AggregateFunction(a, e))
.Subscribe(onNext, onError, onCompleted);
EDIT
Below is a quick example I've knocked up to show how it can be done
public class EventType
{
public string Type { get; set; }
}
public class AggregatedType
{
public string EventType { get; set; }
public int Count { get; set; }
}
class Program
{
public delegate void ExampleEventHandler(EventType e);
public static event ExampleEventHandler Event;
static void Main(string[] args)
{
var subscription = Observable.FromEvent<ExampleEventHandler, EventType>(e => Event += e, e => Event -= e)
.GroupByUntil(e => e.Type, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e
.Select(ev => new AggregatedType { EventType = ev.Type })
.Aggregate(new AggregatedType(), (a, ev) => new AggregatedType { EventType = ev.EventType, Count = a.Count + 1 }))
.Subscribe(OnAggregaredEvent, OnException, OnCompleted);
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
SpinWait.SpinUntil(()=> false, TimeSpan.FromSeconds(2));
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
Console.ReadLine();
}
static void OnAggregaredEvent(AggregatedType aggregated)
{
Console.WriteLine("Type: {0}, Count: {1}", aggregated.EventType, aggregated.Count);
}
static void OnException(Exception ex)
{
}
static void OnCompleted()
{
}
}

Rx operator to distinct sequences

IMPORTANT: for a description of the results and some more details, please have a look also to my answer
I need to group and filter a sequence of objects/events that usually are replicated, buffering them with a TimeSpan interval. I try to explain it better with sort of marble diagrams:
X-X-X-X-X-Y-Y-Y-Z-Z-Z-Z-X-X-Y-Z-Z
would produce
X---Y---Z---X---Y---Z
where X, Y and Z are different event types, and '---' means the interval.
Additionally, I would also like to distinct by a key property that it is available on all types because they have a common base class:
X, Y, Z : A
and A contains a property Key. Using the notation X.a meaning X.Key = a, A final sample would be:
X.a-X.b-X.a-Y.b-Y.c-Z.a-Z.a-Z.c-Z.b-Z.c
would produce
X.a-X.b---Y.b-Y.c-Z.a-Z.c-Z.b
Can anybody help me putting together the required Linq operators (probably DistinctUntilChanged and Buffer) to achieve this behavior?
Thanks
UPDATE 18.08.12:
as requested, I try to give a better explanation.
We have devices collecting and sending events to a web service. These devices have an old logic (and we can't change it due to backward compatibility) and they continuously send an event until they receive an acknowledge; after the acknowledge, they send the next event in their queue, and so on.
Events contain the network address of the unit and some other properties distinguishing events in the queue for each device.
An event looks like this:
class Event
{
public string NetworkAddress { get; }
public string EventCode { get; }
public string AdditionalAttribute { get; }
}
The goal is that of processing every 5 seconds the distinguished events received from all devices, storing information in the database (that's why we don't want to do it in batches) and sending the ack to the device.
Let's make an example with only two devices and some events:
Device 'a':
Event 1 (a1): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'x'
Event 2 (a2): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'y'
Event 3 (a3): NetworkAddress = '1', EventCode = B, AdditionalAttribute = 'x'
Device 'b':
Event 1 (b1): NetworkAddress = '2', EventCode = A, AdditionalAttribute = 'y'
Event 2 (b2): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'x'
Event 3 (b3): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'y'
Event 4 (b4): NetworkAddress = '2', EventCode = C, AdditionalAttribute = 'x'
Pn are the operations done by our server, explained later
Possible marble diagram (input streams + output stream):
Device 'a' : -[a1]-[a1]-[a1]----------------[a2]-[a2]-[a2]-[a3]-[a3]-[a3]-...
Device 'b' : ------[b1]-[b1]-[b2]-[b2]-[b2]------[b3]-[b3]-[b4]-[b4]-[b4]-...
Time : ------------[1s]-----------[2s]------------[3s]------------[4s]-
DB/acks (rx output) : ------------[P1]-----------[P2]------------[P3]------------[P4]-
P1: Server stores and acknowledges [a1] and [b1]
P2: " " " " [b2]
P3: " " " " [a2] and [b3]
P4: " " " " [a3] and [b4]
At the end I think it is probably a simple combination of basic operators, but I'm new to Rx and I'm a bit confused since it seems that there are lots of operators (or combinations of operators) to get the same output stream.
Update 19.08.12:
Please keep in mind that this code runs on a server and it should run for days without memory leaks...I'm not sure about the behavior of subjects. At the moment, for each event I call a push operation on a service, which calls the OnNext of a Subject on top of which I should build the query (if I'm not wrong about the usage of subjects).
Update 20.08.12:
Current implementation, including validation test; this is what I tried and it seems the same suggested by #yamen
public interface IEventService
{
// Persists the events
void Add(IEnumerable<Event> events);
}
public class Event
{
public string Description { get; set; }
}
/// <summary>
/// Implements the logic to handle events.
/// </summary>
public class EventManager : IDisposable
{
private static readonly TimeSpan EventHandlingPeriod = TimeSpan.FromSeconds(5);
private readonly Subject<EventMessage> subject = new Subject<EventMessage>();
private readonly IDisposable subscription;
private readonly object locker = new object();
private readonly IEventService eventService;
/// <summary>
/// Initializes a new instance of the <see cref="EventManager"/> class.
/// </summary>
/// <param name="scheduler">The scheduler.</param>
public EventManager(IEventService eventService, IScheduler scheduler)
{
this.eventService = eventService;
this.subscription = this.CreateQuery(scheduler);
}
/// <summary>
/// Pushes the event.
/// </summary>
/// <param name="eventMessage">The event message.</param>
public void PushEvent(EventMessage eventMessage)
{
Contract.Requires(eventMessage != null);
this.subject.OnNext(eventMessage);
}
/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
/// <filterpriority>2</filterpriority>
public void Dispose()
{
this.Dispose(true);
}
private void Dispose(bool disposing)
{
if (disposing)
{
// Dispose unmanaged resources
}
this.subject.Dispose();
this.subscription.Dispose();
}
private IDisposable CreateQuery(IScheduler scheduler)
{
var buffered = this.subject
.DistinctUntilChanged(new EventComparer())
.Buffer(EventHandlingPeriod, scheduler);
var query = buffered
.Subscribe(this.HandleEvents);
return query;
}
private void HandleEvents(IList<EventMessage> eventMessages)
{
Contract.Requires(eventMessages != null);
var events = eventMessages.Select(this.SelectEvent);
this.eventService.Add(events);
}
private Event SelectEvent(EventMessage message)
{
return new Event { Description = "evaluated description" };
}
private class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
return x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
// Other properties
}
And the test:
public void PushEventTest()
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var eventServiceMock = new Mock<IEventService>();
var scheduler = new TestScheduler();
var target = new EventManager(eventServiceMock.Object, scheduler);
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
scheduler.Schedule(() => target.PushEvent(eventMessageA1));
scheduler.Schedule(TimeSpan.FromSeconds(1), () => target.PushEvent(eventMessageB1));
scheduler.Schedule(TimeSpan.FromSeconds(2), () => target.PushEvent(eventMessageA1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(6).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 2)), Times.Once());
scheduler.Schedule(TimeSpan.FromSeconds(3), () => target.PushEvent(eventMessageB1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(11).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 1)), Times.Once());
}
Additionally, I remark again that it is really important that the software could run for days without problems, handling thousands of messages.
To make it clear: the test doesn't pass with the current implementation.

I'm not sure if this does exactly what you'd like, but you may be to group the elements explicitly using the group keyword, and then to manipulate the various IObservables separately before recombining them.
E.g. if we have class definitions such as
class A
{
public char Key { get; set; }
}
class X : A { }
...
and a Subject<A>
Subject<A> subject = new Subject<A>();
then we can write
var buffered =
from a in subject
group a by new { Type = a.GetType(), Key = a.Key } into g
from buffer in g.Buffer(TimeSpan.FromMilliseconds(300))
where buffer.Any()
select new
{
Count = buffer.Count,
Type = buffer.First().GetType().Name,
Key = buffer.First().Key
};
buffered.Do(Console.WriteLine).Subscribe();
We can test this with the data you provided:
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'b' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
...
subject.OnCompleted();
To get the output you provided:
{ Count = 2, Type = X, Key = a }
{ Count = 1, Type = X, Key = b }
{ Count = 1, Type = Y, Key = b }
{ Count = 1, Type = Y, Key = c }
{ Count = 2, Type = Z, Key = a }
{ Count = 2, Type = Z, Key = c }
{ Count = 1, Type = Z, Key = b }

Not sure if this is exactly what you want, but it seems to support your use cases.
First, let's define the base class to use (you can easily modify this to suit your needs):
public class MyEvent
{
public string NetworkAddress { set; get; }
public string EventCode { set; get; }
}
Let's set up your devices as an array of IObservable<MyEvent> - you may have these available differently, and the below would need to change to accommodate that of course. These devices will each produce a value with a random delay between 0.5 and 1.5 seconds.
var deviceA = new MyEvent[] { new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "2"} };
var deviceB = new MyEvent[] { new MyEvent() {NetworkAddress = "B", EventCode = "1"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "3"} };
var random = new Random();
var deviceARand = deviceA.ToObservable().Select(a => Observable.Return(a).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var deviceBRand = deviceB.ToObservable().Select(b => Observable.Return(b).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var devices = new IObservable<MyEvent>[] { deviceARand, deviceBRand };
Now let's take all of these individual device streams, make them 'distinct', and merge them into a single master stream:
var stream = devices.Aggregate(Observable.Empty<MyEvent>(), (acc, device) => acc.DistinctUntilChanged(a => a.EventCode).Merge(device));
Once you have that, getting this stream to be consumed periodically is just a matter of buffering it with Buffer:
stream.Buffer(TimeSpan.FromSeconds(1)).Subscribe(x => { /* code here works on a list of the filtered events per second */ });

After searches and experiments, I put together some code that produces the output that I expect:
static void Main(string[] args)
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var comparer = new EventComparer();
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 5 };
var list = new[] { eventMessageA1, eventMessageA1, eventMessageB1, eventMessageA2, eventMessageA1, eventMessageA1 };
var queue = new BlockingCollection<EventMessage>();
Observable.Interval(TimeSpan.FromSeconds(2)).Subscribe
(
l => list.ToList().ForEach(m =>
{
Console.WriteLine("Producing {0} on thread {1}", m, Thread.CurrentThread.ManagedThreadId);
queue.Add(m);
})
);
// subscribing
queue.GetConsumingEnumerable()
.ToObservable()
.Buffer(TimeSpan.FromSeconds(5))
.Subscribe(e =>
{
Console.WriteLine("Queue contains {0} items", queue.Count);
e.Distinct(comparer).ToList().ForEach(m =>
Console.WriteLine("{0} - Consuming: {1} (queue contains {2} items)", DateTime.UtcNow, m, queue.Count));
}
);
Console.WriteLine("Type enter to exit");
Console.ReadLine();
}
public class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
var result = x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
return result;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
public override string ToString()
{
const string Format = "{0} ({1}, {2})";
var s = string.Format(Format, this.NetworkAddress, this.EventCode, this.Attribute);
return s;
}
}
Anyway, monitoring the application, it seems that this causes a memory leak.
My question is now:
what is causing the memory leak? [please see the update below]
is this the best way to do it (if I put the distinct on the first observable, I don't get the other events on next buffers, but items in each buffer should be isolated from others)?
how can I write a test using the test scheduler?
UPDATE:
it seems that the memory increment lasts only some minutes, then the value is stable. I will run a long test.
Of course, this would be an absolutely acceptable behavior.
UPDATE 26.08.12:
as I already mentioned in the previous update, the memory usage increases only (and slowly) for some minutes after the startup. After 8 hours the memory consumed was stable, with normal fluctuations in the range of few KB)
this question is very similar to mine and the suggested Drain extension could apply well to my problem (still to be verified)
Anyway, I think that my question is still open for unit tests using the test scheduler.
thanks
Francesco