IMPORTANT: for a description of the results and some more details, please have a look also to my answer
I need to group and filter a sequence of objects/events that usually are replicated, buffering them with a TimeSpan interval. I try to explain it better with sort of marble diagrams:
X-X-X-X-X-Y-Y-Y-Z-Z-Z-Z-X-X-Y-Z-Z
would produce
X---Y---Z---X---Y---Z
where X, Y and Z are different event types, and '---' means the interval.
Additionally, I would also like to distinct by a key property that it is available on all types because they have a common base class:
X, Y, Z : A
and A contains a property Key. Using the notation X.a meaning X.Key = a, A final sample would be:
X.a-X.b-X.a-Y.b-Y.c-Z.a-Z.a-Z.c-Z.b-Z.c
would produce
X.a-X.b---Y.b-Y.c-Z.a-Z.c-Z.b
Can anybody help me putting together the required Linq operators (probably DistinctUntilChanged and Buffer) to achieve this behavior?
Thanks
UPDATE 18.08.12:
as requested, I try to give a better explanation.
We have devices collecting and sending events to a web service. These devices have an old logic (and we can't change it due to backward compatibility) and they continuously send an event until they receive an acknowledge; after the acknowledge, they send the next event in their queue, and so on.
Events contain the network address of the unit and some other properties distinguishing events in the queue for each device.
An event looks like this:
class Event
{
public string NetworkAddress { get; }
public string EventCode { get; }
public string AdditionalAttribute { get; }
}
The goal is that of processing every 5 seconds the distinguished events received from all devices, storing information in the database (that's why we don't want to do it in batches) and sending the ack to the device.
Let's make an example with only two devices and some events:
Device 'a':
Event 1 (a1): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'x'
Event 2 (a2): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'y'
Event 3 (a3): NetworkAddress = '1', EventCode = B, AdditionalAttribute = 'x'
Device 'b':
Event 1 (b1): NetworkAddress = '2', EventCode = A, AdditionalAttribute = 'y'
Event 2 (b2): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'x'
Event 3 (b3): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'y'
Event 4 (b4): NetworkAddress = '2', EventCode = C, AdditionalAttribute = 'x'
Pn are the operations done by our server, explained later
Possible marble diagram (input streams + output stream):
Device 'a' : -[a1]-[a1]-[a1]----------------[a2]-[a2]-[a2]-[a3]-[a3]-[a3]-...
Device 'b' : ------[b1]-[b1]-[b2]-[b2]-[b2]------[b3]-[b3]-[b4]-[b4]-[b4]-...
Time : ------------[1s]-----------[2s]------------[3s]------------[4s]-
DB/acks (rx output) : ------------[P1]-----------[P2]------------[P3]------------[P4]-
P1: Server stores and acknowledges [a1] and [b1]
P2: " " " " [b2]
P3: " " " " [a2] and [b3]
P4: " " " " [a3] and [b4]
At the end I think it is probably a simple combination of basic operators, but I'm new to Rx and I'm a bit confused since it seems that there are lots of operators (or combinations of operators) to get the same output stream.
Update 19.08.12:
Please keep in mind that this code runs on a server and it should run for days without memory leaks...I'm not sure about the behavior of subjects. At the moment, for each event I call a push operation on a service, which calls the OnNext of a Subject on top of which I should build the query (if I'm not wrong about the usage of subjects).
Update 20.08.12:
Current implementation, including validation test; this is what I tried and it seems the same suggested by #yamen
public interface IEventService
{
// Persists the events
void Add(IEnumerable<Event> events);
}
public class Event
{
public string Description { get; set; }
}
/// <summary>
/// Implements the logic to handle events.
/// </summary>
public class EventManager : IDisposable
{
private static readonly TimeSpan EventHandlingPeriod = TimeSpan.FromSeconds(5);
private readonly Subject<EventMessage> subject = new Subject<EventMessage>();
private readonly IDisposable subscription;
private readonly object locker = new object();
private readonly IEventService eventService;
/// <summary>
/// Initializes a new instance of the <see cref="EventManager"/> class.
/// </summary>
/// <param name="scheduler">The scheduler.</param>
public EventManager(IEventService eventService, IScheduler scheduler)
{
this.eventService = eventService;
this.subscription = this.CreateQuery(scheduler);
}
/// <summary>
/// Pushes the event.
/// </summary>
/// <param name="eventMessage">The event message.</param>
public void PushEvent(EventMessage eventMessage)
{
Contract.Requires(eventMessage != null);
this.subject.OnNext(eventMessage);
}
/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
/// <filterpriority>2</filterpriority>
public void Dispose()
{
this.Dispose(true);
}
private void Dispose(bool disposing)
{
if (disposing)
{
// Dispose unmanaged resources
}
this.subject.Dispose();
this.subscription.Dispose();
}
private IDisposable CreateQuery(IScheduler scheduler)
{
var buffered = this.subject
.DistinctUntilChanged(new EventComparer())
.Buffer(EventHandlingPeriod, scheduler);
var query = buffered
.Subscribe(this.HandleEvents);
return query;
}
private void HandleEvents(IList<EventMessage> eventMessages)
{
Contract.Requires(eventMessages != null);
var events = eventMessages.Select(this.SelectEvent);
this.eventService.Add(events);
}
private Event SelectEvent(EventMessage message)
{
return new Event { Description = "evaluated description" };
}
private class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
return x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
// Other properties
}
And the test:
public void PushEventTest()
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var eventServiceMock = new Mock<IEventService>();
var scheduler = new TestScheduler();
var target = new EventManager(eventServiceMock.Object, scheduler);
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
scheduler.Schedule(() => target.PushEvent(eventMessageA1));
scheduler.Schedule(TimeSpan.FromSeconds(1), () => target.PushEvent(eventMessageB1));
scheduler.Schedule(TimeSpan.FromSeconds(2), () => target.PushEvent(eventMessageA1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(6).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 2)), Times.Once());
scheduler.Schedule(TimeSpan.FromSeconds(3), () => target.PushEvent(eventMessageB1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(11).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 1)), Times.Once());
}
Additionally, I remark again that it is really important that the software could run for days without problems, handling thousands of messages.
To make it clear: the test doesn't pass with the current implementation.
I'm not sure if this does exactly what you'd like, but you may be to group the elements explicitly using the group keyword, and then to manipulate the various IObservables separately before recombining them.
E.g. if we have class definitions such as
class A
{
public char Key { get; set; }
}
class X : A { }
...
and a Subject<A>
Subject<A> subject = new Subject<A>();
then we can write
var buffered =
from a in subject
group a by new { Type = a.GetType(), Key = a.Key } into g
from buffer in g.Buffer(TimeSpan.FromMilliseconds(300))
where buffer.Any()
select new
{
Count = buffer.Count,
Type = buffer.First().GetType().Name,
Key = buffer.First().Key
};
buffered.Do(Console.WriteLine).Subscribe();
We can test this with the data you provided:
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'b' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
...
subject.OnCompleted();
To get the output you provided:
{ Count = 2, Type = X, Key = a }
{ Count = 1, Type = X, Key = b }
{ Count = 1, Type = Y, Key = b }
{ Count = 1, Type = Y, Key = c }
{ Count = 2, Type = Z, Key = a }
{ Count = 2, Type = Z, Key = c }
{ Count = 1, Type = Z, Key = b }
Not sure if this is exactly what you want, but it seems to support your use cases.
First, let's define the base class to use (you can easily modify this to suit your needs):
public class MyEvent
{
public string NetworkAddress { set; get; }
public string EventCode { set; get; }
}
Let's set up your devices as an array of IObservable<MyEvent> - you may have these available differently, and the below would need to change to accommodate that of course. These devices will each produce a value with a random delay between 0.5 and 1.5 seconds.
var deviceA = new MyEvent[] { new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "2"} };
var deviceB = new MyEvent[] { new MyEvent() {NetworkAddress = "B", EventCode = "1"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "3"} };
var random = new Random();
var deviceARand = deviceA.ToObservable().Select(a => Observable.Return(a).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var deviceBRand = deviceB.ToObservable().Select(b => Observable.Return(b).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var devices = new IObservable<MyEvent>[] { deviceARand, deviceBRand };
Now let's take all of these individual device streams, make them 'distinct', and merge them into a single master stream:
var stream = devices.Aggregate(Observable.Empty<MyEvent>(), (acc, device) => acc.DistinctUntilChanged(a => a.EventCode).Merge(device));
Once you have that, getting this stream to be consumed periodically is just a matter of buffering it with Buffer:
stream.Buffer(TimeSpan.FromSeconds(1)).Subscribe(x => { /* code here works on a list of the filtered events per second */ });
After searches and experiments, I put together some code that produces the output that I expect:
static void Main(string[] args)
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var comparer = new EventComparer();
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 5 };
var list = new[] { eventMessageA1, eventMessageA1, eventMessageB1, eventMessageA2, eventMessageA1, eventMessageA1 };
var queue = new BlockingCollection<EventMessage>();
Observable.Interval(TimeSpan.FromSeconds(2)).Subscribe
(
l => list.ToList().ForEach(m =>
{
Console.WriteLine("Producing {0} on thread {1}", m, Thread.CurrentThread.ManagedThreadId);
queue.Add(m);
})
);
// subscribing
queue.GetConsumingEnumerable()
.ToObservable()
.Buffer(TimeSpan.FromSeconds(5))
.Subscribe(e =>
{
Console.WriteLine("Queue contains {0} items", queue.Count);
e.Distinct(comparer).ToList().ForEach(m =>
Console.WriteLine("{0} - Consuming: {1} (queue contains {2} items)", DateTime.UtcNow, m, queue.Count));
}
);
Console.WriteLine("Type enter to exit");
Console.ReadLine();
}
public class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
var result = x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
return result;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
public override string ToString()
{
const string Format = "{0} ({1}, {2})";
var s = string.Format(Format, this.NetworkAddress, this.EventCode, this.Attribute);
return s;
}
}
Anyway, monitoring the application, it seems that this causes a memory leak.
My question is now:
what is causing the memory leak? [please see the update below]
is this the best way to do it (if I put the distinct on the first observable, I don't get the other events on next buffers, but items in each buffer should be isolated from others)?
how can I write a test using the test scheduler?
UPDATE:
it seems that the memory increment lasts only some minutes, then the value is stable. I will run a long test.
Of course, this would be an absolutely acceptable behavior.
UPDATE 26.08.12:
as I already mentioned in the previous update, the memory usage increases only (and slowly) for some minutes after the startup. After 8 hours the memory consumed was stable, with normal fluctuations in the range of few KB)
this question is very similar to mine and the suggested Drain extension could apply well to my problem (still to be verified)
Anyway, I think that my question is still open for unit tests using the test scheduler.
thanks
Francesco
Related
Is there a collection in C# that guarantees me that I will have only unique elements? I've read about HashSet, but this collection can contain duplicates. Here is my code:
public class Bean
{
public string Name { get; set; }
public int Id { get; set; }
public override bool Equals(object obj)
{
var bean = obj as Bean;
if (bean == null)
{
return false;
}
return this.Name.Equals(bean.Name) && this.Id == bean.Id;
}
public override int GetHashCode()
{
return Name.GetHashCode() * this.Id.GetHashCode();
}
}
You may complain about using non-readonly properties in my GetHashCode method, but this is a way of doing (not the right one).
HashSet<Bean> set = new HashSet<Bean>();
Bean b1 = new Bean {Name = "n", Id = 1};
Bean b2 = new Bean {Name = "n", Id = 2};
set.Add(b1);
set.Add(b2);
b2.Id = 1;
var elements = set.ToList();
var elem1 = elements[0];
var elem2 = elements[1];
if (elem1.Equals(elem2))
{
Console.WriteLine("elements are equal");
}
And in this case, my set contains duplicates.
So is there a collection in C# that guarantees me that it does not contains duplicates?
So is there a collection in C# that guarantees me that it does not
contains duplicates?
There is no existing collection class in C# that does this. You could write your own, but there is no existing one.
Some extra information regarding the issue you are experiencing
If you change a HashSet entry after adding it to the HashSet, then you need to regenerate the HashSet. My below RegenerateHashSet can be used to do that.
The reason you need to regenerate is that duplicate detection only occurs at insertion time (or, in other words, it relies on you not changing an object after you insert it). Which makes sense, if you think about it. The HashSet has no way to detect that an object it contains has changed.
using System;
using System.Collections.Generic;
using System.Linq;
namespace Test
{
public static class HashSetExtensions
{
public static HashSet<T> RegenerateHashSet<T>(this HashSet<T> original)
{
return new HashSet<T>(original, original.Comparer);
}
}
public class Bean
{
public string Name { get; set; }
public int Id { get; set; }
public override bool Equals(object obj)
{
var bean = obj as Bean;
if (bean == null)
{
return false;
}
return Name.Equals(bean.Name) && Id == bean.Id;
}
public override int GetHashCode()
{
return Name.GetHashCode() * Id.GetHashCode();
}
}
public class Program
{
static void Main(string[] args)
{
HashSet<Bean> set = new HashSet<Bean>();
Bean b1 = new Bean { Name = "n", Id = 1 };
Bean b2 = new Bean { Name = "n", Id = 2 };
set.Add(b1);
set.Add(b2);
b2.Id = 1;
var elements = set.ToList();
var elem1 = elements[0];
var elem2 = elements[1];
if (elem1.Equals(elem2))
{
Console.WriteLine("elements are equal");
}
Console.WriteLine(set.Count);
set = set.RegenerateHashSet();
Console.WriteLine(set.Count);
Console.ReadLine();
}
}
}
Note that the above technique is not bullet-proof - if you add two objects (Object A and Object B) which are duplicates and then change Object B to be different to Object A then the HashSet will still only have one entry in it (since Object B was never added). As such, what you probably want to do is actually store your complete list in a List instead, and then use new HashSet<T>(yourList) whenever you want unique entries. The below class may assist you if you decide to go down that route.
public class RecalculatingHashSet<T>
{
private List<T> originalValues = new List<T>();
public HashSet<T> GetUnique()
{
return new HashSet<T>(originalValues);
}
public void Add(T item)
{
originalValues.Add(item);
}
}
If you don't write your own collection type and handle property changed events to re-evaluate the items, you need to re-evaluate the items at each access. This can be accomplished with LINQ deferred execution:
ICollection<Bean> items= new List<Bean>();
IEnumerable<Bean> reader = items.Distinct();
Rule: only use items to insert or remove elements, use reader for any read access.
Bean b1 = new Bean { Name = "n", Id = 1 };
Bean b2 = new Bean { Name = "n", Id = 2 };
items.Add(b1);
items.Add(b2);
b2.Id = 1;
var elements = reader.ToList();
var elem1 = elements[0];
var elem2 = elements[1]; // throws exception because there is only one element in the result list.
I have started looking at using Reactive Extensions with EventStore. As a proof of concept, I'd like to see if I can get Rx to consume an event stream and output the count of events grouped by type for a window of one second.
So, say that I am consuming a stream with the name "orders", I'd like to see something like the following appear in the console:
OrderCreated 201
OrderUpdated 111
(a second passes..)
OrderCreated 123
OrderUpdated 132
And so on.
So far, I have been able to get an output of the count of all events per second. But can't seem to be able to group them by event type.
The code I am using is based on a gist by James Nugent:
internal class EventStoreRxSubscription
{
public Subject<ResolvedEvent> ResolvedEvents { get; }
public Subject<SubscriptionDropReason> DroppedReasons { get; }
public EventStoreSubscription Subscription { get; }
public EventStoreRxSubscription(EventStoreSubscription subscription, Subject<ResolvedEvent> resolvedEvent, Subject<SubscriptionDropReason> droppedReasons)
{
Subscription = subscription;
ResolvedEvents = resolvedEvent;
DroppedReasons = droppedReasons;
}
}
static class EventStoreConnectionExtensions
{
public static Task<EventStoreRxSubscription> SubscribeTo(this IEventStoreConnection connection, string streamName, bool resolveLinkTos)
{
return Task<EventStoreRxSubscription>.Factory.StartNew(() => {
var resolvedEvents = new Subject<ResolvedEvent>();
var droppedReasons = new Subject<SubscriptionDropReason>();
var subscriptionTask = connection.SubscribeToStreamAsync(streamName, resolveLinkTos,
(subscription, #event) => resolvedEvents.OnNext(#event),
(subscription, dropReason, arg3) => droppedReasons.OnNext(dropReason));
subscriptionTask.Wait();
return new EventStoreRxSubscription(subscriptionTask.Result, resolvedEvents, droppedReasons);
});
}
}
class Program
{
static void Main(string[] args)
{
var connection = EventStoreConnection.Create(new IPEndPoint(IPAddress.Loopback, 1113));
connection.ConnectAsync();
var subscriptionTask = connection.SubscribeTo("orders", true);
subscriptionTask.Wait();
var events = subscriptionTask.Result.ResolvedEvents;
var query = events.Timestamp()
.Buffer(TimeSpan.FromSeconds(1))
.Select(e => e.Count);
query.Subscribe(Console.WriteLine);
Console.ReadLine();
}
}
I have done something similar to this before, I used Throttle to group all events within a set frequency, however you could use Buffer to get the count/collection for every period.
The example below provides an abstract example of how I achieved this, where AggregateType and AggregateFunction would be replaced by your own type and aggregation.
GroupByUntil allows you to group by a type within a set period.
subscription = observable
.GroupByUntil(e => e.Key, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e.Aggregate(new AggregateType(), (a, e) => AggregateFunction(a, e))
.Subscribe(onNext, onError, onCompleted);
EDIT
Below is a quick example I've knocked up to show how it can be done
public class EventType
{
public string Type { get; set; }
}
public class AggregatedType
{
public string EventType { get; set; }
public int Count { get; set; }
}
class Program
{
public delegate void ExampleEventHandler(EventType e);
public static event ExampleEventHandler Event;
static void Main(string[] args)
{
var subscription = Observable.FromEvent<ExampleEventHandler, EventType>(e => Event += e, e => Event -= e)
.GroupByUntil(e => e.Type, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e
.Select(ev => new AggregatedType { EventType = ev.Type })
.Aggregate(new AggregatedType(), (a, ev) => new AggregatedType { EventType = ev.EventType, Count = a.Count + 1 }))
.Subscribe(OnAggregaredEvent, OnException, OnCompleted);
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
SpinWait.SpinUntil(()=> false, TimeSpan.FromSeconds(2));
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
Console.ReadLine();
}
static void OnAggregaredEvent(AggregatedType aggregated)
{
Console.WriteLine("Type: {0}, Count: {1}", aggregated.EventType, aggregated.Count);
}
static void OnException(Exception ex)
{
}
static void OnCompleted()
{
}
}
I tried to compare two lists by using the Except method. But when I did, I got an error saying:
Cannot convert from 'Systems.Collections.Generic.List<>' to 'System.Linq.IQueryable<>'
'System.Collections.Generic.List<> does not contain a definition for 'Except' and the best extension method overload 'System.Linq.Queryable.Except(System.Linq.IQueryable, System.Collections.GEneric.IEnumerable)' has some invalid arguments
I also experienced this when I tried Intersect. I'm trying to compare Sent list and Result list (code and list shown below) and return items that does not have any match. So when I googled for how to do so, I came across the Except method as well as the Intersect.
public class Sent
{
public string Address;
public string Data;
}
public class Result
{
public string AddressOK;
public string DataOK;
}
var sent = new List<Sent>();
sent.Add(new Sent() { Address = linaddr1, Data = lindat1 });
var res = new List<Result>();
res.Add( new Result() { AddressOK = linaddr2, DataOK = lindat2 } );
//linaddr1 and 2, lindat1 and 2 contains the address and data shown in the list below
//taken from another part of the entire program
The lists look like such:
sent res
Address Data Address Data
04004C 55AA55 04004C 55AA55
040004 0720 040004 0720
040037 30
04004A FFFF 04004A FFFF
I only tried using this code:
var diff = sent.Except(res).ToList()
but as I've mentioned, it results with the aforementioned errors above.
EDIT: I edited the list. Sorry for that. It's just only a matter of the res list missing one or two or more items from the original list and then comparing both lists to see which item/s is/are missing from the res list.
Use Any:
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
var sent = new List<Sent>()
{
new Sent { Address = "04004C", Data = "55AA55" },
new Sent { Address = "040004", Data = "0720" },
new Sent { Address = "040037", Data = "31" },
new Sent { Address = "04004A", Data = "FFFF" }
};
var res = new List<Result> () {
new Result { AddressOK = "04004C", DataOK = "55AA55" },
new Result { AddressOK = "040004", DataOK = "0721" },
new Result { AddressOK = "040038 ", DataOK = "31" },
new Result { AddressOK = "04004A", DataOK = "FFFF" }
};
var diff =
sent.Where (s => !res.Any (r => s.Address == r.AddressOK && s.Data == r.DataOK ));
foreach (var item in diff)
{
Console.WriteLine("{0} {1}", item.Address, item.Data);
}
}
}
public class Sent
{
public string Address;
public string Data;
}
public class Result
{
public string AddressOK;
public string DataOK;
}
Output:
040004 0720
040037 31
Live Code: https://dotnetfiddle.net/ZVuiPd
The types Sent and Result are distinct types, but sent.Except(res) expects them to be the same. That's your first mistake.
The following is a simple (but incorrect) fix:
var diff =
sent
.Except(res.Select(x => new Sent() { Address = x.AddressOK, Data = x.DataOK }))
.ToList();
Even though this compiles, and runs, it doesn't remove the duplicates because your Sent doesn't override GetHashCode and Equals, hence it only compares references and not the actual properties.
You can either implement GetHashCode and Equals, or create an IEqualityComparer<Sent> to get this to work.
An IEqualityComparer<Sent> implementation might look like this:
public class SentEqualityComparer : IEqualityComparer<Sent>
{
public int GetHashCode(Sent sent)
{
return sent.Address.GetHashCode() ^ sent.Data.GetHashCode();
}
public bool Equals(Sent left, Sent right)
{
return (left.Address == right.Address) && (left.Data == right.Data);
}
}
And you would use it like so:
var diff =
sent
.Except(
res.Select(x => new Sent() { Address = x.AddressOK, Data = x.DataOK }),
new SentEqualityComparer())
.ToList();
This works as you expect.
The other option, to override GetHashCode and Equals, comes with an additional hurdle. The result of GetHashCode should not ever change throughout the lifetime of the object otherwise you can't use the object in a dictionary or any other data structure that relies on the hash code.
So, to make it work, you need to change Address & Data to be read-only.
Here is an implementation of your Sent class that will work correctly:
public sealed class Sent : IEquatable<Sent>
{
private readonly string _Address;
private readonly string _Data;
public string Address { get { return _Address; } }
public string Data { get { return _Data; } }
public Sent(string Address, string Data)
{
_Address = Address;
_Data = Data;
}
public override bool Equals(object obj)
{
if (obj is Sent)
return Equals((Sent)obj);
return false;
}
public bool Equals(Sent obj)
{
if (obj == null) return false;
if (!EqualityComparer<string>.Default.Equals(_Address, obj._Address)) return false;
if (!EqualityComparer<string>.Default.Equals(_Data, obj._Data)) return false;
return true;
}
public override int GetHashCode()
{
int hash = 0;
hash ^= EqualityComparer<string>.Default.GetHashCode(_Address);
hash ^= EqualityComparer<string>.Default.GetHashCode(_Data);
return hash;
}
}
If you are comfortable using an AOP component to automate the manual code of implementing IEquatable, another approach would be is to use Equals.Fody:
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
var a = new Sent { Address = "04004C", Data = "55AA55" };
var b = new Sent { Address = "04004C", Data = "55AA55" };
Console.WriteLine(a.Equals(b)); // True with use of an AOP, False with no AOP
var sent = new List<Sent>() {
new Sent { Address = "04004C", Data = "55AA55" },
new Sent { Address = "040004", Data = "0720" },
new Sent { Address = "040037", Data = "31" },
new Sent { Address = "04004A", Data = "FFFF" }
};
var res = new List<Result>() {
new Result { AddressOK = "04004C", DataOK = "55AA55" },
new Result { AddressOK = "040004", DataOK = "0721" },
new Result { AddressOK = "040038 ", DataOK = "31" },
new Result { AddressOK = "04004A", DataOK = "FFFF" }
};
var diff =
sent.Except(
res.Select(r => new Sent { Address = r.AddressOK, Data = r.DataOK })
);
foreach (var item in diff)
Console.WriteLine("{0} {1}", item.Address, item.Data);
}
}
[Equals]
public class Sent
{
public string Address;
public string Data;
[CustomEqualsInternal]
bool CustomLogic(Sent other)
{
return other.Address == this.Address && other.Data == this.Data;
}
}
public class Result
{
public string AddressOK;
public string DataOK;
}
Output:
True
040004 0720
040037 31
If you'll do map Result to Sent very often, you can further shorten your Linq query code to..
var diff = sent.Except(res.Select(r => (Sent)r));
..by automating the mapping of Result to Sent, use implicit operator:
[Equals]
public class Sent
{
public string Address;
public string Data;
[CustomEqualsInternal]
bool CustomLogic(Sent other)
{
return other.Address == this.Address && other.Data == this.Data;
}
public static implicit operator Sent(Result r)
{
return new Sent { Address = r.AddressOK, Data = r.DataOK };
}
}
#Kurisuchin
Suppose you have 2 list and in both you have ID property based on which you want to compare both list and want to store non matching item in third list.
In this Situation following Linq Query can help out.
var result = List2.Where(p => !List1.Any(p2 => p2.ID == p.ID)).ToList();
I have a single observable sequence of messages. There is a set of subscribers that can handle these messages. Each subscriber has an execution priority. Each message must be handled once by the highest priority subscriber chosen from the list of currently subscribed subscribers. Subscribers are constantly subscribed/unsubscribed from the sequence, so we don't know the number and priorities of subscribers when constructing the sequence. Is it a possible/viable solution using rx?
To illustrate:
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Trace.WriteLine(priority);
}
};
var s2 = sequence.Priority(2).Subscribe(m => subscriber(m, 2));
var s1 = sequence.Priority(1).Subscribe(m => subscriber(m, 1));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 2
The missing piece to make this solution work is the Priority method which do not exist in Rx library.
This was a very interesting problem...
So, first off: I am not aware of any intrinsic Rx operators that can achieve a "routing" effect similar to what you want in this Priority extension.
That said, I was playing around in LINQPad over lunch today, and came up with a (very) hacky proof of concept that appears to work:
First, your message class
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
Next, the extension method wrapper-class:
public static class Ext
{
public static PrioritizedObservable<T> Prioritize<T>(this IObservable<T> source)
{
return new PrioritizedObservable<T>(source);
}
}
And what is this PrioritizedObservable<T>?
public class PrioritizedObservable<T>
: IObservable<T>, IObserver<T>, IDisposable
{
private IObservable<T> _source;
private ISubject<T,T> _intermediary;
private IList<Tuple<int, Subject<T>>> _router;
public PrioritizedObservable(IObservable<T> source)
{
// Make sure we don't accidentally duplicate subscriptions
// to the underlying source
_source = source.Publish().RefCount();
// A proxy from the source to our internal router
_intermediary = Subject.Create(this, _source);
_source.Subscribe(_intermediary);
// Holds per-priority subjects
_router = new List<Tuple<int, Subject<T>>>();
}
public void Dispose()
{
_intermediary = null;
foreach(var entry in _router)
{
entry.Item2.Dispose();
}
_router.Clear();
}
private ISubject<T,T> GetFirstListener()
{
// Fetch the first subject in our router
// ordered by priority
return _router.OrderBy(tup => tup.Item1)
.Select(tup => tup.Item2)
.FirstOrDefault();
}
void IObserver<T>.OnNext(T value)
{
// pass along value to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnNext(value);
}
void IObserver<T>.OnError(Exception error)
{
// pass along error to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnError(error);
}
void IObserver<T>.OnCompleted()
{
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnCompleted();
}
public IDisposable Subscribe(IObserver<T> obs)
{
return PrioritySubscribe(1, obs);
}
public IDisposable PrioritySubscribe(int priority, IObserver<T> obs)
{
var sub = new Subject<T>();
var subscriber = sub.Subscribe(obs);
var entry = Tuple.Create(priority, sub);
_router.Add(entry);
_intermediary.Subscribe(sub);
return Disposable.Create(() =>
{
subscriber.Dispose();
_router.Remove(entry);
});
}
}
And a test harness:
void Main()
{
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount().Prioritize();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Console.WriteLine(priority);
}
};
var s3 = sequence.PrioritySubscribe(3, Observer.Create<Message>(m => subscriber(m, 3)));
var s2 = sequence.PrioritySubscribe(2, Observer.Create<Message>(m => subscriber(m, 2)));
var s1 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
var s11 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 1
s11.Dispose();
subject.OnNext(new Message()); // output: 2
s2.Dispose();
subject.OnNext(new Message()); // output: 3
sequence.Dispose();
}
Lets say I have a unit test similar to the below, is there a way to write one unit test rather than several but also avoid having a for loop in the unit test?
[Test]
public void RunTestWithMultipleOptions()
{
MyClass code = new MyClass();
code.Prefix = "{DS1}"; //Options are {DS1}, {DS2}, {DS3}, {DS4}
//Property could be set to
//code.Prefix = "{DS1}{DS2}";
//code.Prefix = "{DS1}{DS2}{DS3}";
//And so on
//Based on how many {DS} used a method needs calling
code.InputDataStore(1,"Data1");
//If used {DS1}{DS2} in Prefix then
//code.InputDataStore(1,"Data1");
//code.InputDataStore(2,"Data2");
//If used {DS1}{DS2}{DS3} in Prefix then
//code.InputDataStore(1,"Data1");
//code.InputDataStore(2,"Data2");
//code.InputDataStore(3,"Data3");
string OutputData = String.Empty;
code.Output += delegate(int Id, string Data)
{
if (Id == (int)OutputsEnum.OutputModified)
OutputData = Data;
};
//Call the input method which will raise the Output event which we can assert against
code.Input("hi there");
//Assert that output has replace the prefix {DS} with the data in the datastorecontent list
Assert.AreEqual("Data1hi there", OutputData);
}
I can pass in the property value to the unit test method and use test cases but based on what the property is MyMethod needs to be called x number of times. Without putting a loop in the test I can't think of a way without having all the permetations as separate unit tests.
UPDATE: Here is the main contents of the class:
public event Action<int, string> Output;
public string Prefix { get; set; }
public string Postfix { get; set; }
private List<string> DataStoreContents = new List<string>() { "", "", "", "" };
public void Input(string Data)
{
if (Output != null)
{
if (!String.IsNullOrEmpty(Prefix))
{
Prefix = Prefix.Replace("{DS1}", DataStoreContents[0]);
Prefix = Prefix.Replace("{DS2}", DataStoreContents[1]);
Prefix = Prefix.Replace("{DS3}", DataStoreContents[2]);
Prefix = Prefix.Replace("{DS4}", DataStoreContents[3]);
}
if (!String.IsNullOrEmpty(Postfix))
{
Postfix = Postfix.Replace("{DS1}", DataStoreContents[0]);
Postfix = Postfix.Replace("{DS2}", DataStoreContents[1]);
Postfix = Postfix.Replace("{DS3}", DataStoreContents[2]);
Postfix = Postfix.Replace("{DS4}", DataStoreContents[3]);
}
Output((int)OutputsEnum.OutputBeforeModified, Data);
Output((int)OutputsEnum.OutputModified, Prefix + Data + Postfix);
Output((int)OutputsEnum.OutputAfterModified, Data);
}
}
}
public void InputDataStore(int DataStore, string Data)
{
if (DataStore < 1 || DataStore > 4)
throw new ArgumentOutOfRangeException("Datastore number out of range");
DataStoreContents[DataStore - 1] = Data;
}
}
I want to test that when I call InputDataStore(1,"MyData1"); InputDataStore(2, "MyData"); that Output does in fact replace the relevant {DS1} values with the relevant string and also combine it with any other {DS} values
One options is to test each call apart and all of them together. If you test A&B&C, you can limit yourself to testing A,B,C apart and A&B&C together. One option is the next code(made some assumptions):
[TestFixture]
public class ToTestFixture
{
[SetUp]
public void SetUp()
{
_instance = new ToTest();
_instance.InputDataStore(1, "1");
_instance.InputDataStore(2, "2");
_instance.InputDataStore(3, "3");
_instance.InputDataStore(4, "4");
}
private ToTest _instance;
[TestCase("{DS1}","1")]
[TestCase("{DS2}", "2")]
[TestCase("{DS3}", "3")]
[TestCase("{DS4}", "4")]
[TestCase("{DS1}{DS2}{DS3}{DS4}", "1234")]
[Test]
public void TestPrefixReplacements(string input, string expectedResult)
{
_instance.Prefix = input;
//Call the input method which will raise the Output event which we can test
_instance.Input("Any string goes here as we test only prefix." );
Assert.AreEqual(expectedResult, _instance.Prefix);
}
}
internal enum OutputsEnum
{
OutputBeforeModified,
OutputModified,
OutputAfterModified
}
public class ToTest
{
public event Action<int, string> Output = (x, result) => Console.WriteLine(x.ToString() + result);
public string Prefix { get; set; }
public string Postfix { get; set; }
private List<string> DataStoreContents = new List<string>() {"1", "2", "3", "4"};
public void Input(string Data)
{
if (Output != null)
{
if (!String.IsNullOrEmpty(Prefix))
{
Prefix = Prefix.Replace("{DS1}", DataStoreContents[0]);
Prefix = Prefix.Replace("{DS2}", DataStoreContents[1]);
Prefix = Prefix.Replace("{DS3}", DataStoreContents[2]);
Prefix = Prefix.Replace("{DS4}", DataStoreContents[3]);
}
if (!String.IsNullOrEmpty(Postfix))
{
Postfix = Postfix.Replace("{DS1}", DataStoreContents[0]);
Postfix = Postfix.Replace("{DS2}", DataStoreContents[1]);
Postfix = Postfix.Replace("{DS3}", DataStoreContents[2]);
Postfix = Postfix.Replace("{DS4}", DataStoreContents[3]);
}
Output((int) OutputsEnum.OutputBeforeModified, Data);
Output((int) OutputsEnum.OutputModified, Prefix + Data + Postfix);
Output((int) OutputsEnum.OutputAfterModified, Data);
}
}
public void InputDataStore(int DataStore, string Data)
{
if (DataStore < 1 || DataStore > 4)
throw new ArgumentOutOfRangeException("Datastore number out of range");
DataStoreContents[DataStore - 1] = Data;
}
}
Anyhow I feel there is a bond between "DS1" and the index of the array. (1-0, 2-1). This means next refactoring is possible:
Prefix = Prefix.Replace("{DS"+index+"}", DataStoreContents[index-1]);
More than that I guess output action decision is strange and two if-s are duplicate code. This is what I mean:
public void Input(string Data)
{
if (Output != null)
{
Prefix = ApplyReplaceRules(Prefix);
Postfix = ApplyReplaceRules(Postfix);
Output((int) OutputsEnum.OutputBeforeModified, Data);
Output((int) OutputsEnum.OutputModified, Prefix + Data + Postfix);
Output((int) OutputsEnum.OutputAfterModified, Data);
}
}
private string ApplyReplaceRules(string patternString)
{
if (!String.IsNullOrEmpty(Postfix))
{
patternString = patternString.Replace("{DS1}", DataStoreContents[0]);
patternString = patternString.Replace("{DS2}", DataStoreContents[1]);
patternString = patternString.Replace("{DS3}", DataStoreContents[2]);
patternString = patternString.Replace("{DS4}", DataStoreContents[3]);
}
return patternString;
}