We Keep Coding

Calling back a FormDialog a defined number of times

I am trying to create a bot that will ask questions to the user. At launch, the bot asks which choice the user would take :
Start a quiz (he will have to choose the theme and the difficulty of the quiz after) ;
See the score he got in the last quiz ;
Reset the score.
The content of the quiz is in a XML file. I already stored the questions and the answers associated in a struct.
The FormBuilder is :
[Serializable]
public class QuizQuestionsLoader
{
public QuizQuestion Question { get; set; }
public static IForm<QuizQuestionsLoader> QuestionLoaderForm(QuizQuestion question)
{
return new FormBuilder<QuizQuestionsLoader>()
.Field(new FieldReflector<QuizQuestionsLoader>(nameof(Question))
.SetType(null)
.SetDefine(async (state, field) =>
{
field
.AddDescription(state.Question.QuestionText, state.Question.QuestionText)
.AddTerms(state.Question.QuestionText, state.Question.QuestionText);
return true;
}))
.AddRemainingFields()
.Build();
}
}
And so, I did an IDialog with a switch that determine which choice the user took. The DefaultCase will be activated if the user chose to start a quiz :
new DefaultCase<QuizChoices?, IDialog<string>>((context, value) =>
{
return Chain.From(() => FormDialog.FromForm(QuizStart.QuizForm, FormOptions.PromptInStart))
.Select(c => c.category)
.ContinueWith(async (ctx, res) =>
{
CategoryOptions? category = await res;
IList<QuizQuestion> questions = QuestionsLoader.LoadQuestions(category.Value.ToString().ToLowerInvariant()).ToList();
QuizQuestion currentQuestion = questions[0];
var questionsDialogs = Chain.From(() => FormDialog.FromForm(() => { return QuizQuestionsLoader.QuestionLoaderForm(currentQuestion); })).PostToUser();
for (int i = 1; i < questions.Count(); i++)
{
currentQuestion = questions[i];
questionsDialogs.ContinueWith(async (forctx, fores) =>
{
await fores;
return Chain.From(() => FormDialog.FromForm(() => { return QuizQuestionsLoader.QuestionLoaderForm(currentQuestion); }));
}).PostToUser();
}
return Chain.Return(questionsDialogs).Unwrap();
})
.ContinueWith(async (ctx, res) =>
{
await res;
return Chain.Return("Quiz fini !");
});
})
I want to display 10 questions to the user, that's why I thought that recall the FormBuilder was a good idea, because I don't see how I can do that in another way. When I build and run this, after the choice of the difficulty, the Bot Framework emulator sends a 500 internal server error.
That's why I tried to recall a "test" FormBuilder, which is composed whith a simple message and three choices, to see if I can recall the FormDialog with a for loop.
Here is the FormBuilder :
public enum TestOptions
{
A, B, C
}
[Serializable]
public class Test
{
public TestOptions? choice;
public static IForm<Test> TestForm()
{
return new FormBuilder<Test>()
.Message("Test")
.Field(nameof(choice))
.Build();
}
}
And here is the IDialog :
return Chain.From(() => FormDialog.FromForm(Test.TestForm, FormOptions.PromptInStart))
.ContinueWith(async(ctx, res) =>
{
await res;
var testDialog = Chain.From(() => FormDialog.FromForm(() => { return Test.TestForm(); })).PostToUser();
for (int i = 0; i < 2; i++)
{
testDialog.ContinueWith<Test, Test>(async (forctx, fores) =>
{
await fores;
return Chain.From(() => FormDialog.FromForm(Test.TestForm, FormOptions.PromptInStart));
});
}
return Chain.Return(testDialog);
})
With this, the FormDialog will display once, but I saw that the for loop is executed. However, the testDialog variable is null.
So, do you know how I can proceed to recall correctly my FormBuilder to have 10 questions on the Bot Framework Emulator ?
Thanks a lot !

I added a sample that illustrates how to iterate over questions in a quiz with this commit. It uses a chainable dialog named FoldDialog to invoke a series of dialogs in sequence and aggregate the responses:
var quiz = Chain
.PostToChain()
.Select(_ => "how many questions?")
.PostToUser()
.WaitToBot()
.Select(m => int.Parse(m.Text))
.Select(count => Enumerable.Range(0, count).Select(index => Chain.Return($"question {index + 1}?").PostToUser().WaitToBot().Select(m => m.Text)))
.Fold((l, r) => l + "," + r)
.Select(answers => "your answers were: " + answers)
.PostToUser();
It allows the creation of a script like this:
"hello",
"how many questions?",
"3",
"question 1?",
"A",
"question 2?",
"B",
"question 3?",
"C",
"your answers were: A,B,C"

First, thank you for you answer Will Pornoy !
However, I already succeed to resolve my issue before your answer, like this :
new DefaultCase<QuizChoices?, IDialog<string>>((context, value) =>
{
return Chain.From(() => FormDialog.FromForm(QuizStart.QuizForm, FormOptions.PromptInStart))
.Select(c => new QuizParameters
{
CategoryParameter = c.category.Value.ToString(),
DifficultyParameter = c.difficulty.Value.ToString()
})
.ContinueWith<QuizParameters?, int>(async (ctx, res) =>
{
await res;
IList<QuizQuestion> questions = QuestionsLoader.LoadQuestions(QuizParameters.CategoryParameter, QuizParameters.DifficultyParameter).ToList();
return new QuizQuestionsLoader(questions);
})
Where QuizParameter is a struct which contains the category and the difficulty choices of the user.
I get simply an IList which contains the questions text and its answers.
Finally, I pass it to a new object QuizQuestionLoader. In this class, I make some methods :
[Serializable]
public class QuizQuestionsLoader : IDialog<int>
{
public static int Score { get; private set; }
private IList<QuizQuestion> problems;
private QuizQuestion theQuestion;
private int index;
private int jokerCount = 2;
private const string jokerAnswerText = "Utiliser un joker";
public QuizQuestionsLoader(IList<QuizQuestion> problems)
{
this.problems = problems;
}
A Task method which is called each time the quiz is started :
public async Task StartAsync(IDialogContext context)
{
problems.Shuffle();
DisplayQuestion(context);
}
An DisplayQuestion method overload (the first is for the case where no joker left) :
private void DisplayQuestion(IDialogContext context)
{
DisplayQuestion(context, false);
}
private void DisplayQuestion(IDialogContext context, bool useJoker)
{
theQuestion = problems[index];
string questionText = theQuestion.QuestionText;
IList<Answer> answers = theQuestion.Answers.ToList();
if (useJoker)
{
IList<Answer> randomBadAnswers = answers.Where(a => !a.IsCorrect).ToList();
randomBadAnswers.Shuffle();
randomBadAnswers = randomBadAnswers.Take(2).ToList();
answers = answers.Except(randomBadAnswers).ToList();
}
else if (jokerCount > 0)
{
Answer jokerAnswer = new Answer
{
AnswerText = $"{jokerAnswerText} ({jokerCount}) restant(s)"
};
answers.Add(jokerAnswer);
}
PromptDialog.Choice(context, CheckResponseAsync, answers, questionText, null, 0, PromptStyle.Auto);
}
AND, finally, the loop that will reload this process until the 10 questions are displayed :
public async Task CheckResponseAsync(IDialogContext context, IAwaitable<Answer> argument)
{
Answer answer = await argument;
if (answer.AnswerText.StartsWith(jokerAnswerText))
{
jokerCount--;
await context.PostAsync("Suppression de deux mauvaises réponses...");
DisplayQuestion(context, true);
}
else
{
await context.PostAsync(answer.IsCorrect ? "Bonne réponse !" : "Mauvaise réponse !");
index++;
Answer goodAnswer = theQuestion.Answers.First(a => a.IsCorrect);
if (answer.AnswerText == goodAnswer.AnswerText)
{
Score++;
}
if (index < problems.Count)
{
DisplayQuestion(context);
}
else
{
await context.PostAsync($"Votre score est de {Score}");
context.Done(Score);
}
}
}
Hope it will help ! :)

Static Query Building with NEST

I'm playing around with Elasticsearch and NEST.
I do have some trouble understanding the various classes and interfaces which can be used to create and build static queries.
Here's a simplified example of what I want to achieve:
using Nest;
using System;
using System.Text;
namespace NestTest
{
public class Product
{
public string Name { get; set; }
public int Price { get; set; }
}
public class ProductFilter
{
public string[] IncludeNames { get; set; }
public string[] ExcludeNames { get; set; }
public int MaxPrice { get; set; }
}
class Program
{
static void Main(string[] args)
{
var filter = new ProductFilter();
filter.MaxPrice = 100;
filter.IncludeNames = new[] { "Notebook", "Workstation" };
filter.ExcludeNames = new[] { "Router", "Modem" };
var query = CreateQueryFromFilter(filter);
var client = new ElasticClient();
// Test Serialization
var serialized = Encoding.UTF8.GetString(client.Serializer.Serialize(query));
Console.WriteLine(serialized);
// TODO: How to convert the IQuery to QueryContainer?
//client.Search<Product>(s => s.Query(q => query));
}
private static IQuery CreateQueryFromFilter(ProductFilter filter)
{
var baseBoolean = new BoolQueryDescriptor<Product>();
if (filter.IncludeNames != null && filter.IncludeNames.Length > 0)
{
foreach (var include in filter.IncludeNames)
{
// TODO: This overwrites the previous must
baseBoolean.Must(q => q.Term(t => t.Name, include));
}
}
if (filter.ExcludeNames != null && filter.ExcludeNames.Length > 0)
{
foreach (var exclude in filter.ExcludeNames)
{
// TODO: This overwrites the previous must
baseBoolean.MustNot(q => q.Term(t => t.Name, exclude));
}
}
if (filter.MaxPrice > 0)
{
// TODO: This overwrites the previous must
baseBoolean.Must(q => q.Range(r => r.LowerOrEquals(filter.MaxPrice).OnField(f => f.Price)));
}
return baseBoolean;
}
}
}
As you can see, I'd like to create some kind of query object (most likely BoolQuery) and then fill this object later on. I've added some TODOS in code where I have the actual problems. But in general, there are just too many possibilities (IQuery, QueryContainer, XXXQueryDescriptor, SearchDescriptor, SearchRequest) and I cannot figure out how to successfully "build" a query part by part.
Anybody who could enlighten me?

Combinding boolean queries is described in the documentation here:
http://nest.azurewebsites.net/nest/writing-queries.html
That page is slightly outdated and will be updated soon although most of it still applies I updated your CreateQueryFromFilter method to showcase the several ways you can formulate queries:
private static IQueryContainer CreateQueryFromFilter(ProductFilter filter)
{
QueryContainer queryContainer = null;
if (filter.IncludeNames != null && filter.IncludeNames.Length > 0)
{
foreach (var include in filter.IncludeNames)
{
//using object initializer syntax
queryContainer &= new TermQuery()
{
Field = Property.Path<Product>(p => p.Name),
Value = include
};
}
}
if (filter.ExcludeNames != null && filter.ExcludeNames.Length > 0)
{
foreach (var exclude in filter.ExcludeNames)
{
//using static Query<T> to dispatch fluent syntax
//note the ! support here to introduce a must_not clause
queryContainer &= !Query<Product>.Term(p => p.Name, exclude);
}
}
if (filter.MaxPrice > 0)
{
//fluent syntax through manually newing a descriptor
queryContainer &= new QueryDescriptor<Product>()
.Range(r => r.LowerOrEquals(filter.MaxPrice).OnField(f => f.Price)
);
}
return queryContainer;
}
Here's how you can pass that to a search operation:
static void Main(string[] args)
{
//using the object initializer syntax
client.Search<Product>(new SearchRequest()
{
Query = query
});
//using fluent syntax
client.Search<Product>(s => s.Query(query));
}

Await on item being present in a list

I'm currently looking for a way to monitor a list for an item to be added (although it may already be in the list) with a certain ID. The below example demonstrates what I want to do, however I was hoping there would be a neater approach to this.
var list = new List<string>();
var task = new Task<bool>(() =>
{
for (int i = 0; i < 10; i++)
{
if (list.Contains("Woft"))
return true;
Thread.Sleep(1000);
}
return false;
});
I would appreciate any suggestions, and can elaborate if required.
EDIT: I'm not sure how I would use a CollectionChanged handler for this, still not sure I'm describing the problem well enough, here is a more complete example.
class Program
{
private static List<string> receivedList = new List<string>();
static void Main(string[] args)
{
var thing = new item() { val = 1234, text = "Test" };
var message = new message() { id = "12", item = thing };
//This would be a message send
//in the receiver we would add the message id to the received list.
var task2 = new Task(() =>
{
Thread.Sleep(2000);
receivedList.Add(message.id);
});
task2.Start();
var result = EnsureReceived(thing, message.id);
if (result == null)
{
Console.WriteLine("Message not received!");
}
else
{
Console.WriteLine(result.text + " " + result.val);
}
Console.ReadLine();
}
//This checks if the message has been received
private static item EnsureReceived(item thing, string id)
{
var task = new Task<bool>(() =>
{
for (int i = 0; i < 10; i++)
{
if (receivedList.Contains(id))
return true;
Thread.Sleep(1000);
}
return false;
});
task.Start();
var result = task.Result;
return result ? thing : null;
}
}
class message
{
public string id { get; set; }
public item item { get; set; }
}
class item
{
public string text { get; set; }
public int val { get; set; }
}

It sounds like what you want to do is use an ObservableCollection and subscribe to the CollectionChanged event. That way, you'll have an event fire whenever the collection is modified. At that point, you can check if the collection has the element you're expecting.
Even better, the CollectionChanged event handler will receive NotifyCollectionChangedEventArgs, which contains a NewItems property.
See MSDN for more information.

You can implement own type, what intercept all operations to add items and exposes ReadOnlyList. Then create an event ItemAdded with added item value (you will have to rise up multiple events when adding a collection obviously). Subscribe to it, check, do whatever.

Prioritized subscriptions invocation

I have a single observable sequence of messages. There is a set of subscribers that can handle these messages. Each subscriber has an execution priority. Each message must be handled once by the highest priority subscriber chosen from the list of currently subscribed subscribers. Subscribers are constantly subscribed/unsubscribed from the sequence, so we don't know the number and priorities of subscribers when constructing the sequence. Is it a possible/viable solution using rx?
To illustrate:
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Trace.WriteLine(priority);
}
};
var s2 = sequence.Priority(2).Subscribe(m => subscriber(m, 2));
var s1 = sequence.Priority(1).Subscribe(m => subscriber(m, 1));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 2
The missing piece to make this solution work is the Priority method which do not exist in Rx library.

This was a very interesting problem...
So, first off: I am not aware of any intrinsic Rx operators that can achieve a "routing" effect similar to what you want in this Priority extension.
That said, I was playing around in LINQPad over lunch today, and came up with a (very) hacky proof of concept that appears to work:
First, your message class
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
Next, the extension method wrapper-class:
public static class Ext
{
public static PrioritizedObservable<T> Prioritize<T>(this IObservable<T> source)
{
return new PrioritizedObservable<T>(source);
}
}
And what is this PrioritizedObservable<T>?
public class PrioritizedObservable<T>
: IObservable<T>, IObserver<T>, IDisposable
{
private IObservable<T> _source;
private ISubject<T,T> _intermediary;
private IList<Tuple<int, Subject<T>>> _router;
public PrioritizedObservable(IObservable<T> source)
{
// Make sure we don't accidentally duplicate subscriptions
// to the underlying source
_source = source.Publish().RefCount();
// A proxy from the source to our internal router
_intermediary = Subject.Create(this, _source);
_source.Subscribe(_intermediary);
// Holds per-priority subjects
_router = new List<Tuple<int, Subject<T>>>();
}
public void Dispose()
{
_intermediary = null;
foreach(var entry in _router)
{
entry.Item2.Dispose();
}
_router.Clear();
}
private ISubject<T,T> GetFirstListener()
{
// Fetch the first subject in our router
// ordered by priority
return _router.OrderBy(tup => tup.Item1)
.Select(tup => tup.Item2)
.FirstOrDefault();
}
void IObserver<T>.OnNext(T value)
{
// pass along value to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnNext(value);
}
void IObserver<T>.OnError(Exception error)
{
// pass along error to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnError(error);
}
void IObserver<T>.OnCompleted()
{
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnCompleted();
}
public IDisposable Subscribe(IObserver<T> obs)
{
return PrioritySubscribe(1, obs);
}
public IDisposable PrioritySubscribe(int priority, IObserver<T> obs)
{
var sub = new Subject<T>();
var subscriber = sub.Subscribe(obs);
var entry = Tuple.Create(priority, sub);
_router.Add(entry);
_intermediary.Subscribe(sub);
return Disposable.Create(() =>
{
subscriber.Dispose();
_router.Remove(entry);
});
}
}
And a test harness:
void Main()
{
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount().Prioritize();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Console.WriteLine(priority);
}
};
var s3 = sequence.PrioritySubscribe(3, Observer.Create<Message>(m => subscriber(m, 3)));
var s2 = sequence.PrioritySubscribe(2, Observer.Create<Message>(m => subscriber(m, 2)));
var s1 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
var s11 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 1
s11.Dispose();
subject.OnNext(new Message()); // output: 2
s2.Dispose();
subject.OnNext(new Message()); // output: 3
sequence.Dispose();
}

Rx operator to distinct sequences

IMPORTANT: for a description of the results and some more details, please have a look also to my answer
I need to group and filter a sequence of objects/events that usually are replicated, buffering them with a TimeSpan interval. I try to explain it better with sort of marble diagrams:
X-X-X-X-X-Y-Y-Y-Z-Z-Z-Z-X-X-Y-Z-Z
would produce
X---Y---Z---X---Y---Z
where X, Y and Z are different event types, and '---' means the interval.
Additionally, I would also like to distinct by a key property that it is available on all types because they have a common base class:
X, Y, Z : A
and A contains a property Key. Using the notation X.a meaning X.Key = a, A final sample would be:
X.a-X.b-X.a-Y.b-Y.c-Z.a-Z.a-Z.c-Z.b-Z.c
would produce
X.a-X.b---Y.b-Y.c-Z.a-Z.c-Z.b
Can anybody help me putting together the required Linq operators (probably DistinctUntilChanged and Buffer) to achieve this behavior?
Thanks
UPDATE 18.08.12:
as requested, I try to give a better explanation.
We have devices collecting and sending events to a web service. These devices have an old logic (and we can't change it due to backward compatibility) and they continuously send an event until they receive an acknowledge; after the acknowledge, they send the next event in their queue, and so on.
Events contain the network address of the unit and some other properties distinguishing events in the queue for each device.
An event looks like this:
class Event
{
public string NetworkAddress { get; }
public string EventCode { get; }
public string AdditionalAttribute { get; }
}
The goal is that of processing every 5 seconds the distinguished events received from all devices, storing information in the database (that's why we don't want to do it in batches) and sending the ack to the device.
Let's make an example with only two devices and some events:
Device 'a':
Event 1 (a1): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'x'
Event 2 (a2): NetworkAddress = '1', EventCode = A, AdditionalAttribute = 'y'
Event 3 (a3): NetworkAddress = '1', EventCode = B, AdditionalAttribute = 'x'
Device 'b':
Event 1 (b1): NetworkAddress = '2', EventCode = A, AdditionalAttribute = 'y'
Event 2 (b2): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'x'
Event 3 (b3): NetworkAddress = '2', EventCode = B, AdditionalAttribute = 'y'
Event 4 (b4): NetworkAddress = '2', EventCode = C, AdditionalAttribute = 'x'
Pn are the operations done by our server, explained later
Possible marble diagram (input streams + output stream):
Device 'a' : -[a1]-[a1]-[a1]----------------[a2]-[a2]-[a2]-[a3]-[a3]-[a3]-...
Device 'b' : ------[b1]-[b1]-[b2]-[b2]-[b2]------[b3]-[b3]-[b4]-[b4]-[b4]-...
Time : ------------[1s]-----------[2s]------------[3s]------------[4s]-
DB/acks (rx output) : ------------[P1]-----------[P2]------------[P3]------------[P4]-
P1: Server stores and acknowledges [a1] and [b1]
P2: " " " " [b2]
P3: " " " " [a2] and [b3]
P4: " " " " [a3] and [b4]
At the end I think it is probably a simple combination of basic operators, but I'm new to Rx and I'm a bit confused since it seems that there are lots of operators (or combinations of operators) to get the same output stream.
Update 19.08.12:
Please keep in mind that this code runs on a server and it should run for days without memory leaks...I'm not sure about the behavior of subjects. At the moment, for each event I call a push operation on a service, which calls the OnNext of a Subject on top of which I should build the query (if I'm not wrong about the usage of subjects).
Update 20.08.12:
Current implementation, including validation test; this is what I tried and it seems the same suggested by #yamen
public interface IEventService
{
// Persists the events
void Add(IEnumerable<Event> events);
}
public class Event
{
public string Description { get; set; }
}
/// <summary>
/// Implements the logic to handle events.
/// </summary>
public class EventManager : IDisposable
{
private static readonly TimeSpan EventHandlingPeriod = TimeSpan.FromSeconds(5);
private readonly Subject<EventMessage> subject = new Subject<EventMessage>();
private readonly IDisposable subscription;
private readonly object locker = new object();
private readonly IEventService eventService;
/// <summary>
/// Initializes a new instance of the <see cref="EventManager"/> class.
/// </summary>
/// <param name="scheduler">The scheduler.</param>
public EventManager(IEventService eventService, IScheduler scheduler)
{
this.eventService = eventService;
this.subscription = this.CreateQuery(scheduler);
}
/// <summary>
/// Pushes the event.
/// </summary>
/// <param name="eventMessage">The event message.</param>
public void PushEvent(EventMessage eventMessage)
{
Contract.Requires(eventMessage != null);
this.subject.OnNext(eventMessage);
}
/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
/// <filterpriority>2</filterpriority>
public void Dispose()
{
this.Dispose(true);
}
private void Dispose(bool disposing)
{
if (disposing)
{
// Dispose unmanaged resources
}
this.subject.Dispose();
this.subscription.Dispose();
}
private IDisposable CreateQuery(IScheduler scheduler)
{
var buffered = this.subject
.DistinctUntilChanged(new EventComparer())
.Buffer(EventHandlingPeriod, scheduler);
var query = buffered
.Subscribe(this.HandleEvents);
return query;
}
private void HandleEvents(IList<EventMessage> eventMessages)
{
Contract.Requires(eventMessages != null);
var events = eventMessages.Select(this.SelectEvent);
this.eventService.Add(events);
}
private Event SelectEvent(EventMessage message)
{
return new Event { Description = "evaluated description" };
}
private class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
return x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
// Other properties
}
And the test:
public void PushEventTest()
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var eventServiceMock = new Mock<IEventService>();
var scheduler = new TestScheduler();
var target = new EventManager(eventServiceMock.Object, scheduler);
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
scheduler.Schedule(() => target.PushEvent(eventMessageA1));
scheduler.Schedule(TimeSpan.FromSeconds(1), () => target.PushEvent(eventMessageB1));
scheduler.Schedule(TimeSpan.FromSeconds(2), () => target.PushEvent(eventMessageA1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(6).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 2)), Times.Once());
scheduler.Schedule(TimeSpan.FromSeconds(3), () => target.PushEvent(eventMessageB1));
scheduler.AdvanceTo(TimeSpan.FromSeconds(11).Ticks);
eventServiceMock.Verify(s => s.Add(It.Is<List<Event>>(list => list.Count == 1)), Times.Once());
}
Additionally, I remark again that it is really important that the software could run for days without problems, handling thousands of messages.
To make it clear: the test doesn't pass with the current implementation.

I'm not sure if this does exactly what you'd like, but you may be to group the elements explicitly using the group keyword, and then to manipulate the various IObservables separately before recombining them.
E.g. if we have class definitions such as
class A
{
public char Key { get; set; }
}
class X : A { }
...
and a Subject<A>
Subject<A> subject = new Subject<A>();
then we can write
var buffered =
from a in subject
group a by new { Type = a.GetType(), Key = a.Key } into g
from buffer in g.Buffer(TimeSpan.FromMilliseconds(300))
where buffer.Any()
select new
{
Count = buffer.Count,
Type = buffer.First().GetType().Name,
Key = buffer.First().Key
};
buffered.Do(Console.WriteLine).Subscribe();
We can test this with the data you provided:
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'b' });
Thread.Sleep(100);
subject.OnNext(new X { Key = 'a' });
Thread.Sleep(100);
...
subject.OnCompleted();
To get the output you provided:
{ Count = 2, Type = X, Key = a }
{ Count = 1, Type = X, Key = b }
{ Count = 1, Type = Y, Key = b }
{ Count = 1, Type = Y, Key = c }
{ Count = 2, Type = Z, Key = a }
{ Count = 2, Type = Z, Key = c }
{ Count = 1, Type = Z, Key = b }

Not sure if this is exactly what you want, but it seems to support your use cases.
First, let's define the base class to use (you can easily modify this to suit your needs):
public class MyEvent
{
public string NetworkAddress { set; get; }
public string EventCode { set; get; }
}
Let's set up your devices as an array of IObservable<MyEvent> - you may have these available differently, and the below would need to change to accommodate that of course. These devices will each produce a value with a random delay between 0.5 and 1.5 seconds.
var deviceA = new MyEvent[] { new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "1"},
new MyEvent() {NetworkAddress = "A", EventCode = "2"} };
var deviceB = new MyEvent[] { new MyEvent() {NetworkAddress = "B", EventCode = "1"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "2"},
new MyEvent() {NetworkAddress = "B", EventCode = "3"} };
var random = new Random();
var deviceARand = deviceA.ToObservable().Select(a => Observable.Return(a).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var deviceBRand = deviceB.ToObservable().Select(b => Observable.Return(b).Delay(TimeSpan.FromMilliseconds(random.Next(500,1500)))).Concat();
var devices = new IObservable<MyEvent>[] { deviceARand, deviceBRand };
Now let's take all of these individual device streams, make them 'distinct', and merge them into a single master stream:
var stream = devices.Aggregate(Observable.Empty<MyEvent>(), (acc, device) => acc.DistinctUntilChanged(a => a.EventCode).Merge(device));
Once you have that, getting this stream to be consumed periodically is just a matter of buffering it with Buffer:
stream.Buffer(TimeSpan.FromSeconds(1)).Subscribe(x => { /* code here works on a list of the filtered events per second */ });

After searches and experiments, I put together some code that produces the output that I expect:
static void Main(string[] args)
{
const string Address1 = "A:2.1.1";
const string Address2 = "A:2.1.2";
var comparer = new EventComparer();
var eventMessageA1 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 4 };
var eventMessageB1 = new EventMessage { NetworkAddress = Address2, EventCode = 1, Attribute = 5 };
var eventMessageA2 = new EventMessage { NetworkAddress = Address1, EventCode = 1, Attribute = 5 };
var list = new[] { eventMessageA1, eventMessageA1, eventMessageB1, eventMessageA2, eventMessageA1, eventMessageA1 };
var queue = new BlockingCollection<EventMessage>();
Observable.Interval(TimeSpan.FromSeconds(2)).Subscribe
(
l => list.ToList().ForEach(m =>
{
Console.WriteLine("Producing {0} on thread {1}", m, Thread.CurrentThread.ManagedThreadId);
queue.Add(m);
})
);
// subscribing
queue.GetConsumingEnumerable()
.ToObservable()
.Buffer(TimeSpan.FromSeconds(5))
.Subscribe(e =>
{
Console.WriteLine("Queue contains {0} items", queue.Count);
e.Distinct(comparer).ToList().ForEach(m =>
Console.WriteLine("{0} - Consuming: {1} (queue contains {2} items)", DateTime.UtcNow, m, queue.Count));
}
);
Console.WriteLine("Type enter to exit");
Console.ReadLine();
}
public class EventComparer : IEqualityComparer<EventMessage>
{
public bool Equals(EventMessage x, EventMessage y)
{
var result = x.NetworkAddress == y.NetworkAddress && x.EventCode == y.EventCode && x.Attribute == y.Attribute;
return result;
}
public int GetHashCode(EventMessage obj)
{
var s = string.Concat(obj.NetworkAddress + "_" + obj.EventCode + "_" + obj.Attribute);
return s.GetHashCode();
}
}
public class EventMessage
{
public string NetworkAddress { get; set; }
public byte EventCode { get; set; }
public byte Attribute { get; set; }
public override string ToString()
{
const string Format = "{0} ({1}, {2})";
var s = string.Format(Format, this.NetworkAddress, this.EventCode, this.Attribute);
return s;
}
}
Anyway, monitoring the application, it seems that this causes a memory leak.
My question is now:
what is causing the memory leak? [please see the update below]
is this the best way to do it (if I put the distinct on the first observable, I don't get the other events on next buffers, but items in each buffer should be isolated from others)?
how can I write a test using the test scheduler?
UPDATE:
it seems that the memory increment lasts only some minutes, then the value is stable. I will run a long test.
Of course, this would be an absolutely acceptable behavior.
UPDATE 26.08.12:
as I already mentioned in the previous update, the memory usage increases only (and slowly) for some minutes after the startup. After 8 hours the memory consumed was stable, with normal fluctuations in the range of few KB)
this question is very similar to mine and the suggested Drain extension could apply well to my problem (still to be verified)
Anyway, I think that my question is still open for unit tests using the test scheduler.
thanks
Francesco