We Keep Coding

Create async method for queued timer based HTTP call

I have a class, which holds a queue of requests, which will be collected and send to an Web API via HTTP call after a time interval of max 1 second:
public class AsyncValueTimerIntervalWriter
{
private class ValueRequest
{
public string FullName { get; set; }
public object Value { get; set; }
}
private readonly IValuesClient _valuesClient; // auto generated Swagger HTTP client
private List<ValueRequest> _valueRequests = new List<ValueRequest>();
private object _valuesLock = new object();
private Timer _timer;
public AsyncValueTimerIntervalWriter(IValuesClient valuesClient)
{
_valuesClient = valuesClient;
}
public void Start()
{
_timer = new Timer(o => WriteValuesToServer(), null, 0, 1000);
}
public void Stop()
{
_timer?.Dispose();
_timer = null;
}
public void AddWrite(string fullName, object value)
{
lock (_valuesLock)
{
_valueRequests.Add(new ValueRequest { FullName = fullName, Value = value });
}
}
private async void WriteValuesToServer()
{
IList<ValueRequest> values;
lock (_valuesLock)
{
values = _valueRequests.ToArray();
_valueRequests.Clear();
}
if (values.Any())
{
await _valuesClient.SetValuesAsync(values); // Sends HTTP POST request
}
}
}
Caller example:
var asyncWriter = new AsyncValueTimerIntervalWriter(...);
asyncWriter.AddWrite("My.Var.Tree.VarName", 1234);
asyncWriter.AddWrite("My.Var.Tree.AnotherVar", "Test");
// after max 1 sec the values are written to server
My goal is to write an async method, which also adds a value to write, and returns when the value is written:
await asyncWriter.WriteAsync("My.Var.Tree.VarName", 1234);
// should continue after written to server
Important: I need to handle requests in a queue, because the writer may be stopped at any time and it is not allowed to loose requests. After start the writer again, the added requests needs to be send to server.
I tried to use the ManualResetEvent, but it feels strange:
...
public Task WriteAsync(string fullName, object value)
{
var resetEvent = new ManualResetEvent(false);
lock (_valuesLock)
{
_valueRequests.Add(
new ValueRequest
{
FullName = fullName,
Value = value,
CompletedEvent = resetEvent
});
}
resetEvent.WaitOne();
return Task.CompletedTask;
}
private async void WriteValuesToServer()
{
IList<ValueRequest> values;
lock (_valuesLock)
{
values = _valueRequests.ToArray();
_valueRequests.Clear();
}
if (values.Any())
{
await _valuesClient.SetValuesAsync(values); // Sends HTTP POST request
foreach (var value as values)
value.CompletedEvent?.Set();
}
}
...
Any suggestions?

You can use a TaskCompletionSource within the ValueEntry class to pass a signal from the writer to the caller.
private class ValueEntry
{
public string FullName { get; set; }
public object Value { get; set; }
protected readonly TaskCompletionSource _tcs = new TaskCompleteionSource();
public Task AwaitCompletion()
{
return _tcs.Task;
}
public Task MarkComplete()
{
return _tcs.SetResult();
}
}
Small change to WriteValuesToServer:
public async Task WriteValuesToServer()
{
// snip
if (values.Any())
{
await _emsClient.SetValuesAsync(values); // Sends HTTP POST request
foreach (var value as values)
await value.MarkComplete();
}
}
Now your writer is very simple:
public Task WriteAsync(string fullName, object value)
{
var request = new ValueRequest { FullName = fullName, Value = value };
lock (_valuesLock)
{
_valueRequests.Add(request)
};
await request.AwaitCompletion();
}
Also, I suggest you look into using a BlockingCollection, which is designed to handle a producer/consumer queue, and would allow you to get rid of most of your lock blocks.

TPL Dataflow two-phase commit

I wanted to implement something like two-phase commit protocol for consuming messages.
In order to do it, I implemented ITargetBlock myself:
public class Worker : ITargetBlock<Message>
{
// Is connected to remote server
// Maintaining connection removed for brevity in this example
private bool _isConnectionAlive;
private readonly ActionBlock<MessageWithSource> _action;
public Worker()
{
_action = new ActionBlock<MessageWithSource>(DoWork);
}
public DataflowMessageStatus OfferMessage(
DataflowMessageHeader messageHeader, Message messageValue,
ISourceBlock<Message> source, bool consumeToAccept)
{
if (consumeToAccept || source == null)
{
return DataflowMessageStatus.Declined;
}
if (!_isConnectionAlive)
{
return DataflowMessageStatus.Postponed;
}
var reservedMessage = source.ReserveMessage(messageHeader, this);
if (reservedMessage)
{
_action.Post(new MessageWithSource(messageValue, source, messageHeader));
}
return DataflowMessageStatus.Postponed;
}
// Other methods removed for brevity
private async Task DoWork(MessageWithSource value)
{
try
{
// sending message to the server removed for brevity
// commit that we finished processing without error
var message = value.SourceBlock.ConsumeMessage(value.MessageHeader, this, out _);
if (message != value.Message)
{
// In which cases can we get here?
throw new InvalidOperationException("Consumed some other message... oh my");
}
}
catch (WebSocketException)
{
// Release reservation if we can't finish work, so other Workers can pickup this message and process it
value.SourceBlock.ReleaseReservation(value.MessageHeader, this);
}
}
private class MessageWithSource
{
public Message Message { get; }
public ISourceBlock<Message> SourceBlock { get; }
public DataflowMessageHeader MessageHeader { get; }
}
}
In the docs it says ConsumeMessage can return a different instance than it was previously offered.
I wonder in which cases and way it happens?

c# AsyncSocket Server need locking?

i have written a TCP Socket-Server. In general i want to have the following behaviour:
A Listening Socket can accept N Connections (There a multiple Listeners on different Ports (1337,733) )
A Connection can authenticated itself as a "Client", multiple connections can grouped to one "Client"
Multiple connection can accepted / revice data at the same time (concurrency)
Here my Server-Class:
class Server
{
internal List ConnectionListeners;
internal bool IsListening = false;
internal Server()
{
this.ClientManager = new ClientManager();
this.ConnectionManager = new ConnectionManager();
this.ConnectionListeners = new List();
}
internal void AddConnectionListener(int Port)
{
ConnectionListener c = new ConnectionListener(Port);
c.AcceptedConnection += new ConnectionListener.AcceptedConnectionEventHandler(ConnectionProcessor.AcceptConnection);
ConnectionListeners.Add(c);
}
internal void RemoveConnectionListener(ConnectionListener ConnectionListener)
{
ConnectionListeners.Remove(ConnectionListener);
}
public delegate void OnStartListeningEventHandler();
public event OnStartListeningEventHandler OnStartListening;
internal void StartListening()
{
IsListening = true;
foreach (ConnectionListener cl in this.ConnectionListeners)
{
cl.StartListening();
}
OnStartListening?.Invoke();
}
public delegate void OnStopListeningEventHandler();
public event OnStopListeningEventHandler OnStopListening;
internal void StopListening()
{
ConnectionManager.DisconnectConnections();
foreach (ConnectionListener cl in this.ConnectionListeners)
{
cl.StopListening();
}
IsListening = false;
OnStopListening?.Invoke();
}
}
Method of ConnectionProcessor where i hande a new Accept Connection (ConnectionProcessor.AcceptConnection):
internal void AcceptConnection(Socket Socket)
{
Connection Connection = new Connection(Socket);
Connection.Sent += new Connection.SentEventHandler(onSend);
Connection.Received += new Connection.ReceivedEventHandler(onRecive);
Connection.Disconnected += new Connection.DisconnectedEventHandler(OnDisconnect);
Connection.Recive();
Logger.Instance.AddLog(new LogMessage(LogMessage.LogLevel.Normal, "Connection ("+Connection.ConnectionId+") Accepted"));
ConnectionManager.AddConnection(Connection);
}
ConnectionManager:
class ConnectionManager
{
internal ConnectionManager()
{
this.Connections = new List();
}
internal void AddConnection(Connection Connection)
{
Connections.Add(Connection);
OnAddConnection(Connection);
}
internal void RemoveConnection(Connection Connection)
{
Connections.Remove(Connection);
OnRemoveConnection(Connection);
}
internal void DisconnectConnections()
{
foreach (Connection c in Connections)
{
c.Disconnect();
}
}
}
Everything seems to work, but I am unsure about concurrency.
As you can see in the ConnectionManger, i store each Connection in a List (Connections.Add(Connection)). Its enough do to this? I have to care about that a normal "List" is not Thread safe?
Is my "design" in gerneal the right way to solve my requirements?

Since all you do is adding\removing\enumerating connections in your list - you can use thread-safe collection, without any locks. Unfortunately, there is no ConcurrentList or ConcurrentHashSet, but you can use ConcurrentDictionary with dummy keys, either directly or by wrapping in separate class:
class BasicConcurrentSet<T> : IEnumerable<T> {
private readonly ConcurrentDictionary<T, byte> _items = new ConcurrentDictionary<T, byte>();
public void Add(T item) {
_items.TryAdd(item, 0);
}
public void Remove(T item) {
byte tmp;
_items.TryRemove(item, out tmp);
}
IEnumerator IEnumerable.GetEnumerator() {
return GetEnumerator();
}
public IEnumerator<T> GetEnumerator() {
foreach (var kv in _items) {
yield return kv.Key;
}
}
}
Adding, removing and enumerating items from such concurrent collection is thread safe.
class ConnectionManager {
private readonly BasicConcurrentSet<Connection> _connections = new BasicConcurrentSet<Connection>();
internal ConnectionManager() {
}
internal void AddConnection(Connection connection) {
_connections.Add(connection);
OnAddConnection(Connection);
}
internal void RemoveConnection(Connection connection) {
_connections.Remove(connection);
OnRemoveConnection(connection);
}
internal void DisconnectConnections() {
foreach (var connection in _connections) {
connection.Disconnect();
}
}
}

Even if you incapsulate Connections (should be private) inside ConnectionManager class and situation when RemoveConnection or DisconnectConnections can be earlier then AddConnection is impossible - no, this is not thread safe. So my suggestion is make all 3 functions thread safe like this :
private Object lck;
internal ConnectionManager()
{
lck = new Object();
this.Connections = new List();
}
internal void AddConnection(Connection Connection)
{
lock (lck)
{
Connections.Add(Connection);
OnAddConnection(Connection);
}
}

Making asynchronous API synchronous

I am connecting to an API to get some data that is defined like this:
A client object ClientConnection, which allows one to send requests.
A IApi interface that needs to be passed to the ClientConnection to receive callbacks.
Schematically it looks like this:
// defined in the API dll
public class ClientConnection {
public ClientConnection(IApi api) { ... }
public void request(int reqid, string reqdetails) { ... }
}
interface IApi
{
void receiveData(int reqid, string ans);
}
Now, obviously this is a fairly standard asynchronous way of doing things: send requests through a global object, with a requestid, and receive answers tagged with that requestid.
I want to create a wrapper that is synchronous. What would be the most natural way of doing this? Is there a smart way of using async await, instead of using thread locking and stuff?
class MyWrapper : IApi
{
private ClientConnection _client;
private int _reqToken = 0;
public MyWrapper()
{
_client = new ClientConnection(this);
}
public string getData(string reqdetails)
{
_client.request(_reqToken++, reqdetails);
// what to do here?
}
public void receiveData(int reqid, string data) {
// what to do here?
}
}

Didn't test the code below, but it should give you the idea. Basically you can use ManualResetEvent to be signalled when you receive your result (and don't ever call this without proper timeout):
class MyWrapper : IApi {
private ClientConnection _client;
// here you store your requests
private Dictionary<int, PendingRequest> _pendingRequests = new Dictionary<int, PendingRequest>();
private int _reqToken = 0;
public MyWrapper() {
_client = new ClientConnection(this);
}
public string getData(string reqdetails, TimeSpan timout) {
// if this is multithreaded - lock over _pendingRequests when you add\remove requests there
// and when you increment your _reqToken, or use concurrent collection
using (var token = new PendingRequest()) {
var id = _reqToken;
// lock here
_pendingRequests.Add(id, token);
_client.request(id, reqdetails);
// and here use Interlocked.Increment
_reqToken++;
if (!token.Signal.WaitOne(timout)) {
// and here
_pendingRequests.Remove(id);
// timeout
throw new Exception("timout");
}
// if we are here - we have the result
return token.Result;
}
}
public void receiveData(int reqid, string data) {
// here you might need to lock too
if (_pendingRequests.ContainsKey(reqid)) {
var token = _pendingRequests[reqid];
_pendingRequests.Remove(reqid);
token.Complete(data);
}
}
private class PendingRequest : IDisposable {
public PendingRequest() {
Signal = new ManualResetEvent(false);
}
public ManualResetEvent Signal { get; private set; }
public string Result { get; private set; }
public void Complete(string result) {
this.Result = result;
Signal.Set();
}
public void Dispose() {
Signal.Dispose();
}
}
}

Pattern for 'routing' events through several subscribers?

I have a group of classes with the following interface:
public interface RoutedEventReceiver<T>
{
IDisposable Apply(IObservable<T> stream);
bool ShouldForwardEvent(T anEvent);
}
What I would like to do is to maintain a stack of these classes, with each event being filtered through the ShouldForwardEvent(T) predicate, and the resulting IObservable<T> passed to the next receiver. I also want to be able to push and pop new receivers while my program is running (at some point I may want to move from a stack to some other collection but for now a stack is sufficient).
What I have currently does work, but I don't feel like it is very "Rx". I am sure there must be a way to do what I want without all this imperative logic:
private void Refresh()
{
// _subscriptions is a list of previous subscriptions
foreach (var subscription in _subscriptions)
subscription.Dispose();
_subscriptions.Clear();
// _stream is my stream of incoming events
if (_stream != null)
{
var stream = _stream;
foreach (var eventReceiver in _eventReceivers)
{
// add the subscription so it can be disposed next Refresh()
_subscriptions.Add(eventReceiver.Apply(stream));
// filter the stream for the next event receiver
stream = stream.Where(eventReceiver.ShouldForwardEvent);
}
}
}
The above method is called whenever I Push or Pop on the stack.
Is there a cleaner, more functional way to express the above intent? I have tried .Publish() but with little success - perhaps I don't know it well enough.

I have managed to make the Publish approach work, but it doesn't afford me much other than getting rid of the need to keep a list of IDisposables:
private void Refresh()
{
_published.DisposeIfNotNull();
if (_stream != null)
{
var connectable = _stream.Publish();
_published = connectable.Connect();
var stream = connectable.AsObservable();
foreach (var eventReceiver in _eventReceivers)
{
eventReceiver.Apply(stream);
stream = stream.Where(eventReceiver.ShouldForwardEvent);
}
}
}

The class below (named CORStack for Chain Of Responsibility* Stack), tries to do what you're after. Internally it adds an ShouldHandle bool to the stream and uses this to determine whether to process. It exposes the standard Push, Pop, and Peek methods.
public sealed class CORStack<T>
{
Stack<StackFrame> _handlers;
public CORStack(IObservable<T> source)
{
_handlers = new Stack<StackFrame>();
_handlers.Push(new StackFrame(
source.Select(t => new ShouldHandleWrapper(t, true)),
new Handler<T>(new Action<T>(t => { }), true)));
}
public void Push(Handler<T> handler)
{
_handlers.Push(new StackFrame(_handlers.Peek().Observable, handler));
}
public Handler<T> Peek()
{
return _handlers.Peek().Handler;
}
public Handler<T> Pop()
{
var frame = _handlers.Pop();
frame.Dispose();
return frame.Handler;
}
class StackFrame : IDisposable
{
IDisposable _unsub;
public IObservable<ShouldHandleWrapper> Observable { get; private set; }
public Handler<T> Handler { get; private set; }
public StackFrame(IObservable<ShouldHandleWrapper> topOfStack, Handler<T> handler)
{
_unsub = topOfStack.Subscribe(shouldHandle =>
{
if (shouldHandle.ShouldHandle)
handler.Action.Invoke(shouldHandle.Value);
});
Observable = topOfStack.Select(shouldHandle =>
new ShouldHandleWrapper(shouldHandle.Value, shouldHandle.ShouldHandle && handler.Forward));
Handler = handler;
}
public void Dispose()
{
_unsub.Dispose();
}
}
class ShouldHandleWrapper
{
public readonly T Value;
public readonly bool ShouldHandle;
public ShouldHandleWrapper(T value, bool shouldHandle)
{
Value = value;
ShouldHandle = shouldHandle;
}
}
}
public class Handler<T>
{
public Action<T> Action { get; set; }
public bool Forward { get; set; }
public Handler(Action<T> action, bool forward)
{
Action = action;
Forward = forward;
}
}
*I realised that it's not a chain of responsibility, but can't think of a better name atm.

This is a case where I'd actually use Subjects. Create a subject for each handler, then subscribe to the stream and loop through the handlers passing the event as required. This avoids continually unsubscribing/resubscribing to the stream (and thus the Refresh method), which is not always appropriate. We use lock to guard against a new receiver being added or removed at the same moment as a new value is coming through the stream. If you can guarantee that cannot happen, then you can remove the lock statements.
public class YourClass<T> : IDisposable
{
private readonly Stack<Tuple<Subject<T>, RoutedEventReceiver<T>, IDisposable> _handlers;
private readonly IObservable<T> _stream;
private readonly IDisposable _streamSubscription;
public YourClass(IObservable<T> stream)
{
_handlers = new Stack<Tuple<Subject<T>, RoutedEventReceiver<T>, IDisposable>();
_stream = stream;
_streamSubscription = stream.Subscribe(OnNext, OnError, OnCompleted);
}
public void Dispose()
{
_streamSubscription.Dispose();
lock (_handlers)
{
foreach (var h in _handlers)
{
h.Item3.Dispose();
h.Item1.Dispose();
}
_handlers.Clear();
}
}
private void OnNext(T value)
{
lock (_handlers)
{
for (var h in _handlers)
{
h.Item1.OnNext(value);
if (!h.Item2.ShouldForwardEvent(value)) break;
}
}
}
private void OnError(Exception e)
{
lock (_handlers)
{
for (var h in _handlers) { h.Item1.OnError(e); }
}
}
private void OnCompleted()
{
lock (_handlers)
{
for (var h in _handlers) { h.Item1.OnCompleted(); }
}
}
public void Push(RoutedEventReceiver<T> handler)
{
lock (_handlers)
{
var subject = new Subject<T>;
_handlers.Push(Tuple.Create(subject, handler, handler.Apply(subject)));
}
}
public RoutedEventReceiver<T> Pop()
{
lock (_handlers)
{
var handler = _handlers.Pop();
handler.Item3.Dispose();
handler.Item1.Dispose();
return handler.Item2;
}
}
}