We Keep Coding

Directing messages to consumers

My client is attempting to send messages to the receiver. However I noticed that the receiver sometimes does not receive all the messages sent by the client thus missing a few messages (not sure where the problem is ? Client or the receiver).
Any suggestions on why that might be happening. This is what I am currently doing
On the receiver side this is what I am doing.
This is the Event Processor
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
}
}
This is how the client connects to the event hub
var StrBuilder = new EventHubsConnectionStringBuilder(eventHubConnectionString)
{
EntityPath = eventHubName,
};
this.eventHubClient = EventHubClient.CreateFromConnectionString(StrBuilder.ToString());
How do I direct my messages to specific consumers

I'm using this sample code from eventhub official doc, for sending and receiving.
And I have 2 consumer groups: $Default and newcg. Suppose you have 2 clients, the client_1 are using the default consumer group($Default), and client_2 are using the other consumer group(newcg)
First, after create the send client, in the SendMessagesToEventHub method, we need to add a property with value. The value should be the consumer group name. Sample code like below:
private static async Task SendMessagesToEventHub(int numMessagesToSend)
{
for (var i = 0; i < numMessagesToSend; i++)
{
try
{
var message = "444 Message";
Console.WriteLine($"Sending message: {message}");
EventData mydata = new EventData(Encoding.UTF8.GetBytes(message));
//here, we add a property named "cg", it's value is the consumer group. By setting this property, then we can read this message via this specified consumer group.
mydata.Properties.Add("cg", "newcg");
await eventHubClient.SendAsync(mydata);
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
}
await Task.Delay(10);
}
Console.WriteLine($"{numMessagesToSend} messages sent.");
}
Then in the client_1, after create the receiver project, which use the default consumer group($Default)
-> in the SimpleEventProcessor class -> ProcessEventsAsync method, we can filter out the unnecessary event data. Sample code for ProcessEventsAsync method:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here
if (eventData.Properties["cg"].ToString() == "$Default")
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
Console.WriteLine($"Message received. Partition: '{context.PartitionId}', Data: '{data}'");
Console.WriteLine(context.ConsumerGroupName);
}
}
return context.CheckpointAsync();
}
And in another client, like client_2, which use another consumer group, like it's name is newcg, we can follow the steps in client_1, just a little changes in ProcessEventsAsync method, like below:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here, using another consumer group name
if (eventData.Properties["cg"].ToString() == "newcg")
{
//other code
}
}
return context.CheckpointAsync();
}

This happens only when there are 2 or more Event Processor Host reading from same consumer group.
If you have event hub with 32 partitions and 2 event processor host reading from same consumer group. Then each event processor host will read from 16 partition and so on.
Similarly if 4 Event processor host parallelly reading from same consumer group then each will read from 8 partitions.
Check if you have 2 or more event processor host running on same consumer group.

I have tested your code and slightly modified it(different overload of EventProcessorHost constructor, and added CheckpointAsync after consuming the messages), and then did some tests.
By using the default implementation and default EventProcessorOptions(EventProcessorOptions.DefaultOptions) I can say that I did experience some latency when it comes to consuming messages, but all messages were processed successfully.
So, sometimes it seems like I am not getting the messages from the certain partition, but after a certain period of time, all messages arrive:
Here you can find the actual modified code that worked for me. It is a simple console app that prints to the console if something arrives.
string processorHostName = Guid.NewGuid().ToString();
var Options = new EventProcessorOptions()
{
MaxBatchSize = 1, //not required to make it working, just for testing
};
Options.SetExceptionHandler((ex) =>
{
System.Diagnostics.Debug.WriteLine($"Exception : {ex}");
});
var eventHubCS = "event hub connection string";
var storageCS = "storage connection string";
var containerName = "test";
var eventHubname = "test2";
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventHubname, "$Default", eventHubCS, storageCS, containerName);
eventProcessorHost.RegisterEventProcessorAsync<MyEventProcessor>(Options).Wait();
For sending the messages to the event hub and testing I used this message publisher app.

Signalr - It's possible to wait reponse from client?

I am a beginner in using Signalr and am checking out some examples.
Is it possible to send a message to the client from the server and wait for a return from it? Or is it possible to guarantee that after the answer the same session will be used?
My question is because in a given process, within a transaction, I need to ask the user if he wants to continue with the changes. I have not been able to ask this question before because validations should be done in the same session where changes have been made (but not yet confirmed).

Reiterating the comment from Jaime Yule, WebSockets are bidirectional communication and do not follow the Request/Response architecture for messaging. Given the very fluid nature of communication around WebSockets, these bullet points are good to keep in mind for your current (& future) scenarios:
SignalR is great if you're going to use it for fire & forget (Display a pop-up to a user and that's it)
It's not designed around request-response like you're asking, and trying to use it as such is an anti-pattern
Messages may be sent from either end of the connection at any time,
and there is no native support for one message to indicate it is
related to another
This makes the protocol poorly suited for transactional requirements

It is possible, but i would not recommend (relying on) it.
And it's not a pretty solution (using a static event and being pretty complex for such a simple thing).
Story goes like this:
Make sure client and server know the connectionId - They probably know that already, but i could not figure out a way to access it.
Await NotificationService.ConfirmAsync
... which will call confirm on the client
... which will await the user supplied answer
... and send it back to the server using Callback from The hub.
... which will notify the Callback from the NotificationService over a static event
... which will hand off the message back to ConfirmAsync (using a AutoResetEvent)
... which is hopefully still waiting :)
Client and server both have a 10 second timeout set.
The hub:
// Setup as /notification-hub
public class NotificationHub : Hub {
public string ConnectionId() => Context.ConnectionId;
public static event Action<string, string> Response;
public void Callback(string connectionId, string message) {
Response?.Invoke(connectionId, message);
}
}
Service:
// Wire it up using DI
public class NotificationService {
private readonly IHubContext<NotificationHub> _notificationHubContext;
public NotificationService(IHubContext<NotificationHub> notificationHubContext) {
_notificationHubContext = notificationHubContext;
}
public async Task<string> ConfirmAsync(string connectionId, string text, IEnumerable<string> choices) {
await _notificationHubContext.Clients.Client(connectionId)
.SendAsync("confirm", text, choices);
var are = new AutoResetEvent(false);
string response = null;
void Callback(string connId, string message) {
if (connectionId == connId) {
response = message;
are.Set();
}
}
NotificationHub.Response += Callback;
are.WaitOne(TimeSpan.FromSeconds(10));
NotificationHub.Response -= Callback;
return response;
}
}
Client side js:
var conn = new signalR.HubConnectionBuilder().withUrl("/notification-hub").build();
var connId;
// using Noty v3 (https://ned.im/noty/)
function confirm(text, choices) {
return new Promise(function (resolve, reject) {
var n = new Noty({
text: text,
timeout: 10000,
buttons: choices.map(function (b) {
return Noty.button(b, 'btn', function () {
resolve(b);
n.close();
});
})
});
n.show();
});
}
conn.on('confirm', function(text, choices) {
confirm(text, choices).then(function(choice) {
conn.invoke("Callback", connId, choice);
});
});
conn.start().then(function() {
conn.invoke("ConnectionId").then(function (connectionId) {
connId = connectionId;
// Picked up by a form and posted to the server
document.querySelector(".connection-id-input").value = connectionId;
});
});
For me this is way to complex to put it into the project i am working on.
It really looks like something that will come back and bite you later...

Is it possible to send a message to the client from the server and wait for a return from it? Or is it possible to guarantee that after the answer the same session will be used?
None of this is possible. Currently there's no way to wait for the client's response or even to get to know if the client received the message. There's some discussion implementing this on GitHub. Also here's the feature request.
Until then, the workaround is to send a "notification" from the server with a fire and forget attitude and let the client get the required data via a HTTP request to the server.

This is now possible with .NET 7 using Client Results.
Today, I've highlighted this issue in dotnet's Github page and got a good response from one of the developers of SignalR.
This requires the server to use ISingleClientProxy.InvokeAsync to be able to make request to the client and wait for response.
Quote from the documentation
In addition to making calls to clients, the server can request a
result from a client. This requires the server to use
ISingleClientProxy.InvokeAsync and the client to return a result from
its .On handler.
From the client (js/ts)
hubConnection.on("GetMessage", async () => {
let promise = new Promise((resolve, reject) => {
setTimeout(() => {
resolve(new { data: "message" });
}, 100);
});
return promise;
});
From the server (C#)
//By calling Client(...) on an instance of IHubContext<T>
async Task<object> SomeMethod(IHubContext<MyHub> context)
{
string result = await context.Clients.Client(connectionID).InvokeAsync<string>(
"GetMessage");
return result;
}
//---------------------------//
//Or by calling Client(...) or Caller on the Clients property in a Hub method
public class ChatHub : Hub
{
public async Task<string> WaitForMessage(string connectionId)
{
var message = await Clients.Client(connectionId).InvokeAsync<string>(
"GetMessage");
return message;
}
}

Using the following form with Invoke waits for and returns the response directly (just like a "real" synchronous method call)
var persons = hubProxy.Invoke<IEnumerable<Person>>("GetPersonsSynchronous", SearchCriteria, noteFields).Result;
foreach (Person person in persons)
{
Console.WriteLine($"{person.LastName}, {person.FirstName}");
}

How to perform database operation independently?

I have 1 exe which is nothing bit a Windows form which will continuously run in background and will watch my serial port and I have 1 event data receive event which fires as my serial port receive data.
As soon as I receive data in this event I will pass this data to another event handler which saves this data in database through web api method.
But data to my serial port will be coming frequently so I want to save this data to my database independently so that my database insert operation doesn't block my incoming serial port data.
This is my code:
void _serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)//Fires as my serial port receives data
{
int dataLength = _serialPort.BytesToRead;
byte[] data = new byte[dataLength];
int nbrDataRead = _serialPort.Read(data, 0, dataLength);
if (nbrDataRead == 0)
return;
// Send data to whom ever interested
if (NewSerialDataRecieved != null)
{
NewSerialDataRecieved(this, new SerialDataEventArgs(data)); //pass serial port data to new below event handler.
}
}
void _spManager_NewSerialDataRecieved(object sender, SerialDataEventArgs e) //I want this event handler to run independently so that database save operation doenst block incoming serial port data
{
if (this.InvokeRequired)
{
// Using this.Invoke causes deadlock when closing serial port, and BeginInvoke is good practice anyway.
this.BeginInvoke(new EventHandler<SerialDataEventArgs>(_spManager_NewSerialDataRecieved), new object[] { sender, e });
return;
}
//data is converted to text
string str = Encoding.ASCII.GetString(e.Data);
if (!string.IsNullOrEmpty(str))
{
//This is where i will save data to through my web api method.
RunAsync(str).Wait();
}
}
static async Task RunAsync(string data)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:33396/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var content = new StringContent(data);
var response = await client.PostAsJsonAsync<StringContent>("api/Service/Post", content);//nothing happens after this line.
}
}
Web api controller:
public class MyController : ApiController
{
[HttpPost]
public HttpResponseMessage Post(HttpRequestMessage request)
{
var someText = request.Content.ReadAsStringAsync().Result;
return new HttpResponseMessage() { Content = new StringContent(someText) };
}
}
But here problem is:
var response = await client.PostAsJsonAsync<StringContent>("api/Service/Post", content);
Nothing happens after this line that is operation blocks on this line.
So can anybody guide me with this?

By independently we determined in the SO C# chat room that you really mean "Asynchronously".
Your solution is the code above, saving this data to a WebAPI endpoint so any solution to the problem needs to be in 2 parts ...
PART 1: The Client Part
On the client all we need to do is make the call asynchronously in order to free up the current thread to carry on receiving data on the incoming serial port, we can do that like so ...
// build the api client, you may want to keep this in a higher scope to avoid recreating on each message
var api = new HttpClient();
api.BaseAddress = new Uri(someConfigVariable);
// asynchronously make the call and handle the result
api.PostAsJsonAsync("api/My", str)
.ContinueWith(t => HandleResponseAsync(t.Result))
.Unwrap();
...
PART 2: The Server Part
Since you have web api i'm also going to assume you are using EF too, the common and "clean" way to do this, with all the extras stripped out (like model validation / error handling) might look something like this ...
// in your EF code you will have something like this ...
Public async Task<User> SaveUser(User userModel)
{
try
{
var newUser = await context.Users.AddAsync(userModel);
context.SavechangesAsync();
return newUser;
}
catch(Exception ex) {}
}
// and in your WebAPI controller something like this ...
HttpPost]
public async Task<HttpResponseMessage> Post(User newUser)
{
return Ok(await SaveUser(newUser));
}
...
Disclaimer:
The concepts involved here go much deeper and as I hinted above, much has been left out here like validation, error checking, ect but this is the core to getting your serial port data in to a database using the technologies I believe you are using.
Key things to read up on for anyone wanting to achieve this kind of thing might include: Tasks, Event Handling, WebAPI, EF, Async operations, streaming.

From what you describe it seems like you might want to have a setup like this:
1) your windows form listens for serial port
2) when new stuff comes to port your windows forms app saves it to some kind of a queue (msmq, for example)
3) you should have separate windows service that checks queue and as it finds new messages in a queue it sends request to web api

Best solution for this problem is to use ConcurrentQueue.
Just do search on google and you will get planty of samples.
ConcurrentQueue is thread safe and it support writing and reading from multiple threads.
So the component listening to the searal port can write data to the queue. And you can have 2 or more tasks running parallel which listening to this queue and update db as soon as it receives data.

Not sure if it's the problem, but you shouldn't block on async code. You are doing RunAsync(str).Wait(); and I believe that's the problem. Have a look at this blog post by Stephen Cleary:
http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html

Azure Service Bus SubscriptionClient.OnMessage always completes message when it shouldnt

I am trying to receive all messages for a given subscription to a Service Bus Topic, but for the context of this app I do not want them dead lettered at this time, I just want to view them and leave them on the subscription. Despite instantiating the Client as
SubscriptionClient sc = SubscriptionClient.CreateFromConnectionString(connectionString, sub.topicName, sub.subscriptionName, ReceiveMode.PeekLock);
and making sure that I am using message.Abandon() rather than message.Complete() the message always gets Dead-lettered after accessing the message. I also have options.AutoComplete set to false
full method code below:
public List<ServiceBusMessage> RetrieveSubscriptionMessages(Subscription sub) {
ServiceBusMessage sbm;
List<ServiceBusMessage> list = new List<ServiceBusMessage>();
String connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"].ToString();
SubscriptionClient sc = SubscriptionClient.CreateFromConnectionString(connectionString, sub.topicName, sub.subscriptionName, ReceiveMode.PeekLock);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
sc.OnMessage((message) => {
try {
sbm = new ServiceBusMessage() {
topicName = sub.topicName,
messageText = message.GetBody<String>()
};
list.Add(sbm);
message.Abandon();
}
catch (Exception) {
message.Abandon();
throw;
}
}, options);
return list;
}
Am I missing something ? Or is there an issue with auto dead-lettering with the onMessage() method?
Thanks !

When a message is abandoned the service bus will immediately make it available for re-delivery to any subscriber of the topic.
If you are trying to configure a multicast mechanism in which multiple listeners all receive the same message, then understand that all listeners on a given subscription will be competing for the same message. In order for every listener to receive its own copy of the message, then simply create a unique subscription to the topic for each listener.
If your intent is to delay re-delivery of the abandoned message, you might look at the SO question: What's the proper way to abandon an Azure SB Message so that it becomes visible again in the future in a way I can control?

SignalR notification system

This is my first time playing around with SignalR. I am trying to build a notification system where the server checks at regular intervals to see if there is something (query database) to broadcast and if there is then it broadcasts it to all the clients.
I came across this post on Stackoverflow and was wondering if modifying the code to make a DB call at a particular interval was indeed the right way to do it. If not is there a better way to do it?
I did see a lot of Notification related questions posted here but none with any code in it. Hence this post.
This is the exact code that I am using:
public class NotificationHub : Hub
{
public void Start()
{
Thread thread = new Thread(Notify);
thread.Start();
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification()
{
Server = "Server A", Application = "Some App",
Message = "This is a long ass message and amesaadfasd asdf message",
ImgURL = "../Content/Images/accept-icon.png"
});
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
I am already seeing some weird behaviour where the notifications come more often than they are supposed to. Even though I am supposed to get it every 20s I get it around 4-5 secs and I get multiple messages.
Here is my client:
var notifier = $.connection.notificationHub;
notifier.shownotification = function (data) {
$.each(data, function (i, sample) {
var output = Mustache.render("<img class='pull-left' src='{{ImgURL}}'/> <div><strong>{{Application}}</strong></div><em>{{Server}}</em> <p>{{Message}}</p>", sample);
$.sticky(output);
});
};
$.connection.hub.start(function () { notifier.start(); });

Couple of notes:
As soon as a second client connects to your server there will be 2 threads sending the notifications, therefore if you ave more than one client you will have intervals smaller than 20s
Handling thread manually within ASP.NET is considered bad practice, you should avoid this if possible
In general this smells a lot like polling which is kind of the thing SignalR lets you get rid of since you don't need to signal the server/client
In order to solve this you need todo something like this (again, threads in a web application are generally not a good idea):
public class NotificationHub : Hub
{
public static bool initialized = false;
public static object initLock = new object();
public void Start()
{
if(initialized)
return;
lock(initLock)
{
if(initialized)
return;
Thread thread = new Thread(Notify);
thread.Start();
initialized = true;
}
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification() { Server = "Server A", Application = "Some App", Message = "This is a long ass message and amesaadfasd asdf message", ImgURL = "../Content/Images/accept-icon.png" });
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
The static initialized flag prevents multiple threads from being created. The locking around it is to ensure that the flag is only set once.

I am working on the same task over here. Instead of continuously checking the database, I created my own events and listener, where an event is RAISED when a NOTIFICATION IS ADDED :) What do you think about that?