We Keep Coding

Directing messages to consumers

My client is attempting to send messages to the receiver. However I noticed that the receiver sometimes does not receive all the messages sent by the client thus missing a few messages (not sure where the problem is ? Client or the receiver).
Any suggestions on why that might be happening. This is what I am currently doing
On the receiver side this is what I am doing.
This is the Event Processor
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
}
}
This is how the client connects to the event hub
var StrBuilder = new EventHubsConnectionStringBuilder(eventHubConnectionString)
{
EntityPath = eventHubName,
};
this.eventHubClient = EventHubClient.CreateFromConnectionString(StrBuilder.ToString());
How do I direct my messages to specific consumers

I'm using this sample code from eventhub official doc, for sending and receiving.
And I have 2 consumer groups: $Default and newcg. Suppose you have 2 clients, the client_1 are using the default consumer group($Default), and client_2 are using the other consumer group(newcg)
First, after create the send client, in the SendMessagesToEventHub method, we need to add a property with value. The value should be the consumer group name. Sample code like below:
private static async Task SendMessagesToEventHub(int numMessagesToSend)
{
for (var i = 0; i < numMessagesToSend; i++)
{
try
{
var message = "444 Message";
Console.WriteLine($"Sending message: {message}");
EventData mydata = new EventData(Encoding.UTF8.GetBytes(message));
//here, we add a property named "cg", it's value is the consumer group. By setting this property, then we can read this message via this specified consumer group.
mydata.Properties.Add("cg", "newcg");
await eventHubClient.SendAsync(mydata);
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
}
await Task.Delay(10);
}
Console.WriteLine($"{numMessagesToSend} messages sent.");
}
Then in the client_1, after create the receiver project, which use the default consumer group($Default)
-> in the SimpleEventProcessor class -> ProcessEventsAsync method, we can filter out the unnecessary event data. Sample code for ProcessEventsAsync method:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here
if (eventData.Properties["cg"].ToString() == "$Default")
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
Console.WriteLine($"Message received. Partition: '{context.PartitionId}', Data: '{data}'");
Console.WriteLine(context.ConsumerGroupName);
}
}
return context.CheckpointAsync();
}
And in another client, like client_2, which use another consumer group, like it's name is newcg, we can follow the steps in client_1, just a little changes in ProcessEventsAsync method, like below:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here, using another consumer group name
if (eventData.Properties["cg"].ToString() == "newcg")
{
//other code
}
}
return context.CheckpointAsync();
}

This happens only when there are 2 or more Event Processor Host reading from same consumer group.
If you have event hub with 32 partitions and 2 event processor host reading from same consumer group. Then each event processor host will read from 16 partition and so on.
Similarly if 4 Event processor host parallelly reading from same consumer group then each will read from 8 partitions.
Check if you have 2 or more event processor host running on same consumer group.

I have tested your code and slightly modified it(different overload of EventProcessorHost constructor, and added CheckpointAsync after consuming the messages), and then did some tests.
By using the default implementation and default EventProcessorOptions(EventProcessorOptions.DefaultOptions) I can say that I did experience some latency when it comes to consuming messages, but all messages were processed successfully.
So, sometimes it seems like I am not getting the messages from the certain partition, but after a certain period of time, all messages arrive:
Here you can find the actual modified code that worked for me. It is a simple console app that prints to the console if something arrives.
string processorHostName = Guid.NewGuid().ToString();
var Options = new EventProcessorOptions()
{
MaxBatchSize = 1, //not required to make it working, just for testing
};
Options.SetExceptionHandler((ex) =>
{
System.Diagnostics.Debug.WriteLine($"Exception : {ex}");
});
var eventHubCS = "event hub connection string";
var storageCS = "storage connection string";
var containerName = "test";
var eventHubname = "test2";
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventHubname, "$Default", eventHubCS, storageCS, containerName);
eventProcessorHost.RegisterEventProcessorAsync<MyEventProcessor>(Options).Wait();
For sending the messages to the event hub and testing I used this message publisher app.

Multiple Conversations For Direct Line Client

I'm trying to use the Microsoft.Bot.Connector.DirectLine .NET client to connect to my Direct Line Channel. My client application will have many conversations open at once (like 1000+).
What I'm trying to do is efficiently create a single Direct Line client object which can receive messages for all my conversations and NOT have a single client per conversation.
This below code is from:
https://learn.microsoft.com/en-us/azure/bot-service/bot-service-channel-directline-extension-net-client?view=azure-bot-service-4.0
The problem is that to create a new conversation I need to create a new client which I think would eventually exhaust use up a lot of sockets. Does anyone know if I can create a single connection and then listen for multiple conversations?
Thanks
static async Task Main(string[] args)
{
Console.WriteLine("What is your name:");
var UserName = Console.ReadLine();
var tokenClient = new DirectLineClient(
new Uri(endpoint),
new DirectLineClientCredentials(secret));
var conversation = await tokenClient.Tokens.GenerateTokenForNewConversationAsync();
var client = new DirectLineClient(
new Uri(endpoint),
new DirectLineClientCredentials(conversation.Token));
await client.StreamingConversations.ConnectAsync(
conversation.ConversationId,
ReceiveActivities);
var startConversation = await client.StreamingConversations.StartConversationAsync();
var from = new ChannelAccount() { Id = startConversation.ConversationId, Name = UserName };
var message = Console.ReadLine();
while (message != "end")
{
try
{
var response = await client.StreamingConversations.PostActivityAsync(
startConversation.ConversationId,
new Activity()
{
Type = "message",
Text = message,
From = from,
ChannelData = new Common.ChannelData() { FromNumber = "+17081234567"}
});
}
catch (OperationException ex)
{
Console.WriteLine(
$"OperationException when calling PostActivityAsync: ({ex.StatusCode})");
}
message = Console.ReadLine();
}
Console.ReadLine();
}
public static void ReceiveActivities(ActivitySet activitySet)
{
if (activitySet != null)
{
foreach (var a in activitySet.Activities)
{
if (a.Type == ActivityTypes.Message && a.From.Id == "MyBotName")
{
Console.WriteLine($"<Bot>: {a.Text}");
}
}
}
}

I think using the Direct Line streaming extensions would be problematic for your purposes. I'm guessing your custom SMS channel would itself be an app service. Since an app service can (and probably should, in your case) be scaled so that multiple instances are running simultaneously, suppose two SMS messages from the same conversation go to two instances of your channel. In addition to having each instance of your channel using many web sockets to talk to many bots, multiple instances of your channel may use duplicated web sockets to talk to the same bot. There's also the problem of each bot itself needing to support streaming extensions.
Rather than using using Direct Line streaming extensions, you might consider using traditional Direct Line. This would involve receiving activities from the bots by polling a Direct Line endpoint.
Since Direct Line is a channel itself that you'd be using on top of your own channel, you might also consider cutting out Direct Line altogether. That way you wouldn't have two channels between the user and the bot. You could send HTTP requests to each bot's endpoint directly, and the activities the bots would receive would contain the service URL for your channel, allowing your channel to receive messages from the bots.

Signalr - It's possible to wait reponse from client?

I am a beginner in using Signalr and am checking out some examples.
Is it possible to send a message to the client from the server and wait for a return from it? Or is it possible to guarantee that after the answer the same session will be used?
My question is because in a given process, within a transaction, I need to ask the user if he wants to continue with the changes. I have not been able to ask this question before because validations should be done in the same session where changes have been made (but not yet confirmed).

Reiterating the comment from Jaime Yule, WebSockets are bidirectional communication and do not follow the Request/Response architecture for messaging. Given the very fluid nature of communication around WebSockets, these bullet points are good to keep in mind for your current (& future) scenarios:
SignalR is great if you're going to use it for fire & forget (Display a pop-up to a user and that's it)
It's not designed around request-response like you're asking, and trying to use it as such is an anti-pattern
Messages may be sent from either end of the connection at any time,
and there is no native support for one message to indicate it is
related to another
This makes the protocol poorly suited for transactional requirements

It is possible, but i would not recommend (relying on) it.
And it's not a pretty solution (using a static event and being pretty complex for such a simple thing).
Story goes like this:
Make sure client and server know the connectionId - They probably know that already, but i could not figure out a way to access it.
Await NotificationService.ConfirmAsync
... which will call confirm on the client
... which will await the user supplied answer
... and send it back to the server using Callback from The hub.
... which will notify the Callback from the NotificationService over a static event
... which will hand off the message back to ConfirmAsync (using a AutoResetEvent)
... which is hopefully still waiting :)
Client and server both have a 10 second timeout set.
The hub:
// Setup as /notification-hub
public class NotificationHub : Hub {
public string ConnectionId() => Context.ConnectionId;
public static event Action<string, string> Response;
public void Callback(string connectionId, string message) {
Response?.Invoke(connectionId, message);
}
}
Service:
// Wire it up using DI
public class NotificationService {
private readonly IHubContext<NotificationHub> _notificationHubContext;
public NotificationService(IHubContext<NotificationHub> notificationHubContext) {
_notificationHubContext = notificationHubContext;
}
public async Task<string> ConfirmAsync(string connectionId, string text, IEnumerable<string> choices) {
await _notificationHubContext.Clients.Client(connectionId)
.SendAsync("confirm", text, choices);
var are = new AutoResetEvent(false);
string response = null;
void Callback(string connId, string message) {
if (connectionId == connId) {
response = message;
are.Set();
}
}
NotificationHub.Response += Callback;
are.WaitOne(TimeSpan.FromSeconds(10));
NotificationHub.Response -= Callback;
return response;
}
}
Client side js:
var conn = new signalR.HubConnectionBuilder().withUrl("/notification-hub").build();
var connId;
// using Noty v3 (https://ned.im/noty/)
function confirm(text, choices) {
return new Promise(function (resolve, reject) {
var n = new Noty({
text: text,
timeout: 10000,
buttons: choices.map(function (b) {
return Noty.button(b, 'btn', function () {
resolve(b);
n.close();
});
})
});
n.show();
});
}
conn.on('confirm', function(text, choices) {
confirm(text, choices).then(function(choice) {
conn.invoke("Callback", connId, choice);
});
});
conn.start().then(function() {
conn.invoke("ConnectionId").then(function (connectionId) {
connId = connectionId;
// Picked up by a form and posted to the server
document.querySelector(".connection-id-input").value = connectionId;
});
});
For me this is way to complex to put it into the project i am working on.
It really looks like something that will come back and bite you later...

Is it possible to send a message to the client from the server and wait for a return from it? Or is it possible to guarantee that after the answer the same session will be used?
None of this is possible. Currently there's no way to wait for the client's response or even to get to know if the client received the message. There's some discussion implementing this on GitHub. Also here's the feature request.
Until then, the workaround is to send a "notification" from the server with a fire and forget attitude and let the client get the required data via a HTTP request to the server.

This is now possible with .NET 7 using Client Results.
Today, I've highlighted this issue in dotnet's Github page and got a good response from one of the developers of SignalR.
This requires the server to use ISingleClientProxy.InvokeAsync to be able to make request to the client and wait for response.
Quote from the documentation
In addition to making calls to clients, the server can request a
result from a client. This requires the server to use
ISingleClientProxy.InvokeAsync and the client to return a result from
its .On handler.
From the client (js/ts)
hubConnection.on("GetMessage", async () => {
let promise = new Promise((resolve, reject) => {
setTimeout(() => {
resolve(new { data: "message" });
}, 100);
});
return promise;
});
From the server (C#)
//By calling Client(...) on an instance of IHubContext<T>
async Task<object> SomeMethod(IHubContext<MyHub> context)
{
string result = await context.Clients.Client(connectionID).InvokeAsync<string>(
"GetMessage");
return result;
}
//---------------------------//
//Or by calling Client(...) or Caller on the Clients property in a Hub method
public class ChatHub : Hub
{
public async Task<string> WaitForMessage(string connectionId)
{
var message = await Clients.Client(connectionId).InvokeAsync<string>(
"GetMessage");
return message;
}
}

Using the following form with Invoke waits for and returns the response directly (just like a "real" synchronous method call)
var persons = hubProxy.Invoke<IEnumerable<Person>>("GetPersonsSynchronous", SearchCriteria, noteFields).Result;
foreach (Person person in persons)
{
Console.WriteLine($"{person.LastName}, {person.FirstName}");
}

Can't get NetMQ pub-sub pattern to work with ReceiveReady

I'm trying my hands on NetMQ (3.3.3.4) and creating a pub-sub pattern.
I want a host/server to listen to all incoming data on one port (9000) and forward the data to all clients/subscribers on another port (9001).
The clients will then send data on 9000 and receive all messages sent (by whomever) on 9001.
Following the documentation I created something like the code below, but I can't get it to work. Mainly, I believe, because ReceiveReady is never called!
How I believe it should work:
client.Publish should cause the first line in host.SubscriberSocket_ReceiveReady to unblock and pass the data along to the other socket
When data has been passed along it should appear in the infinite running Task in the client
Results:
Breakpoints on // This line is never reached are never reached
There are no exceptions anywhere.
Switching the ports on the host so that publish = 9000 and subscribe = 9001 has no effect
Windows Firewall is turned off, so there should not be any blocking
It makes no difference if I'm putting the address into PublisherSocket constructor, or if I'm using _publisherSocket.Bind(address) in Host or _publisherSocket.Connect(address) in Client
What am I doing wrong?
Host
public class MyNetMQHost {
private NetMQSocket _publishSocket;
private NetMQSocket _subscribeSocket;
private NetMQPoller _poller;
public MyNetMQHost(string publishAddress = "#tcp://localhost:9001", string subscribeAddress = "#tcp://localhost:9000") {
Task.Factory.StartNew(() => {
using (_publishSocket = new PublisherSocket(publishAddress))
using (_subscribeSocket = new SubscriberSocket(subscribeAddress))
using (_poller = new NetMQPoller { _publishSocket, _subscribeSocket }) {
_subscriberSocket.ReceiveReady += SubscriberSocket_ReceiveReady;
_poller.Run();
}
});
}
private void SubscriberSocket_ReceiveReady(object sender, NetMQSocketEventArgs e) {
var data = e.Socket.ReceiveMultipartBytes(); // This line is never reached
_publishSocket.SendMultipartBytes(data);
}
}
Client
public class MyNetMQClient {
private readonly NetMQSocket _publishSocket;
private readonly NetMQSocket _subscribeSocket;
public MyNetMQClient(string publishAddress = ">tcp://localhost:9000", string subscribeAddress = ">tcp://localhost:9001") {
_publishSocket = new PublisherSocket(publishAddress);
_subscribeSocket = new SubscriberSocket(subscribeAddress);
Task.Factory.StartNew(() => {
while (true) {
byte[] frameBytes = _subscribeSocket.ReceiveFrameBytes();
int one = 1; // This line is never reached
}
});
}
public void Publish(byte[] data) {
_publishSocket.SendFrame(data);
}
}
Tester
public class Tester {
public void MyTester() {
MyNetMQHost host = new MyNetMQHost();
MyNetMQClient client = new MyNetMQClient();
client.Publish(Encoding.Unicode.GetBytes("Hello world!");
}
}

Both your broker and client never call suscribe.
On the broker call suscriber.Subscribe("") to subscribe for all. On your client subscribe to what ever you want.
In your broker you should actually use XSubscriber and XPublisher to move susvriptions around. That way you dont need the subscribe all. You can use Proxy class for that.

Why does a bound SUB receive only one message from a connecting PUB?

I'm making examples for my ZeroMQ CLR namespace, however I have a problem with PUB/SUB.
Why do I get only the first message? Sometimes I get no message, if I debug through the client (on PubSub_Client(arg);) I get some messages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Security.Cryptography;
using ZeroMQ;
namespace ZeroMQ.Test
{
static partial class Program
{
static string PubSub_FrontendAddress = "tcp://127.0.0.1:2772";
public static void Main(string[] args)
{
if (args == null || args.Length < 1)
{
// say here were some arguments...
args = new string[] { "World" };
}
// Setup the ZContext
context = ZContext.Create();
CancellationTokenSource cancellor0 = null;
{
// Create the "Server" cancellor and threads
cancellor0 = new CancellationTokenSource();
var serverThread = new Thread(PubSub_Server);
serverThread.Start(cancellor0.Token);
serverThread.Join(64);
}
{
Thread.Sleep(1000);
Console.WriteLine("Starting...");
// foreach arg we are the Client, asking the Server
foreach (string arg in args)
{
PubSub_Client(arg);
// Thread.Sleep(1000);
}
Console.WriteLine("Ended...");
}
if (cancellor0 != null)
{
// Cancel the Server
cancellor0.Cancel();
}
// we could have done here context.Terminate()
}
static void PubSub_Server(object cancelluS)
{
var cancellus = (CancellationToken)cancelluS;
using (var socket = ZSocket.Create(context, ZSocketType.SUB))
{
socket.Bind(PubSub_FrontendAddress);
socket.SubscribeAll();
/* var poller = ZPollItem.Create(socket, (ZSocket _socket, out ZMessage message, out ZError _error) =>
{
while (null == (message = _socket.ReceiveMessage(/* ZSocketFlags.DontWait, * out _error)))
{
if (_error == ZError.EAGAIN)
{
_error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(_error);
}
return true;
}); /**/
while (!cancellus.IsCancellationRequested)
{
ZError error;
ZMessage request;
/* if (!poller.TryPollIn(out request, out error, TimeSpan.FromMilliseconds(512)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
if (null == (request = socket.ReceiveMessage(ZSocketFlags.DontWait, out error)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
foreach (ZFrame frame in request)
{
string strg = frame.ReadString();
Console.WriteLine("{0} said hello!", strg);
}
}
socket.Unbind(PubSub_FrontendAddress);
}
}
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
using (var crypto = new RNGCryptoServiceProvider())
{
var identity = new byte[8];
crypto.GetBytes(identity);
socket.Identity = identity;
}
socket.Connect(PubSub_FrontendAddress);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
}
}

I'm having trouble with your design which seems just wrong:
A single subscriber and multiple publishers is an odd choice. I trust you have a good reason for it, but you should have said what that is. When sending messages from multiple clients to a single server, it is normal to use DEALER/ROUTER sockets instead. PUB/SUB is intended for a small set of publishers to a large number of subscribers.
A client that connects, sends one message, then immediately disconnects, is another very unusual use case that I hope is just an example:
For one thing, you are open to linger problems whereby the message will get dropped on the disconnect it is isn't sent within the linger timeout. [I don't know what the default linger is for your language binding, so that may or may not be an issue, but you should at least check to ensure that it isn't.]
For another, as you've already found, there are issues around the time it takes to connect to a socket, which may lead to PUB messages getting dropped if they are sent before the socket has properly connected.
If you insist on using PUB/SUB in this manner, you will need an out of band protocol to synchronise the PUB and SUB threads before the pub messages are sent. There are examples of how to do this reliable pub/sub in the zeromq guide. This will involve a second set of sockets in the same threads to send the synchronisation messages; DEALER sockets don't drop messages which is why they are suitable for that purpose...
But, DEALER/ROUTER sockets would appear to be a better choice than PUB/SUB unless there is some design requirement that hasn't been disclosed.

Well... There was a comment by Martin Sustrik: "The problem is that connecting is asynchronous and takes certain amount of time."
Now there is Thread.Sleep(64) - and it works...:
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
socket.Connect(PubSub_FrontendAddress);
Thread.Sleep(64);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
Do you know any better way to get the connection established?