We Keep Coding

Retained MQTT messages being missed

I am trying to create a simple application that will publish some messages to a topic with MQTT (library I am using is M2Mqtt.net) and then I want to subscribe to the topic once the messages have already been sent and then have them all be received and then discarded, because they have been received.
I am using mosquitto 2.0.12 as the broker
This is the publisher:
public class MessagePublisher : IMessagePublisher
{
private readonly MqttClient _client;
public MessagePublisher()
{
_client = new MqttClient("localhost");
// clean session needs to be set to false so that it retains all the missed messages, not just the last one
_client.Connect(Guid.NewGuid().ToString(), "username", "password", false, byte.MaxValue);
}
public void Publish(string topic, string message, bool retain = false)
{
Console.Write($"Sent: {topic}, {message}");
_client.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, retain);
Total.SentAndReceived.Add(message, null);
}
}
This is the listener:
public class MessageReceiver : IMessageReceiver
{
private readonly MqttClient _client;
public MessageReceiver()
{
_client = new MqttClient("localhost");
}
public void Subscribe(params string[] topics)
{
_client.Subscribe(topics, new[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
_client.MqttMsgPublishReceived += client_receivedMessage;
}
public void Connect()
{
// clean session needs to be set to false so that it retains all the missed messages, not just the last one
_client.Connect(Guid.NewGuid().ToString(), "username", "password", false, byte.MaxValue);
}
public void Disconnect()
{
_client.Disconnect();
}
static void client_receivedMessage(object sender, MqttMsgPublishEventArgs e)
{
var message = Encoding.Default.GetString(e.Message);
Console.WriteLine($"Message Received: {message}");
if (Total.SentAndReceived.ContainsKey(message))
Total.SentAndReceived[message] = message;
}
}
And this is the main application:
public static class Program
{
public static void Main(string[] args)
{
var messageReceiver = new MessageReceiver();
var publisher = new MessagePublisher();
for (var i = 1; i <= 10000; i++)
{
publisher.Publish("Devices/", i.ToString(), true);
}
messageReceiver.Subscribe("Devices/");
messageReceiver.Connect();
Thread.Sleep(5000);
var b = Total.SentAndReceived.Where(x => x.Value == null);
Console.WriteLine($"{b.Count()} Missed Messages");
}
}
The problem I am having is that there are missed messages. And the number of missed messages always changes when I run the application. And it's not that last n messages being missed it's the first n messages.
I hope that if I was to build a service that would listen to the published messages. If the services stops for any reason. Once the service comes back online, the messages sent in that downtime would be received.

I think you have a misunderstanding around some terms here.
First, MQTT does not generally queue messages. The only time the broker will queue messages is if the receiving client has already already been connected and subscribed to the topic at QOS > 0. If that client then disconnects before the publisher sends the messages the broker will queue the messages. They will then only be sent to the receiving client if they then reconnect with the same client id and have the clean session flag set to false. This is the only way that messages will be queued up.
Since you appear to be using randomly generated client ids (Guid.NewGuid().ToString()) this will not work. You also appear to be trying to subscribe before you connect, again that won't work.
Secondly, retained messages have nothing to do with message queuing as described above. A message is retained if the retained flag is set at the point of publishing. The broker will then store that specific message and deliver it ever time a client subscribes to the matching topic. This message will be sent before any other messages on the topic. If another message with the retained flag is published it will replace the previous message, there can only be 1 retained message per topic.

Integration Test to publish to a topic and subscribe to another in Azure Service Bus is unreliable is there a race condition?

I am trying to write an integration / acceptance test to test some code in azure, the code in the question ATM simply subscribes to one topic and publishes to another.
I have written the test but it doesn't always pass, seems as though there could be a race condition in place. I've tried writing it a couple of ways including using OnMessage and also using Receive (example I show here).
When using OnMessage the test seemed to always exit prematurely (around 30 seconds), which I guess perhaps means its inappropriate for this test anyway.
My query concerning my example specifically, I assumed that once I created the subscription to the target topic, that any message sent to it I would be able to pickup using Receive(), whatever point in time that message arrived meaning, if the message arrives at the target topic before I call Receive(), I would still be able to read the message afterward by calling Receive(). Could anyone please shed any light on this?
namespace somenamespace {
[TestClass]
public class SampleTopicTest
{
private static TopicClient topicClient;
private static SubscriptionClient subClientKoEligible;
private static SubscriptionClient subClientKoIneligible;
private static OnMessageOptions options;
public const string TEST_MESSAGE_SUB = "TestMessageSub";
private static NamespaceManager namespaceManager;
private static string topicFleKoEligible;
private static string topicFleKoIneligible;
private BrokeredMessage message;
[ClassInitialize]
public static void BeforeClass(TestContext testContext)
{
//client for publishing messages
string connectionString = ConfigurationManager.AppSettings["ServiceBusConnectionString"];
string topicDataReady = ConfigurationManager.AppSettings["DataReadyTopicName"];
topicClient = TopicClient.CreateFromConnectionString(connectionString, topicDataReady);
topicFleKoEligible = ConfigurationManager.AppSettings["KnockOutEligibleTopicName"];
topicFleKoIneligible = ConfigurationManager.AppSettings["KnockOutIneligibleTopicName"];
//create test subscription to receive messages
namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.SubscriptionExists(topicFleKoEligible, TEST_MESSAGE_SUB))
{
namespaceManager.CreateSubscription(topicFleKoEligible, TEST_MESSAGE_SUB);
}
if (!namespaceManager.SubscriptionExists(topicFleKoIneligible, TEST_MESSAGE_SUB))
{
namespaceManager.CreateSubscription(topicFleKoIneligible, TEST_MESSAGE_SUB);
}
//subscriber client koeligible
subClientKoEligible = SubscriptionClient.CreateFromConnectionString(connectionString, topicFleKoEligible, TEST_MESSAGE_SUB);
subClientKoIneligible = SubscriptionClient.CreateFromConnectionString(connectionString, topicFleKoIneligible, TEST_MESSAGE_SUB);
options = new OnMessageOptions()
{
AutoComplete = false,
AutoRenewTimeout = TimeSpan.FromMinutes(1),
};
}
[TestMethod]
public void E2EPOCTopicTestLT50()
{
Random rnd = new Random();
string customerId = rnd.Next(1, 49).ToString();
FurtherLendingCustomer sentCustomer = new FurtherLendingCustomer { CustomerId = customerId };
BrokeredMessage sentMessage = new BrokeredMessage(sentCustomer.ToJson());
sentMessage.CorrelationId = Guid.NewGuid().ToString();
string messageId = sentMessage.MessageId;
topicClient.Send(sentMessage);
Boolean messageRead = false;
//wait for message to arrive on the ko eligible queue
while((message = subClientKoEligible.Receive(TimeSpan.FromMinutes(2))) != null){
//read message
string messageString = message.GetBody<String>();
//Serialize
FurtherLendingCustomer receivedCustomer = JsonConvert.DeserializeObject<FurtherLendingCustomer>(messageString.Substring(messageString.IndexOf("{")));
//assertion
Assert.AreEqual(sentCustomer.CustomerId, receivedCustomer.CustomerId,"verify customer id");
//pop message
message.Complete();
messageRead = true;
//leave loop after processing one message
break;
}
if (!messageRead)
Assert.Fail("Didn't receive any message after 2 mins");
}
}
}

As the official document states about SubscriptionClient.Receive(TimeSpan):
Parameters
serverWaitTime
TimeSpan
The time span the server waits for receiving a message before it times out.
A Null can be return by this API if operation exceeded the timeout specified, or the operations succeeded but there are no more messages to be received.
Per my test, if a message sent to the topic and then delivered to your subscription within your specific serverWaitTime, then you could receive a message no matter whether the message arrives at the target topic before or after you call Receive.
When using OnMessage the test seemed to always exit prematurely (around 30 seconds), which I guess perhaps means its inappropriate for this test anyway.
[TestMethod]
public void ReceiveMessages()
{
subClient.OnMessage(msg => {
System.Diagnostics.Trace.TraceInformation($"{DateTime.Now}：{msg.GetBody<string>()}");
msg.Complete();
});
Task.Delay(TimeSpan.FromMinutes(5)).Wait();
}
For Subscription​Client.​On​Message, I assumed that it basically a loop invoking Receive. After calling OnMessage, you need to wait for a while and stop this method to exit. Here is a blog about the Event-Driven message programming for windows Azure Service Bus, you could refer to here.
Additionally, I found that your topicClient for sending messages and the subClientKoEligible for receiving a message are not targeted at the same topic path.

NetMQ - messages only work on reply to incoming message, not to other port

On initialize of my router application I call the following code. It binds fine, receives messages fine but refuses to work for the On_ReceiveXXX methods unless it's a direct response. I want to know why
public void Initialize(string frontEndAddress, string backEndAddress)
{
_poller = new Poller();
_timeAllowedBetweenPings = TimeSpan.FromMinutes(1);
_lastPingResponse = DateTime.Now;
using (var ctx = NetMQContext.Create())
{
_frontEnd = ctx.CreateRouterSocket();
_backEnd = ctx.CreateRouterSocket();
_frontEnd.Bind(frontEndAddress);
Console.WriteLine(string.Format("[Router]: Connected to {0}", frontEndAddress));
_backEnd.Bind(backEndAddress);
Console.WriteLine(string.Format("[Router]: Connected to {0}", backEndAddress));
_frontEnd.ReceiveReady += On_ReceiveFrontEnd;
_backEnd.ReceiveReady += On_ReceiveBackEnd;
_poller.AddSocket(_frontEnd);
_poller.AddSocket(_backEnd);
var timer = new NetMQTimer(TimeSpan.FromSeconds(1));
timer.Elapsed += On_Ping;
_poller.AddTimer(timer);
_poller.PollTillCancelled();
}
}
This fails to call the dealer ReceiveReady event:
private void On_ReceiveFrontEnd(object sender, NetMQSocketEventArgs e)
{
_lastPingResponse = DateTime.Now;
var frontEndMsg = e.Socket.ReceiveMultipartBytes();
var streamData = frontEndMsg.Last();
ApplicationMessage msg = PackageHelper.DeserializeOutgoing(streamData);
Console.WriteLine(string.Format("Command received: {0}", msg.CO));
_backEnd.SendMultipartBytes(frontEndMsg);
}
BUT if I change the line
_backEnd.SendMultipartBytes(frontEndMsg);
to
_frontEnd.SendMultipartBytes(frontEndMsg);
It suddenly works... so messages coming from my front end application can only be responded to, not passed on to the back end application. The same is true the other way round, for the back end messages.

When working with router the first frame is the routing id and it specific to the socket. So you can't pass the entire message from router to router. Change the backend to dealer and it will work, or prefix the message with routing id of the backend socket.

Can anyone tell me how to create a durable publisher and subscriber for active mq topic message in c#?

This is my existing code from subscriber but it is not working
_connection = factory.CreateConnection();
_connection.ClientId = "ID";
_connection.Start();
_session = _connection.CreateSession();
ITopic dest = _session.GetTopic(QUEUE_DESTINATION);
using (IMessageConsumer consumer = _session.CreateDurableConsumer(dest,"myDurable",null,false))
{
IMessage message;
while(true)
{
while ((message = consumer.Receive(TimeSpan.FromMilliseconds(5000))) != null)
{
var objectMessage = message as IObjectMessage;
if (objectMessage != null)
{
person = objectMessage.Body as Person;
if (person != null)
{
Console.WriteLine(person.FirstName);
Console.WriteLine(person.LastName);
}
}
else
{
Console.WriteLine("Object Message is null");
}
}
}
I am also not sure if anything needs to be done on the publisher side.
This code perfectly works when both publisher and subscriber are UP and RUNNING,
But when the subscriber is DOWN and again UP and RUNNING the messages that are enqueued are not dequeued. I also verified this in active mq console. A durable consumer is created, but i don't receive the messages in the subscriber.

The problem was i did not run the subscriber program first. It is necessary to run so that your subcriber first registers with activemq. You can find the durable consumer in your active mq console under the subcriber tab.

SignalR notification system

This is my first time playing around with SignalR. I am trying to build a notification system where the server checks at regular intervals to see if there is something (query database) to broadcast and if there is then it broadcasts it to all the clients.
I came across this post on Stackoverflow and was wondering if modifying the code to make a DB call at a particular interval was indeed the right way to do it. If not is there a better way to do it?
I did see a lot of Notification related questions posted here but none with any code in it. Hence this post.
This is the exact code that I am using:
public class NotificationHub : Hub
{
public void Start()
{
Thread thread = new Thread(Notify);
thread.Start();
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification()
{
Server = "Server A", Application = "Some App",
Message = "This is a long ass message and amesaadfasd asdf message",
ImgURL = "../Content/Images/accept-icon.png"
});
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
I am already seeing some weird behaviour where the notifications come more often than they are supposed to. Even though I am supposed to get it every 20s I get it around 4-5 secs and I get multiple messages.
Here is my client:
var notifier = $.connection.notificationHub;
notifier.shownotification = function (data) {
$.each(data, function (i, sample) {
var output = Mustache.render("<img class='pull-left' src='{{ImgURL}}'/> <div><strong>{{Application}}</strong></div><em>{{Server}}</em> <p>{{Message}}</p>", sample);
$.sticky(output);
});
};
$.connection.hub.start(function () { notifier.start(); });

Couple of notes:
As soon as a second client connects to your server there will be 2 threads sending the notifications, therefore if you ave more than one client you will have intervals smaller than 20s
Handling thread manually within ASP.NET is considered bad practice, you should avoid this if possible
In general this smells a lot like polling which is kind of the thing SignalR lets you get rid of since you don't need to signal the server/client
In order to solve this you need todo something like this (again, threads in a web application are generally not a good idea):
public class NotificationHub : Hub
{
public static bool initialized = false;
public static object initLock = new object();
public void Start()
{
if(initialized)
return;
lock(initLock)
{
if(initialized)
return;
Thread thread = new Thread(Notify);
thread.Start();
initialized = true;
}
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification() { Server = "Server A", Application = "Some App", Message = "This is a long ass message and amesaadfasd asdf message", ImgURL = "../Content/Images/accept-icon.png" });
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
The static initialized flag prevents multiple threads from being created. The locking around it is to ensure that the flag is only set once.

I am working on the same task over here. Instead of continuously checking the database, I created my own events and listener, where an event is RAISED when a NOTIFICATION IS ADDED :) What do you think about that?