We Keep Coding

Losing messages with MQTT with C# uPLibrary.Networking.M2Mqtt

I have a problem that I lose messages with MQTT although I send them with "QOS_LEVEL_EXACTLY_ONCE".
The loss is only when the receiver is not running and then starts later.
These messages are then not collected.
Version of M2Mqtt is 4.3.0
If both clients, i.e. receiver and transmitter, are running, no messages are lost.
Only if the receiver is not running, the messages are prefetched during this time and do not arrive at the receiver.
I can't find any setting on the server(broker) for how long messages should be saved
sender
public class Programm
{
static MqttClient mqttClient;
static async Task Main(string[] args)
{
var locahlost = true;
var clientName = "Sender 1";
Console.WriteLine($"{clientName} Startet");
var servr = locahlost ? "localhost" : "test.mosquitto.org";
mqttClient = new MqttClient(servr);
mqttClient.Connect(clientName);
Task.Run(() =>
{
if (mqttClient != null && mqttClient.IsConnected)
{
for (int i = 0; i < 100; i++)
{
var Message = $"{clientName} ->Test {i}";
mqttClient.Publish("Application1/NEW_Message", Encoding.UTF8.GetBytes($"{Message}"), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, true);
Console.WriteLine(Message);
Thread.Sleep(i * 1000);
}
}
});
Console.WriteLine($"{clientName} End");
}
}
Server
public class Programm
{
static async Task Main(string[] args)
{
Console.WriteLine("Server");
MqttServerOptionsBuilder options = new MqttServerOptionsBuilder()
// set endpoint to localhost
.WithDefaultEndpoint()
// port used will be 707
.WithDefaultEndpointPort(1883);
// handler for new connections
// creates a new mqtt server
IMqttServer mqttServer = new MqttFactory().CreateMqttServer();
// start the server with options
mqttServer.StartAsync(options.Build()).GetAwaiter().GetResult();
// keep application running until user press a key
Console.ReadLine();
}
}
Receiver
public class Programm
{
static MqttClient mqttClient;
static async Task Main(string[] args)
{
var clientName = "Emfänger 1";
var locahlost = true;
Console.WriteLine($"Start of {clientName}");
Task.Run(() =>
{
var servr = locahlost ? "localhost" : "test.mosquitto.org";
mqttClient = new MqttClient(servr);
mqttClient.MqttMsgPublishReceived += MqttClient_MqttMsgPublishReceived;
mqttClient.Subscribe(new string[] { "Application1/NEW_Message" }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
mqttClient.Connect(clientName);
});
// client.UseConnecedHandler(e=> {Console.WriteLine("Verbunden") });
Console.ReadLine();
Console.WriteLine($"end of {clientName}");
Console.ReadLine();
}
private static void MqttClient_MqttMsgPublishReceived(object sender, uPLibrary.Networking.M2Mqtt.Messages.MqttMsgPublishEventArgs e)
{
var message = Encoding.UTF8.GetString(e.Message);
Console.WriteLine(message);
}
}

The default value for the Clean session flag when connecting to the broker with M2MQTT is true.
This means that the broker will discard any queued messages.
https://m2mqtt.wordpress.com/using-mqttclient/
You need to set this to false to ensure the client receives the queued messages.
mqttClient.Connect(clientName, false);

I found the error, saving was missing.
Here is the new code from the server
static async Task Main(string[] args)
{
Console.WriteLine("Server");
MqttServerOptionsBuilder options = new MqttServerOptionsBuilder()
.WithDefaultEndpoint()
.WithDefaultEndpointPort(1883)
.WithConnectionValidator(OnNewConnection)
.WithApplicationMessageInterceptor(OnNewMessage)
.WithClientMessageQueueInterceptor(OnOut)
.WithDefaultCommunicationTimeout(TimeSpan.FromMinutes(5))
.WithMaxPendingMessagesPerClient(10)
.WithPersistentSessions()
.WithStorage(storage);
// creates a new mqtt server
IMqttServer mqttServer = new MqttFactory().CreateMqttServer();
// start the server with options
mqttServer.StartAsync(options.Build()).GetAwaiter().GetResult();
// keep application running until user press a key
Console.ReadLine();
}

Kafka consume message and then produce to another topic

I have to consume from a Kafka topic, get the message and do some json clean and filter job, then I need to produce the new message to another Kafka topic, my code is like this:
public static YamlMappingNode configs;
public static void Main(string[] args)
{
using (var reader = new StreamReader(Path.Combine(Directory.GetCurrentDirectory(), ".gitlab-ci.yml")))
{
var yaml = new YamlStream();
yaml.Load(reader);
//find variables
configs = (YamlMappingNode)yaml.Documents[0].RootNode;
configs = (YamlMappingNode)configs.Children.Where(k => k.Key.ToString() == "variables")?.FirstOrDefault().Value;
}
CancellationTokenSource cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => {
e.Cancel = true; // prevent the process from terminating.
cts.Cancel();
};
Run_ManualAssign(configs, cts.Token);
}
public static async void Run_ManualAssign(YamlMappingNode configs, CancellationToken cancellationToken)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topics = configs.Where(k => k.Key.ToString() == "input_kfk_topic")?.FirstOrDefault().Value.ToString();
var config = new ConsumerConfig
{
// the group.id property must be specified when creating a consumer, even
// if you do not intend to use any consumer group functionality.
GroupId = new Guid().ToString(),
BootstrapServers = brokerList,
// partition offsets can be committed to a group even by consumers not
// subscribed to the group. in this example, auto commit is disabled
// to prevent this from occurring.
EnableAutoCommit = true
};
using (var consumer =
new ConsumerBuilder<Ignore, string>(config)
.SetErrorHandler((_, e) => Console.WriteLine($"Error: {e.Reason}"))
.Build())
{
//consumer.Assign(topics.Select(topic => new TopicPartitionOffset(topic, 0, Offset.Beginning)).ToList());
consumer.Assign(new TopicPartitionOffset(topics, 0, Offset.End));
//var producer = new ProducerBuilder<Null, string>(config).Build();
try
{
while (true)
{
try
{
var consumeResult = consumer.Consume(cancellationToken);
/// Note: End of partition notification has not been enabled, so
/// it is guaranteed that the ConsumeResult instance corresponds
/// to a Message, and not a PartitionEOF event.
//filter message
var result = ReadMessage(configs, consumeResult.Message.Value);
//send to kafka topic
await Run_ProducerAsync(configs, result);
}
catch (ConsumeException e)
{
Console.WriteLine($"Consume error: {e.Error.Reason}");
}
}
}
catch (OperationCanceledException)
{
Console.WriteLine("Closing consumer.");
consumer.Close();
}
}
}
#endregion
#region Run_Producer
public static async Task Run_ProducerAsync(YamlMappingNode configs, string message)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topicName = configs.Where(k => k.Key.ToString() == "target_kafka_topic")?.FirstOrDefault().Value.ToString();
var config = new ProducerConfig {
BootstrapServers = brokerList,
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
try
{
/// Note: Awaiting the asynchronous produce request below prevents flow of execution
/// from proceeding until the acknowledgement from the broker is received (at the
/// expense of low throughput).
var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });
producer.Flush(TimeSpan.FromSeconds(10));
Console.WriteLine($"delivered to: {deliveryReport.TopicPartitionOffset}");
}
catch (ProduceException<string, string> e)
{
Console.WriteLine($"failed to deliver message: {e.Message} [{e.Error.Code}]");
}
}
}
#endregion
Am I doing something wrong here? The program existed immediately when executing var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });, no error message, no error code.
In the meanwhile I used Python and config the same for Producer, it works well.

Run_ManualAssign(configs, cts.Token);
For this line in the Main function, you are calling async without await in a sync function. Thus the program exit immediately after this invoke started (not finished as it is async)
You could have 2 options
Use async Main function and add await in front of this invoke.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-7.1/async-main
If you really want to call async function in sync function
Run_ManualAssign(configs, ts.Token).ConfigureAwait(false).GetAwaiter().GetResult();

I solved this problem but I don't know why actually. I opened an issue here.

Backgroundtask is activated by SocketActivityTrigger, but the socket in the trigger returns null

I want to be able to send and receive data over a socket using bluetooth when my app is suspended. I register my BackgroundReader once a connection is established in the foreground app.
Code run once a connection is established:
private void OnConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
{
BackgroundTaskRegistration receiveTask = RegisterBackgroundTask("X.ReceiverTask", "ReceiverTask", new SocketActivityTrigger(), null);
BackgroundTaskRegistration sendTask = RegisterBackgroundTask("X.Z", "SenderTask", new ApplicationTrigger(), null);
var socket = args.Socket;
socket.EnableTransferOwnership(receiveTask.TaskId, SocketActivityConnectedStandbyAction.DoNotWake);
socket.TransferOwnership("RfcommSocket");
}
Method to register a Background task:
private static BackgroundTaskRegistration RegisterBackgroundTask(string taskEntryPoint, string name, IBackgroundTrigger trigger, IBackgroundCondition condition)
{
foreach(var cur in BackgroundTaskRegistration.AllTasks)
{
if(cur.Value.Name == name)
{
//task is already registered
return (BackgroundTaskRegistration)(cur.Value);
}
}
var builder = new BackgroundTaskBuilder();
builder.Name = name;
if(taskEntryPoint != null && taskEntryPoint != String.Empty)
{
builder.TaskEntryPoint = taskEntryPoint;
}
builder.SetTrigger(trigger);
if(condition != null)
{
builder.AddCondition(condition);
}
BackgroundTaskRegistration task = builder.Register();
return task;
}
When I try to send data the Receiver background task activates as expected. Only when I try to read the stream socket from its properties, it returns null and throws an Exception when I try to access it:
public async void Run(IBackgroundTaskInstance taskInstance)
{
taskDeferral = taskInstance.GetDeferral();
taskInstance.Canceled += OnTaskCanceled;
var details = taskInstance.TriggerDetails as SocketActivityTriggerDetails;
var socketInformation = details.SocketInformation;
switch (details.Reason)
{
...
case SocketActivityTriggerReason.SocketActivity:
Debug.WriteLine(socketInformation.SocketKind.ToString());
var socket = socketInformation.StreamSocket; //This returns null
DataReader reader = new DataReader(socket.InputStream);
reader.InputStreamOptions = InputStreamOptions.Partial;
await reader.LoadAsync(250);
var dataString = reader.ReadString(reader.UnconsumedBufferLength);
Debug.WriteLine(dataString);
socket.TransferOwnership(socketInformation.Id); /* Important! */
break;
...
}
taskDeferral.Complete();
}
The backgroundTaskHost.exe exits and, supposedly, takes the socket with it.
Thanks in advance!

Backgroundtask is activated by SocketActivityTrigger, but the socket in the trigger returns null
I checked your code, The problem is you used SocketActivityTrigger in Bluetooth RF communication. In UWP we often use RfcommConnectionTrigger to detect Bluetooth message notification. The RfcommConnectionTrigger has more features that support Bluetooth RF communication. And I have test with official code sample and it could get the socket instance correctly and could load the message with reader.

BrokeredMessage disposed after accessing from different thread

This might be a duplicate of this question but that's confused with talk about batching database updates and still has no proper answer.
In a simple example using Azure Service Bus queues, I can't access a BrokeredMessage after it's been placed on a queue; it's always disposed if I read the queue from another thread.
Sample code:
class Program {
private static string _serviceBusConnectionString = "XXX";
private static BlockingCollection<BrokeredMessage> _incomingMessages = new BlockingCollection<BrokeredMessage>();
private static CancellationTokenSource _cancelToken = new CancellationTokenSource();
private static QueueClient _client;
static void Main(string[] args) {
// Set up a few listeners on different threads
Task.Run(async () => {
while (!_cancelToken.IsCancellationRequested) {
var msg = _incomingMessages.Take(_cancelToken.Token);
if (msg != null) {
try {
await msg.CompleteAsync();
Console.WriteLine($"Completed Message Id: {msg.MessageId}");
} catch (ObjectDisposedException) {
Console.WriteLine("Message was disposed!?");
}
}
}
});
// Now set up our service bus reader
_client = GetQueueClient("test");
_client.OnMessageAsync(async (message) => {
await Task.Run(() => _incomingMessages.Add(message));
},
new OnMessageOptions() {
AutoComplete = false
});
// Now start sending
Task.Run(async () => {
int sent = 0;
while (!_cancelToken.IsCancellationRequested) {
var msg = new BrokeredMessage();
await _client.SendAsync(msg);
Console.WriteLine($"Sent {++sent}");
await Task.Delay(1000);
}
});
Console.ReadKey();
_cancelToken.Cancel();
}
private static QueueClient GetQueueClient(string queueName) {
var namespaceManager = NamespaceManager.CreateFromConnectionString(_serviceBusConnectionString);
if (!namespaceManager.QueueExists(queueName)) {
var settings = new QueueDescription(queueName);
settings.MaxDeliveryCount = 10;
settings.LockDuration = TimeSpan.FromSeconds(5);
settings.EnableExpress = true;
settings.EnablePartitioning = true;
namespaceManager.CreateQueue(settings);
}
var factory = MessagingFactory.CreateFromConnectionString(_serviceBusConnectionString);
factory.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(0.1), maxBackoff: TimeSpan.FromSeconds(30), maxRetryCount: 100);
var queueClient = factory.CreateQueueClient(queueName);
return queueClient;
}
}
I've tried playing around with settings but can't get this to work. Any ideas?

Answering my own question with response from Serkant Karaca # Microsoft here:
Very basic rule and I am not sure if this is documented. The received message needs to be processed in the callback function's life time. In your case, messages will be disposed when async callback completes, this is why your complete attempts are failing with ObjectDisposedException in another thread.
I don't really see how queuing messages for further processing helps on the throughput. This will add more burden to client for sure. Try processing the message in the async callback, that should be performant enough.
Bugger.

Azure: How to Delete "DeadLettered" Messages from Service Bus queue

I would like to delete the Dead Lettered messages from the service bus queue. In particular, this value is called DeadLetterMessageCount and you can find out this by right-clicking the "Properties" of the SB queue in the Server Explorer of your project (in case of using a SB queue).
The reason I would like to do this is because I've set up an Autoscale of the cloud service. So, when the SB queue is quite big, it adds some more cores in order to proceed the messages faster (it enables more worker roles). I realized that when you set up the scaling depending on the number of messages in the queue, it counts the DeadLettered messages as well (messages that cannot be consumed).
So that's a waste of money, as more instances are enabled that are not needed.
Any queries, please let me know.
Thanks for your help

You read and delete messages from dead letter queue the same way you read from normal queues or subscriptions.
You can use this method to get the path of the queue: QueueClient.FormatDeadLetterPath(queuePath).
Also see this previous answer: How do I delete a DeadLetter message on an Azure Service Bus Topic

This is a code to delete a Dead-Letter messages from Queues.
public async void DeleteMessagesFromQueueAsync()
{
bool isDeadLetter=true;
long SequenceNumber = 12;
string queuePath='queue name';
string connectionString='connection string of ASB Namespace';
BrokeredMessage _srcMessage = null;
DeleteMessageResponse _msgDeletionStatus = new DeleteMessageResponse();
MessageReceiver fromQueueClient = null;
try
{
MessagingFactory factory = MessagingFactory.CreateFromConnectionString(connectionString);
string _fromEntityPath = !isDeadLetter ? queuePath : QueueClient.FormatDeadLetterPath(queuePath);
fromQueueClient = await factory.CreateMessageReceiverAsync(_fromEntityPath, ReceiveMode.PeekLock);
BrokeredMessage _message = await fromQueueClient.ReceiveAsync(SequenceNumber);
if (_message != null)
_srcMessage= _message;
if (_srcMessage != null )
{
await _srcMessage.CompleteAsync();
}
}
catch (Exception ex)
{
}
finally
{
if (fromQueueClient != null)
await fromQueueClient.CloseAsync();
}
}

You can use 'ReceiveAndDelete' mode and 'ReceiveBatchAsync' to delete quickly from DeadLetterQueue
private async void button1_Click(object sender, EventArgs e)
{
try
{
var DLQPath = "/$DeadLetterQueue"; ///**** Important - Pointing to DLQ'
var topicName = "message";
var sub = "message-subscription";
int batchSize = 100;
runProcess = true;
_subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionSt, topicName, sub + DLQPath, ReceiveMode.ReceiveAndDelete);
int cnt = 0;
do
{
var messages = await _subscriptionClient.ReceiveBatchAsync(batchSize);
var msgCount = messages.Count();
if (msgCount == 0)
{
break;
}
cnt += msgCount;
labelCount.Text = cnt.ToString();
}
while (runProcess);
_subscriptionClient.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.GetBaseException().Message);
return;
}
}