I have a problem that I lose messages with MQTT although I send them with "QOS_LEVEL_EXACTLY_ONCE".
The loss is only when the receiver is not running and then starts later.
These messages are then not collected.
Version of M2Mqtt is 4.3.0
If both clients, i.e. receiver and transmitter, are running, no messages are lost.
Only if the receiver is not running, the messages are prefetched during this time and do not arrive at the receiver.
I can't find any setting on the server(broker) for how long messages should be saved
sender
public class Programm
{
static MqttClient mqttClient;
static async Task Main(string[] args)
{
var locahlost = true;
var clientName = "Sender 1";
Console.WriteLine($"{clientName} Startet");
var servr = locahlost ? "localhost" : "test.mosquitto.org";
mqttClient = new MqttClient(servr);
mqttClient.Connect(clientName);
Task.Run(() =>
{
if (mqttClient != null && mqttClient.IsConnected)
{
for (int i = 0; i < 100; i++)
{
var Message = $"{clientName} ->Test {i}";
mqttClient.Publish("Application1/NEW_Message", Encoding.UTF8.GetBytes($"{Message}"), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, true);
Console.WriteLine(Message);
Thread.Sleep(i * 1000);
}
}
});
Console.WriteLine($"{clientName} End");
}
}
Server
public class Programm
{
static async Task Main(string[] args)
{
Console.WriteLine("Server");
MqttServerOptionsBuilder options = new MqttServerOptionsBuilder()
// set endpoint to localhost
.WithDefaultEndpoint()
// port used will be 707
.WithDefaultEndpointPort(1883);
// handler for new connections
// creates a new mqtt server
IMqttServer mqttServer = new MqttFactory().CreateMqttServer();
// start the server with options
mqttServer.StartAsync(options.Build()).GetAwaiter().GetResult();
// keep application running until user press a key
Console.ReadLine();
}
}
Receiver
public class Programm
{
static MqttClient mqttClient;
static async Task Main(string[] args)
{
var clientName = "Emfänger 1";
var locahlost = true;
Console.WriteLine($"Start of {clientName}");
Task.Run(() =>
{
var servr = locahlost ? "localhost" : "test.mosquitto.org";
mqttClient = new MqttClient(servr);
mqttClient.MqttMsgPublishReceived += MqttClient_MqttMsgPublishReceived;
mqttClient.Subscribe(new string[] { "Application1/NEW_Message" }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
mqttClient.Connect(clientName);
});
// client.UseConnecedHandler(e=> {Console.WriteLine("Verbunden") });
Console.ReadLine();
Console.WriteLine($"end of {clientName}");
Console.ReadLine();
}
private static void MqttClient_MqttMsgPublishReceived(object sender, uPLibrary.Networking.M2Mqtt.Messages.MqttMsgPublishEventArgs e)
{
var message = Encoding.UTF8.GetString(e.Message);
Console.WriteLine(message);
}
}
The default value for the Clean session flag when connecting to the broker with M2MQTT is true.
This means that the broker will discard any queued messages.
https://m2mqtt.wordpress.com/using-mqttclient/
You need to set this to false to ensure the client receives the queued messages.
mqttClient.Connect(clientName, false);
I found the error, saving was missing.
Here is the new code from the server
static async Task Main(string[] args)
{
Console.WriteLine("Server");
MqttServerOptionsBuilder options = new MqttServerOptionsBuilder()
.WithDefaultEndpoint()
.WithDefaultEndpointPort(1883)
.WithConnectionValidator(OnNewConnection)
.WithApplicationMessageInterceptor(OnNewMessage)
.WithClientMessageQueueInterceptor(OnOut)
.WithDefaultCommunicationTimeout(TimeSpan.FromMinutes(5))
.WithMaxPendingMessagesPerClient(10)
.WithPersistentSessions()
.WithStorage(storage);
// creates a new mqtt server
IMqttServer mqttServer = new MqttFactory().CreateMqttServer();
// start the server with options
mqttServer.StartAsync(options.Build()).GetAwaiter().GetResult();
// keep application running until user press a key
Console.ReadLine();
}
Related
I'm trying to connect to Mosquitto using MQTTnet. I am able to receive messages but IsConnected property is false. Anyone know why?
Only property iStart is set to true.
The problem seems to be with the Mosquitto host.
mosquitto.conf is mostly defaults. I only have these parameters set:
allow_anonymous true
password_file C:\mosquitto\passwd
tls_version tlsv1.2
log_dest file C:\mosquitto\log\mosquitto.log
Log file is empty.
class Program
{
static async Task Main()
{
var mqttClientId = "MyClientId"; // Unique ClientId or pass a GUID as string for something random
var mqttBrokerAddress = "localhost"; // hostname or IP address of your MQTT broker
var mqttBrokerUsername = "guest"; // Broker Auth username if using auth
var mqttBrokerPassword = "guest"; // Broker Auth password if using auth
var topic = "topic"; // topic to subscribe to
var mqttClient = new MqttFactory().CreateManagedMqttClient();
var mqttClientOptions = new ManagedMqttClientOptionsBuilder()
.WithAutoReconnectDelay(TimeSpan.FromSeconds(5))
.WithClientOptions(new MqttClientOptionsBuilder()
.WithTcpServer(mqttBrokerAddress, 1883)
.WithCredentials(mqttBrokerUsername, mqttBrokerPassword) // Remove this line if no auth
.WithCleanSession().WithKeepAlivePeriod(new TimeSpan(1))//.WithTls(tlsOptions)
.Build()
)
.Build();
mqttClient.ApplicationMessageReceivedAsync += async e => MqttOnNewMessage(e);
mqttClient.ConnectedAsync += async e => MqttOnConnected(e);
mqttClient.DisconnectedAsync += async e => MqttOnDisconnected(e);
mqttClient.SynchronizingSubscriptionsFailedAsync += async e => test(e);
mqttClient.ConnectingFailedAsync += async e => test2(e);
var aaa = new List<MqttTopicFilter>();
var bbb = new MqttTopicFilterBuilder().WithTopic(topic).Build();
aaa.Add(bbb);
try
{
await mqttClient.SubscribeAsync(aaa);
await mqttClient.StartAsync(mqttClientOptions);
System.Console.WriteLine(mqttClient.IsConnected);
}
catch (Exception ex)
{
throw;
}
Console.ReadLine();
}
private static void test2(ConnectingFailedEventArgs e)
{
Console.WriteLine();
}
private static void test(ManagedProcessFailedEventArgs e)
{
Console.WriteLine();
}
private static void MqttOnNewMessage(MqttApplicationMessageReceivedEventArgs e)
{
// Do something with each incoming message from the topic
Console.WriteLine($"MQTT Client: OnNewMessage Topic: {e.ApplicationMessage.Topic} / Message: {e.ApplicationMessage.Payload}");
}
private static void MqttOnConnected(MqttClientConnectedEventArgs e) => Console.WriteLine($"MQTT Client: Connected with result: {e.ConnectResult.ResultCode}");
private static void MqttOnDisconnected(MqttClientDisconnectedEventArgs e) => Console.WriteLine($"MQTT Client: Broker connection lost with reason: {e.Reason}.");
}
I'm using UdpClient to grab UDP datagrams from Forza, a car racing game using their built in data out function.
I've set the data out target endpoint on Forza to be 127.0.0.1:5300
I can verify that these are being sent, and in fact are able to be received by my application.
The problem is that they are only received after adding an internal heartbeat.
public const int FORZA_DATA_OUT_PORT = 5300;
public const int FORZA_HOST_PORT = 5200;
private static void Main(string[] args)
{
var ipEndPoint = new IPEndPoint(IPAddress.Loopback, FORZA_DATA_OUT_PORT);
var senderClient = new UdpClient(FORZA_HOST_PORT);
// Heartbeat task - remove this and nothing is received
var senderTask = Task.Run(async () =>
{
while (true)
{
Console.WriteLine("Sending heartbeat");
await senderClient.SendAsync(new byte[1], 1, ipEndPoint);
await Task.Delay(5000);
}
});
int count = 0;
var receiverTask = Task.Run(async () =>
{
Console.WriteLine("Packet Forwarding UP!");
var client = new UdpClient(FORZA_DATA_OUT_PORT);
Console.WriteLine("Listening... ");
while (true)
{
var receive = await client.ReceiveAsync();
count++;
Console.WriteLine($"{count:D5} - {receive.Buffer.Length}B - {receive.RemoteEndPoint.Address} - {receive.RemoteEndPoint.AddressFamily} - {receive.RemoteEndPoint.Port}");
}
});
Task.WaitAll(senderTask, receiverTask);
}
If I remove the senderClient.SendAsync then no packets are received.
Packets are being detected by the LiveTcpUdpWatch program both with and without the line.
Is there any reason for this?
I have to consume from a Kafka topic, get the message and do some json clean and filter job, then I need to produce the new message to another Kafka topic, my code is like this:
public static YamlMappingNode configs;
public static void Main(string[] args)
{
using (var reader = new StreamReader(Path.Combine(Directory.GetCurrentDirectory(), ".gitlab-ci.yml")))
{
var yaml = new YamlStream();
yaml.Load(reader);
//find variables
configs = (YamlMappingNode)yaml.Documents[0].RootNode;
configs = (YamlMappingNode)configs.Children.Where(k => k.Key.ToString() == "variables")?.FirstOrDefault().Value;
}
CancellationTokenSource cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => {
e.Cancel = true; // prevent the process from terminating.
cts.Cancel();
};
Run_ManualAssign(configs, cts.Token);
}
public static async void Run_ManualAssign(YamlMappingNode configs, CancellationToken cancellationToken)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topics = configs.Where(k => k.Key.ToString() == "input_kfk_topic")?.FirstOrDefault().Value.ToString();
var config = new ConsumerConfig
{
// the group.id property must be specified when creating a consumer, even
// if you do not intend to use any consumer group functionality.
GroupId = new Guid().ToString(),
BootstrapServers = brokerList,
// partition offsets can be committed to a group even by consumers not
// subscribed to the group. in this example, auto commit is disabled
// to prevent this from occurring.
EnableAutoCommit = true
};
using (var consumer =
new ConsumerBuilder<Ignore, string>(config)
.SetErrorHandler((_, e) => Console.WriteLine($"Error: {e.Reason}"))
.Build())
{
//consumer.Assign(topics.Select(topic => new TopicPartitionOffset(topic, 0, Offset.Beginning)).ToList());
consumer.Assign(new TopicPartitionOffset(topics, 0, Offset.End));
//var producer = new ProducerBuilder<Null, string>(config).Build();
try
{
while (true)
{
try
{
var consumeResult = consumer.Consume(cancellationToken);
/// Note: End of partition notification has not been enabled, so
/// it is guaranteed that the ConsumeResult instance corresponds
/// to a Message, and not a PartitionEOF event.
//filter message
var result = ReadMessage(configs, consumeResult.Message.Value);
//send to kafka topic
await Run_ProducerAsync(configs, result);
}
catch (ConsumeException e)
{
Console.WriteLine($"Consume error: {e.Error.Reason}");
}
}
}
catch (OperationCanceledException)
{
Console.WriteLine("Closing consumer.");
consumer.Close();
}
}
}
#endregion
#region Run_Producer
public static async Task Run_ProducerAsync(YamlMappingNode configs, string message)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topicName = configs.Where(k => k.Key.ToString() == "target_kafka_topic")?.FirstOrDefault().Value.ToString();
var config = new ProducerConfig {
BootstrapServers = brokerList,
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
try
{
/// Note: Awaiting the asynchronous produce request below prevents flow of execution
/// from proceeding until the acknowledgement from the broker is received (at the
/// expense of low throughput).
var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });
producer.Flush(TimeSpan.FromSeconds(10));
Console.WriteLine($"delivered to: {deliveryReport.TopicPartitionOffset}");
}
catch (ProduceException<string, string> e)
{
Console.WriteLine($"failed to deliver message: {e.Message} [{e.Error.Code}]");
}
}
}
#endregion
Am I doing something wrong here? The program existed immediately when executing var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });, no error message, no error code.
In the meanwhile I used Python and config the same for Producer, it works well.
Run_ManualAssign(configs, cts.Token);
For this line in the Main function, you are calling async without await in a sync function. Thus the program exit immediately after this invoke started (not finished as it is async)
You could have 2 options
Use async Main function and add await in front of this invoke.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-7.1/async-main
If you really want to call async function in sync function
Run_ManualAssign(configs, ts.Token).ConfigureAwait(false).GetAwaiter().GetResult();
I solved this problem but I don't know why actually. I opened an issue here.
I am just trying to connect my device to cloud using Azure IOT Hub. But I am getting an error as below.
MessagingEntityNotFoundException: The messaging entity 'ihsuprodsgres029dednamespace:eventhub:iothub-ehub-' could not be found. TrackingId:4772b610-8ff3-4709-8ea9-ffcd5784fe1c_B4, SystemTracker:ihsuprodsgres029dednamespace:eventhub:iothub-ehub-sibeeshiot-176205-a588b66686~16383|team01, Timestamp:6/23/2017 3:07:54 PM TrackingId:41110b704d814af497fd9924da6714d8_G4, SystemTracker:gateway2, Timestamp:6/23/2017 3:07:55 PM, referenceId: 41110b704d814af497fd9924da6714d8_G4
Can you please help me with it, if you have ever faced the same issue. Below is the code I am trying.
static void Main(string[] args) {
Console.WriteLine("Receive messages. Ctrl-C to exit.\n");
eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, iotHubD2cEndpoint);
var d2cPartitions = eventHubClient.GetRuntimeInformation().PartitionIds;
CancellationTokenSource cts = new CancellationTokenSource();
System.Console.CancelKeyPress += (s, e) = >{
e.Cancel = true;
cts.Cancel();
Console.WriteLine("Exiting...");
};
var tasks = new List < Task > ();
foreach(string partition in d2cPartitions) {
tasks.Add(ReceiveMessagesFromDeviceAsync(partition, cts.Token));
}
Task.WaitAll(tasks.ToArray());
}
private static async Task ReceiveMessagesFromDeviceAsync(string partition, CancellationToken ct) {
var eventHubReceiver = eventHubClient.GetConsumerGroup("Team01").CreateReceiver(partition, DateTime.UtcNow);
while (true) {
if (ct.IsCancellationRequested) break;
EventData eventData = await eventHubReceiver.ReceiveAsync();
if (eventData == null) continue;
string data = Encoding.UTF8.GetString(eventData.GetBytes());
Console.WriteLine("Message received. Partition: {0} Data: '{1}'", partition, data);
}
}
It looks like your iotHubD2cEndpoint value is not correct Event Hub-compatible name (probably you are using the messages/events such as an Azure IoT Hub Endpoint).
The following screen snippet shows the Event Hub compatible endpoint for Events:
The other option is to use an Azure IoT Hub Connection string and Endpoint Events, see the following example:
iotHubD2cEndpoint = "messages/events"
connectionString = "HostName=*****.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=******"
This might be a duplicate of this question but that's confused with talk about batching database updates and still has no proper answer.
In a simple example using Azure Service Bus queues, I can't access a BrokeredMessage after it's been placed on a queue; it's always disposed if I read the queue from another thread.
Sample code:
class Program {
private static string _serviceBusConnectionString = "XXX";
private static BlockingCollection<BrokeredMessage> _incomingMessages = new BlockingCollection<BrokeredMessage>();
private static CancellationTokenSource _cancelToken = new CancellationTokenSource();
private static QueueClient _client;
static void Main(string[] args) {
// Set up a few listeners on different threads
Task.Run(async () => {
while (!_cancelToken.IsCancellationRequested) {
var msg = _incomingMessages.Take(_cancelToken.Token);
if (msg != null) {
try {
await msg.CompleteAsync();
Console.WriteLine($"Completed Message Id: {msg.MessageId}");
} catch (ObjectDisposedException) {
Console.WriteLine("Message was disposed!?");
}
}
}
});
// Now set up our service bus reader
_client = GetQueueClient("test");
_client.OnMessageAsync(async (message) => {
await Task.Run(() => _incomingMessages.Add(message));
},
new OnMessageOptions() {
AutoComplete = false
});
// Now start sending
Task.Run(async () => {
int sent = 0;
while (!_cancelToken.IsCancellationRequested) {
var msg = new BrokeredMessage();
await _client.SendAsync(msg);
Console.WriteLine($"Sent {++sent}");
await Task.Delay(1000);
}
});
Console.ReadKey();
_cancelToken.Cancel();
}
private static QueueClient GetQueueClient(string queueName) {
var namespaceManager = NamespaceManager.CreateFromConnectionString(_serviceBusConnectionString);
if (!namespaceManager.QueueExists(queueName)) {
var settings = new QueueDescription(queueName);
settings.MaxDeliveryCount = 10;
settings.LockDuration = TimeSpan.FromSeconds(5);
settings.EnableExpress = true;
settings.EnablePartitioning = true;
namespaceManager.CreateQueue(settings);
}
var factory = MessagingFactory.CreateFromConnectionString(_serviceBusConnectionString);
factory.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(0.1), maxBackoff: TimeSpan.FromSeconds(30), maxRetryCount: 100);
var queueClient = factory.CreateQueueClient(queueName);
return queueClient;
}
}
I've tried playing around with settings but can't get this to work. Any ideas?
Answering my own question with response from Serkant Karaca # Microsoft here:
Very basic rule and I am not sure if this is documented. The received message needs to be processed in the callback function's life time. In your case, messages will be disposed when async callback completes, this is why your complete attempts are failing with ObjectDisposedException in another thread.
I don't really see how queuing messages for further processing helps on the throughput. This will add more burden to client for sure. Try processing the message in the async callback, that should be performant enough.
Bugger.