We Keep Coding

M2Mqtt in C# is not connected, doesn't give data

I have a project creating a websocket client side (Subscriber) to a MQTT publisher. I am quite new to C# and MQTT protocol. I follow some youtube video to make finish my very first lines connecting to this MQTT publisher to get all the train going in and out Helsinki station.
broker: "rata.digitraffic.fi"
Port: 80
Topic: trains-by-station/HKI (HKI abbr for Helsinki)
I use M2Mqtt library in dotnet to build the subscriber, somehow the client_MqttMsgPublishReceived function is never triggered. the client.IsConnected always returned false value!
You can find info of this mqtt protocol in the url below.
https://www.digitraffic.fi/rautatieliikenne/#websocket-mqtt
It gives me example in JavaScripts and it seems to run fine with the example of each MQTT. But when I tried to do it with my PC, it doesn't give me any thing, but
Hello World!!!
False
and the cmd window on hold.
SOOOO FRUSTRATING right now. it would be much appreciate if anyone can help me out.
BTW, I am using win10, I tried with dotnet 4/5/6 and m2mqtt 4.3.0.
using System.Text;
using uPLibrary.Networking.M2Mqtt;
using uPLibrary.Networking.M2Mqtt.Messages;
using System;
namespace m2qttSubscriber
{
internal class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!!!!");
MqttClient client = new MqttClient("rata.digitraffic.fi",
80,
false,
MqttSslProtocols.None,
null,
null);
client.MqttMsgPublishReceived += client_MqttMsgPublishReceived;
string clientID = "myclientid_" + RandomDigits(4);
client.Connect(clientID);
Console.WriteLine(client.IsConnected);
client.Subscribe(new string[] { "trains-by-station/HKI" },
new byte[] { MqttMsgBase.QOS_LEVEL_AT_MOST_ONCE}) ;
}
static void client_MqttMsgPublishReceived(object sender, MqttMsgPublishEventArgs e)
{
Console.WriteLine("SOme thing is received");
string payload = Encoding.Default.GetString(e.Message);
Console.WriteLine("Recevied {0} from", payload);
}
static public string RandomDigits(int length)
{
var random = new Random();
string s = string.Empty;
for (int i = 0; i < length; i++)
s = String.Concat(s, random.Next(10).ToString());
return s;
}
}
}

As per the comments the library used (uPLibrary.Networking.M2Mqtt) does not support MQTT over Websockets (which is what rata.digitraffic.fi:80 offers).
If you are able to use standard MQTT (over TCP/IP) then rata-mqtt.digitraffic.fi:1883 works (OP succeeded with MqttClient client = new MqttClient("rata-mqtt.digitraffic.fi", 1883, false, MqttSslProtocols.None, null, null);) and this is generally preferable to using WebSockets (there are some situations where you have to use WebSockets; e.g. code running in a browser or to bypass some filters/proxies).
Alternatively there are other libraries that do offer support for MQTT over Websockets.

NetMQ (ZeroMQ) how to make "Brokerless Reliability (Freelance Pattern)" works

I facing some problems with the example I got from the ZeroMQ Guide, looks like the class ZSocket and ZContext doesn't exist.
I'm totally new with ZeroMQ (just start lo learn) and I'm following the "ØMQ - The Guide". The first example about REQ-REP, which is very simple, worked well. But now I'm trying something more similar to my objective, the "Brokerless Reliability (Freelance Pattern)" and this one didn't work.
I'm using Visual Studio 2019 with C# code, I created a new project, added NetMQ V4.0.1.6 via Nuget and copied the server code to my project. I got errors with ZContext and ZSocket. I already check the API V3 and API V4, they are clear different. The guide is totally based on version 3 and I'm using V 4. I didn't find any document about the changes or updates or equivalent function/classes/methods and I don't know how to convert the example to the NetMQ V4.
This is my test code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using NetMQ;
namespace Examples
{
static partial class Program
{
public static void FLServer1(string[] args)
{
//
// Freelance server - Model 1
// Trivial echo service
//
// Author: metadings
//
if (args == null || args.Length < 1)
{
Console.WriteLine();
Console.WriteLine("Usage: ./{0} FLServer1 [Endpoint]", AppDomain.CurrentDomain.FriendlyName);
Console.WriteLine();
Console.WriteLine(" Endpoint Where FLServer1 should bind on.");
Console.WriteLine(" Default is tcp://127.0.0.1:7780");
Console.WriteLine();
args = new string[] { "tcp://127.0.0.1:7780" };
}
using (var context = new ZContext())
using (var server = new ZSocket(context, ZSocketType.REP))
{
server.Bind(args[0]);
Console.WriteLine("I: echo service is ready at {0}", args[0]);
ZMessage message;
ZError error;
while (true)
{
if (null != (message = server.ReceiveMessage(out error)))
{
using (message)
{
server.Send(message);
}
}
else
{
if (error == ZError.ETERM)
return; // Interrupted
throw new ZException(error);
}
}
}
}
}
}

After long hours trying to understand that logic, I found a list of differences from ZeroMQ V3 and V4:
https://github.com/zeromq/netmq/wiki/Migrating-to-v4
Also, accidentally I found the example I was looking for:
https://github.com/NetMQ/Samples/tree/master/src/Brokerless%20Reliability%20(Freelance%20Pattern)/Model%20One

How to use multiple consumers in different programming language for same group ID in Kafka

I wanted to create a load balancing in Kafka (multiple programming languages) for a topic. So I did the following.
Created a topic with 4 partitions.
Created a producer in C# (producing messages every second)
Created one consumer(consumer1) in C# (consumer group: testConsumerGrp)
Created one more consumer(consumer2) in NodeJs (consumer group: testConsumerGrp)
I used confluent.kafka in C# and kafkajs in NodeJs.
I Open the producer and keep it running.
If I run only C# consumer, it works fine.
If I run only NodeJs consumer, it works fine.
If I run multiple C# consumer (only c# and less than 4 instances), it works fine.
If I run multiple NodeJs consumer (only NodeJs and less than 4 instances), it works fine.
If I run one C# and one NodeJs consumer then I am getting Inconsistent group protocol error
Can't we use two programming languages for a same consumer group?
Producer in C# - windows form
using System;
using System.Collections.Generic;
using System.Windows.Forms;
using Confluent.Kafka;
namespace KafkaProducer
{
public partial class frmProducer : Form
{
const string TOPIC = "testTopic";
private IProducer<Null, string> pBuilder;
public frmProducer()
{
InitializeComponent();
}
private async void timer1_Tick(object sender, EventArgs e)
{
try
{
// instead of sending some value, we send current DateTime as value
var dr = await pBuilder.ProduceAsync(TOPIC, new Message<Null, string> { Value = DateTime.Now.ToLongTimeString() });
// once done, add the value into list box
listBox1.Items.Add($"{dr.Value} - Sent to Partition: {dr.Partition.Value}");
listBox1.TopIndex = listBox1.Items.Count - 1;
}
catch (ProduceException<Null, string> err)
{
MessageBox.Show($"Failed to deliver msg: {err.Error.Reason}");
}
}
private void frmProducer_Load(object sender, EventArgs e)
{
ProducerConfig config = new ProducerConfig { BootstrapServers = "localhost:9092" };
pBuilder = new ProducerBuilder<Null, string>(config).Build();
timer1.Enabled = true;
}
private void frmProducer_FormClosing(object sender, FormClosingEventArgs e)
{
timer1.Enabled = false;
pBuilder.Dispose();
}
}
}
Consumer in C# - windows form
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
using Confluent.Kafka;
namespace KafkaConsumer
{
public partial class frmConsumer : Form
{
CancellationTokenSource cts = new CancellationTokenSource();
public frmConsumer()
{
InitializeComponent();
}
private void StartListen()
{
var conf = new ConsumerConfig
{
GroupId = "test-consumer-group",
BootstrapServers = "localhost:9092",
AutoOffsetReset = AutoOffsetReset.Earliest
};
using (var c = new ConsumerBuilder<Ignore, string>(conf).Build())
{
c.Subscribe("testTopic");
//TopicPartitionTimestamp tpts = new TopicPartitionTimestamp("testTopic", new Partition(), Timestamp. )
//c.OffsetsForTimes()
try
{
while (true)
{
try
{
var cr = c.Consume(cts.Token);
// Adding the consumed values into the UI
listBox1.Invoke(new Action(() =>
{
listBox1.Items.Add($"{cr.Value} - from Partition: {cr.Partition.Value}" );
listBox1.TopIndex = listBox1.Items.Count - 1;
}));
}
catch (ConsumeException err)
{
MessageBox.Show($"Error occured: {err.Error.Reason}");
}
}
}
catch (OperationCanceledException)
{
// Ensure the consumer leaves the group cleanly and final offsets are committed.
c.Close();
}
}
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
cts.Cancel();
}
private async void frmConsumer_Load(object sender, EventArgs e)
{
await Task.Run(() => StartListen());
}
}
}
Consumer in NodeJs
const { Kafka } = require("kafkajs");
const kafka = new Kafka({
clientId: 'my-app',
brokers: ["localhost:9092"]
});
const consumer = kafka.consumer({ groupId: "test-consumer-group" });
const run = async () => {
// Consuming
await consumer.connect();
await consumer.subscribe({ topic: "testTopic", fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log(message.value.toString() + " - from Partition " + partition);
}
});
};
run().catch(console.error);
If I run C# and NodeJs consumer at same time then getting Inconsistent group protocol error.
How to use multiple consumer from different programming languages in Kafka?

Short answer:
This may not have as much to do with the different languages as you might think. This is happening due to the differences in the protocols of the 2 consumer clients (and their libraries).
Try setting the following property in both the consumer clients:
partition.assignment.strategy = round-robin
Note: I've just supplied the general property so you'll need to look at the language specific versions for your clients. You could even set this to range but keep it consistent.
The explanation goes like this:
Reading through the protocol on Kafka's wiki to find out the root cause of Inconsistent group protocol - it turns out that this is returned when:
There is an active consumer group with active/running consumers
And a new consumer arrives to join this group with a protocol type (or a set of protocols) that is not compatible with that of the current group
Now, there could be various aspects in the ConsumerGroupProtocolMetadata but one of the aspects that does seem to differ in the libraries of the clients that you're using is the partition.assignment.strategy.
The dotnet client is a wrapper around librdkafka defaults the value of the above property to range. Here's the reference.
where as
kafkajs as per the documentation defaults it to round-robin - hence causing the inconsistency.
Hope this helps.

I know this comes one year too late but this happens because of the same group naming
When you start the C# client it creates a group for its consumers.
E.g. group-1 (group-1-consumer-1,group-1-consumer-2, etc) - These names are automatically allocated so don't bother. I think you can set these manually but is not recommended to avoid potential name collision.
Now, when you set this in motion you cannot add the same group from a different group runner (from another microservice).
See what Lalit quoted from Kafka wiki:
There is an active consumer group with active/running consumers
Now, when you will start the nodeJs one, you should use a different group name as most likely will carry out other tasks with that data.
Yes, you can subscribe both groups to the same topics as Kafka will keep an offset for each group and where they left of.

What exception is thrown by ReadBufferAsync when there is no internet connectivity

When getting data from the net, how can I tell if I failed to connect?
I am using GetBufferAsync, which throws an error when I pull the ethernet cable out of the computer.
I can find no documentation about this error, and I don't know if this is the best method to use.
I am loading a text file that is up to 60Kb. Not very much data.
I am developing in Visual Studio 2019.
The code below loads some data from www.google.com, mostly 0's.
When I disconnect the ethernet cable, I get exception thrown at the ReadBufferAsync line.
I tried catching specific exceptions: InvalidOperationException, HttpRequestException
Error: 0x8002ee7 (decimal 12007).
Text: The text associated with this error code could not be found. The server name or address could not be resolved.
InnerException: null
The MS documentation https://learn.microsoft.com/en-us/windows/desktop/WinInet/wininet-errors does not include the error I'm getting, though my exception is in the correct range (see https://learn.microsoft.com/en-us/windows/desktop/Debug/system-error-codes--12000-15999-)
using System;
using Windows.Storage.Streams;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Windows.Web.Http;
namespace ReadBufferAsyncTest
{
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.Loaded += MyProgram_Loaded;
}
private async void MyProgram_Loaded(object sender, RoutedEventArgs e)
{
try
{
// Connecting to internet
var uriBing = new Uri("http://www.google.com");
var client = new HttpClient();
IBuffer morseBuffer = await client.GetBufferAsync(uriBing);
DataReader dataReader = DataReader.FromBuffer(morseBuffer);
byte[] morseBytes = new byte[morseBuffer.Length];
dataReader.ReadBytes(morseBytes);
}
catch(Exception ex)
{
return; // breakpoint goes here
}
}
}
}
Is the error I am getting (0x80072ee7) the expected error?
Are there other errors to look for?
What is the best way to handle the errors when there is limited documentation?
(other note: is this the best way to download a small text file?)

how can I tell if I failed to connect?
For your requirement, you could check if the internet is available before getting buffer via NetworkHelper.
// Detect if Internet can be reached
if (NetworkHelper.Instance.ConnectionInformation.IsInternetAvailable)
{
}

Why does a bound SUB receive only one message from a connecting PUB?

I'm making examples for my ZeroMQ CLR namespace, however I have a problem with PUB/SUB.
Why do I get only the first message? Sometimes I get no message, if I debug through the client (on PubSub_Client(arg);) I get some messages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Security.Cryptography;
using ZeroMQ;
namespace ZeroMQ.Test
{
static partial class Program
{
static string PubSub_FrontendAddress = "tcp://127.0.0.1:2772";
public static void Main(string[] args)
{
if (args == null || args.Length < 1)
{
// say here were some arguments...
args = new string[] { "World" };
}
// Setup the ZContext
context = ZContext.Create();
CancellationTokenSource cancellor0 = null;
{
// Create the "Server" cancellor and threads
cancellor0 = new CancellationTokenSource();
var serverThread = new Thread(PubSub_Server);
serverThread.Start(cancellor0.Token);
serverThread.Join(64);
}
{
Thread.Sleep(1000);
Console.WriteLine("Starting...");
// foreach arg we are the Client, asking the Server
foreach (string arg in args)
{
PubSub_Client(arg);
// Thread.Sleep(1000);
}
Console.WriteLine("Ended...");
}
if (cancellor0 != null)
{
// Cancel the Server
cancellor0.Cancel();
}
// we could have done here context.Terminate()
}
static void PubSub_Server(object cancelluS)
{
var cancellus = (CancellationToken)cancelluS;
using (var socket = ZSocket.Create(context, ZSocketType.SUB))
{
socket.Bind(PubSub_FrontendAddress);
socket.SubscribeAll();
/* var poller = ZPollItem.Create(socket, (ZSocket _socket, out ZMessage message, out ZError _error) =>
{
while (null == (message = _socket.ReceiveMessage(/* ZSocketFlags.DontWait, * out _error)))
{
if (_error == ZError.EAGAIN)
{
_error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(_error);
}
return true;
}); /**/
while (!cancellus.IsCancellationRequested)
{
ZError error;
ZMessage request;
/* if (!poller.TryPollIn(out request, out error, TimeSpan.FromMilliseconds(512)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
if (null == (request = socket.ReceiveMessage(ZSocketFlags.DontWait, out error)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
foreach (ZFrame frame in request)
{
string strg = frame.ReadString();
Console.WriteLine("{0} said hello!", strg);
}
}
socket.Unbind(PubSub_FrontendAddress);
}
}
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
using (var crypto = new RNGCryptoServiceProvider())
{
var identity = new byte[8];
crypto.GetBytes(identity);
socket.Identity = identity;
}
socket.Connect(PubSub_FrontendAddress);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
}
}

I'm having trouble with your design which seems just wrong:
A single subscriber and multiple publishers is an odd choice. I trust you have a good reason for it, but you should have said what that is. When sending messages from multiple clients to a single server, it is normal to use DEALER/ROUTER sockets instead. PUB/SUB is intended for a small set of publishers to a large number of subscribers.
A client that connects, sends one message, then immediately disconnects, is another very unusual use case that I hope is just an example:
For one thing, you are open to linger problems whereby the message will get dropped on the disconnect it is isn't sent within the linger timeout. [I don't know what the default linger is for your language binding, so that may or may not be an issue, but you should at least check to ensure that it isn't.]
For another, as you've already found, there are issues around the time it takes to connect to a socket, which may lead to PUB messages getting dropped if they are sent before the socket has properly connected.
If you insist on using PUB/SUB in this manner, you will need an out of band protocol to synchronise the PUB and SUB threads before the pub messages are sent. There are examples of how to do this reliable pub/sub in the zeromq guide. This will involve a second set of sockets in the same threads to send the synchronisation messages; DEALER sockets don't drop messages which is why they are suitable for that purpose...
But, DEALER/ROUTER sockets would appear to be a better choice than PUB/SUB unless there is some design requirement that hasn't been disclosed.

Well... There was a comment by Martin Sustrik: "The problem is that connecting is asynchronous and takes certain amount of time."
Now there is Thread.Sleep(64) - and it works...:
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
socket.Connect(PubSub_FrontendAddress);
Thread.Sleep(64);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
Do you know any better way to get the connection established?