We Keep Coding

Kafaka Subscriber not receiving the messages

I am writing sample applications in .Net core to interact with Kafka.
I have downloaded Kafka and Zookeeper official docker images to my machine.
I am using Confluent.Kafka nuget package for both producer and consumer. I am able to produce the message to Kafka. But my consumer part is not working.
Below is my producer and consumer code snippet. I am not sure what mistake I'm doing here.
Do we need to Explicitly create a consumer group?
Consumer Code (This code not working. Thread waiting at consumer.Consume(cToken);)
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "myroupidd",
AutoOffsetReset = AutoOffsetReset.Earliest,
};
var cToken = ctokenSource.Token;
var consumerBuilder = new ConsumerBuilder<Null, string>(config);
consumerBuilder.SetPartitionsAssignedHandler((consumer, partitionlist) =>
{
consumer.Assign(new TopicPartition("myactual-toppics", 0) { });
Console.WriteLine("inside SetPartitionsAssignedHandler action");
});
using var consumer = consumerBuilder.Build();
consumer.Subscribe("myactual-toppics");
while (!cToken.IsCancellationRequested)
{
var consumeResult = consumer.Consume(cToken);
if (consumeResult.Message != null)
Console.WriteLine(consumeResult.Message.Value);
}
Producer Code (This is working fine. I am able to see the messages using Conduckto Tool)
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
ClientId = Dns.GetHostName(),
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
while (!stoppingToken.IsCancellationRequested)
{
var top = new TopicPartition("myactual-toppics", 0);
var result = await producer.ProduceAsync(top, new Message<Null, string> { Value = "My First Message" });
Console.WriteLine($"Publishedss1234" );
await Task.Delay(5000, stoppingToken);

How can I create a topic on Event Hub/Kafka using the AdminClient?

I'm trying to create a topic (an Event Hub) programmatically from the Kafka interface using AdminClient.CreateTopicsAsync. This works when connecting to Kafka, but not to Event Hub. I'm running into the following error:
Default partition count (KIP-464) not supported by broker, requires
broker version <= 2.4.0
using Confluent.Kafka;
using Confluent.Kafka.Admin;
var adminClient =
new AdminClientBuilder(
new[] {
("sasl.mechanism","PLAIN"),
("security.protocol","SASL_SSL"),
("bootstrap.servers", Address),
("sasl.username", "$ConnectionString"),
("sasl.password", ConnectionString),
}.Select((kvp) => new KeyValuePair<string, string>(kvp.Item1, kvp.Item2))
)
.Build();
await adminClient.CreateTopicsAsync(new[] {
new TopicSpecification {
Name = "test-topic"
}
});
It complains that using a default number of partitions is not supported, but as far as I can tell, I can't provide one as the underlying librdkafka does not support it.
The only information I could find by googling this is that someone in 2021 did make it work.

This code works for me on both Kafka and EventHub.
using (var kafkaProducer = new ProducerBuilder<string, string>(producerConfig)
.Build())
{
using (var adminClient = new DependentAdminClientBuilder(kafkaProducer.Handle).Build())
{
var metaData = adminClient.GetMetadata(TimeSpan.FromSeconds(5));
var topicInfo = metaData.Topics.Where(tp => string.Equals(fullTopicName, tp.Topic, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
if (topicInfo == null)
{
var t = new Confluent.Kafka.Admin.TopicSpecification
{
Name = fullTopicName,
// at least 2 partitions
NumPartitions = kafkaTestConfig.CreateTopicOptions.NumPartitions, // 3, //1,
//// at least 1 replication factor
ReplicationFactor = kafkaTestConfig.CreateTopicOptions.ReplicationFactor, // 3, //(short)numberOfBrokers,
Configs = kafkaTestConfig.CreateTopicOptions.DynamicConfigs,
};
var o = new CreateTopicsOptions { OperationTimeout = TimeSpan.FromMilliseconds(_timeout), };
AssertES.True(adminClient.CreateTopicsAsync(new List<Confluent.Kafka.Admin.TopicSpecification> { t, }
, o).Wait(_timeout), "Failed to create topic in time: " + fullTopicName);
}
}
}

Task WhenAll Method Get List of Completed Tasks for Azure ServiceBus Send Bulk Messages

I have written a Code below to send Bulk Messages to Azure Service Bus but I am getting Error in between and I wanted to know the list of Ids that are Pushed to Azure Service Bus.
public async Task ProcessProfile(NIFFUNDbContext dbContext)
{
List<Task> concurrentTasks = new List<Task>();
queueClient = new QueueClient(sbConnectionString, sbQueueName, ReceiveMode.ReceiveAndDelete, RetryPolicy.Default);
List<int> personIds = new List<int>();
for (int i = 0; i < 10; i++)
{
queueClient.OperationTimeout = TimeSpan.FromSeconds(10);
var list = dbContext.TpPerson.Include("TpPersonContactInformation").Take(10000).ToList();
foreach (var item in list)
{
var dfObjectMessage = NIF.FUN.Framework.DAL.MSSQL.Converter.TransformToDFMessage(item, ISDObjectTypeEnum.TpPerson, EntityChangeStateEnum.Added, DateTime.Now);
var message = SB.Converter.ToMessage(dfObjectMessage);
concurrentTasks.Add(queueClient.SendAsync(message));
}
await Task.WhenAll(concurrentTasks);
//await queueClient.CloseAsync();
}
}

var concurrentTasks = new List<Task>();
var succeededMessages = new ConcurrentBag<your entity type>();
var failedMessages = new ConcurrentBag<your entity type>();
for (int i = 0; i < 10; i++) // just for the sake of showing iteration
{
var entity = <your logic of getting entity>;
concurrentTasks.Add(Task.Run(async () =>
{
try
{
var message = <message from your entity>;
await queueClient.SendAsync(message);
succeededMessages.Add(entity);
}
catch
{
failedMessages.Add(entity);
}
}));
}
await Task.WhenAll(concurrentTasks);

you can utilize the following snippet to send messages in batch so that if it fails it fails for the entire batch you dont have to worry about which message is failed and its also the suggested approach to send messages in batch.
var messagelist = new List<Message>();
messagelist.Add(new Message(Encoding.ASCII.GetBytes("Test1")));
messagelist.Add(new Message(Encoding.ASCII.GetBytes("Test2")));
var sender = new QueueClient("connectionstring","entitypath");
sender.SendAsync(messagelist);

ZeroMQ Broadcast with Proxy losing messages

I am doing some performance tests on ZeroMQ in order to compare it with others like RabbitMQ and ActiveMQ.
In my broadcast tests and to avoid "The Dynamic Discovery Problem" as referred by ZeroMQ documentation I have used a proxy. In my scenario, I am using 50 concurrent publishers each one sending 500 messages with 1ms delay between sends. Each message is then read by 50 subscribers. And as I said I am losing messages, each of the subscribers should receive a total of 25000 messages and they are each receiving between 5000 and 10000 messages only.
I am using Windows and C# .Net client clrzmq4 (4.1.0.31).
I have already tried some solutions that I found on other posts:
I have set linger to TimeSpan.MaxValue
I have set ReceiveHighWatermark to 0 (as it is presented as infinite, but I have tried also Int32.MaxValue)
I have set checked for slow start receivers, I made receivers start some seconds before publishers
I had to make sure that no garbage collection is made to the socket instances (linger should do it but to make sure)
I have a similar scenario (with similar logic) using NetMQ and it works fine. The other scenario does not use security though and this one does (and that's also the reason why I use clrzmq in this one because I need client authentication with certificates that is not yet possible on NetMQ).
EDIT:
public class MCVEPublisher
{
public void publish(int numberOfMessages)
{
string topic = "TopicA";
ZContext ZContext = ZContext.Create();
ZSocket publisher = new ZSocket(ZContext, ZSocketType.PUB);
//Security
// Create or load certificates
ZCert serverCert = Main.GetOrCreateCert("publisher");
var actor = new ZActor(ZContext, ZAuth.Action, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
publisher.CurvePublicKey = serverCert.PublicKey;
publisher.CurveSecretKey = serverCert.SecretKey;
publisher.CurveServer = true;
publisher.Linger = TimeSpan.MaxValue;
publisher.ReceiveHighWatermark = Int32.MaxValue;
publisher.Connect("tcp://127.0.0.1:5678");
Thread.Sleep(3500);
for (int i = 0; i < numberOfMessages; i++)
{
Thread.Sleep(1);
var update = $"{topic} {"message"}";
using (var updateFrame = new ZFrame(update))
{
publisher.Send(updateFrame);
}
}
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure publisher is not garbage collected
ulong Affinity = publisher.Affinity;
}
}
public class MCVESubscriber
{
private ZSocket subscriber;
private List<string> prints = new List<string>();
public void read()
{
string topic = "TopicA";
var context = new ZContext();
subscriber = new ZSocket(context, ZSocketType.SUB);
//Security
ZCert serverCert = Main.GetOrCreateCert("xpub");
ZCert clientCert = Main.GetOrCreateCert("subscriber");
subscriber.CurvePublicKey = clientCert.PublicKey;
subscriber.CurveSecretKey = clientCert.SecretKey;
subscriber.CurveServer = true;
subscriber.CurveServerKey = serverCert.PublicKey;
subscriber.Linger = TimeSpan.MaxValue;
subscriber.ReceiveHighWatermark = Int32.MaxValue;
// Connect
subscriber.Connect("tcp://127.0.0.1:1234");
subscriber.Subscribe(topic);
while (true)
{
using (var replyFrame = subscriber.ReceiveFrame())
{
string messageReceived = replyFrame.ReadString();
messageReceived = Convert.ToString(messageReceived.Split(' ')[1]);
prints.Add(messageReceived);
}
}
}
public void PrintMessages()
{
Console.WriteLine("printing " + prints.Count);
}
}
public class Main
{
static void Main(string[] args)
{
broadcast(500, 50, 50, 30000);
}
public static void broadcast(int numberOfMessages, int numberOfPublishers, int numberOfSubscribers, int timeOfRun)
{
new Thread(() =>
{
using (var context = new ZContext())
using (var xsubSocket = new ZSocket(context, ZSocketType.XSUB))
using (var xpubSocket = new ZSocket(context, ZSocketType.XPUB))
{
//Security
ZCert serverCert = GetOrCreateCert("publisher");
ZCert clientCert = GetOrCreateCert("xsub");
xsubSocket.CurvePublicKey = clientCert.PublicKey;
xsubSocket.CurveSecretKey = clientCert.SecretKey;
xsubSocket.CurveServer = true;
xsubSocket.CurveServerKey = serverCert.PublicKey;
xsubSocket.Linger = TimeSpan.MaxValue;
xsubSocket.ReceiveHighWatermark = Int32.MaxValue;
xsubSocket.Bind("tcp://*:5678");
//Security
serverCert = GetOrCreateCert("xpub");
var actor = new ZActor(ZAuth.Action0, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
xpubSocket.CurvePublicKey = serverCert.PublicKey;
xpubSocket.CurveSecretKey = serverCert.SecretKey;
xpubSocket.CurveServer = true;
xpubSocket.Linger = TimeSpan.MaxValue;
xpubSocket.ReceiveHighWatermark = Int32.MaxValue;
xpubSocket.Bind("tcp://*:1234");
using (var subscription = ZFrame.Create(1))
{
subscription.Write(new byte[] { 0x1 }, 0, 1);
xpubSocket.Send(subscription);
}
Console.WriteLine("Intermediary started, and waiting for messages");
// proxy messages between frontend / backend
ZContext.Proxy(xsubSocket, xpubSocket);
Console.WriteLine("end of proxy");
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure xpubSocket and xsubSocket are not garbage collected
ulong Affinity = xpubSocket.Affinity;
int ReceiveHighWatermark = xsubSocket.ReceiveHighWatermark;
}
}).Start();
Thread.Sleep(5000); //to make sure proxy started
List<MCVESubscriber> Subscribers = new List<MCVESubscriber>();
for (int i = 0; i < numberOfSubscribers; i++)
{
MCVESubscriber ZeroMqSubscriber = new MCVESubscriber();
new Thread(() =>
{
ZeroMqSubscriber.read();
}).Start();
Subscribers.Add(ZeroMqSubscriber);
}
Thread.Sleep(10000);//to make sure all subscribers started
for (int i = 0; i < numberOfPublishers; i++)
{
MCVEPublisher ZeroMqPublisherBroadcast = new MCVEPublisher();
new Thread(() =>
{
ZeroMqPublisherBroadcast.publish(numberOfMessages);
}).Start();
}
Thread.Sleep(timeOfRun);
foreach (MCVESubscriber Subscriber in Subscribers)
{
Subscriber.PrintMessages();
}
}
public static ZCert GetOrCreateCert(string name, string curvpath = ".curve")
{
ZCert cert;
string keyfile = Path.Combine(curvpath, name + ".pub");
if (!File.Exists(keyfile))
{
cert = new ZCert();
Directory.CreateDirectory(curvpath);
cert.SetMeta("name", name);
cert.Save(keyfile);
}
else
{
cert = ZCert.Load(keyfile);
}
return cert;
}
}
This code also produces the expected number of messages when security is disabled, but when turned on it doesn't.
Does someone know another thing to check? Or has it happened to anyone before?
Thanks

How to trigger some functionality for incoming messages on RabbitMQ

Whenever rabbitMq Receives any messages it should trigger some functionality.
How can i achieve this ?

We can use receive event to do the functionality.
Below code sample shows how to do the same.
ConnectionFactory oFactory = new ConnectionFactory();
oFactory.UserName = oRabbitMQData.userName;
oFactory.Password = oRabbitMQData.password;
oFactory.VirtualHost = oRabbitMQData.virtualHost;
oFactory.HostName = oRabbitMQData.hostName;
oFactory.Port = oRabbitMQData.port;
var connection = oFactory.CreateConnection();
var channel = connection.CreateModel();
// setup signal
var signal = new ManualResetEvent(true);
oRabbitMQConnections.Add(connection);
oRabbitMQChannels.Add(channel);
var consumer = new EventingBasicConsumer(channel);
byte[] messageBody = null;
consumer.Received += (sender, args) =>
{
messageBody = args.Body;
string sMsg = UTF8Encoding.UTF8.GetString(messageBody);
// process your message or store for later
// set signal
channel.BasicAck(args.DeliveryTag, false);
// Do something
signal.Set();
};
// start consuming
channel.BasicConsume(oRabbitMQData.queueName, false, consumer);