I'm experimenting with RabbitMQ and I hope someone can give me some hints.
I try to do the following: Processing events, which are time consuming to process, as quickly as possible with one consumer.
The test: 5 events and each one takes 2 seconds to process.
First attempt: 'Hello world' example from RabbitMQ (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html).
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost"};
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 5; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Thread.Sleep(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: Too slow because events are processed synchronously.
Second attempt: Asynchronous processing using AsyncEventingBasicConsumer.
Code changes:
DispatchConsumersAsync-property is set in the ConnectionFactory
AsyncEventingBasicConsumer is used
consumer.Received is async
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost", DispatchConsumersAsync = true };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 5; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += async (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
await Task.Delay(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: Too slow because events are still processed synchronously.
Third attempt: Async event handler for the consumer.
Code changes:
DispatchConsumersAsync-property is removed in the ConnectionFactory.
EventingBasicConsumer is used again
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost" };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 101; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new EventingBasicConsumer(channel);
consumer.Received += async (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
await Task.Delay(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: This is the result I want. The messages are processed very fast even when 1000 events are published.
But is this a proper solution for usage in a production environment?
I hope someone can help to clarify this or share their experience for processing events fast in a production environment.
Related
I wrote these code lines, but I couldn't figure out how I can print some values on console in consumer.receive. This code is working because I checked some values on RabbitMQ CloudAMQP, but I cannot see any changes on console.
Problem is here:
Console.WriteLine(Encoding.UTF8.GetString(e.Body.ToArray()) + " received");
Full code:
// publisher
ConnectionFactory factory = new ConnectionFactory();
factory.Uri = new Uri("amqps://guest:guest#localhost");
using (IConnection connection = factory.CreateConnection())
using (IModel channel = connection.CreateModel())
{
channel.ExchangeDeclare("kuyruk", type: ExchangeType.Fanout);
for (int i = 1; i <= 100; i++)
{
byte [] bytemessage = Encoding.UTF8.GetBytes($"is - {i}");
IBasicProperties properties = channel.CreateBasicProperties();
properties.Persistent = true;
channel.BasicPublish(exchange: "kuyruk", routingKey: "", basicProperties: properties, body: bytemessage);
}
}
// Consumer
ConnectionFactory factory = new ConnectionFactory();
factory.Uri = new Uri("amqps://guest:guest#localhost");
using (IConnection connection = factory.CreateConnection())
using (IModel channel = connection.CreateModel())
{
channel.ExchangeDeclare("kuyruk", type: ExchangeType.Fanout);
// Here consumer İçin Oluşturulacak Kuyruklara Random İsim Oluşturma
string queueName = channel.QueueDeclare().QueueName;
channel.QueueBind(queue: queueName, exchange: "kuyruk", routingKey: "");
channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
EventingBasicConsumer consumer = new EventingBasicConsumer(channel);
channel.BasicConsume(queueName, false, consumer);
consumer.Received += (sender, e) =>
{
Thread.Sleep(500);
Console.WriteLine(Encoding.UTF8.GetString(e.Body.ToArray()) + " received");
channel.BasicAck(e.DeliveryTag, false);
};
Console.Read();
}
First you have to create the queue. When you publish a message after this point, it will be queued. If you publish the message first, the message will disappear because there is no queue to send the message to.
Another bug in your code is this: you are creating a consumer channel. You define the action to be taken when the message arrives. Then, when your code run into to the publishing process below, you exit the using block which is wrap your consumer channel. For this reason, consumer is closed and cannot consume message.
//consumer
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
var factory = new ConnectionFactory
{
Uri = new Uri("amqp://guest:guest#localhost")
};
using (IConnection connection = factory.CreateConnection())
using (IModel consumeChannel = connection.CreateModel())
{
consumeChannel.ExchangeDeclare("kuyruk", type: ExchangeType.Fanout);
//#region Her Consumer İçin Oluşturulacak Kuyruklara Random İsim Oluşturma
string queueName = consumeChannel.QueueDeclare().QueueName;
consumeChannel.QueueBind(queue: queueName, exchange: "kuyruk", routingKey: "");
//#endregion
consumeChannel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
var consumer = new EventingBasicConsumer(consumeChannel);
consumeChannel.BasicConsume(queueName, false, consumer);
consumer.Received += (sender, e) =>
{
Thread.Sleep(500);
Console.WriteLine(Encoding.UTF8.GetString(e.Body.ToArray()) + " recieved");
consumeChannel.BasicAck(e.DeliveryTag, false);
};
//publisher
using (IModel publishChannel = connection.CreateModel())
{
publishChannel.ExchangeDeclare("kuyruk", type: ExchangeType.Fanout);
for (int i = 1; i <= 100; i++)
{
var str = $"is - {i}";
byte[] byteMessage = Encoding.UTF8.GetBytes(str);
IBasicProperties properties = publishChannel.CreateBasicProperties();
properties.Persistent = true;
publishChannel.BasicPublish(exchange: "kuyruk", routingKey: "", basicProperties: properties, body: byteMessage);
}
}
Console.Read();
}
Following is the method to receive the message on RabbitMq Queue. Method works fine but the unit testing this method has problem mentioned below
public void GetMessage(Action<string> action)
{
//create a connection factory
var factory = new ConnectionFactory();
if (_connectionString != null)
{
//assign connection string of rabbitmq
factory.Uri = new Uri(_connectionString);
}
//create connection and channel from connectionfactory
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
//declare the queque on the channel
channel.QueueDeclare(queue: _queque,
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
//create consumer using the channel
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
//When message is recieved on the consumer this will be triggered
var body = ea.Body.ToArray();
_message = Encoding.UTF8.GetString(body);
//call the method passed by the caller method
action(_message);
};
//add the consumer on this channel
channel.BasicConsume(queue: _queque,
autoAck: true,
consumer: consumer);
Console.ReadLine();
}
}
How to write a unit test test above message. Below unit test loading forever inside the call back method and the execution never goes to Assert.Equal line.
[Fact]
public async Task AddMessageAsync_ShouldAddToQueue()
{
//Arrange
string messageToSend = "Test";
string recievedMessage = null;
var queue = GetQueueManager();
queue.Message = messageToSend;
await queue.AddMessageAsync();
//Act
queue.GetMessage((string message) =>
{
//Hit here and recieved the message but does not exist from this method
recievedMessage = message;
});
//Assert
Assert.Equal(messageToSend, recievedMessage);
}
One way to adress the issue would be a synchronization barrier.
Here is an example: (There may be more efficient ways, though.)
... // Your code with some changes:
//create connection and channel from connectionfactory
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
//declare the queque on the channel
channel.QueueDeclare(queue: _queque,
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
// Set up a synchronization barrier.
using (var barrier = new ManualResetEventSlim(false))
{
//create consumer using the channel
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
//When message is recieved on the consumer this will be triggered
var body = ea.Body.ToArray();
_message = Encoding.UTF8.GetString(body);
try // you don't want "injected" code to break yours.
{
//call the method passed by the caller method
action(_message);
}
catch(Exception)
{
// at least log it!
}
finally
{
barrier.Set(); // Signal Event fired.
}
};
//add the consumer on this channel
channel.BasicConsume(queue: _queque,
autoAck: true,
consumer: consumer);
barrier.Wait(); // Wait for Event to fire.
}
}
I would imagine that the RabbitMQ API has had an update to support async/await instead of event-based? If so, then using that would of course be preferable.
If not: You may also want to explore Tasks and the Event-based Asynchronous Pattern (EAP)
I am just playing with RabbitMQ and trying to get a test sender and receiver set up in two C# projects.
TestSender.cs
using System;
using RabbitMQ.Client;
using System.Text;
public class TestSender
{
public TestSender()
{
}
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "Test Queue",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
Console.WriteLine(" Press [S] to send a message, [Enter] to exit.");
ConsoleKey key;
int messageId = 0;
while (true)
{
key = Console.ReadKey(true).Key;
if (key == ConsoleKey.Enter)
break;
if (key == ConsoleKey.S)
{
string message = "Message " + (++messageId).ToString();
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: "",
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine("Sent {0}", message);
}
}
}
}
}
}
TestReceiver.cs
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System;
using System.Text;
public class TestReceiver
{
public TestReceiver()
{
}
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// Declare the queue here because we might start the consumer before the publisher
// so the queue should exist before we try to consume messages from it.
channel.QueueDeclare(queue: "Test Queue",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
Console.WriteLine(" Press [R] to receive a message, [Enter] to exit.");
// Register a consumer to listen to a specific queue.
channel.BasicConsume(queue: "Test Queue",
autoAck: true,
consumer: consumer);
ConsoleKey key;
while (true)
{
key = Console.ReadKey(true).Key;
if (key == ConsoleKey.Enter)
break;
if (key == ConsoleKey.R)
{
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("Received {0}", message);
};
}
}
}
}
}
}
I set it up in such a way that if you press the S key, it sends a message to the queue and if you press the R key, it reads from the queue. The sending works but the receiving does absolutely nothing. I removed the while loop and keypress code and the receiving works. However, I would like to figure out how to get messages consumed one by one and not all at once. In addition, does anyone know if RabbitMQ can persist messages in the queue or do they have to be dequeued when consumed?
RabbitMQ sends a new message to consumer/subscriber when you Acknowledge the current message. RabbitMQ will persist message in the queue(durable mode only) until not acknowledged.
Remove autoAck: true options from channel.BasicConsume function. use channel.BasicAck function to ack a message when R key pressed.
I'm trying to consume messages from a RabbitMq queue on button click.
When I put this code in a console app it works but when I am using a button click event it won't work.
I hope you can help me.
This is the server code:
public void PushToQueue(Message message)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
string messageToSend = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(messageToSend);
//var properties = channel.CreateBasicProperties();
//properties.Persistent = true;
channel.BasicPublish(exchange: "",
routingKey: "task_queue",
basicProperties: null,
body: body);
Debug.WriteLine(" [x] Verzonden {0}", messageToSend);
}
}
This is the client code:
public void GetFromQueue()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "task_queue",
autoAck: true,
consumer: consumer);
}
}
This is te button click code:
private void buttonGetFromQueue_Click(object sender, EventArgs e)
{
rabbitManager.GetFromQueue();
}
I stepped trough the code and it seems that it won't reach this part:
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
This is the rabbit manager class:
public class RabbitMqManager
{
public List<Message> allMessages = new List<Message>();
public RabbitMqManager()
{
}
public void GetFromQueue()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "task_queue",
autoAck: true,
consumer: consumer);
}
}
public List<Message> ReturnAllMessages()
{
List<Message> returnList = new List<Message>();
foreach(Message m in allMessages)
{
returnList.Add(m);
}
return returnList;
}
}
The messages get pushed to the queue but I can't get them out of it.
To Retrive Message from RabbitMQ we need to first connect to the RabbitMQ server:
public WebClient GetRabbitMqConnection(string userName, string password)
{
var client = new WebClient();
client.Credentials = new NetworkCredential(userName, password);
return client;
}
Now retrieve messages from RabbitMQ using:
public string GetRabbitMQMessages(string domainName, string port, string
queueName, string virtualHost, WebClient client, string
methodType)
{
string messageResult = string.Empty;
string strUri = "http://" + domainName + ":" + port +
"/api/queues/" + virtualHost + "/";
var data = client.DownloadString(strUri + queueName + "/");
var queueInfo = JsonConvert.DeserializeObject<QueueInfo>(data);
if (queueInfo == null || queueInfo.messages == 0)
return string.Empty;
if (methodType == "POST")
{
string postbody = "
{\"ackmode\":\"ack_requeue_true\",\"count\":
\"$totalMessageCount\",\"name\":\"${DomainName}\",
\"requeue\":\"false\",\"encoding\":\"auto\",\"vhost\" :
\"${QueueName}\"}";
postbody = postbody.Replace("$totalMessageCount",
queueInfo.messages.ToString()).Replace("${DomainName}",
domainName).Replace("${QueueName}", queueName);
messageResult = client.UploadString(strUri + queueName +
"/get", "POST", postbody);
}
return messageResult;
}
I think this will help you to implement RabbitMQ.
I'm working through my first steps with RabbitMQ and have a question on why this didn't work.
The basic tutorial (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html) has one executable to send to the broker and another to receive it.
I ran this code through a single console app in Visual Studio and couldn't receive any messages.
If I take the "Receive" code and put it into a separate console app and open that, I get the message (no other code changes).
Can someone explain why I'm not able to have both in the same process? I had figured that the connection factory would handle independent connections accordingly regardless of whether it was the same process or not.
For the sake of completeness (though I doubt it's required), here's the code that didn't work until I pulled out the "Receiver" code and put it into it's own console app:
class Program
{
static void Main(string[] args) {
Receiver.Receive();
Console.WriteLine("receiver set up");
System.Threading.Thread.Sleep(5000);
Console.WriteLine("sending...");
Test.Send();
// can also reverse order of send/receive methods, same result
Console.ReadKey();
}
}
public class Receiver
{
public static void Receive() {
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) => {
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
System.Diagnostics.Debug.WriteLine("=====================");
System.Diagnostics.Debug.WriteLine(message);
System.Diagnostics.Debug.WriteLine("=====================");
};
channel.BasicConsume("hello", true, consumer);
}
}
}
}
public class Test
{
public static void Send() {
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
channel.QueueDeclare("hello", false, false, false, null);
string message = "Check it!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", "hello", null, body);
}
}
}
}
I think code has a problem. When you send message then Receive connection and channel already dead (Dispose called).
Standard code from RabbitMQ site:
using(var connection = factory.CreateConnection())
using(var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "hello",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "hello",
noAck: true,
consumer: consumer);
Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();
}
connection and channel are yet alive when client received message.