We Keep Coding

RabbitMQ: Fast processing of time consuming events

I'm experimenting with RabbitMQ and I hope someone can give me some hints.
I try to do the following: Processing events, which are time consuming to process, as quickly as possible with one consumer.
The test: 5 events and each one takes 2 seconds to process.
First attempt: 'Hello world' example from RabbitMQ (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html).
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost"};
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 5; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Thread.Sleep(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: Too slow because events are processed synchronously.
Second attempt: Asynchronous processing using AsyncEventingBasicConsumer.
Code changes:
DispatchConsumersAsync-property is set in the ConnectionFactory
AsyncEventingBasicConsumer is used
consumer.Received is async
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost", DispatchConsumersAsync = true };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 5; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += async (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
await Task.Delay(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: Too slow because events are still processed synchronously.
Third attempt: Async event handler for the consumer.
Code changes:
DispatchConsumersAsync-property is removed in the ConnectionFactory.
EventingBasicConsumer is used again
static void Main(string[] args)
{
// Sender
var factory = new ConnectionFactory { HostName = "localhost" };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: false, arguments: null);
const string message = "Hello World!";
for (int i = 1; i <= 101; i++)
{
var body = Encoding.UTF8.GetBytes($"{message} {i}");
channel.BasicPublish(exchange: string.Empty,
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine($"Sent {message} {i}");
}
// Consumer
var consumer = new EventingBasicConsumer(channel);
consumer.Received += async (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
await Task.Delay(2000);
Console.WriteLine($"Received {message}");
};
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
Console.WriteLine("Press [enter] to exit.");
Console.ReadLine();
}
Conclusion: This is the result I want. The messages are processed very fast even when 1000 events are published.
But is this a proper solution for usage in a production environment?
I hope someone can help to clarify this or share their experience for processing events fast in a production environment.

C#, RabbitMq How can I send back the processed message

I just started learning RabbitMQ yesterday and I don't understand how it can be implemented.
Here is my producer
public class RabbitMQProducerService: IMessageSender
{
public void SendMessage<T>(T message)
{
var factory = new ConnectionFactory { HostName = "localhost" };
var connection = factory.CreateConnection();
var channel = connection.CreateModel();
channel.QueueDeclare("Input", exclusive: false);
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish(exchange: "", routingKey: "Input", basicProperties: null, body: body);
}
}
And my subscriber
string exepction = "";
StatisticOfTest statistic;
var factory = new ConnectionFactory
{
HostName = "localhost"
};
var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare("Input", exclusive: false);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, eventArgs) =>
{
SolutionService solution = new SolutionService();
var body = eventArgs.Body.ToArray();
var jsonStr = Encoding.UTF8.GetString(body);
SolutionAddModel solutionAdd = JsonConvert.DeserializeObject<SolutionAddModel>(jsonStr);
if (String.IsNullOrEmpty(solution.Validating(solutionAdd)))
{
statistic = solution.Create(solutionAdd); //I want to return statistic
}
else
{
exepction = solution.Validating(solutionAdd); // Or Exception
}
Console.WriteLine($"Message received: {jsonStr}");
};
channel.BasicConsume(queue: "Input", autoAck: true, consumer: consumer);
Console.ReadKey();
Based on the result, I want to return statistic or exception. But to another queue, for example, to some Output queue

consumer.Received += (model, eventArgs) => { ... }
The ... is the place when you declare what will happen, when a new message is received. The code there would be an event handler for a Received event.
In your case you want each new message to be published to a queue on some condition. Easy to do, just publish a message in your event handler code:
...
// Create a separate channel for sending messages to the output queue
var outputChannel = connection.CreateModel();
// Declare the output queue, you want to send messages to
outputChannel.QueueDeclare("Output", exclusive: false);
// Specify how the actual processing is going to happen
consumer.Received += (model, eventArgs) =>
{
var body = GetMessageBody(eventArgs.Body); // This is where you get new message content based on the received message.
outputChannel.BasicPublish(exchange: "", routingKey: "Input", basicProperties: null, body: body);
};
...
Also, looking at your producer code: you don't need to create a new connection, a new channel, and declare a queue every time you want to send a message. You could do it beforehand just once. It would really save RabbitMQ's resources.
RabbitMQ also has a connection limit, so sometime in the future your program would stop working. It is better if an application actually uses a single connection for everything, if possible, just spawning channels where needed.
Also, don;t forget, that IConnection and IModel are implementing IDisposable, so you should dispose of them properly.
This is how you should do it:
// 'using' tells the compiler to generate a Dispose call for this object at the end of the current block.
using var connection = connectionFactory.CreateConnection();
uisng var channel = connection.CreateModel();
channel.QueueDeclare("Input", exclusive: false);
for (var i = 0; i < 100l i++)
producer.SendMessage(i);

Get messages from RabbitMQ on button click

I'm trying to consume messages from a RabbitMq queue on button click.
When I put this code in a console app it works but when I am using a button click event it won't work.
I hope you can help me.
This is the server code:
public void PushToQueue(Message message)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
string messageToSend = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(messageToSend);
//var properties = channel.CreateBasicProperties();
//properties.Persistent = true;
channel.BasicPublish(exchange: "",
routingKey: "task_queue",
basicProperties: null,
body: body);
Debug.WriteLine(" [x] Verzonden {0}", messageToSend);
}
}
This is the client code:
public void GetFromQueue()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "task_queue",
autoAck: true,
consumer: consumer);
}
}
This is te button click code:
private void buttonGetFromQueue_Click(object sender, EventArgs e)
{
rabbitManager.GetFromQueue();
}
I stepped trough the code and it seems that it won't reach this part:
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
This is the rabbit manager class:
public class RabbitMqManager
{
public List<Message> allMessages = new List<Message>();
public RabbitMqManager()
{
}
public void GetFromQueue()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "task_queue",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Debug.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "task_queue",
autoAck: true,
consumer: consumer);
}
}
public List<Message> ReturnAllMessages()
{
List<Message> returnList = new List<Message>();
foreach(Message m in allMessages)
{
returnList.Add(m);
}
return returnList;
}
}
The messages get pushed to the queue but I can't get them out of it.

To Retrive Message from RabbitMQ we need to first connect to the RabbitMQ server:
public WebClient GetRabbitMqConnection(string userName, string password)
{
var client = new WebClient();
client.Credentials = new NetworkCredential(userName, password);
return client;
}
Now retrieve messages from RabbitMQ using:
public string GetRabbitMQMessages(string domainName, string port, string
queueName, string virtualHost, WebClient client, string
methodType)
{
string messageResult = string.Empty;
string strUri = "http://" + domainName + ":" + port +
"/api/queues/" + virtualHost + "/";
var data = client.DownloadString(strUri + queueName + "/");
var queueInfo = JsonConvert.DeserializeObject<QueueInfo>(data);
if (queueInfo == null || queueInfo.messages == 0)
return string.Empty;
if (methodType == "POST")
{
string postbody = "
{\"ackmode\":\"ack_requeue_true\",\"count\":
\"$totalMessageCount\",\"name\":\"${DomainName}\",
\"requeue\":\"false\",\"encoding\":\"auto\",\"vhost\" :
\"${QueueName}\"}";
postbody = postbody.Replace("$totalMessageCount",
queueInfo.messages.ToString()).Replace("${DomainName}",
domainName).Replace("${QueueName}", queueName);
messageResult = client.UploadString(strUri + queueName +
"/get", "POST", postbody);
}
return messageResult;
}
I think this will help you to implement RabbitMQ.

RabbitMQ Can't Send and Receive from same process

I'm working through my first steps with RabbitMQ and have a question on why this didn't work.
The basic tutorial (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html) has one executable to send to the broker and another to receive it.
I ran this code through a single console app in Visual Studio and couldn't receive any messages.
If I take the "Receive" code and put it into a separate console app and open that, I get the message (no other code changes).
Can someone explain why I'm not able to have both in the same process? I had figured that the connection factory would handle independent connections accordingly regardless of whether it was the same process or not.
For the sake of completeness (though I doubt it's required), here's the code that didn't work until I pulled out the "Receiver" code and put it into it's own console app:
class Program
{
static void Main(string[] args) {
Receiver.Receive();
Console.WriteLine("receiver set up");
System.Threading.Thread.Sleep(5000);
Console.WriteLine("sending...");
Test.Send();
// can also reverse order of send/receive methods, same result
Console.ReadKey();
}
}
public class Receiver
{
public static void Receive() {
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) => {
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
System.Diagnostics.Debug.WriteLine("=====================");
System.Diagnostics.Debug.WriteLine(message);
System.Diagnostics.Debug.WriteLine("=====================");
};
channel.BasicConsume("hello", true, consumer);
}
}
}
}
public class Test
{
public static void Send() {
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
channel.QueueDeclare("hello", false, false, false, null);
string message = "Check it!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", "hello", null, body);
}
}
}
}

I think code has a problem. When you send message then Receive connection and channel already dead (Dispose called).
Standard code from RabbitMQ site:
using(var connection = factory.CreateConnection())
using(var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "hello",
durable: false,
exclusive: false,
autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume(queue: "hello",
noAck: true,
consumer: consumer);
Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();
}
connection and channel are yet alive when client received message.

Why declare Exchange in RabbitMQ?

I am working on a project with RabbitMQ. My code is below.
Producer:
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", "hello", null, body);
Console.WriteLine(" [x] Sent {0}", message);
}
}
}
Consumer with Exchange declared:
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("hello", "direct",false, false, false, null);
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
Consumer without Exchange declared:
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
Both consumer code works well, so what's the main use of declaring exchange? I am confused. Can anyone clarify?

Publishing to queues lets you only implement basic publish-subscribe scenarios, where the producer and consumer use the exact queue. In case of multiple consumers a single queue of messages is distributed between multiple consumers.
Publishing to exchanges lets you create complicated scenarios, because of routing between exchanges and queues.
For example, a fanout exchange routes messages to all bound queues. This way, you can have one producer and multiple consumers and each message is copied to all bound queues independently and received independently.
Another example of exchange, a topic exchange routes messages to bound queues based on routing key in a message and a pattern on a queue. This introduces an interesting possibility of tagging messages and delivering them conditionally.
For a complete reference of exchange types and their profiles refer to the documentation:
https://www.rabbitmq.com/tutorials/amqp-concepts.html
https://www.rabbitmq.com/getstarted.html