My application's bottleneck has become sending and receiving messages over MSMQ with MassTransit. The send and receive are both happening within the same application, but there are often too many messages to use an in-memory queue.
Here is my simple queue setup:
messageBus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom("msmq://localhost/msg_queue");
sbc.Subscribe(subs =>
{
subs.Handler<EventMessage>(msg => Enqueue(msg));
});
});
For my experiments, MSMQ currently has ~1 million messages and we are not pushing any new messages to it. We are not doing any work in Enqueue() except time how quickly messages are being sent.
With that information, we can only grab between 150 and 200 messages per second from MSMQ, when I'd hoped it would be at least 1000 messages per second when there is no network latency. Each messages is <2kb.
How can we speed up how quickly MSMQ passes messages to an application over MassTransit while maintaining the message ordering enforced by a queue?
I did addressed something similar before. If I remember it correctly,
We specified message expiration through TimeToBeReceived (The total time for a sent message to be received from the destination queue. The default is InfiniteTimeout.) . Refer this msdn link.
Related
I have a RabbitMQ queue in which I post thousands of messages. I need a background Service that will:
Retrieve the messages in batches of 100 messages
Store those 100 messages in a Database
Ack all 100 messages at once
Proceed with the next batch of 100 messages
I'm using the RabbitMQ Client to listen for messages, but I don't know how to "batch" them.
Does anyone has a working example on how to get my messagess 100-100 at a time and ACK them all at once after they have been saved in a DB?
Thanx a lot in advance
You want to use the "prefetch" and "multi ack" features -
Prefetch - https://www.rabbitmq.com/consumer-prefetch.html
Multi-ack - https://www.rabbitmq.com/confirms.html#consumer-acks-multiple-parameter
This is what you'll do in your code:
Open a connection and channel
Set the channel prefetch to 100
Start a consumer. The callback you register for your consumer will start being called with messages. You will have to save these messages in a list or other data structure
Once you have received 100 messages, save the delivery tag of the 100th message and do your database work.
Ack the 100 messages by setting "multi ack" to true and use the delivery tag of the 100th message.
RabbitMQ will send the next 100 messages in the same manner
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Is it possible to bulk read messages from Solace queue rather than receiving them one by one on callback?
Currently MessageEventHandler receives about 20 messages per minute, this is too slow for our application.
Does anyone have a better solution to speed things up in Solace?
This is a C# application.
We used
ISession.CreateFlow(FlowProperties, IEndpoint, ISubscription,
EventHandler<MessageEventArgs>, EventHandler<FlowEventArgs>)
Passing in a MessageEventHandler which gets the message via MessageEventArgs.Message
queue = CreateQueue();
Flow = Session.CreateFlow(flowProperties, queue, null, OnHandleMessageEvent, OnHandleFlowEvent);
..
void OnHandleMessageEvent(object sender, MessageEventArgs args)
{
var msgObj = args.Message.BinaryAttachment;
..
}
```
No, there is no API call for a user to read messages in bulk.
By default, the API already obtaining messages from the message broker in batches, with each message being individualy delivered to the application in the message receive callback.
FlowProperties.WindowSize and FlowProperties.MaxUnackedMessages can change this behavior.
20 messages per minute is extremely slow.
One common reason for slowness is that the application is taking a long time to process messages in the message receive callback ("OnHandleMessageEvent").
Blocking in the message receive callback will prevent the API from delivering another message to the application.
Refer to Do Not Block in Callbacks for details.
I'd like to write parallel execution module based on Solace. And I use request-reply schema for this.
I have:
Multiple message consumers, which publish messages into the same queue.
Multiple message producers, which read queue and create reply messages.
Message execution time is between 10 seconds to 10 minutes.
Queue access type is non-exclusive (e.g. it does round-robin between all consumers).
Each producer and consumer is asynchronous, e.g. Solace API blocks execution during the connection only.
What I'd like to have: if produces works on the message, it should not receive any other messages. This is extremely important, because some tasks blocks executor for several minutes, however other executors can be free after couple of seconds.
Scheme below can be workable (possible), however blocking code appears below. I'd like to avoid it.
while(true)
{
var inputMessage = flow.ReceiveMsg( /*timeout 1s*/1_000); // <--- blocking code, I'd like to avoid it
flow.Ack(inputMessage.ADMessageId);
var reply = await ProcessMessageAsync(inputMessage); // execute plus handle exceptions
session.SendReply(inputMessage, reply)
}
Messages are only pushed to the consuming applications.
That being said, your desired behavior can be obtained by setting the "max-delivered-unacked-msgs-per-flow" on your queue to 1.
This means that each consumer bound to the queue is only allowed to have 1 outstanding unacknowledged messages.
The next message will be only sent to the consumer after it has acknowledged the message.
Details about this feature can be found here.
Do note that your code snippet does not appear to be valid.
IFlow.ReceiveMsg is only used in transacted sessions, which makes use of ITransactedSession.Commit to acknowledge messages.
I have a couple of queues where certain information is queued. Let us say I have "success" and "failed" queues in which Server side component has continuously written some data to these queues for clients.
Clients read this data and display it on a UI for end users. Now, I have a situation to purge any message in these queues older than 30 days. Clients would then only be able to see only 30 days of information at any point of time.
I have searched a lot and could see some command line options to purge whole queue but could not find a relevant suggestion.
Any help in the right direction is appreciated. Thanks
I don't think this is possible; looks like you're trying to use RabbitMq as data storage instead of message server.
The only way to understand if a message is "older" than 30, is to process the message, and by doing this you are removing the messagge from the queue.
Best thing to do here is to process the messages and store them in a long term storage; then you can implement a deletion policy to eliminate the older elements.
If you really want to go down this path, RabbitMQ implements TTL at queue level or message level; take a look at this: https://www.rabbitmq.com/ttl.html
[As discussed in comments]
To keep the message in the queue you can try to use a NACK instead of ACK as confirmation; this way RabbitMQ will consider the message undelivered and it will try to deliver it again and again. Remember to create a durable queue (https://www.rabbitmq.com/confirms.html).
You can also check this answer: Rabbitmq Ack or Nack, leaving messages on the queue
I am creating a simple Publisher/Subscriber using MassTransit and RabbitMQ.
The Publisher has the following code to initialize the bus:
/** create the bus */
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
});
/** start the bus and publish */
bus.Start();
bus.Publish<IPersonLogin>(new {FirstName = "John", LastName = "Smith"});
And the Subscriber has this code for initialization:
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ReceiveEndpoint(host, "person_login", e =>
{
e.Consumer<PersonLoginConsumer>();
});
});
If I shut-down the Subscriber and publish 2 messages, the messages are not lost and as soon as the Subscriber comes back to life the Messages are processed.
So my questions are:
How do I ensure that a Message stays in the Queue of RabbitMQ until one Subscriber comes up and pick it up?
What happen if the Server is reboot and some Messages were not processed by any Subscriber, do they get lost or do they get processed as soon as the Subscriber come alive after reboot?
Is this the correct pattern to ensure that every single message is processed or should I use a different strategy?
by default, any message sitting in a queue will remain there until one of three things happens:
the message is consumed
the message "time to live" expires (default is to live forever)
the server crashes or restarts
if you have a queue full of messages, the messages will generally stick around until one of those three things happens. hopefully you will have your consumers online soon enough that you can consume the messages and process them.
you would only set a time to live (ttl) if you want messages to be automatically deleted after a period of time (assuming they are not consumed first)
for crashes... a message can survive a crash / restart if you make the message persistent to disk. there's still a chance that the message will be lost if the server crashes before the message is routed from the exchange to the queue, though.
On top of mind.
If there arent any subscribers RabbitMQ wont know to which queue a message should be delivered. Then a message will be undeliverable.(Not sure if this will be moved to a error queue or skipped)
If the exchanges are already there it will be placed in the queue of consumer that has subscribed to the event. So you endpoint hosting your consumer can be down the message will still be delivered.
When the message is delivered to the queue the consumer will pick up your message and process it. If a exception occurs while processing your message it will be moved to the endpoint_error queue. (Depending on your RetryPolicy). Deploy a fix and move you message back in to the main queue and the messages will be processed as if nothing has happend.
Good read for common issues on common gotcha's
Under the Hood
Common Gotcha's