I'm using ASB Topics. I'm connecting the service by using Microsoft .NET ServiceBus nuget (namespace Microsoft.Azure.ServiceBus.Core)
When a message arrive, my consumer either handles it and release the message or resending it to the topic with a delay.
The problem is that when the delay is less than 15 seconds, sometimes the message only arrive after 15 seconds.
e.g setting the delay to 3s or 10s usually works fine, but some of the messages arrive only after 15s (in both 3s or 10s cases).
When setting the delay to 20 seconds it works just fine with no exceptions.
It's for sure not load on the consumer because in some cases it was idle during the wait time.
I tried using prefetchCount but it had no effect.
I wanted to track the scheduled message in Azure UI but it seems that this option available for queues (not topics) only.
Any idea why is that happening and what can I do? thanks!
I'm using premium tier, my receiver runs in azure k8s. i'm quite sure
this number 15s is defined somewhere to delay messages in certain
cases. was wondering if someone knows about this.
You can queue or subject messages for later processing; for example, you can plan a job to be ready for processing by a system at a specific time. This functionality allows for the creation of a dependable distributed time-based scheduler.
Scheduled messages do not appear in the queue until the enqueue time has passed. Scheduled messages can be cancelled before that time. The communication is deleted when you cancel it.
You can use any of our clients to schedule messages in one of two ways:
Use the standard send API, but before sending, set the ScheduledEnqueueTimeUtc property on the message.
Pass both the standard message and the planned time to the schedule message API. This will return the SequenceNumber of the planned message, which you can use to cancel it later if necessary.
For more information please refer the below links:
MICROSOFT DOCUMENTATION:- Scheduled messages & Best Practices for performance improvements using Service Bus Messaging
Related
I've built a bot using botframework V4 for .Net that replies to user for both email and directline channels.
However, some of the request takes more than 15 seconds to complete, therefore I'd receive a GatewayTimeout error:
These requests are heavy (fetch some data from the database, fetch other data from another server via API calls, process the data, generate HTML and send them back to the user...) therefore nothing can be done to shorten the process.
I am aware that the gateway timeout delay is by design (the 15 seconds), but the problem is that the channel automatically retries the request after a small period of time and I end up receiving multiple emails for the same query (approx. 1 minute apart each).
I noticed as well that the directline replies are much faster than email ones (websocket vs SMTP), therefore this is mainly occurring with the email channel only. Noting that the emails are kept under 300KB as per this comment but can easily have a size close to this limit.
Therefore, is there a way to:
Increase the timeout delay?
Disable the automatic retries?
Or perhaps a certain workaround to prevent this issue?
Remember that your bot is a web app that exposes an HTTP endpoint, and every activity sent to your bot is an API call. Long-running API calls should be designed to return a response immediately and do their processing asynchronously. For example, consider the Recognize Text Computer Vision API. It just returns an Operation-Location where the actual result will become available later.
For bot Framework bots, all you have to do to send a message to the channel after the turn already ended is to send a proactive message. It's often also a good idea to design your bot to give the user an indication that the result is coming, such as by sending a preliminary "processing" message or a typing indicator, but that's probably unwanted in the case of the email channel. Eric Dahlvang explained this in the issue you linked to:
If the developer knows the response will take longer than 15 seconds, it is possible, depending on the channel, to start a separate thread to handle the long running process, return a valid status code on the receiving thread, and when the process finishes, send a proactive message from the background thread.
I've got a simple project that use M2mqtt client library to connect to HiveMQ broker. When a message arrives an event will fire, the fact is I can receive up to 100 messages per second and the program is able to processing just 20 messages per second.
HiveMQClient.MqttMsgPublishReceived += HiveMQClient_MqttMsgPublishReceived;
So, I have all the HiveMQ logs and telemetry and i can clearly see that the messages arrives in my application with the right rate (100 per second) but the strange thing is that the CPU of the PC where the client program is hosted runs at 10% of its capacity.
I was wondering if I need to "multi thread the event" or there is something that I'm missing.
Thank you all
EDIT
Inside the MqttMsgPublishReceived event i've got a ThreadPool that stores the messages that i receive inside an azure blob account. After some reviews I understood that this is the problem (thanks #Hans Kilian ).
Now i've got an azure blob storage account, standard configuration) that accepts only 30 calls per second. I tried to update into the premium tier but it is only for virtual machine VHD images.
Anybody knows how to improve these numbers?
The MqttMsgPublishReceived call back runs on the clients network thread, if you are interested in performance then you should not be doing any real work in this callback.
For high performance applications the model is normally to use the MqttMsgPublishReceived method to place the incoming message in a local queue in the client and have a thread pool consuming messages from that queue.
This becomes even more important when using QOS 1 or 2 messages as the broker will not send the next message until the MqttMsgPublishReceived has returned and the QOS handshake completes.
As #HandKilian says in the comments things like databases can also be a bottle neck, but using a thread pool combined with a database connection pool can help as it makes sure you are not building and tearing down a connection to the database for each message.
When a manager creates a task and sets the activation date in the future, it's supposed to be stored in the DB. No message is being dispatched out to the regarded workers, until a day or two before it's due. When the time's approaching, an email's being sent out to the subordinates.
Previously I've resolved that using a locally run Windows Service that scheduled the messaging. However, as I'm implementing something similar in the Azure, I'm not sure how to resolve it (other than actually hosting my own Windows Server in the cloud, of course, but kind of defeats the whole point).
Since my MVC application is strictly event driven, I've browsed around in the Azure portal to find a utility to schedule or postpone a method being invoked. No luck. So at the moment, all the emails are dispensed immediately and the scheduling is performed by keeping the message in the inbox until it's time (or manually setting up an appointment).
How should I approach the issue?
Other possible solution is to use Queueing mechanism. You can use Azure Storage Queues or Service Bus Queues.
The way it would work is when a task is created and saved in the database, you will write a message in a queue. This message will contain details about the task (may be a task id). However that message will be invisible by default and will only become visible after certain amount of time (you will calculate this period based on when you would need to send out the email). When the visibility timeout period expires, the message will become available to be consumed in the queue. Then you will have a WebJob with a Queue trigger (i.e. the WebJob will become alive when there's a message in the queue). In your WebJob code, you will fetch the task information from the database and send the notification to concerned person.
If you're using Azure Storage Queue, the property you would be interested in is InitialVisibilityTimeout. Please see this thread for more details: Azure storage queue message (show at specific time).
If you're using Azure Service Bus Queue, the property you would be interested in is BrokeredMessage.ScheduledEnqueueTimeUtc. You can read more about this property here: https://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.scheduledenqueuetimeutc.aspx.
One solution to run background tasks is to use Web Jobs. Web Jobs can run on a schedule (let's say once per day), manually or triggered by a message in a queue.
You can use Azure WebJobs. Basically, create a WebJob and schedule it to regularly check the data in your database for upcoming tasks and then notify people.
What is the best way to implement in service bus messages that are requiring once a week or once a day etc.
I am thinking of having a separate windows service that just drops in messages from the database into the service bus but is there another way?
In simple terms i want a message that once it is processed, it will appear again in the queue in a specified amount of time to be processed again.Obviously once i process a message i can tell service bus to delete the message or appear again in the queue.
You will need to have some external process (e.g. your windows service) which sends the message in the first place, on schedule. You can use Azure Scheduler to do that, see http://www.prasadthinks.com/blog/2015/07/11/azure-scheduler-can-post-to-azure-service-bus-queue-and-topic/
When you are processing your message, you can do what you are describing i.e. re-send a copy of the message, using BrokeredMessage.ScheduledEnqueueTimeUtc property so that it arrives at the time you want. But I wouldn't do that, does not feel right. If you have your external processing already sending messages on schedule, just rely on that 100%.
Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.