What is the best way to implement in service bus messages that are requiring once a week or once a day etc.
I am thinking of having a separate windows service that just drops in messages from the database into the service bus but is there another way?
In simple terms i want a message that once it is processed, it will appear again in the queue in a specified amount of time to be processed again.Obviously once i process a message i can tell service bus to delete the message or appear again in the queue.
You will need to have some external process (e.g. your windows service) which sends the message in the first place, on schedule. You can use Azure Scheduler to do that, see http://www.prasadthinks.com/blog/2015/07/11/azure-scheduler-can-post-to-azure-service-bus-queue-and-topic/
When you are processing your message, you can do what you are describing i.e. re-send a copy of the message, using BrokeredMessage.ScheduledEnqueueTimeUtc property so that it arrives at the time you want. But I wouldn't do that, does not feel right. If you have your external processing already sending messages on schedule, just rely on that 100%.
Related
I'm using ASB Topics. I'm connecting the service by using Microsoft .NET ServiceBus nuget (namespace Microsoft.Azure.ServiceBus.Core)
When a message arrive, my consumer either handles it and release the message or resending it to the topic with a delay.
The problem is that when the delay is less than 15 seconds, sometimes the message only arrive after 15 seconds.
e.g setting the delay to 3s or 10s usually works fine, but some of the messages arrive only after 15s (in both 3s or 10s cases).
When setting the delay to 20 seconds it works just fine with no exceptions.
It's for sure not load on the consumer because in some cases it was idle during the wait time.
I tried using prefetchCount but it had no effect.
I wanted to track the scheduled message in Azure UI but it seems that this option available for queues (not topics) only.
Any idea why is that happening and what can I do? thanks!
I'm using premium tier, my receiver runs in azure k8s. i'm quite sure
this number 15s is defined somewhere to delay messages in certain
cases. was wondering if someone knows about this.
You can queue or subject messages for later processing; for example, you can plan a job to be ready for processing by a system at a specific time. This functionality allows for the creation of a dependable distributed time-based scheduler.
Scheduled messages do not appear in the queue until the enqueue time has passed. Scheduled messages can be cancelled before that time. The communication is deleted when you cancel it.
You can use any of our clients to schedule messages in one of two ways:
Use the standard send API, but before sending, set the ScheduledEnqueueTimeUtc property on the message.
Pass both the standard message and the planned time to the schedule message API. This will return the SequenceNumber of the planned message, which you can use to cancel it later if necessary.
For more information please refer the below links:
MICROSOFT DOCUMENTATION:- Scheduled messages & Best Practices for performance improvements using Service Bus Messaging
I'm looking the best way for know if a RabbitMq message is processed or exists in queue. I have my Web Management that sends messages to my queue with a loot of traffic. My app checks in the database, during 1 minute if the consumer insert the new data on the database but if is not inserted the app shows a error, that the data is not inserted. The problem is in the case of high demand the delay can be a lot of time, and the data can be inserted after of the delay and the user of the Web Management don´t know if the data was inserted and the user can try to send another message to the queue. I need to know if is possible check if a message exists in the queue or know if the message was processed.
Simplest answer.... no, you can't because it's against the architecture of an queue. It's build to post messages in and consume it constantly.
The only chance, if you are just working with 1 consumer is, interrupt it, consume ALL messages and repost them to the Queue and check if the desired message was consumed. Be aware, it will mess up the queue! I think a better solution will be another queue, where you consumer can publish a "finished" message on an own queue, which your original publisher can consume.
When a manager creates a task and sets the activation date in the future, it's supposed to be stored in the DB. No message is being dispatched out to the regarded workers, until a day or two before it's due. When the time's approaching, an email's being sent out to the subordinates.
Previously I've resolved that using a locally run Windows Service that scheduled the messaging. However, as I'm implementing something similar in the Azure, I'm not sure how to resolve it (other than actually hosting my own Windows Server in the cloud, of course, but kind of defeats the whole point).
Since my MVC application is strictly event driven, I've browsed around in the Azure portal to find a utility to schedule or postpone a method being invoked. No luck. So at the moment, all the emails are dispensed immediately and the scheduling is performed by keeping the message in the inbox until it's time (or manually setting up an appointment).
How should I approach the issue?
Other possible solution is to use Queueing mechanism. You can use Azure Storage Queues or Service Bus Queues.
The way it would work is when a task is created and saved in the database, you will write a message in a queue. This message will contain details about the task (may be a task id). However that message will be invisible by default and will only become visible after certain amount of time (you will calculate this period based on when you would need to send out the email). When the visibility timeout period expires, the message will become available to be consumed in the queue. Then you will have a WebJob with a Queue trigger (i.e. the WebJob will become alive when there's a message in the queue). In your WebJob code, you will fetch the task information from the database and send the notification to concerned person.
If you're using Azure Storage Queue, the property you would be interested in is InitialVisibilityTimeout. Please see this thread for more details: Azure storage queue message (show at specific time).
If you're using Azure Service Bus Queue, the property you would be interested in is BrokeredMessage.ScheduledEnqueueTimeUtc. You can read more about this property here: https://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.scheduledenqueuetimeutc.aspx.
One solution to run background tasks is to use Web Jobs. Web Jobs can run on a schedule (let's say once per day), manually or triggered by a message in a queue.
You can use Azure WebJobs. Basically, create a WebJob and schedule it to regularly check the data in your database for upcoming tasks and then notify people.
What design has someone successfully used to implement job processing on Windows Azure?
Requirements:
Ability to push a Job into a queue.
N workers can consume Jobs from the queue and process them.
Invoker of the job should be able to be alerted (push, not polling) of the job being completed.
Research thus far:
Create a "Job" Queue using Azure Service Bus Queues (http://blogs.msdn.com/b/appfabric/archive/2011/05/17/an-introduction-to-service-bus-queues.aspx)
Web front-end pushes Jobs to the queue, workers block on Receive() indefinitely (see http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.aspx) until a Job is ready (to avoid "null" long polling, which costs money due to API call transaction costs)
With regards to being notified of Job completion:
There is no apparent ability to be alerted to when a Job has been completed.
I thought I could leverage Service Bus Topics/Subscriptions (https://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/) and have a caller "subscribe to" a "Job Finished Notifications" topic, however:
You apparently can't subscribe more than once to the same topic, unless you create multiple "Subscription" entries (which does not scale)
Unless we did create a "Subscription" for each Job Id, and had the caller block on a Receive() API call (using I/O completion ports) on that subscription, we can't get real time notifications of when a Job has been processed.
Has anyone had any experience implementing this sort of Job system (real time, low latency, with completion notifications for the caller) before?
Thanks
Actually, queue does not stand by push. The whole idea about queue is the receiver does not need to receive the message in real time, and wants to check the message periodically. If you need real time communication, you can create an HTTP/TCP listener on the receiver side, and let the sender make an HTTP/TCP request.
Thus, one approach is to create a web service on the web role, using internal endpoints. You send the service's address along with the message to worker role using queue. When the job is finished, the worker role invokes the service to notify the web role that job is done.
This approach is fine, but it does not provide much value. It cannot display something on the UI(unless you implement web socket), since a server cannot notify the browser. So if you want to display a notification in a browser client, I would like to suggest you to use a pull solution (unless you implement web socket). If you're using a rich client, you can host a web service on the client machine, and let the worker role notify the client by invoking the service.
Best Regards,
Ming Xu.
Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.