I'm looking the best way for know if a RabbitMq message is processed or exists in queue. I have my Web Management that sends messages to my queue with a loot of traffic. My app checks in the database, during 1 minute if the consumer insert the new data on the database but if is not inserted the app shows a error, that the data is not inserted. The problem is in the case of high demand the delay can be a lot of time, and the data can be inserted after of the delay and the user of the Web Management don´t know if the data was inserted and the user can try to send another message to the queue. I need to know if is possible check if a message exists in the queue or know if the message was processed.
Simplest answer.... no, you can't because it's against the architecture of an queue. It's build to post messages in and consume it constantly.
The only chance, if you are just working with 1 consumer is, interrupt it, consume ALL messages and repost them to the Queue and check if the desired message was consumed. Be aware, it will mess up the queue! I think a better solution will be another queue, where you consumer can publish a "finished" message on an own queue, which your original publisher can consume.
Related
What is the best way to implement in service bus messages that are requiring once a week or once a day etc.
I am thinking of having a separate windows service that just drops in messages from the database into the service bus but is there another way?
In simple terms i want a message that once it is processed, it will appear again in the queue in a specified amount of time to be processed again.Obviously once i process a message i can tell service bus to delete the message or appear again in the queue.
You will need to have some external process (e.g. your windows service) which sends the message in the first place, on schedule. You can use Azure Scheduler to do that, see http://www.prasadthinks.com/blog/2015/07/11/azure-scheduler-can-post-to-azure-service-bus-queue-and-topic/
When you are processing your message, you can do what you are describing i.e. re-send a copy of the message, using BrokeredMessage.ScheduledEnqueueTimeUtc property so that it arrives at the time you want. But I wouldn't do that, does not feel right. If you have your external processing already sending messages on schedule, just rely on that 100%.
I have a web application that is processing some documents and emailing users and this could take some time, so I thought I'd use MSMQ to accomplish this. I've never used MSMQ and I have a few questions about how the queuing system would process the messages I send.
If I send a message can I setup the program to start processing the message right away?
If there are 10 message in the queue can the program start automatically and process each message --one at a time--?
How does the program know to start processing the queue?
If you know the answers to my questions can you also give an example?
Thanks!
I have the following code which send out SMS to the subscribers. However, some SMS were rejected from the SMSGateway because I'm sending too many SMS at one time. So I'm thinking to make a delay in between.
Sending out the SMS like this -
foreach (DataRow row in dt.Rows)
{
//Gets Subscriber number
smsSender.destinationNum = Convert.ToInt64(row["callerID"]);
foreach (DataRow articleRow in dtArticle.Rows)
{
//Gets SMS content
smsSender.smsMessage = articleRow["news"].ToString();
//Then send out the SMS
smsSendder.sendSMS();
}
}
Please advice because I have no experience with the threads and timers
It would depend on the architecture of the application.
Assuming this is a service-style app, with no user interface, that simply gets data out of the database and sends it to SMS, then Thread.Sleep(...) is fine.
If this app has a user interface, and you're running this SMS sending code on the UI thread, then Thread.Sleep(...) will block your UI. Actually, smsSender.sendSMS is probably already blocking your UI in this case!
Refactoring so that this code is off the UI thread is the answer. And you can do that simply by using a timer, although you will have to refactor the code so that the result set is cached in a local object and the timer iterates through the set sending one SMS out at a time.
In either case, I hope you don't have a lock on the database while you're sending SMSes.
Your question is tagged [asp.net] so I assume that you have a webpage that when requested will send a number of SMS messages (e.g. when a user presses a "submit" button or something similar in a HTML form).
In that case you can have multiple users requesting the webpage simultaneously. Also, you don't want to sleep in the thread serving the web page to the user. If you do that then there will be a delay where the user waits for the web page to respond while the SMS messages are sent.
I would suggest something like this:
When you need to send SMS messages you store the messages in a table in your database.
After storing new messages in the database you start a task (Task.Factory.StartNew) to process the SMS messages in the database.
You need to make sure that no more than one task is running in the ASP.NET application. Storing new messages in the database involves checking if the task is running and if not starting it.
The task will process all remaining messages in the database and send them using the appropriate delay (done by Thread.Sleeep).
When the task has sent an SMS message it is removed from the database.
This solution offloads the sending of messages to a background task that can be as slow as required and introduces persistence using the database to avoid loosing messages even if say the application pool is recycled.
Thread.Sleep is more appropriate, because it models better the waiting aspect.
Thread.Sleep() should be a good choice to delay calling to SMS gateway to prevent server reject your request.
I don't think it's Thread.Sleep() that's tying up the CPU.
Thread.Sleep seems bad design. Please refer http://blogs.msmvps.com/peterritchie/2007/04/26/thread-sleep-is-a-sign-of-a-poorly-designed-program/ about why Thread.sleep is a bad.
Timer are more accurate, Thread.Sleep is only guaranteed to wait at LEAST as long as the amount of time you specify (the OS may put it to sleep for much longer). .
I am developing an Application where I am submitting POST Requests to a .NET Web Service.
Current implementation is to process the request instantly and give response. In actual deployment, there will be huge amount of data that needs to be processed and thus the request must be processed offline.
What are the strategies that can have the task accomplished
Should I implement a Windows Service, or a scheduled task that invokes an application to perform the desired task.
This might be a good case for MSMQ. Your webservice can fill the queue with incoming data, and another process can read those messages and perform the necessary processing.
Here's a good overview of MSMQ:
http://www.primaryobjects.com/CMS/Article77.aspx
If you have so much data it cannot be processed in real-time, I would probably setup the service to do the following:
ProcessRecordViaPost
Create new record in "Queue" database with UniqueID, and all other info to be processed
Return UniqueID to client immediatly
ReadRecordViaGet
Check queue, if processed return data if not return status code (number of items in queue before it?)
I would also have a windows service that continually grabs the oldest item from the Queue, and processes it and moves on to the next oldest.
Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.