I have an asp.net web service which is like a reservation system and cannot reserve the same seat by multiple persons and the eligibility for the reservation is based on some other table values in SQL server. I plan to use SQL server queue processing as mentioned here. My customer wants to do this in a synchronized call, means we want the result in the same web service method call. My question is how efficient it to allow a synchronized method call to process a queue and wait until the queue return values (by means of a loop within a time span or so). Please advise the best possible approaches to achieve this.
You are stepping in the XY problem trap. Your primary goal is to make sure no seat can be reserved twice. You should rethink your approach of using a queue is the appropriate solution to this problem. A queue is great to efficiently use your processing resources on a background task. For real-time processing (like yours) it will create more problems that it solves.
It seems like you want to avoid the race condition that occurs when multiple users try to access the same seat twice. The queue doesn't solve the problem, you just move it to the enqueing phase. The one who enters the queue first wins. At the end you have added an unnecessary complication that doesn't bring you any benefit.
A much simpler solution to your problem is to create a unique key on your DB that makes sure that no seat can be reserved twice. Once you try to reserve a seat that has been taken just a moment ago, you will get an SQLException with the error number 2627. You can then prompt the user that the seat is taken.
We're using RabbitMQ for storing lightweight messages that we eventually want to store in our SQL Server database. There will be times when the queue is empty and times when there is a spike of traffic - 30,000 messages.
We have a C# console app running in the same server.
Do we have the console app run every minute or so and grab a designated number of items off the queue for insertion into the database? (taking manageable bites)
OR
Do we have the console app always "listen" and hammer items into the database as they come in? (more aggressive approach)
Personally I'd go for the first approach. During those "spike" times, you're going to be hammering the database with potentially 30,000 inserts. Whilst this potentially could complete quite quickly (depending on many variables outside the scope of this question), we could do this a little smarter.
Firstly, by periodically polling, you can grab "x" messages from the queue and bulk insert them in a single go (performance-wise, you might want to tweak the the 2 variables here... polling time and how many you take from the queue).
One problem with this approach is that you might end up falling behind during busy periods. So you could make your application change it's polling time based on how many it is receiving, whilst keeping between some min/max thresholds. E.g. if you suddenly get a spike and grab 500 messages... you might decrease your poll time. If the next poll, you can still get thousand, do it again, decrease poll time. As the number you are able to get drops off, you can then begin increasing your polling time under a particular threshold.
This would give you the best of both world imho and be reactive to the spikes/lull periods.
It depends a bit on your requirement but I would create a service that calls SQLBulkCopy to that bulk inserts every couple of minutes. This is by far the fastests approach. Also if your Spike is 30k records I would not worry too much about falling behind.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx
We have a C# console app running in the same server.
Why not a Service?
What I would do is have the console app always listen to the rabbitmq and then in the console app build your own queue for inserting into the database that way you can throttle the database insertion. By doing this you can control the flow in busy a time by only allowing so many tasks at once and then in slow times you get a faster reaction then polling every so often. The way I would do this is by raising an event and the you know there is something to do in the queue and you can check the queue length to see how many transactions you want to process.
Instead of using a Console Application, you could set up a Windows Service, and set up a timer on the service to poll every n minutes. Take a look at the links below:
http://www.codeproject.com/Questions/189250/how-to-use-a-timer-in-windows-service
http://msdn.microsoft.com/en-us/library/zt39148a.aspx
With a Windows Service, if the server is re-booted, the service can be set up to restart.
So, I've got a WCF application that accepts requests to do work at a specific time. I could have a list of thousands of things to do in the future at varying times. Is there an existing framework that we can leverage to do this? The current implementation polls a database, looking for things to do based on a datetime, which smells.
A few ideas.
Timers. Set a timer when the request comes in that fires are the appropriate time. This seems like I could have too many threads floating around.
Maintain a list of objects with a datetime in memory, poll this for things to do.
Use a library like quartz. I have concerns as to whether this can handle the volume.
If you keep a list of tasks sorted by their trigger times (Your database should be able to do this without any issues. If you want to keep it in-memory, Power Collections has a priority queue you could use), you can get by with a single timer that always activates for the first one in the list.
If I had a lot of messages in a database that I wanted to send, and each row specified a date and time to send the message, and a flag for if it has been sent.
These won't always be at fixed intervals, and more than 1 message may want to be sent at the same time.
In this case it would just queue them up and send in order of when they were created.
Is the easiest thing to do just to have a function that runs over and over again, once it completes it just runs again
So it would:
Start Running and check the current date/time
Check for any unsent messages
Send all the messages due to go out before and up to the time it started running
Start all over again and take the current date/time
My problem with this is, would it just be horribly inefficient to continuously have a method running, possibly for hours or days without actually sending a message.
The main strain in this case I think would be put on the database, it would constantly be getting hit with a query.
Is there a better way to schedule something like this to happen.
Or just do the above but every time it runs make it wait for 5 minutes before running again.
Does Workflow 4 offer anything suitable for scheduling perhaps?
You could always do a pre-emptive read of the next time value in the series and do a single sleep until then, instead of looping through short sleeps over and over.
Not sure if that's as elaborate as you want though
Maybe have a compiled view in the database which returns messages that are not sent (I assume there's a flag on each record?) and for which the intended send time is prior to the current time. Then a Windows Service or console application on a scheduled interval can hit that view (which can be performance-tuned in the database pretty well, I'd imagine) and send any messages returned by it.
You could use a windows service to accomplish this. Or if you're using MSSQL, you could even use a SQL Server Agent Job.
Several answers has suggested sending some messages then calling sleep until the next message is due to be sent.
How you sleep in this instance is all important.
You can - in theory - tell a thread to sleep for hours, however if during that time the app (or service) needs to shut down then you're in trouble. The process will be terminated, no cleanup will be executed. This is a less than ideal.
Don't get confused between the concept of polling for work to do, and sleeping between polls.
If you've to wait 5 minutes (or 5 hours) before next polling the database, that's fine, however you never want to *sleep for more than a second or two at a time.
What I'd do . . .
Write a windows service. The service has one active thread that polls the database, see's are any messages due to send, and sends them.
It will then poll on a configurable delay (1 minute, 5 minutes, 1 hour, what ever suits).
However it will never sleep for more than a second while it's waiting to poll the database.
If you can be assured that messages can only be added to send after the last message in the DB? If so you can check the time of the next message and not poll until that time.
However, if I find that the next message doesn't need to be sent for 5 hours, is it possible that while I'm waiting a message was added that should be sent in 30 minutes?
If so then you can never trust the "Next message time" and not poll until then, you have to continuously poll on your fixed interval NB worth saying again, your polling interval and your sleep interval are not the same thing.
How about writing a windows service which does this for you. This Windows service will run in the background and check the current time with your db records in a purticular interval (ex : every 5 minutes) and send emails to people and update corresponding records in your tables to set the Email Sent Flag to true
You can even have an SQL job which selects records which are not sent and matches wtih the current time and call a stored procedure which calls dot net assembly to send email . The dot net assembly can use SMTPClient to send emails.
It depends on what you use. Using a scheduled task or a service is perfectly acceptable for the scenario you describe.
You have to be careful though that you do not tie up resources if the process runs too often. It might be more efficient for it to run less often at peak times and more often during off-peak times.
Whatever method you prefer (make a Windows Service, use Task Scheduler, etc..), please bear in mind that your initial suggestion is exactly what is called busy waiting, which is something you should avoid unless you really know what you're doing.
What you describe isn't that bad if you extend it with
"when there are no messages due select the next time a message will be due and sleep till then".
Alternatively use a DB with "notification support" making the whole thing event-driven i.e. the DB sends you an event whenever a message is due.
you can use this one .NET Scheduled Timer for checking timeinervals and running the function(sending messages) at specific time intervals ....
I would say create a windows service with the timer. it may sleep for configured amount of seconds and then compare the datetime from the database. if it matched then send an e-mail & set the flag in the database for sent e-mails.
I recently implemented a windows service which utilized a class called IntervalHeap in the C5 collection class library. I then added a persistence layer which keeps tracks of the items and their intervals in case the service is stopped/crashed.
Has been in production for a few months and has been working very well.
We do this at a financial institution to send out internal e-mails from our intranet applications. Once every 15 minutes, a scheduling software (enterprise scheduler, not a Windows scheduled task) fires off a job. We have a view called PendingEmail on top of a table called EmailQueue that only lists out what needs to be sent this go around (the EmailQueue table has a PopDate, which is an effective date as to when the e-mail should get sent). The application fires off e-mails for whatever it found in the PendingEmails view.
The job sends out a maximum batch size of emails every 15 minutes, marking each record with whether it was successfully sent or whether there was an error (invalid email address, etc.) and what the Exception was, and whether we would like to try re-sending it the next time around. It updates that EmailQueue table all at once, not each record individually. The batch size was put in place to prevent the job from taking more than 15 minutes and stomping on itself.
I don't know that polling every so often is really consuming all that many resources, unless you're going to do it every 5 seconds or something. If you're sending out millions of messages you may need to distribute the work across multiple machines. If you're going to write some custom code, I would use a Timer over Thread.Sleep(), and set the Timer to tick every 5 minutes or whatever interval you'd like to perform work. An event fires on every tick that would subscribe to to start the routine that sends your messages.
See this post on Thread.Sleep() vs. the Timer class:
Compare using Thread.Sleep and Timer for delayed execution
Many databases allow events to be fired by triggers, eg. 'after insert'. The trigger is run by the database process/thread and the actions it can take are database-specific. It could, for instance, call a C or java procedure that signals a named semaphore upon which you emailer is waiting or exec. an emailer app directly. Look at 'trigger' or 'create trigger' for your database.
We are looking to develop an email notification service where emails can be scheduled daily, weekly, or hourly based on certain actions that happen within our system (User Registration) or summary emails sent every Friday for example.
What is the best way to handle making sure duplicate emails are not sent? We thought about maybe having the application write a record to a queue whenver a system action happens but this would seem to have one more point of failure. Or maybe just making all notifications data driven, for example, select all users where created on date is greater than now. But with this scenario we need a way to make sure if the service runs again duplicate emails are not sent..
any ideas would be great!
My 2 cents
1) Queues. Queues are great for the tasks where you want to have a 'single entry and single exit' type of architecture. Queues decouple the systems and allow you to load balance the system. They are usually used with the multiple workers on one and another end. You will just add (maybe a lot) of messages to the queue, and later run a bulk dequeue. IMO that is irrational memory and resource consumption.
2) Data-driven via Users. Much easier to implement, however for each notification you will check every user and will put heavy load on db.
3) Data-driven via UserNotifications. Alternatively, you can create a separate table UserNotifications, where each user will be added once he has registered. It is much easier to select the needed users within given time frame and you don't store them in memory. Once notification sent, you remove the user from UserNotifications table.