We Keep Coding

Windows service to read from database and invoke applications based on dates

I am developing a data optimization project. It has a client side web app to receive tasks from users. Let's say the tasks are some heavy calculations that cannot be done easily by normal systems. The program takes a long while for big amount of data to be calculated. So what I am trying to do is receive calculation orders from my web application and have a windows service on my server side to listen for new tasks to be done.
I would like my service to be listening to the data that is being inserted into my Tasks table and run the calculator based on the time of those dates. I will of course would have to deal with some multi-threading. And maybe if the program is busy, the other processes would have to wait.
I also don't mind having a small GUI for my application to see which orders are now being processed and whether my service is busy or idle.
I first thought about adding SQL Server jobs in my data base to query the table frequently and run the application based on the dates. But that does not look like a nice solution to me. What I want is a nimble ready-to-serve service who becomes aware when we have new data in the database and decides what to do.
I don't insist on Windows Services particularly here. So any good idea is welcome.

I've done something similar in Powershell.
What I have:
A "queue" table where requests gather
Requests come in by clients calling a procedure
A "queue history" table - request already "treated" go from "queue" to "queue_history" (delete trigger)
A powershell loop reading queue table
Requests from queue are started independently via start-job command, those are actually separate Powershell instances (my tasks are mostly some external exe calls, but I think could be usefull for stored procedure calls also).
The powershell loop also report his "heartbeat" to a table.
Also watches a table and file existance for STOP flag so I can stop it.

Increasing the performance with service and multi threading in C#

I am doing a project that needs to communicate with 20 small computer boards. I will need to keep check of their connections and they will also return some data to me. So my aim is to build a control/monitoring system for these boards.
I will be using Visual Studio 2010 and C# WPF.
My idea/ plan would be like this:
On the main thread:
There will be only one control window, so a main thread will be created mainly to update the data to be displayed. Datas of each board will be display and refreshed at a time interval of 1s. The source of data will be from a database where the main thread will look for the latest data(I have not decided on which kind of database to use yet).
There will be control buttons on the control window too. I already have a .dll library, so I will only need to call the functions inside to direct the boards to action (by starting another thread).
There will be two services:
(Timer service) One will be a scheduled timer to turn the boards on/ off at a specific time. Users would be able to change the on/ off time. It would read from the database to get the on/ off time.
(Connection service) Another one will be responsible to ask and receive information/ status from the board every 30s or less. The work would be including connecting with the board through internet, asking for data, receiving the data and then writing the data to the database. And also writing down the exceptions thrown if the internet connection failed.
My questions:
1) For the connection service, I am wondering if I should be starting 20 threads to do this, one thread per connection to a board. Because if the connections were made by only one thread, the next board connection must wait for the first to finish, which may add up to 1-2 mins for the whole process to end. So I would need around 20 - 40 mins to get all the data back. But if I separate the connection to 20 threads, will it make a big difference in the performance? As the 20 threads never dies, it keeps asking for data every 30s if possible. Besides, does that mean I will have to have 20 database, as it would clash the database if 20 threads are writing in at the same time?
2) For updating the display of data on the main thread for every 1s, should I also start a service to do this? And as the connection service is also accessing the same database, will this clash the database?
There will be more than 100 boards to control and monitor in the future, so I would like to make the program as light as possible.
Thank you very much! Comments and ideas very much appreciated!

Starting 20 threads would be the best bet. (Or as Ralf said, use a thread when needed, in your specific case, it would probably be 20 at some point). Most databases are thread safe, meaning you can write into them from separate threads. If you use a "real" database, this isn't any issue at all.
No, use a Timer on the main thread to update your UI. The UI can easily read from the DB. As long as the update action itself is not taking a lot of time, it is OK to do it on the UI thread.

1) Why not use threads when needed. You can use one DBMS they are build to processing large amounts of information.
2) Not sure what you mean by start a service for the UI thread. As with 1) Database Management Systems are build to process data.

Database Insert Performance When Grabbing Items From Queue

We're using RabbitMQ for storing lightweight messages that we eventually want to store in our SQL Server database. There will be times when the queue is empty and times when there is a spike of traffic - 30,000 messages.
We have a C# console app running in the same server.
Do we have the console app run every minute or so and grab a designated number of items off the queue for insertion into the database? (taking manageable bites)
OR
Do we have the console app always "listen" and hammer items into the database as they come in? (more aggressive approach)

Personally I'd go for the first approach. During those "spike" times, you're going to be hammering the database with potentially 30,000 inserts. Whilst this potentially could complete quite quickly (depending on many variables outside the scope of this question), we could do this a little smarter.
Firstly, by periodically polling, you can grab "x" messages from the queue and bulk insert them in a single go (performance-wise, you might want to tweak the the 2 variables here... polling time and how many you take from the queue).
One problem with this approach is that you might end up falling behind during busy periods. So you could make your application change it's polling time based on how many it is receiving, whilst keeping between some min/max thresholds. E.g. if you suddenly get a spike and grab 500 messages... you might decrease your poll time. If the next poll, you can still get thousand, do it again, decrease poll time. As the number you are able to get drops off, you can then begin increasing your polling time under a particular threshold.
This would give you the best of both world imho and be reactive to the spikes/lull periods.

It depends a bit on your requirement but I would create a service that calls SQLBulkCopy to that bulk inserts every couple of minutes. This is by far the fastests approach. Also if your Spike is 30k records I would not worry too much about falling behind.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx
We have a C# console app running in the same server.
Why not a Service?

What I would do is have the console app always listen to the rabbitmq and then in the console app build your own queue for inserting into the database that way you can throttle the database insertion. By doing this you can control the flow in busy a time by only allowing so many tasks at once and then in slow times you get a faster reaction then polling every so often. The way I would do this is by raising an event and the you know there is something to do in the queue and you can check the queue length to see how many transactions you want to process.

Instead of using a Console Application, you could set up a Windows Service, and set up a timer on the service to poll every n minutes. Take a look at the links below:
http://www.codeproject.com/Questions/189250/how-to-use-a-timer-in-windows-service
http://msdn.microsoft.com/en-us/library/zt39148a.aspx
With a Windows Service, if the server is re-booted, the service can be set up to restart.

What's the easiest way to schedule a function to run at a specific time using C#

If I had a lot of messages in a database that I wanted to send, and each row specified a date and time to send the message, and a flag for if it has been sent.
These won't always be at fixed intervals, and more than 1 message may want to be sent at the same time.
In this case it would just queue them up and send in order of when they were created.
Is the easiest thing to do just to have a function that runs over and over again, once it completes it just runs again
So it would:
Start Running and check the current date/time
Check for any unsent messages
Send all the messages due to go out before and up to the time it started running
Start all over again and take the current date/time
My problem with this is, would it just be horribly inefficient to continuously have a method running, possibly for hours or days without actually sending a message.
The main strain in this case I think would be put on the database, it would constantly be getting hit with a query.
Is there a better way to schedule something like this to happen.
Or just do the above but every time it runs make it wait for 5 minutes before running again.
Does Workflow 4 offer anything suitable for scheduling perhaps?

You could always do a pre-emptive read of the next time value in the series and do a single sleep until then, instead of looping through short sleeps over and over.
Not sure if that's as elaborate as you want though

Maybe have a compiled view in the database which returns messages that are not sent (I assume there's a flag on each record?) and for which the intended send time is prior to the current time. Then a Windows Service or console application on a scheduled interval can hit that view (which can be performance-tuned in the database pretty well, I'd imagine) and send any messages returned by it.

You could use a windows service to accomplish this. Or if you're using MSSQL, you could even use a SQL Server Agent Job.

Several answers has suggested sending some messages then calling sleep until the next message is due to be sent.
How you sleep in this instance is all important.
You can - in theory - tell a thread to sleep for hours, however if during that time the app (or service) needs to shut down then you're in trouble. The process will be terminated, no cleanup will be executed. This is a less than ideal.
Don't get confused between the concept of polling for work to do, and sleeping between polls.
If you've to wait 5 minutes (or 5 hours) before next polling the database, that's fine, however you never want to *sleep for more than a second or two at a time.
What I'd do . . .
Write a windows service. The service has one active thread that polls the database, see's are any messages due to send, and sends them.
It will then poll on a configurable delay (1 minute, 5 minutes, 1 hour, what ever suits).
However it will never sleep for more than a second while it's waiting to poll the database.
If you can be assured that messages can only be added to send after the last message in the DB? If so you can check the time of the next message and not poll until that time.
However, if I find that the next message doesn't need to be sent for 5 hours, is it possible that while I'm waiting a message was added that should be sent in 30 minutes?
If so then you can never trust the "Next message time" and not poll until then, you have to continuously poll on your fixed interval NB worth saying again, your polling interval and your sleep interval are not the same thing.

How about writing a windows service which does this for you. This Windows service will run in the background and check the current time with your db records in a purticular interval (ex : every 5 minutes) and send emails to people and update corresponding records in your tables to set the Email Sent Flag to true
You can even have an SQL job which selects records which are not sent and matches wtih the current time and call a stored procedure which calls dot net assembly to send email . The dot net assembly can use SMTPClient to send emails.

It depends on what you use. Using a scheduled task or a service is perfectly acceptable for the scenario you describe.
You have to be careful though that you do not tie up resources if the process runs too often. It might be more efficient for it to run less often at peak times and more often during off-peak times.

Whatever method you prefer (make a Windows Service, use Task Scheduler, etc..), please bear in mind that your initial suggestion is exactly what is called busy waiting, which is something you should avoid unless you really know what you're doing.

What you describe isn't that bad if you extend it with
"when there are no messages due select the next time a message will be due and sleep till then".
Alternatively use a DB with "notification support" making the whole thing event-driven i.e. the DB sends you an event whenever a message is due.

you can use this one .NET Scheduled Timer for checking timeinervals and running the function(sending messages) at specific time intervals ....

I would say create a windows service with the timer. it may sleep for configured amount of seconds and then compare the datetime from the database. if it matched then send an e-mail & set the flag in the database for sent e-mails.

I recently implemented a windows service which utilized a class called IntervalHeap in the C5 collection class library. I then added a persistence layer which keeps tracks of the items and their intervals in case the service is stopped/crashed.
Has been in production for a few months and has been working very well.

We do this at a financial institution to send out internal e-mails from our intranet applications. Once every 15 minutes, a scheduling software (enterprise scheduler, not a Windows scheduled task) fires off a job. We have a view called PendingEmail on top of a table called EmailQueue that only lists out what needs to be sent this go around (the EmailQueue table has a PopDate, which is an effective date as to when the e-mail should get sent). The application fires off e-mails for whatever it found in the PendingEmails view.
The job sends out a maximum batch size of emails every 15 minutes, marking each record with whether it was successfully sent or whether there was an error (invalid email address, etc.) and what the Exception was, and whether we would like to try re-sending it the next time around. It updates that EmailQueue table all at once, not each record individually. The batch size was put in place to prevent the job from taking more than 15 minutes and stomping on itself.
I don't know that polling every so often is really consuming all that many resources, unless you're going to do it every 5 seconds or something. If you're sending out millions of messages you may need to distribute the work across multiple machines. If you're going to write some custom code, I would use a Timer over Thread.Sleep(), and set the Timer to tick every 5 minutes or whatever interval you'd like to perform work. An event fires on every tick that would subscribe to to start the routine that sends your messages.
See this post on Thread.Sleep() vs. the Timer class:
Compare using Thread.Sleep and Timer for delayed execution

Many databases allow events to be fired by triggers, eg. 'after insert'. The trigger is run by the database process/thread and the actions it can take are database-specific. It could, for instance, call a C or java procedure that signals a named semaphore upon which you emailer is waiting or exec. an emailer app directly. Look at 'trigger' or 'create trigger' for your database.

how to synchronize near realtime reads from a sql server table

We have a reporting app thats needs to update it's charts as the data gets written to it's corresponding table. (the report is based off just one table). Currently we just keep the last read sessionid + rowid (unique combo) in memory and a polling timer just does a select where rowid > what we have in memory (to get the latest rows added). Timer runs every second or so and the fast sql reader does it's job well. So far so good. However I feel this is not optimal because sometimes there are pauses in the data writes due to the process by design. (user clicking the pause button on the system that writes data ..). Meanwhile our timer keeps hitting the db and does not get any new rows. No errors or anything. How is this situation normally handled. The app that writes the data is separate from the reporting app. The 2 apps run on different machines. Bottomline : How to get data into a c# app as and when it is written into a sql server table without polling unnecessarily. thank you

SQL Server has the capability to notify a waiting application for changes, see The Mysterious Notification. This is how SqlDependency works. But this will only work up to a certain threshold of data change rate. If your data changes too frequently then the cost of setting up a query notification just to be immediately invalidated by receiving the notification is too much. For really high end rates of changes the best place is to notify the application directly from the writer, usually achieved via some forms of a pub-sub infrastructure.
You could also attempt a mixed approach: pool for changes in your display application and only set up a query notification if there are no changes. This way you avoid the cost of constantly setting up Query Notifications when the rate of changes is high, but you also get the benefits of non-pooling once the writes settle down.

Unfortunately the only 'proper' way is to poll, however you can reduce the cost of this polling by having SQL wait in a loop (make sure you WAITFOR something like 30ms each loop pass) until data is available (or a set time period elapses, e.g. 10s). This is commonly used when writing SQL pseudoqueues.
You could use extended procs - but that is fragile, or, you could drop messages into MSMQ.

If your reporting application is running on a single server then you can have the application that is writing the data to SQL Server also send a message to the reporting app letting it know that new data is available.
However, having your application connect to the server to see if new records have been added is the most common way of doing it. As long as you do the polling on a background thread, it shouldn't effect the performance of your application at all.

you will need to push the event out of the database into the realm of your application.
The application will need to listen for the message. (you will need to decide what listening means - what port, what protocol, what format etc.)
The database will send the message based on the event through a trigger. (you need to look up how to use external application logic in triggers)