I am developing a data optimization project. It has a client side web app to receive tasks from users. Let's say the tasks are some heavy calculations that cannot be done easily by normal systems. The program takes a long while for big amount of data to be calculated. So what I am trying to do is receive calculation orders from my web application and have a windows service on my server side to listen for new tasks to be done.
I would like my service to be listening to the data that is being inserted into my Tasks table and run the calculator based on the time of those dates. I will of course would have to deal with some multi-threading. And maybe if the program is busy, the other processes would have to wait.
I also don't mind having a small GUI for my application to see which orders are now being processed and whether my service is busy or idle.
I first thought about adding SQL Server jobs in my data base to query the table frequently and run the application based on the dates. But that does not look like a nice solution to me. What I want is a nimble ready-to-serve service who becomes aware when we have new data in the database and decides what to do.
I don't insist on Windows Services particularly here. So any good idea is welcome.
I've done something similar in Powershell.
What I have:
A "queue" table where requests gather
Requests come in by clients calling a procedure
A "queue history" table - request already "treated" go from "queue" to "queue_history" (delete trigger)
A powershell loop reading queue table
Requests from queue are started independently via start-job command, those are actually separate Powershell instances (my tasks are mostly some external exe calls, but I think could be usefull for stored procedure calls also).
The powershell loop also report his "heartbeat" to a table.
Also watches a table and file existance for STOP flag so I can stop it.
Related
The task is to create Windows Service which should periodically connect to the SQL Server database (it contains GPS data from hundreds or thousands of cars), read data from the table, process it and write result to another table.
The problem is that depends on how much data are there in the database processing time can vary from milliseconds to several hours.
If there are a lot of data it should wait until previous processing will end and then start another iteration.
If there are not much data it should accumulate at least 500 GPS records, process it and start new iteration.
Please provide your examples with C#.
P.S.
Processing of GPS data means generating complex car events, for example, defining car overspeed, stop points, entering specific geographical zone and so on...
From the algorithmic point of view generating some of these events can be resource intensive.
P.P.S
I have already create it but as console application with infinite cycle, but I'm new to windows services and I don't know how to realize such functionality as windows service correctly.
I would
create a watchdog service that checks if the processing
application is running every X seconds/minutes/hours (depending on
how often you want to process the data).
If it is running then wait until the next scheduled event time and check again.
If it's not running then start the processing application.
I have researched a lot and I haven't found anything that meets my needs. I'm hoping someone from SO can throw some insight into this.
I have an application where the expected load is thousands of jobs per customer and I can have 100s of customers. Currently it is 50 customers and close to 1000 jobs per each. These jobs are time sensitive (scheduled by customer) and can run up to 15 minutes (each job).
In order to scale and match the schedules, I'm planning to run this as multi threaded on a single server. So far so good. But the business wants to scale more (as needed) by adding more servers into the mix. Currently the way I have it is when it becomes ready in the database, a console application picks up first 500 and uses Task Parallel library to spawn 10 threads and waits until they are complete. I can't scale this to another server because that one could pick up the same records. I can't update a status on the db record as being processed because if the application crashes on one server, the job will be in limbo.
I could do a message queue and have multiple machines pick from it. The problem with this is the queue has to be transactional to support handling for any crashes. MSMQ supports only MS DTC transaction since it involves database and I'm not really comfortable with DTC transactions, especially with multi threads and multiple machines. Too much maintenance and set up and possibly unknown issues.
Is SQL service broker a good approach instead? Has anyone done something like this in a production environment? I also want to keep the transactions short (A job could run for 15,20 minutes - mostly streaming data from a service). The only reason I'm doing a transaction is to keep the message integrity of queue. I need the job to be re-picked if it crashes (re-appear in the queue)
Any words of wisdom?
Why not having an application receive the jobs and insert them in a table that will contain the queue of jobs. Each work process can then pick up a set of jobs and set the status as processing, then complete the work and set the status as done. Other info such as server name that processed each job, start and end time-stamp could also be logged. Moreover, instead of using multiple threads, you could use independent work processes so as to make your programming easier.
[EDIT]
SQL Server supports record level locking and lock escalation can also be prevented. See Is it possible to force row level locking in SQL Server?. Using such mechanism, you can have your work processes take exclusive locks on jobs to be processed, until they are done or crash (thereby releasing the lock).
First, I apologize for the seemingly dumb question I'm not very strong with databases.
I'm re-designing a desktop application in C# for use over a local network that basically is a highly specialized ticket tracking system. Essentially when a user launches the application they'll be asked for their credentials to gain access to the system and then the application will query the central database for data (currently a MySQL server running on a local machine), displaying it on the screen.
My question is if four users are connected and two users enter new data, what is the most efficient method of letting each user know of the new data? Would it be simply to query the database and update the application with the new data on a timer? Or would creating a server application to sit in between the user and the database server to perform queries itself and notify each connected user of updated data?
See it all depends how important is it to notify the clients in real time about the changes in your database. If your clients have no issue with a delay of minute or two you can probably go for the timer approach. But if they really wish the data to be real time (delay of less than 1-2 sec), go for the other approach. Create a separate service which polls the database and notify the client application for any update. For this you can make use of socket listners.
Hope that helps !!
4 users? On a local LAN? Using simple, indexed queries? Just poll the DB from the clients. Kick off a thread at application start up and have it run a query every 2-5 seconds, then notify the user using whatever is appropriate for background threads updating GUIs in .NET.
This is straightforward, don't over think it. The "hardest" part is the asynchronous notification of the user (which depending on your GUI layout and required actions is probably not a big deal either, thus the quotes).
We have a reporting app thats needs to update it's charts as the data gets written to it's corresponding table. (the report is based off just one table). Currently we just keep the last read sessionid + rowid (unique combo) in memory and a polling timer just does a select where rowid > what we have in memory (to get the latest rows added). Timer runs every second or so and the fast sql reader does it's job well. So far so good. However I feel this is not optimal because sometimes there are pauses in the data writes due to the process by design. (user clicking the pause button on the system that writes data ..). Meanwhile our timer keeps hitting the db and does not get any new rows. No errors or anything. How is this situation normally handled. The app that writes the data is separate from the reporting app. The 2 apps run on different machines. Bottomline : How to get data into a c# app as and when it is written into a sql server table without polling unnecessarily. thank you
SQL Server has the capability to notify a waiting application for changes, see The Mysterious Notification. This is how SqlDependency works. But this will only work up to a certain threshold of data change rate. If your data changes too frequently then the cost of setting up a query notification just to be immediately invalidated by receiving the notification is too much. For really high end rates of changes the best place is to notify the application directly from the writer, usually achieved via some forms of a pub-sub infrastructure.
You could also attempt a mixed approach: pool for changes in your display application and only set up a query notification if there are no changes. This way you avoid the cost of constantly setting up Query Notifications when the rate of changes is high, but you also get the benefits of non-pooling once the writes settle down.
Unfortunately the only 'proper' way is to poll, however you can reduce the cost of this polling by having SQL wait in a loop (make sure you WAITFOR something like 30ms each loop pass) until data is available (or a set time period elapses, e.g. 10s). This is commonly used when writing SQL pseudoqueues.
You could use extended procs - but that is fragile, or, you could drop messages into MSMQ.
If your reporting application is running on a single server then you can have the application that is writing the data to SQL Server also send a message to the reporting app letting it know that new data is available.
However, having your application connect to the server to see if new records have been added is the most common way of doing it. As long as you do the polling on a background thread, it shouldn't effect the performance of your application at all.
you will need to push the event out of the database into the realm of your application.
The application will need to listen for the message. (you will need to decide what listening means - what port, what protocol, what format etc.)
The database will send the message based on the event through a trigger. (you need to look up how to use external application logic in triggers)
I have a require ment to read data from a table(SQL 2005) and send that data to other application for every 5 seconds. I am looking for the best approach to do the same.
Right now I am planning to write a console application(.NET and C#) which will read the data from sql server 2005(QUEUE table which will be filled through different applications) and send to other application through TCP/IP(Central server). Run that console application under schedule task for every 5 seconds. I am assuming scheduled task will take care to discard new run event if task is already running(avoid to run concurrent executions).
Does any body come accross similar situation? Please share your experience and advice me for best approach.
Thanks in advance for your valuable time spending for my request.
-Por-hills-
We have done simliar work. If you are going to query a sql database every 5 seconds, be sure to use a stored procedure that is optimized to be very fast. It should not update data unless aboslutely necessary. This approach is typically called 'polling' and I've found that it is acceptable if your sqlserver is not otherwise bogged down with too many other calls.
In approaches we've used, a Windows Service that does the polling works well.
To communicate results to another app, it all depends on what your other app is doing and what type of interface you can make into it, and how quickly you need the results. The WCF class libraries from Microsoft provide many workable approaches for real time communication. My preference is to write to the applications database, and then have the application read the data (if it works for that app). If you need something real time, WCF is the way to go, and I'd suggest using a stateless protocol like http if < 5 sec response time is required, (using standard HTTP posts), or TCP/IP if subsecond response time is required.
since I assume your central storage is also SQL 2005, have you considered using what SQL Server 2005 offers out of the box to achieve your requirements? Rather than pool every 5 seconds, marshal and unmarshal TCP/IP, implement authentication and authorization for the TCP/IP pipe, scale TCP transmission with boxcaring, manage message acknowledgments and retries, deal with central site availability, fragment large messages, implement fairness in transmission and so on and so forth, why not simply use Service Broker? It does all you need and more, out of the box, already tested, already tuned for performance and scalability.
Getting reliable messaging right is not trivial and you should focus your efforts in meeting your business specifics, not reiventing the wheel.
I would recommend writing a Windows Service (since you are C#) that has some timer which runs every 5 seconds. That way you wont be starting and stopping an application all the time, it can run even when there is no one logged into the machine, and it will automatically start when the machine is restarted.
For one of my projects, I needed to do something periodically. I opted for a service and set up a timer that takes care of reading the data. You might consider that solution. It has worked well for me.
I suggest to create a windows service and not an application and to perform the timing yourself - create a timer and execute one step on each timer event. For the communication you have many choices - I would consider using standard technologies like a webservice or Winows Communication Foundation.
Besides this custom solution I would evaluate if the task can be solved using Microsoft Integration Services .
Finally other question comes to mind - why do you need this application? Why doesn't/don't the application(s) consuming the data query the database? Is the expensive polling required? Is it possible for the data producers to signal the availibilty of new data directly to the data consumers?
I am not sure about the details of your project, specifically related to security but maybe it would be better to create an SSIS package and schedule it as a job?