Here's an interesting problem. I have an ETL script written in c# that I have been running manually on a somewhat regular basis. It is used to update my web app's database.
I want to automate the ETL process AND create an interface for the web app admins to manually start the ETL process.
I could have sql server kick off the ETL process on a schedule and implement a button or something on a web page that will do the same thing, but I don't want to put my code in 2 different places because I don't want to update it in 2 places when it changes. But I don't know how to make my web app tell SQL server to manually start a scheduled process. Can this be done?
OR
I could somehow implement the scheduling in the web app itself, but by now most people are familiar with the problems that are faced when trying that (app may not be running at certain times, must wait on request to start a process (without some trickery)). Also, since the ETL process takes a while, I don't want to make some poor end user wait on a response, so it would definitely have to use a new thread.
What else could I do? What would you do?
You mentioned you are using SQL Server. What version is it? Is SQL Server Integration Services (which is the full-blown ETL tool for the platform) an option for you? If you want scheduled Extraction, Transformation and Load jobs, SSIS is a great tool on the Microsoft platform and is included in most SQL Server licenses.
Or, check out Quartz.
Related
I just started my internship as dev student.
I need to write a program in C# that will read a CSV productfeed, then parse it, map it and insert OR update this into a SQL Server database.
The program will run on a Windows VPS and has to be executed daily as the CSV will also be updated daily. I have to automate this process.
SO to come to my questions:
1. Can I go with a console application? (because I don't need any GUI)
2. Will a program like this with nested loops for insert/update crash or be too heavy on the DB as the CSV can contain +100k records?
Any guidance in the right direction is much apreciated
YES. You can develop a Console Application indeed. Doing something with Windows Forms or similar (i.e a C# app with a form and a single button with code.) would be a bad idea as you don't need a GUI and would be much harder to automate.
What options do you have?
Console application: Easy to develop. You can schedule it using Windows Task Scheduler.
Windows Service: A little bit harder to develop. It is much easier to control if it is running or not.
WCF or Restful Web Service: Harder to develop but more open if it needs to evolve in the future. Easier to interconnect with other servers as execution can be triggered externally.
NO, potentially no, although with bad code you can manage to create problems in SQL Server.
My advice: If you don't have much time/skills, go for Console App. If you can, create an asynchronous RESTFUL web service implementing GET to see the status of last/current execution, POST to trigger a new execution and return correct HTTP Response codes.
1. Can I go with a console application? (because i don't need any GUI)
Yes you can you Console Application and Task schedule for run this program daily.
Also you can write Windows Service - in this case your program will run in background.
2. Will a program like this with nested loops for insert/update crash or be too heavy on the DB as the CSV can contain +100k records?
100K+ rows its nothing for MS SQL. But sure you need write code correct. I will parse its and then Insert in db in multi thread. But it depends on what you need to do.
Technically you could use any application type you want, but If I'm gonna write this application I will pick Windows Service Application , because you want the application to run repeatedly in an automated way.
and for the performance, I don't think that hitting the database for every record it the best practice here may be you should look for Bulk Insert.
I have some C# code and i want that this C# should run exactly at 5 in the evening for that i need to create a job on a server. For creating job i need an ETL package.
How to make ETL of C# code?
The obvious answer would be to use a script task in the control flow. You can put your C# code there and have it execute. The tricky bit is if your code needs access to non-standard libraries in which case you'll need those deployed to the GAL on the server.
You could also use the execute process task if you have a compiled executable but in that case I would imagine you could just copy it to the server and use windows scheduler.
You can also directly run executables using the SQL agent: How execute exe file from sql agent or job?
I'm in the process of building a Scheduler in C sharp. The scheduler should ping the DataBase Queue and if anything new is written in the Database it should pick that up and send it to the remote server for processing. (The files will be uploaded to the database by the client using a WCF webservice) how do you guys think i should handle this situation? any type of help would be highly appreciated. I'm still in the design phase.
Thanks.
As this is a periodic task, I'd consider writing an app that does the job when run (with no scheduling). Then add it to the Windows Task Scheduler.
Take a look at Quartz.NET. It easily lets you set up a scheduler and plug in Jobs (i.e. Send the files to the remote server) and Triggers (How often do you want it to run..) It has an excellent Fluent API as well as support for CRON style expressions.
There are also tutorials on the site, and google will find you a bunch more.
#spender's idea is easy to implement. Another option is using a database trigger to call CLR proc
http://msdn.microsoft.com/en-us/library/ms131094%28v=SQL.90%29.aspx
Can I call a C# function by a SQL Server trigger?
I am writing a software for a company in c# which is intended to run on windows platform.
One of my requirements is to allow the user to schedule back ups.
That is, the user will set a time where the database will be backed up automatically by the computer.
On the linux platform I would have use crons but I am a bit lost on the windows platform. I do not want the software itself to be actually opened for the back up to run. I want it to be carried out even if the software itself is not running.
My best bet is to use windows scheduler or create a custom service which will run at start up.
Can anyone point me to how to actually achieve this? Any constructive suggestions are welcomed.
Thanks.
For info the Windows "AT" command is somewhat similar to cron. You can get help from the command line thus:
AT /?
I wouldn't necessarily recommend it for a db backup. Either create a Windows scheduled task, or to backup a SQL Server database, use SQL server's built in scheduler.
Another alternative would be to create a windows service to handle the task. Then you could write any code needed (e.g. Backup / Email logs, etc) quickly and easily, and it would work w/o your application running.
There are ways to accomplish the same task with Task Scheduler built into windows, but just an alternative that I would prefer.
Let me give a back ground for everybody before I go to my problem. My company hosts website for many clients, my company also contracts some of the work to another company.
So when we first set up a website with all the informations to our clients, we pass that information to the other company we contracted and three of us have the same data. Problem is once the site is up and running, our clients will change some data and when ever they do that we should be able to update our contracted company.
The way we transfer data to the contracted company is by using a web service (httppost, xml data). Now my question is what it the best way to write a program which sends updated data to the contracted company everytime our clients change some data.
1) Write a windows service having a timer inside my code where every 30min or so connects to the database and find all changes and send it to the contracted company
2) Write the same code as #1 (with out the timer in it) but this time make it a simple program and let windows scheduler wake it every 30min
3) Any other suggestion you may have
Techenologies available for me are VS 2008, SQLServer 2005
Scheduled task is the way to go. Jon wrote up a good summary of why services are not well suited for this sort of thing: http://weblogs.asp.net/jgalloway/archive/2005/10/24/428303.aspx
A service is easy to create and install and is more "professional" feeling so why not go that way? Using a non-service EXE would also work of course and would be slightly easier to get running (permissions, etc.) but I think the difference in setup between the two is nearly negligible.
One possible solution would be to add a timestamp column to your data tables.
Once this is done, you can have one entry in each table that has the last collected time by your contracted company. They can pull all records since that last time and update their records accordingly.
A Windows Service is more self contained, and you can easily configure it to start up automatically when the OS is starting up. You might also need to create additional configuration options, as well as some way to trigger the synchronization immediately.
It will also give you more room to grow your functionality for the service in the future.
A standalone app should be easier to develop though, however you are reliant on the windows scheduler to execute the task always. My experience has been that it is easier to mess up things with the windows scheduler and have it not run, for example in cases where you reboot the OS but no user has logged in.
If you want a more professional approach go with the service, even though it might mean a little bit more work.
A windows service makes more sense in this case. Think about what happens after your server is restarted:
With a Windows Application you need to have someone restart the application, or manually copy a shortcut to the startup folder to make sure the application gets launched
OR,
With a Windows Service you set it to start automatically and forget about it. When the machine reboots your service starts up and continues processing.
One more consideration, what happens when there is an error? A Windows application would likely show an error dialog and wait for input before continuing; whereas a service would log the error in the event log and carry on.