I have an application where whenever a file is uploaded to a directory, I have to call SSIS to parse the XML file.
Can I call a SSIS directly from a .NET Windows service?
Running SSIS package programmatically.
I prefer the second method:
Start DTEXEC.EXE process. DTEXEC is command line utility for executing SSIS packages. See its command line options here: http://msdn2.microsoft.com/en-us/library/ms162810.aspx
Benefits: running package out of process gains reliability. Can be used from any programming language (including .NET 1.1 :)). Easy to pass parameters by setting variables values.
Drawbacks: Also local only. Harder to get information about package progress (but SSIS logging can give you most functionality). Some overhead on starting new process (likely minimal compared to execution time for big packages).
ASP.NET specific: Win32 CreateProcess function ignores the thread impersonation. So if you want DTEXEC to run under account different from ASP.NET process account, you should either make user enter name/password and pass it to Process.Start, or use method described in the following KB to run child process under impersonated account http://support.microsoft.com/kb/889251.
you can run your SSIS package programmatically, as follow:
using System;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
namespace ConsoleApplicationSSIS
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Loading SSIS Service...");
//Application object allows load your SSIS package
Application app = new Application();
//In order to retrieve the status (success or failure) after running SSIS Package
DTSExecResult result ;
//Specify the location of SSIS package - dtsx file
string SSISPackagePath = #"C:\Microsofts\BI\SSIS\ConsoleApplicationSSIS\IntegrationServiceScriptTask\Package.dtsx";
//Load your package
Package pckg = (Package)app.LoadPackage(SSISPackagePath,true,null);
//Execute the package and retrieve result
result = pckg.Execute();
//Print the status success or failure of your package
Console.WriteLine("{0}", result.ToString());
Console.ReadLine();
}
}
}
if you want a complete sample, go to :http://hassanboutougha.wordpress.com/2012/10/13/run-your-ssis-package-progammatically/
I explain how create a simple SSIS package and after how to call it programmatically from a console application. Don't forget to have this assembly :C:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SQLServer.DTSRuntimeWrap.dll to reference runtime ssis namespace
you can also pass your variables programmatically and change also source and destination connections of your ssis package
You can call SSIS programtically, execute the package and change the configuration from a .NET code using DTS runtime. Here is complete code of how you can do it.
You can call the SSIS package from your windows service. But Microsoft.SqlServer.Dts should be installed into the system where windows services are going to run. If you have installed DTS installed in that machine, directly call the SSIS package. If it is not installed then you should do the following.
Create the SSIS package
Create the job which runs the SSIS package
In your ADO.NET[resides in windows services code], Call stored
procedure which runs job[configured to run the SSIS package].
Following is an example should be called from your .NET code.
EXEC msdb.dbo.sp_start_job N'YourJobName'
Hope this helps!
Updating this pretty old question:
On SQL Server 2012 you can do this simply by creating stored procedure that will call to create_execution and set_execution_parameter
Step-by-step guide can be found here: https://blogs.msdn.microsoft.com/biblog/2013/05/07/step-by-step-of-executing-ssis-2012-package-through-stored-procedure/
Related
I have the following code which seems to works OK for executing an SSIS package from c# BUT every time the variable "Resp" returns a Failure even if the package passes when I execute it directly in SSIS.
Again, the package contains a Script component that writes to an SQL server table. All that works OK when the package is executed directly in SSIS but nothing happens when the same package is called via C# with the code below. I cant work out what I am doing wrong. help!
string packageLocation = #"c:\packageLocationPath";
Package pkg;
Application app;
app = new Application();
pkg = app.LoadPackage(packageLocation, null);
var Resp = pkg.Execute();
Detecting error
First you have to read the errors raised by the package. There are two options to detect these errors:
(1) loop over errors
You can loop over the package errors by accessing Errors property. As example:
if(p.Errors.Count > 0){
foreach(DtsError err in p.Errors){
Messagebox.Show(err.Description);
}
}
More information at:
How to get all errors of all SSIS packages in a solution
(2) Enable logging from package
You can capture all errors, warning and information from a package by enabling logging option:
Add and configure logging
Possible failure causes
Make sure that if you are using windows authentication to connect to SQL, that the user account used to execute the application via C# is granted to make a connection.
If the package is accessing a file system, make sure that the user has the required permissions to access these files
Setup
I have a vs2010 SSIS package that I can run from visual studio. I have deployed it to my local 2012 server, and I can also execute that package from SSMS. In SSMS, I see the package here:
\Integration Services Catalogs\SSISDB\DAT_Load\Projects\UploadSYS.dtsx
Note: vs2010 does not give me an option to deploy a package anywhere but in a server, and then only in Integration Services Catalogs. Once in there, the MSDB database does not have an entry in the sysssispackages table.
Previously, it was adequate to bring up SSMS and run the package from there(right-click & execute). Now, I have to execute this from a C# web application. Furthermore, I need to trap progress messages and such through events.
Effort
I was able to determine how to set up the event trapping and I got myself to the point where I should have been able to execute the package from code:
public DTSExecResult ExecutePackage(string packageName, HttpContextBase context)
{
string ppath = ConfigurationManager.AppSettings[packageName + "Package"];
string pserv = ConfigurationManager.AppSettings[packageName + "Server"];
string puser = ConfigurationManager.AppSettings[packageName + "User"];
string ppass = ConfigurationManager.AppSettings[packageName + "Pwd"];
_context = context;
_pkgLocation = "";
_app = new Application();
_pkg = _app.LoadFromSqlServer(ppath, pserv, puser, ppass, _SSISEvents);
_pkgResults = _pkg.Execute(_connections, _variables, _SSISEvents, _log, null);
return _pkgResults;
}
Problem
I cannot locate the package. When I reach the LoadFromSqlServer statement, I receive an error that says:
Cannot find folder "\Integration Services Catalogs\SSISDB\DAT_Load\Projects\UploadSYS.dtsx"
The same thing happens for variations in the path (variable = ppath):
\Integration Services Catalogs\SSISDB\DAT_Load\Projects\UploadSYS.dtsx
\SSISDB\DAT_Load\Projects\UploadSYS.dtsx
\DAT_Load\Projects\UploadSYS.dtsx
etc.
Running this from the command line or a stored procedure is not an option.
So, can anyone tell what I am missing here? Does the Application object need to initialize something? Is this even possible?
Taking another bite at the problem but see Set SSIS database package path and SSIS Organization for background reading.
Until SSIS 2012, if packages were deployed to SQL Server, they lived in the msdb. The .NET API for interacting with them was the same across versions.
With the 2012 release of SSIS, we have two different deployment models. Package deployment, which is the "classic" model is alive and fully supported. The same code for running a package on 2005 will work for 2012 package deployment model projects. This is the Microsoft.SqlServer.Dts.Runtime Namespace
Your code is attempting to load a 2012 solution built using the "project deployment model" with the "package deployment model" API. This is the Microsoft.SqlServer.Management.IntegrationServices Namespace and the two don't mix.
Your options are to switch your project back to the Package deployment model or update your code. In the first linked question, I provided the VB.NET implementation for running an SSIS package in the SSISDB catalog. There is some way of running a .ispac file because I see the option in dtexec but I'm not seeing the specific method. This is mechanism VS/SSDT uses when it runs the packages locally.
Problem!
I have a little python script, which goes throught a web page (http-crawling). This web-page is hosted inside the intranet and uses NTLM authentication to gather access to it.
So, I found this task (retrieve http-content) easily programmable using python, instead of trying to re-write the whole python script to C# and then use it througth "Script Task" on SSIS, in order to complete the task.
Hint!
I've looked up closely to SSIS tools and I found that there is a Control Flow named "Execute Process Task", which lets you to execute Win32 executables.
But the problem resides in how to call my python script since it's not executable and needs to be interpreted by the python interpreter (if you'll forgive the repetition). So, I could easily end up building a simple ".bat" file that calls both the python script and the interpreter. And then execute that file through SSIS "Execute Process Task".
Question!
Is there any other way to implement this? (neat way)
Edit #1
Usage
The information retrieved from the script will be storing that information into a table from a database, So that information will be accessed trough the database table from another SSIS process.
I'm retrieving the information from different sources (flat files, database tables, http request, ...) in order to archive that information into a database that could be posted in a web services and then accessed from a Excel project.
Thanks in Advance!
The easiest, at least to my brain, mechanism for using IronPython from the confines of SSIS would be to invoke the external process and dump to a file and then use that as a source for a dataflow.
That said, I was able to host an IronPython app from C# and use the returned data to populate the output buffers and interact with that data in the pipeline. I've only had one machine to perform this on so I'm listing everything I recall doing until the package went green.
Prerequisites
This article set me down the path of how to make this work. Hosting IronPython in a C# 4.0 program I would strongly urge you to create a C#/VB.NET console app and get your IronPython integration working there first as SSIS is going to add an additional layer to everything.
There may be the ability to host older versions of IronPython within C# without requiring the 4.0 framework but that's far beyond the realm of my competency. What I can say is that to use the 4.0 framework, you are looking at SQL Server 2012. A 2008 package can target up to the 3.5 framework (default is 2.0).
Global Assembly Cache, GAC for short. It is a special place in Windows where signed assemblies can live. SSIS may be able to use assemblies that aren't in the GAC, but I've not had luck doing so. This case was no different. My Console app worked fine but when I copied that code into SSIS, it'd tank with Could not load file or assembly 'Microsoft.Scripting... error messages. Blessedly, IronPython-2.7.2.1 (and probably previous versions) are strongly signed dlls. That means you can and must add them into the GAC.
In your Visual Studio directory, look for the Visual Studio Command Prompt (2010).
Assuming your IronPython installation folder is C:\tmp\IronPython-2.7.2.1\IronPython-2.7.2.1 you would type cd C:\tmp\IronPython-2.7.2.1\IronPython-2.7.2.1 Then I registered the following 3 assemblies
C:\tmp\IronPython-2.7.2.1\IronPython-2.7.2.1>gacutil -if Microsoft.Dynamic.dll
Microsoft (R) .NET Global Assembly Cache Utility. Version 4.0.30319.1
Copyright (c) Microsoft Corporation. All rights reserved.
Assembly successfully added to the cache
C:\tmp\IronPython-2.7.2.1\IronPython-2.7.2.1>gacutil -if IronPython.dll
Microsoft (R) .NET Global Assembly Cache Utility. Version 4.0.30319.1
Copyright (c) Microsoft Corporation. All rights reserved.
Assembly successfully added to the cache
C:\tmp\IronPython-2.7.2.1\IronPython-2.7.2.1>gacutil -if Microsoft.Scripting.dll
Microsoft (R) .NET Global Assembly Cache Utility. Version 4.0.30319.1
Copyright (c) Microsoft Corporation. All rights reserved.
Assembly successfully added to the cache
My SSIS project, I had set the Run64bitRuntime to False but in retesting, it does not matter. The default it True and that seems to work fine.
Python script - I don't have enough of a background to make the integration between C# and .NET DLR languages more graceful. It'd have been nice to supply a string or something containing the script I wanted to execute and perhaps that's what a script block is about but I don't have time to investigate. So, this solution requires a script file sitting out somewhere on disk. I had trouble with the imports working from a hosted script (no module named X exceptions). Undoubtedly there's some magic with class paths and all that stuff that needs to provided to the host to make it work well. That's probably a different SO question btw.
Set up
I have a file sitting at C:\ssisdata\simplePy.py
# could not get a simple import to work from hosted
# works fine from "not hosted"
#import os
def GetIPData():
#os.listdir(r'C:\\')
return range(0,100)
After adding a script task to the Data Flow, I configured it to have a single column on the output buffer (wstr 1000). I then used this as my source code.
using System;
using System.Collections.Generic;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using IronPython.Hosting;
using Microsoft.Scripting.Hosting;
/// <summary>
/// Attempt to use IP script as a source
/// http://blogs.msdn.com/b/charlie/archive/2009/10/25/hosting-ironpython-in-a-c-4-0-program.aspx
/// </summary>
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
/// <summary>
/// Create data rows and fill those buckets
/// </summary>
public override void CreateNewOutputRows()
{
foreach (var item in this.GetData())
{
Output0Buffer.AddRow();
Output0Buffer.Content = item;
}
}
/// <summary>
/// I've written plenty of code, but I'm quite certain this is some of the ugliest.
/// There certainly must be more graceful means of
/// * feeding your source code to the ironpython run-time than a file
/// * processing the output of the code the method call
/// * sucking less at life
/// </summary>
/// <returns>A list of strings</returns>
public List<string> GetData()
{
List<string> output = null;
var ipy = Python.CreateRuntime();
dynamic test = ipy.UseFile(#"C:\ssisdata\simplePy.py");
output = new List<string>();
var pythonData = test.GetIPData();
foreach (var item in pythonData)
{
output.Add(item.ToString());
}
return output;
}
}
Quick shot of what my references look like
Click the run button and great success
how neat do you want to be? I think you option is fine and you wont find any simpler way to solve your problem. You have a list of files and need to execute them, that's it.
One thing I can think of is adding all the file paths to a sql table and execute them with xp_cmdshell
you need to enable it on the sql instance (I assume you have one since you are considering using SSIS)
EXEC sp_configure 'show advanced options', 1
GO
reconfigure
go
EXEC sp_configure 'xp_cmdshell', 1
GO
reconfigure
go
than you can loop on the table and for each row execute:
exec master.dbo.xp_cmdshell 'your_script'
How about the simple solution (without .bat):
In the Editor for "Execute Process Task" set Exectutable to your Python INTERPRETER
C:\...\Python34\python.exe
for Arguments set your script path first followed by any args for the script
H:\...\test\helloworld.py -a 1 -b 2
don't forget your WorkingDirectory
H:\...\test
By refering msdn website and other community i was able to run my packages programmatically. I am using a console .net application to run my ssis package. i have hardcorded the location package. it runs smoothly
DTSExecResult pkgResults;
Application app = new Application();
Package p = app.LoadPackage(#"C:\somelocation");
pkgResults = p.Execute();
if (pkgResults == DTSExecResult.Success)
Console.WriteLine("Package ran successfully");
else
Console.WriteLine("Package failed");
the problem i am facing is when i fails i am unable to record which ran successfully and which has failed or stopped in between.
is there anyway to log my progress programmatically in C#, should i use custom method or use their inbuild methods like DTS.LogProvider.
is there anyway i could send a email stating the it has succeeded or failed.
This might be helpful.
There is a code snippet also.
http://msdn.microsoft.com/en-us/library/ms136023.aspx
Logging is available in ssis. You can program any number of tasks to execute on a given event, including a 'send mail' task. See this article to configure logging options in ssis
http://msdn.microsoft.com/en-us/library/ms167456.aspx
I have scheduled sql agent task which runs an SSIS package. I want to be able to run the SSIS package from .net. Is there a way to either run the SSIS package directly or at least run the SQL agent task which would in turn run the SSIS package.
If it helps it is for a .net 3.5 web app written in C#
Thanks!
The options that are available to run a SSIS package are -
Run package programmatically using SSIS Object Model. This is discussed in details in Books Online here.
An Example:
using System;
using Microsoft.SqlServer.Dts.Runtime;
namespace RunFromClientAppCS
{
class Program
{
static void Main(string[] args)
{
string pkgLocation;
Package pkg;
Application app;
DTSExecResult pkgResults;
pkgLocation = "<package path>\CalculatedColumns.dtsx";
app = new Application();
pkg = app.LoadPackage(pkgLocation, null);
pkgResults = pkg.Execute();
Console.WriteLine(pkgResults.ToString());
Console.ReadKey();
}
}
}
Start DTEXEC.EXE process. DTEXEC is command line utility for executing SSIS packages. See its command line options here.
Use SQL Agent. You can configure an Agent job to run your package (either do it manually in advance if the package is static, or programmatically using SMO or using SQL stored procedures just before running the package), and then start it programmatically using SMO or sp_start_job.
Use some other utility to start DTEXEC for you.
Create a custom application that will run the package (either using OM as described in method #1, or using DTEXEC as in method #2). Expose it as a web service or DCOM class, call this service from your program.
Invent your own :)
Reference: Running SSIS Package Programmatically
Yes. Look into Microsoft.SqlServer.Dts.Runtime namespace. The Package class will provide the methods to run it.