I am new to both VC# and SSIS. But here is my scenario, I have data from multiple projects within a single SQL database i.e. multiproject mode (One SQL database storing data from multiple projects). This data is separated based on proj_ID field. I am trying to create C# application which pulls this proj_id in one of it's combobox fields and runs the SSIS package on the project after I click on Export data button. Now I want to use this project id in SSIS package so that package dataflow should only execute on that project.
I have added this in C# code and not sure if this is correct:
public void button2_Click(object sender, EventArgs e)
{
//Start the SSIS Here
try
{
Microsoft.SqlServer.Dts.Runtime.Application app = new Microsoft.SqlServer.Dts.Runtime.Application();
Package package = null;
package = app.LoadPackage(#"C:\SSIS_Projects\XXX_Project\XXXX_Project\Package.dtsx", null);
Microsoft.SqlServer.Dts.Runtime.Variables myVars = package.Variables;
myVars["projectroot"].Value = projectroot;
myVars["path8"].Value =path8;
myVars["PROJ_NAME"].Value = comboBox1.ValueMember;
myVars["PROJ_ID"].Value = comboBox2.ValueMember;
//Excute Package
// working Microsoft.SqlServer.Dts.Runtime.DTSExecResult results = package.Execute();
Microsoft.SqlServer.Dts.Runtime.DTSExecResult results = package.Execute(null, myVars, null, null, null);
if (results == Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure)
{
foreach (Microsoft.SqlServer.Dts.Runtime.DtsError local_DtsError in package.Errors)
{
Console.WriteLine("Package Execution results: {0}", local_DtsError.Description.ToString());
Console.WriteLine();
}
}
}
catch (DtsException ex)
{
throw new FileNotFoundException("[Package.dtsx not found in directory]", ex);
}
}
P.S: PROJ_ID is main variable to call in package but I might need projectroot, name and path as well.
If this is correct, then How do I define and use these variable in SSIS package?
how to proceed further in SSIS package? I can guess that I have to write script task but again it will take me lot of time to learn and write. If you can guide me and provide some sample code then it will be very helpful.
Thanks in advance.
Vishal
I don't remember why, but I do remember that when I needed to execute an SSIS package with variables from C#, here is the approach that finally worked for me:
Create a job that calls the SSIS package.
Create a table that holds the values of the variables used by the package. The table must also include some kind of "Job Result" column.
Include a step at the beginning of the package that reads the values of its variables from the first row in the table that has NULL for JobResult.
Include a step at the end of the package that updates the JobResult column in the variables table, with "Success", "Failure" or whatever you want, as long as it's not NULL.
In C#, populate the table with the variable values you want, then start the job.
Hope this helps.
Related
There is a requirement to extract source,destination and column names of source and destination. Why am I trying to do this is because I have thousands of packages and opening each package has on an average 60 to 75 of columns and listing all required info will take huge amount of time and its not a single time requirement and this task is done manually every two months in my organization currently.
I'm looking for some ways to reverse engineer keeping all packages in a single folder and then go through each package and get the info and put it in some spreadsheet.
I thought of opening package in xml and get the info of interested node and put in spreadsheet which is little cumbersome. Please suggest what are the available libraries to start with it.
SQL server provide assemblies to manipulate packages programmatically.
To do a reverse engineering (deserialize a dtsx package), You have to do this by looping over packages and read them programmatically, just follow this detailed link
Reading DTS and SSIS packages programmatically
There is another way (harder way and not recommended) to achieve this , by reading dtsx as text file and parse the xml content. check my answer at the following question to get an example:
Automate Version number Retrieval from .Dtsx files
Hint:
just open the package in visual studio. go to the package explorer Tab (near control flow and data flow tabs) you will find a treeview. it will leads you the way you have to search for the component you need
Update 1 - C# Script # 2019-07-08
If you are looking for a script that list all package objects you can use a similar script:
using System;
using DtsRuntime = Microsoft.SqlServer.Dts.Runtime;
using DtsWrapper = Microsoft.SqlServer.Dts.Pipeline.Wrapper;
public void Main()
{
string pkgLocation;
DtsRuntime.Package pkg;
DtsRuntime.Application app;
DtsRuntime. DTSExecResult pkgResults;
pkgLocation =
#"D:\Test\Package 1.dtsx";
app = new DtsRuntime.Application();
pkg = app.LoadPackage(pkgLocation, null);
//List Executables (Tasks)
foreach(DtsRuntime.Executable tsk in pkg.Executables)
{
DtsRuntime.TaskHost TH = (DtsRuntime.TaskHost)tsk;
MessageBox.Show(TH.Name + "\t" + TH.HostType.ToString());
//Data Flow Task components
if (TH.InnerObject.ToString() == "System.__ComObject")
{
try
{
DtsWrapper.MainPipe m = (DtsWrapper.MainPipe)TH.InnerObject;
DtsWrapper.IDTSComponentMetaDataCollection100 mdc = m.ComponentMetaDataCollection;
foreach (DtsWrapper.IDTSComponentMetaData100 md in mdc)
{
MessageBox.Show(TH.Name.ToString() + " - " + md.Name.ToString());
}
}
catch {
// If it is not a data flow task then continue foreach loop
}
}
}
//Event Handlers
foreach(DtsRuntime.DtsEventHandler eh in pkg.EventHandlers)
{
MessageBox.Show(eh.Name + " - " + CM.HostType);
}
//Connection Manager
foreach(DtsRuntime.ConnectionManager CM in pkg.Connections)
{
MessageBox.Show(CM.Name + " - " + CM.HostType);
}
//Parameters
foreach (DtsRuntime.Parameter Param in pkg.Parameters)
{
MessageBox.Show(Param.Name + " - " + Param.DataType.ToString());
}
//Variables
foreach (DtsRuntime.Variable Var in pkg.Variables)
{
MessageBox.Show(Var.Name + " - " + Var.DataType.ToString());
}
//Precedence Constraints
foreach (DtsRuntime.PrecedenceConstraint PC in pkg.PrecedenceConstraints)
{
MessageBox.Show(PC.Name);
}
}
References
Loading and Running a Local Package Programmatically
Update 2 - SSISPackageExplorer Project # 2019-07-10
I started a small project called SSISPackageExplorer on Git-Hub which allow the user to read the package objects in a TreeView, It is very basic right now but i will try to improve it in a while:
GitHub - SSISPackageExplorer
Some of the properties in dtsx Microsoft.SqlServer.Dts.Pipeline are not CLS-compliant.
ColumnInformation Constructors
ColumnInformation Class
Definition
Namespace:
Microsoft.SqlServer.Dts.Pipeline
Assembly:
Microsoft.SqlServer.PipelineHost.dll
Important
This API is not CLS-compliant.
C++
Copy
public ref class ColumnInformation
otherwise try this.
Just open your dtsx package in notepad++. Find table name then do the same search on the property name in all packages( find ion all files). I think that even if you search for the column in dtsx opened in a text editor it will give you everything. It's manual but can be updated with Regex and c#. I never did it with regex. I just did notepad++ and one package once.
I am programmatically generating and executing SSIS packages using C# and SQL Server 2012. The generated packages each contain one Data Flow Task with a Flat File Source for reading CSVs, and an OLE DB Destination connected to SQL Server. In between them is a Row Count component with a SSIS Variable hooked up.
After package execution ends, I want to get the value from the Row Count back to my calling application.
It seems that simply creating the variable & Row Count in code as follows:
[...]
// Row count: Create a package variable to store the row count value
var ssisRowCountVariable = package.Variables.Add("MySsisVar", false, "User", 0);
// Row count: Create Row component
IDTSComponentMetaData100 componentRowCount = dataFlowTask.ComponentMetaDataCollection.New();
componentRowCount.Name = "RowCount";
componentRowCount.ComponentClassID = "DTSTransform.RowCount.4";
// Row count: Get row count design-time instance, and initialize component
CManagedComponentWrapper instanceRowCount = componentRowCount.Instantiate();
instanceRowCount.ProvideComponentProperties();
// Row count: Set the variable name property
instanceRowCount.SetComponentProperty("VariableName", "User::MySsisVar");
// Hooking up pipeline Paths
[...]
// Execute package
package.Execute()
and then trying to read the value after package execution:
int Rows = Convert.ToInt32(ssisRowCountVariable.Value);
does not work.
How can I get the value of the Row Count component back to the calling application?
After playing with the second approach some more, I found that the best way to do this seems to be to hook up the Row Count component as described in the question, and add code to let it fire an event when its value gets set or changes:
ssisRowCountVariable.RaiseChangeEvent = true;
Now set up an event handler that derives from the standard DefaultEvents class to capture the OnVariableValueChanged event:
class MySsisEvents : DefaultEvents
{
public int Rows { get; private set; }
public override void OnVariableValueChanged(DtsContainer DtsContainer, Variable variable, ref bool fireAgain)
{
this.Rows = Convert.ToInt32(variable.Value);
}
}
The call to package.Execute() has to be modified to hook up the event handler as follows:
// Execute package
var mySsisEventHandler = new MySsisEvents();
package.Execute(null, null, mySsisEventHandler, null, null);
The number of processed rows is now available as mySsisEventHandler.Rows.
Unfortunately, you cannot get runtime values of Package variables from package.Execute() call directly. However, your task of getting # of processed records can be achieved differently with one of the following ways:
Add a task to the Package which saves value from processed rows variable to some storage. Task can be either following DataFlow or in its PostExecute handler.
Storage can be SQL Database or something else like Web service; value will be saved with either Execute SQL Task for SQL DB or Script Task/Web Service Task for Web service. Here is a good review of this approach.
Use DataFlow standard PostExecute informational message [SSIS.Pipeline] Information: "<your destination name>" wrote N rows. Execute your package capturing events as described in MSDN, and then check results.
Please mind that the second approach only will work if you run package as package.Execute in SSIS without using SSIS Catalog. If you are generating packages yourself and executing it right after generation - its ok, you are not generating and deploying SSISDB Projects.
We have a fully running database server, say serverA, whose data are refreshed daily.
We want to duplicate this database on a different server, says serverB, so that we have a test environment. The databases has been restored to serverB.
Like serverA, we want serverB's data to be refreshed daily also, so the tests we conduct on serverB can be said as fully accurate since they will have the same data as serverA. We deployed the SSIS packages used in serverA in serverB and copied the SQL Server Agent Jobs in serverB also.
I am trying to modify these jobs and packages so that they can run smoothly on serverB, I'm changing directory paths, server names, etc.
Now, there is this job that always fails because of a package, zip.dtsx.
zip.dtsx retrieves files from directoryA, compresses them and saves the compressed file to directoryB, then deletes the file in directoryA. However, I cannot figure out why it's having a runtime error.
zip.dtsx has a script task named Zip files.
The script language is Microsoft Visual C# 2010
The ReadOnlyVariables set are User::DestinationPath, User::NamePart, User::SourcePath,$Package::filename
The script is,
public void Main()
{
String sourcePath = Convert.ToString(Dts.Variables["SourcePath"].Value);
String namePart = Convert.ToString(Dts.Variables["NamePart"].Value);
String destinationPath = Convert.ToString(Dts.Variables["DestinationPath"].Value);
FileStream sourceFile = File.OpenRead(#sourcePath + namePart);
FileStream destFile = File.Create(#destinationPath + namePart);
GZipStream compStream = new GZipStream(destFile, CompressionMode.Compress);
try
{
int theByte = sourceFile.ReadByte();
while (theByte != -1)
{
compStream.WriteByte((byte)theByte);
theByte = sourceFile.ReadByte();
}
}
finally
{
compStream.Dispose();
sourceFile.Close();
destFile.Close();
File.Delete(#sourcePath + namePart);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
The error I'm getting, when I execute the task in Microsoft Visual Studio -> Right click Script Task object -> Execute task
I am not familiar with Microsoft Visual C# and I have just also begun using SSIS packages, so I'm really at a loss.
UPDATE:
I tried commenting out different lines in the C# script. Finally, when I commented out File.Delete(#sourcePath + namePart);, the job calling zip.dtsx has succeeded. However, I am not sure why I'm having an error with this line. I'm not sure if it is because of permissions or any other else.
Is it possible to use the mssql2012 import wizard for databases and tables withing the c# code?
I wanted to write a program with which i can import databases and tables of an access db to a mssql db. Therefore, as a gui, the mssql2012 import wizard works fine, i just need to implement it in c# Code in my project. Are there any prebuilt classes in the .net framework?
Regards
I will suggest you use Bulk Insert. This has already been managed into a .NET class SqlBulkCopy .
If you just want to import data into the database, use SqlBulkCopy. The import wizard does more than just send a batch of data to the server.
Actually, the Import wizard creates and executes an SSIS package with data transformation that extracts data from the source and sends it to the target. It doesn't do anything by itself. The last step of the wizard allows you to save the generated package and use it again, as you would with any other SSIS package.
You can load a saved package and execute it using the code provider in "Loading and Running a Local Package Programmatically". The sample code is very simple:
string pkgLocation;
Package pkg;
Application app;
DTSExecResult pkgResults;
pkgLocation =
#"C:\Program Files\Microsoft SQL Server\100\Samples\Integration Services" +
#"\Package Samples\CalculatedColumns Sample\CalculatedColumns\CalculatedColumns.dtsx";
app = new Application();
pkg = app.LoadPackage(pkgLocation, null);
pkgResults = pkg.Execute();
Console.WriteLine(pkgResults.ToString());
If you want to create a new package each time, you can use the EzAPI library to create a package programmatically, save it and/or execute it. The MSDN blog contains a sample for creating a simple dataflow task:
public class EzOleDbToFilePackage : EzSrcDestPackage<EzOleDbSource, EzSqlOleDbCM, EzFlatFileDestination, EzFlatFileCM>
{
public EzOleDbToFilePackage(Package p) : base(p) { }
public static implicit operator EzOleDbToFilePackage(Package p) { return new EzOleDbToFilePackage(p); }
public EzOleDbToFilePackage(string srv, string db, string table, string file)
: base()
{
SrcConn.SetConnectionString(srv, db);
Source.Table = table;
DestConn.ConnectionString = file;
Dest.Overwrite = true;
// This method defines the columns in FlatFile connection manager which have the same
// datatypes as flat file destination
Dest.DefineColumnsInCM();
}
[STAThread]
static void Main(string[] args)
{
// DEMO 2
EzOleDbToFilePackage p2 = new EzOleDbToFilePackage("localhost", "AdventureWorks", "Address", "result.txt");
p2.DataFlow.Disable = true;
p2.Execute();
Console.Write(string.Format("Package2 executed with result {0}\n", p2.ExecutionResult));
}
}
You should note though that if the generated default mappings aren't suitable for your task, you'll have to define each one separately in code.
We have an existing MQ custom connection manager that's currently being used by several existing SSIS packages.
I want to add a new property and modify the code a little bit, but if I do that, it looks like I am breaking everything else (have to redo all of them).
Is there a way where I can get around this without disrupting the existing packages that use it?
Assuming that your you need to edit the SSIS package by adding the property and RUN it. After that you don't need the SSIS package. Following code method should be added to your application and call this method. So all the changes will be applied only to the new package without modifying another package.
Get the SSIS package
Create the copy of it by appending the GUID. so your package name is like PackageName_GUID.dtsx
Add your property.
RUN your SSIS package
OnSuccess full execution. Delete the SSIS package.
Code
public static DtsErrors RunSSISPackage(string packagePath, string MQProperty)
{
* Append the auto generated GUID with the package name for running the SSIS package
*/
string uniqueId = Guid.NewGuid().ToString();
string uniquePackage = Path.GetDirectoryName(packagePath) + #"\" + Path.GetFileNameWithoutExtension(packagePath) + "_" + uniqueId + ".dtsx";
File.Copy(packagePath, uniquePackage);
Package pkg;
Microsoft.SqlServer.Dts.Runtime.Application app = new Microsoft.SqlServer.Dts.Runtime.Application();
pkg = app.LoadPackage(uniquePackage, null);
//MessageBox.Show(srcFileName);
//MessageBox.Show(TPODBConnection);
pkg.Connections["MQConnection"].<<YourPropertyName>> = MQProperty;
//Uncomment this to overwrite the existing file
//Do nothing until you are using a version control system
//app.SaveToXml(packagePath, package, null);
DTSExecResult result = pkg.Execute();
if (result == DTSExecResult.Failure)
{
return pkg.Errors;
}
File.Delete(uniquePackage);
return null;
}