We Keep Coding

SSIS split flat file(TXT) to multiple based on the number of records it has

i have the following c# script on ssis project in visual studio that generates files based on a column value of a sql table and i would like to add a part that also splits the files when the record count reaches 200..Example of an extraction would be nameoffile_columnvalue_datetime_1.txt for the first 200,nameoffile_columnvalue_datetime_2.txt for the next 200 etc... thank you in advance!
public void Main()
{
// TODO: Add your code here
string datetime_1 = DateTime.Now.ToString("dd/MM/yyyy");
string datetime = datetime_1.Replace("/", String.Empty);
try
{
//Declare Variables
string FileNamePart = Dts.Variables["User::FlatFileNamePart"].Value.ToString();
string DestinationFolder = Dts.Variables["User::DestinationFolder"].Value.ToString();
string TableName = Dts.Variables["User::TableName"].Value.ToString();
string FileDelimiter = Dts.Variables["User::FileDelimiter"].Value.ToString();
string FileExtension = Dts.Variables["User::FileExtension"].Value.ToString();
string ColumnNameForGrouping = Dts.Variables["User::ColumnNameForGrouping"].Value.ToString();
string SubFolder = Dts.Variables["User::SubFolder"].Value.ToString();
Int32 RecordCntPerFile = (Int32)Dts.Variables["User::RecordsPerFile"].Value;
string RecordCntPerFileDecimal = RecordCntPerFile + ".0";
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["AR_GSLO_OLTP"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Read distinct Group Values for each flat file
string query = "Select distinct " + ColumnNameForGrouping + " from " + TableName;
//MessageBox.Show(query.ToString());
SqlCommand cmd = new SqlCommand(query, myADONETConnection);
//myADONETConnection.Open();
DataTable dt = new DataTable();
dt.Load(cmd.ExecuteReader());
myADONETConnection.Close();
//Loop through values for ColumnNameForGroup
foreach (DataRow dt_row in dt.Rows)
{
string ColumnValue = "";
object[] array = dt_row.ItemArray;
ColumnValue = array[0].ToString();
//Load Data into DataTable from SQL ServerTable
string queryString =
"SELECT * from " + TableName + " Where " + ColumnNameForGrouping + "='" + ColumnValue + "'";
SqlDataAdapter adapter = new SqlDataAdapter(queryString, myADONETConnection);
DataSet ds = new DataSet();
adapter.Fill(ds);
foreach (DataTable d_table in ds.Tables)
{
string FileFullPath = SubFolder + "\\" + FileNamePart + "_" + ColumnValue + "_" + datetime + FileExtension;
StreamWriter sw = null;
sw = new StreamWriter(FileFullPath, false);
// Write the Header Row to File an thelw na exei kai header ta kanw uncomment apo 152 mexri 160 grammi
int ColumnCount = d_table.Columns.Count;
// for (int ic = 0; ic < ColumnCount; ic++)
//{
// sw.Write(d_table.Columns[ic]);
// if (ic < ColumnCount - 1)
// {
// sw.Write(FileDelimiter);
// }
//}
// sw.Write(sw.NewLine);
// Write All Rows to the File
foreach (DataRow dr in d_table.Rows)
{
for (int ir = 0; ir < ColumnCount; ir++)
{
if (!Convert.IsDBNull(dr[ir]))
{
sw.Write(dr[ir].ToString());
}
if (ir < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
}
sw.Close();
}
}
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["User::LogFolder"].Value.ToString() + "\\" +
"ErrorLog_" + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
}

So assuming/understanding you have a script task which is the source of your rows?
You can add a Rowcount Task to capture the amount of rows in a variable for later validation.
after that, you might need a new script task where you can do the validation of how many records there is. something like a case statement/if statement or a type of loop for each 200.. odd records to change a value/filename.
Using that loop, you can change the filename/filenamevariable.
So you would need your ConnectionManager File Destination to have an expression with this variable filename.
Thus as soon as it hits 200.. records, the script should change the filename and the output should create a new file.
I am not sure if it will like the in flight name change. but think this might be a way to point your thoughts.

i did it first i used a foreach loop container to take the name of the txts as input variables
then inside the loop i created a script with the following c# code
public void Main()
{
// TODO: Add your code here
string datetime_1 = DateTime.Now.ToString("dd/MM/yyyy");
string datetime = datetime_1.Replace("/", String.Empty);
StreamWriter writer = null;
try
{
//Declare Variables
string filename= Dts.Variables["User::FileName"].Value.ToString();
//MessageBox.Show(filename);
using (StreamReader inputfile = new System.IO.StreamReader(filename))
{
int count = 0;
int filecount = 0;
string line;
while ((line = inputfile.ReadLine()) != null)
{
if (writer == null || count > 199)
{
count = 0;
if (writer != null)
{
writer.Close();
writer = null;
}
++filecount;
writer = new System.IO.StreamWriter(filename.Replace(".txt", "_") + filecount.ToString() + ".txt", true);
}
writer.WriteLine(line);
++count;
}
}
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["User::LogFolder"].Value.ToString() + "\\" +
"ErrorLog_" + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
finally
{
if (writer != null)
writer.Close();
}
Dts.TaskResult = (int)ScriptResults.Success;
}
and that was it!

String.Format Method to align database output in a text file

I know there a some other questions related to mine but still they do not help me very much. I tried to understand the concept of the String.Format Method but i failed over and over again and still have no clue how to use it in my case. I know it is asked very much but if someone has an idea how to get the alignment in place i would appreciate it a lot.
I'm saving the output of different database tables to different text files. I want to align the "name" of the column to its "value". This makes very much sense because in some tables there a lot of "null" values and that leads to a poorly formatted output. Because the "null" is not exported to that file and therefore the values are moved to the left.
private void WriteSQLQueryOutputToTextFile(string DBUser, string DBUserPassword, string sqlQuery, string databaseName, string nameOfOutputFile)
{
string pathOfOutputFile = dWTestResult + "\\DatabaseUpgradeCheck\\" + nameOfOutputFile;
using (SqlConnection sqlCon = new SqlConnection("Data Source=" +
GetEnvironmentVariable.MachineName + "; Initial Catalog=" + databaseName + "; User ID=" + DBUser + "; Password=" + DBUserPassword + ";"))
{
SqlDataAdapter adapter = new SqlDataAdapter(sqlQuery, sqlCon);
try
{
sqlCon.Open();
DataSet dataset = new DataSet();
adapter.Fill(dataset, "nameOfDataset");
string currentLine = "";
foreach (var col in dataset.Tables[0].Columns)
{
currentLine += " " + col.ToString();
}
OutputHandler.AppendDataToFile(pathOfOutputFile, currentLine);
foreach (DataRow row in dataset.Tables[0].Rows)
{
currentLine = "";
foreach (var item in row.ItemArray)
{
currentLine += " " + item.ToString();
}
OutputHandler.AppendDataToFile(pathOfOutputFile, currentLine);
}
}
catch (Exception ex)
{
logger.Debug(ex, "Writing Database Output to the " + "\"" + nameOfOutputFile + "\"" + " file failed");
}
finally
{
sqlCon.Close();
}
}
}

string.Format is good when you know your required format in advance, less good if you have dynamic sources like the results of a query. Try something like this:
private void WriteSQLQueryOutputToTextFile(string DBUser, string DBUserPassword, string sqlQuery, string databaseName, string nameOfOutputFile)
{
string pathOfOutputFile = dWTestResult + "\\DatabaseUpgradeCheck\\" + nameOfOutputFile;
using (SqlConnection sqlCon = new SqlConnection("Data Source=" +
GetEnvironmentVariable.MachineName + "; Initial Catalog=" + databaseName + "; User ID=" + DBUser + "; Password=" + DBUserPassword + ";"))
{
SqlDataAdapter adapter = new SqlDataAdapter(sqlQuery, sqlCon);
try
{
sqlCon.Open();
DataSet dataset = new DataSet();
adapter.Fill(dataset, "nameOfDataset");
string currentLine = "";
var nameLengths = new int[dataset.Tables[0].Columns.Count];
var i = 0;
foreach (var col in dataset.Tables[0].Columns)
{
var colName = col.ToString();
nameLengths[i] = colName.Length;
currentLine += " " + colName;
i++;
}
OutputHandler.AppendDataToFile(pathOfOutputFile, currentLine);
foreach (DataRow row in dataset.Tables[0].Rows)
{
currentLine = "";
i = 0;
foreach (var item in row.ItemArray)
{
var field = item.ToString();
currentLine += " " + field.PadRight(nameLengths[i], ' ');
i++;
}
OutputHandler.AppendDataToFile(pathOfOutputFile, currentLine);
}
}
catch (Exception ex)
{
logger.Debug(ex, "Writing Database Output to the " + "\"" + nameOfOutputFile + "\"" + " file failed");
}
finally
{
sqlCon.Close();
}
}
}

If you get the length of the string representation of every column name and data value, you can work out the minimum width for each column and prepare a format string for that column which sets its minimum width.
Something like this:
using System;
using System.Data;
using System.Linq;
using System.Text;
namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
DataTable dt = SampleData();
// Prepare the column formats
int nCols = dt.Columns.Count;
var dataWidths = dt.Columns.Cast<DataColumn>().Select(x => x.ColumnName.Length).ToList();
foreach (DataRow d in dt.Rows)
{
for (int i = 0; i < nCols; i++)
{
dataWidths[i] = Math.Max(dataWidths[i], d.ItemArray[i].ToString().Length);
}
}
var colFormats = dataWidths.Select(x => $"{{0,{-x}}}").ToList();
//
var sb = new StringBuilder();
// Add the column names
sb.AppendLine(string.Join(" ", dt.Columns.Cast<DataColumn>().Select((x, i) => string.Format(colFormats[i], x.ColumnName))));
// Add in the data
foreach (DataRow d in dt.Rows)
{
sb.AppendLine(string.Join(" ", d.ItemArray.Select((x, i) => string.Format(colFormats[i], x))));
}
Console.WriteLine(sb.ToString());
Console.ReadLine();
}
static DataTable SampleData()
{
DataTable sdt = new DataTable();
string[] cn = "ARCHIVE DBDATETIME NEXTDOCID HIGHESTDISK SYSTEMTYPE FLAGS VERSION SINGLEUSER".Split(' ');
foreach (string n in cn)
{
sdt.Columns.Add(n);
}
DataRow drn = sdt.NewRow();
drn["ARCHIVE"] = "Hello";
drn["DBDATETIME"] = 1316859;
drn["NEXTDOCID"] = 1;
drn["HIGHESTDISK"] = "Nothing";
drn["SYSTEMTYPE"] = 1;
drn["FLAGS"] = "ABC";
drn["VERSION"] = "Hello";
drn["SINGLEUSER"] = 256;
sdt.Rows.Add(drn);
drn = sdt.NewRow();
drn["ARCHIVE"] = "Value longer than header";
// No value for drn["DBDATETIME"] etc.
drn["SINGLEUSER"] = 256;
sdt.Rows.Add(drn);
return sdt;
}
}
}
Sample output:
ARCHIVE DBDATETIME NEXTDOCID HIGHESTDISK SYSTEMTYPE FLAGS VERSION SINGLEUSER
Hello 1316859 1 Nothing 1 ABC Hello 256
Value longer than header 256

Dot is implicitly converted into hash while converting data from Excel file to XML

I have succeeded in creating XML file from an Excel file using the following C# code:
protected void Button5_Click(object sender, EventArgs e)
{
if (FileUpload1.HasFile)
{
OleDbConnection ole = new OleDbConnection();
string s = Server.MapPath("../admin/ProductOptions");
s = s + "\\" + FileUpload1.FileName;
System.IO.File.Delete(s);
FileUpload1.PostedFile.SaveAs(s);
string path = s;
ole.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=" + path + ";" + "Extended Properties=" + "\"" + "Excel 12.0;HDR=YES;" + "\"";
OleDbCommand command = new OleDbCommand("select * from[SHEET1$]", ole);
DataSet ds = new DataSet();
OleDbDataAdapter adapter = new OleDbDataAdapter(command);
adapter.Fill(ds);
GridView1.DataSource = ds.Tables[0];
GridView1.DataBind();
GridView1.Visible = true;
string filepath = Server.MapPath("ProductOptions") + "\\" + DDLproduct.SelectedValue + ".xml";
Session["ss"] = ds;
write_to_xml(ds,filepath);
}
else
{
Label2.Visible = true;
Label2.Text="[Please Select a file]";
}
}
But the problem is when this code is converting the Excel Data to XML data, then dots are itself converted into Hash(Only First Row). I know the reason but don't know the solution.
It`s happening because of dots in Excel file when converted into XML tags them implicitly converted to HASH.......
Kindly suggest me, how can I stop this conversion?

Finally got the solution:
When OLEDB Adapter fills the data in DataSet, it converts DOT into HASH.
Now I have stored that data into a DataTable(dt) and then accessed the column name and replace HASH with DOT (using Replace method of String) and create a new DataTable(dt2) with new column names.
After this using two for loops, I have inserted data from first DataTable(dt) to new Datatable(dt2).
(*one loop for rows and another one for columns)
Finally bind the grid with new DataTable(dt2)
Following is the full code for that function:
if (FileUpload1.HasFile)
{
OleDbConnection ole = new OleDbConnection();
string s = Server.MapPath("../admin/ProductOptions");
s = s + "\\" + FileUpload1.FileName;
FileUpload1.PostedFile.SaveAs(s);
string path = s;
ole.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=" + path + ";" + "Extended Properties=" + "\"" + "Excel 12.0;HDR=YES;IMEX=2;READONLY=FALSE;" + "\" ";
OleDbCommand command = new OleDbCommand("select * from[SHEET1$]", ole);
DataSet ds = new DataSet();
OleDbDataAdapter adapter = new OleDbDataAdapter(command);
adapter.Fill(ds);
DataTable dt = (DataTable)ds.Tables[0];
DataTable dt2 = new DataTable("dt2");
Session["dt"] = null;
for (int i = 0; i < dt.Columns.Count; i++)
{
string s2 = dt.Columns[i].ToString();
s2 = s2.Replace("#", ".");
string ProductName = s2.ToString();
if (Session["dt"] == null)
{
DataColumn dCol1 = new DataColumn(ProductName, typeof(System.String));
dt2.Columns.Add(dCol1);
}
}
for (int i = 0; i < dt.Rows.Count; i++)
{
dt2.Rows.Add();
for (int x = 0; x < dt.Columns.Count; x++)
{
dt2.Rows[i][x] = dt.Rows[i][x];
}
}
System.IO.File.Delete(s);
GridView1.DataSource = dt2;
GridView1.DataBind();
GridView1.Visible = true;
string filepath = Server.MapPath("ProductOptions") + "\\" + DDLproduct.SelectedValue + ".xml";
// Session["ss"] = ds;
write_to_xml(dt2,filepath);
}
else
{
Label2.Visible = true;
Label2.Text="[Please Select a file]";
}
Following is the code for write_to_xml() :
public void write_to_xml(DataTable dt, string path)
{
dt.WriteXml(path);
}
Any query or alternative solution would be appreciated... :)

Turn your headers off by HDR=No in your connection string and get your job done.
Before feeding them back to excel with HDR=Yes replace .s with #s using regex or any tool you want in the first row.

Instead of your solution I use Encoding.UTF8 in this way:
using (var fs = new FileStream(xmlFile, FileMode.CreateNew))
{
using (var xw = new XmlTextWriter(fs, Encoding.UTF8))
{
ds.WriteXml(xw);
}
}
And had no problem, this also converts < to < and > to >.

File Permissions Error When Connecting to Excel via OLDB

This following code works correctly the first time I access the file to do a line count. However, when I attempt to open the same file to read the data, I get an error thrown on ExcelConnection.Open() which states that I do not have permission to access the file or it is already being used. Any thought as to why the file isn't being released after the first connection to it?
public static DataSet GetExcelWorkSheet(string pathName, string fileName)
{
string fileExtention = System.IO.Path.GetExtension(fileName).ToLower();
OleDbConnection ExcelConnection = new OleDbConnection(fileExtention == ".xls" ?
#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + pathName + #"\" + fileName + ";Extended Properties=\"Excel 8.0;HDR=No;IMEX=1;ReadOnly=true;\"" :
#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + pathName + #"\" + fileName + ";Extended Properties=\"Excel 12.0;HDR=No;IMEX=1;ReadOnly=true;\"");
OleDbCommand ExcelCommand = new OleDbCommand();
ExcelCommand.Connection = ExcelConnection;
OleDbDataAdapter ExcelAdapter = new OleDbDataAdapter(ExcelCommand);
ExcelConnection.Open();
// DataTable ExcelSheets = ExcelConnection.GetOleDbSchemaTable(System.Data.OleDb.OleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
DataTable ExcelSheets = ExcelConnection.GetOleDbSchemaTable(System.Data.OleDb.OleDbSchemaGuid.Tables, null);
string CheckSheetName = ExcelSheets.Rows[0]["TABLE_NAME"].ToString();
string SpreadSheetName;
int useMe = -1;
if (CheckSheetName.ToUpper() != "SHEET1$") // Ok, they have renamed things
{
if (CheckSheetName.Substring(0, 5).ToUpper() == "SHEET")
// if it does START with sheet then look for anything that
// DOES NOT start with sheet
{
for (int x = 0; x < ExcelSheets.Rows.Count; x++)
{
if (ExcelSheets.Rows[x]["TABLE_NAME"].ToString().Substring(0, 5).ToUpper() != "SHEET") // If is does not equal sheet then use it
{
useMe = x;
}
}
}
}
SpreadSheetName = string.Format("[{0}]", useMe == -1 ? CheckSheetName : ExcelSheets.Rows[useMe]["TABLE_NAME"].ToString());
try
{
DataSet ExcelDataSet = new DataSet();
ExcelCommand.CommandText = #"SELECT * FROM " + SpreadSheetName;
ExcelAdapter.Fill(ExcelDataSet);
return ExcelDataSet;
}
catch (Exception)
{
ExcelConnection.Close();
return new DataSet();
}
finally
{
// Clean up.
if (ExcelConnection != null)
{
ExcelConnection.Close();
ExcelConnection.Dispose();
}
if (ExcelSheets != null)
{
ExcelSheets.Dispose();
}
}
}
}

You must close the connetion after the file has been read or the operation you have odne is completed....
ExcelConnection.Close();

Sending changes from multiple tables in disconnected dataset to SQLServer

We have a third party application that accept calls using an XML RPC mechanism for calling stored procs.
We send a ZIP-compressed dataset containing multiple tables with a bunch of update/delete/insert using this mechanism. On the other end, a CLR sproc decompress the data and gets the dataset.
Then, the following code gets executed:
using (var conn = new SqlConnection("context connection=true"))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
try
{
foreach (DataTable table in ds.Tables)
{
string columnList = "";
for (int i = 0; i < table.Columns.Count; i++)
{
if (i == 0)
columnList = table.Columns[0].ColumnName;
else
columnList += "," + table.Columns[i].ColumnName;
}
var da = new SqlDataAdapter("SELECT " + columnList + " FROM " + table.TableName, conn);
var builder = new SqlCommandBuilder(da);
builder.ConflictOption = ConflictOption.OverwriteChanges;
da.RowUpdating += onUpdatingRow;
da.Update(ds, table.TableName);
}
}
catch (....)
{
.....
}
}
Here's the event handler for the RowUpdating event:
public static void onUpdatingRow(object sender, SqlRowUpdatingEventArgs e)
{
if ((e.StatementType == StatementType.Update) && (e.Command == null))
{
e.Command = CreateUpdateCommand(e.Row, sender as SqlDataAdapter);
e.Status = UpdateStatus.Continue;
}
}
and the CreateUpdateCommand method:
private static SqlCommand CreateUpdateCommand(DataRow row, SqlDataAdapter da)
{
string whereClause = "";
string setClause = "";
SqlConnection conn = da.SelectCommand.Connection;
for (int i = 0; i < row.Table.Columns.Count; i++)
{
char quoted;
if ((row.Table.Columns[i].DataType == Type.GetType("System.String")) ||
(row.Table.Columns[i].DataType == Type.GetType("System.DateTime")))
quoted = '\'';
else
quoted = ' ';
string val = row[i].ToString();
if (row.Table.Columns[i].DataType == Type.GetType("System.Boolean"))
val = (bool)row[i] ? "1" : "0";
bool isPrimaryKey = false;
for (int j = 0; j < row.Table.PrimaryKey.Length; j++)
{
if (row.Table.PrimaryKey[j].ColumnName == row.Table.Columns[i].ColumnName)
{
if (whereClause != "")
whereClause += " AND ";
if (row[i] == DBNull.Value)
whereClause += row.Table.Columns[i].ColumnName + "=NULL";
else
whereClause += row.Table.Columns[i].ColumnName + "=" + quoted +
val + quoted;
isPrimaryKey = true;
break;
}
}
/* Only values for column that is not a primary key can be modified */
if (!isPrimaryKey)
{
if (setClause != "")
setClause += ", ";
if (row[i] == DBNull.Value)
setClause += row.Table.Columns[i].ColumnName + "=NULL";
else
setClause += row.Table.Columns[i].ColumnName + "=" + quoted +
val + quoted;
}
}
return new SqlCommand("UPDATE " + row.Table.TableName + " SET " + setClause + " WHERE " + whereClause, conn);
}
However, this is really slow when we have a lot of records.
Is there a way to optimize this or an entirely different way to send lots of udpate/delete on several tables? I would really much like to use TSQL for this but can't figure a way to send a dataset to a regular sproc.
Additional notes:
We cannot directly access the SQLServer database.
We tried without compression and it was slower.

If you're using SQL Server 2008, you should look into MERGE
Look here for more info on MERGE