We Keep Coding

C# - Double spaces in DB field causing unwanted newlines when writing to a text file

Let me preface this by saying I have never really coded in C# (I'm a Python programmer) and I coded in C++ back in college a little bit
I have an SSIS package that has several C# script components (found the script online somewhere) that essentially reads data from a database and writes it to a text file. When I started data validation I noticed my file had about 3500 more rows in it than the database table itself. After writing a python script to compare the data, I realized that we have a database field called "REMARKS" that has been appended to several times. Whenever it is appended it apparently causes a double space (hitting space bar twice).
When the script encounters this double white space issue it will write to a new line. I do NOT want it to do this. I need all those remarks to stay in its original field. Please see my script below.
public void Main()
{
// TODO: Add your code here
string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
string date = DateTime.Now.ToString("yyyyMMdd");
try
{
//Declare Variables
string FileNamePart = Dts.Variables["$Project::pvarWorkorderFlatFileNamePart"].Value.ToString();
string DestinationFolder = Dts.Variables["$Project::pvarDestinationFolder"].Value.ToString();
string TableName = Dts.Variables["$Project::pvarWorkorderTableName"].Value.ToString();
string FileDelimiter = Dts.Variables["$Project::pvarFileDelimiter"].Value.ToString();
string FileExtension = Dts.Variables["$Project::pvarFileExtension"].Value.ToString();
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["DB_CNXN"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Read data from table or view to data table
string query = "SELECT * FROM " + TableName;
SqlCommand cmd = new SqlCommand(query, myADONETConnection);
//myADONETConnection.Open();
DataTable d_table = new DataTable();
d_table.Load(cmd.ExecuteReader());
myADONETConnection.Close();
string FileFullPath = DestinationFolder + "\\" + FileNamePart + "_" + date + FileExtension;
StreamWriter sw = null;
sw = new StreamWriter(FileFullPath, false);
// Write the Header Row to File
int ColumnCount = d_table.Columns.Count;
for (int ic = 0; ic < ColumnCount; ic++)
{
sw.Write(d_table.Columns[ic]);
if (ic < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
// Write All Rows to the File
foreach (DataRow dr in d_table.Rows)
{
for (int ir = 0; ir < ColumnCount; ir++)
{
if (!Convert.IsDBNull(dr[ir]))
{
sw.Write(dr[ir].ToString());
}
if (ir < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
}
sw.Close();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["$Project::pvarLogFolder"].Value.ToString() + "\\" +
"ErrorLog_" + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
So the database table row might look something like this:
tip|date |remarks |completed|device_id
n |2018-01-03|1st att 9/7/17 2nd att 9/12/17 3rd att 11/7/17 need code |Y |123
(Notice double spaces in remarks field between each of those entries including after the last one)
When the script runs and outputs to the file this is what it would look like (file is pipe (|) delimitted):
Line 1:n|2018-01-03|
Line 2:1st att 9/7/17
Line 3:2nd att 9/12/17
Line 4:3rd att 11/7/17 need code
Line 5:
Line 6:|Y|1234
I just need the script to take the rows as-is and keep everything on one line like this:
Line 1:n|2018-01-03|1st att 9/7/17 2nd att 9/12/17 3rd att 11/7/17 need code |Y|1234
I am pulling the data from a SQL database and if there are no double spaces in the remarks field, then the script functions as intended.
If someone could PLEASE help I would be forever grateful! Thank you in advance!

I was mistaken that it was double spaces, as it actually was CRLF (\r\n) that was in the database. So taking what #Ali Maleki provided above, I used this to solve my issue:
sw.Write((dr[ir].ToString()).Replace("\r\n","_"));
Thanks for your guys' help!

Find this line in your code : sw.Write(dr[ir].ToString());
Replace with : sw.Write((dr[ir].ToString()).Replace(' ', '_'));
Result :
Line 1:n|2018-01-03|1st_att_9/7/17__2nd_att_9/12/17__3rd_att_11/7/17_need code__|Y|1234

How to reference List<Data>

I am new to C# and I have taken a small task on. The StackOverflow entry for reading a text file and saving to a list is a great start for me. I need to read a text file and send the data to an SQL database.
How to Read This Text File and store in a list using C#
The Data in List<Data> list = new List<Data>(); just keeps staying in red.
How can I stop this please?
I am a PLC engineer and I'm trying to collate data that cannot be handled by PLC. I am just trying to read the file so that I can then show the data in a Grid, with a view to populating the SQL database later.
The text file is held on a remote Linux machine. The collator is a WIndows 10 panel. The SQL can reside on teh Windows panel or remotely.
static void Main(string[] args)
{
List<Data> list = new List<Data>();
var dd = File.ReadAllLines(#"C:\Users\XXXX\Desktop\test.txt")
.Skip(1)
.Where(s => s.Length > 1).ToList();
foreach (var item in dd)
{
var columns = item.Split('\t').Where(c => c.Trim() != string.Empty).ToList();
if (columns != null && columns.Count > 0)
{
int id;
if (int.TryParse(columns[0], out id))
{
list.Add(new Data()
{
id = Convert.ToInt32(columns[0]),
Name = columns[1],
Description = columns[2],
Quantity = Convert.ToInt32(columns[3]),
Rate = Convert.ToDouble(columns[4]),
Discount = Convert.ToInt32(columns[5]),
Amount = int.Parse(columns[6])
});
}
else
{
list.Last().Description += columns[0];
}
}
}
Console.ReadLine();
}
I just keep receiving red squiggly lines on <Data. within Visual Studio

I got the code to work and I read the DAT/text file straight into a DatagridView. I am now writing the Grid to SQL.
Many thanks, sorry for latency as I've been away on-site.
String sLine = "";
try
{
//Pass the file you selected with the OpenFileDialog control to
//the StreamReader Constructor.
System.IO.StreamReader FileStream = new System.IO.StreamReader(openFileDialog1.FileName);
//You must set the value to false when you are programatically adding rows to
//a DataGridView. If you need to allow the user to add rows, you
//can set the value back to true after you have populated the DataGridView
dataGridView1.AllowUserToAddRows = false;
// Clear the DataGridView prior to reading a new text file
dataGridView1.Rows.Clear();
dataGridView1.Columns.Clear();
//Read the first line of the text file
sLine = FileStream.ReadLine();
//The Split Command splits a string into an array, based on the delimiter you pass.
//I chose to use a semi-colon for the text delimiter.
//Any character can be used as a delimeter in the split command.
//string[] s = sLine.Split(';');
string[] s = sLine.Split('\t');
//In this example, I placed the field names in the first row.
//The for loop below is used to create the columns and use the text values in
//the first row for the column headings.
for (int i = 0; i <= s.Count() - 1; i++)
{
DataGridViewColumn colHold = new DataGridViewTextBoxColumn();
colHold.Name = "col" + System.Convert.ToString(i);
colHold.HeaderText = s[i].ToString();
dataGridView1.Columns.Add(colHold);
}
//Read the next line in the text file in order to pass it to the
//while loop below
sLine = FileStream.ReadLine();
//The while loop reads each line of text.
while (sLine != null)
{
//Adds a new row to the DataGridView for each line of text.
dataGridView1.Rows.Add();
//This for loop loops through the array in order to retrieve each
//line of text.
for (int i = 0; i <= s.Count() - 1; i++)
{
//Splits each line in the text file into a string array
//s = sLine.Split(';');
s = sLine.Split('\t');
//Sets the value of the cell to the value of the text retreived from the text file.
dataGridView1.Rows[dataGridView1.Rows.Count - 1].Cells[i].Value = s[i].ToString();
}
sLine = FileStream.ReadLine();
}
//Close the selected text file.
FileStream.Close();
}
catch (Exception err)
{
//Display any errors in a Message Box.
System.Windows.Forms.MessageBox.Show("Error "+ err.Message, "Program Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
}

How to pick specific column from txt file with delimiters in c# Console application

Hello I've got a data(in txt file) which contains 1,4 million rows ++ and i need to separate the data based on ID. So if in the data has 10 differents ID, then the console application will create 10 different files which each files contains data who has same ID.
My problem is , the data that have been given to me , not all columns to be insert, so if the data has has 10 columns, i only need to take 8 columns
Here's the code that i use to write and separate data into files with differents id
string appPath = AppDomain.CurrentDomain.BaseDirectory;
string dirTxt = appPath + "VA_" + tglskrg;
string dirZip = appPath + "VA_" + tglskrg + "\\ZIP";
var writers = new Dictionary<string, TextWriter>();
string header = "COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS"; //its still 10 columns because my code still write all the columns in the old data .
string inputFile = appPath + "va_txn_success_daily_"+tglkemarin+".txt";
string outputPath = dirTxt;
string outputPathh = dirZip;
TextWriter writer;
using (var reader = File.OpenText(inputFile))
{
//skip header
reader.ReadLine();
try
{
while (!reader.EndOfStream)
{
//read one line and separate key and value
var line = reader.ReadLine();
var separatorIndex = line.IndexOf(';');
var id = line.Substring(0, separatorIndex);
var value = line.Substring(separatorIndex - 5);
//get a writer or create one
if (!writers.TryGetValue(id, out writer))
{
writer = File.CreateText(dirTxt + "\\" + "va_txn_success_" + id + "_" + tglskrg + ".txt");
writer.WriteLine(header);
writers.Add(id, writer);
}
//write the line to the correct file
writer.WriteLine(value);
}
}
finally
{
reader.Close();
//dispose all the writers
foreach (var writerr in writers.Values)
{
writerr.Dispose();
}
}
I cant show the data , because the data its restricted
if i can give example so will be
COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS;
A;01;Ricky;15-Jan;ABC01;1000;0;BCD123;Success;Trans success
B;02;John;15-Feb;ABC02;1500;1000;BCC122;Success;Trans success
A;02;Ricky;1-Jan;Abc03;2000;0;BCC;123;Success;Trans success
So it will be create 2 file , for A company Code and B company Code
And i want to take the company code, cust number, cust name, insert date, trans amount, trans ref, flag status ,and trans status only
Thankyou

Read the items of the line into an array. Easiest way: String.Split
string[] items = String.Split(";");
Then you are able to acces the columns;
string company = item[0};
string customer = item[1];
....

Is there a way for extracting an Excel file from an ado.net query and adding custom columns to it using c#

I have an Excel file template, and I need to extract the data from a SQL Server database to this Excel file using C# as same as the template file.
The problem is that I need to add another column using C# to the Excel file, to make this extracted file look like the template file.
So I will not extract the Excel file's data from my web-form application directly.
I need to add some additional columns first.

Convert SQL to CSV while writing SQL result add your custom column data as well. CSV will open in excel by default.
private void SQLToCSV(string query, string Filename)
{
SqlConnection conn = new SqlConnection(connection);
conn.Open();
SqlCommand cmd = new SqlCommand(query, conn);
SqlDataReader result = cmd.ExecuteReader();
using (System.IO.StreamWriter fs = new System.IO.StreamWriter(Filename))
{
// Loop through the fields and add headers
for (int i = 0; i < result.FieldCount; i++)
{
string colval = result.GetColumnName(i);
if (colval.Contains(","))
colval = "\"" + colval + "\"";
fs.Write(colval + ",");
}
//CONCATENATE THE COLUMNS YOU WANT TO ADD IN RESULT HERE
fs.WriteLine();
// Loop through the rows and output the data
while (result.Read())
{
for (int i = 0; i < result.FieldCount; i++)
{
string value = result[i].ToString();
if (value.Contains(","))
value = "\"" + value + "\"";
fs.Write(value + ",");
}
fs.WriteLine();
}
fs.Close();
}
}
You can covert csv to excel
using Excel = Microsoft.Office.Interop.Excel;
private void Convert_CSV_To_Excel()
{
// Rename .csv To .xls
System.IO.File.Move(#"d:\Test.csv", #"d:\Test.csv.xls");
var _app = new Excel.Application();
var _workbooks = _app.Workbooks;
_workbooks.OpenText("Test.csv.xls",
DataType: Excel.XlTextParsingType.xlDelimited,
TextQualifier: Excel.XlTextQualifier.xlTextQualifierNone,
ConsecutiveDelimiter: true,
Semicolon: true);
// Convert To Excle 97 / 2003
_workbooks[1].SaveAs("NewTest.xls", Excel.XlFileFormat.xlExcel5);
_workbooks.Close();
}

Insert data from text file into a table SQL Server

I have a large text file, i.e. Samples.txt file, every 8 line is a one row to be inserted into a table in sql server and the data is in the following format in the mentioned text file,
Company Name:Xpress Care
Sector:Transportation and storage
Operation Type:Logistic Services
License Number:D-39277
Expiry Date:2012-07-18
Contact Numbers:0771709155 / 0789444211
Email:naikmalemail#hotmail.com
Address:House 119, Street 4, Taemany, District 4
So far I wrote the following code in an attempt to bring it into a format so that I can insert into the table like the following.
insert into table(company, sector, operation, license, expiry, contact, email, address) Values ('Xpress Care','Transportation and storage','Logistic Services','D-39277','2012-07-18', '0771709155 / 0789444211','naikmalemail#hotmail.com','House 119, Street 4, Taemany, District 4');
Here is the code I wrote:
static void Main(string[] args)
{
int counter = 0;
int linecounter = 1;
string line;
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\sample.txt");
while ((line = file.ReadLine()) != null)
{
Console.WriteLine(line);
// splite with the : delimeter
string[] values = line.Split(':');
//Console.WriteLine("column name:- {0} value:- {1}",values[0],values[1]);
//hashtable to store the key value pairs from the text file
Hashtable myHT = new Hashtable();
// I AM STUCK here!!! I want to add to and keep the values for 8 lines
myHT.Add(values[0], values[1]);
//if linecounter is 8 then I have the values for one new row to be inserted in the table
if (linecounter == 8)
{
Console.WriteLine("\n\r code to insert the values in the query example below from the hashtable\n\r");
// insert into table(company, sector, operation, license, expiry, contact, email, address) Values ('Xpress Care','Transportation and storage','Logistic Services','D-39277','2012-07-18', '0771709155 / 0789444211','naikmalemail#hotmail.com','House 119, Street 4, Taemany, District 4');
// reset the linecounter and empty the hashtable here for the next row to insert
linecounter = 0;
}
linecounter++;
counter++;
}
file.Close();
// Suspend the screen.
Console.ReadLine();
}
What I am trying to do with the code is that I want to add and keep the key value pairs into a HashTable for 8 lines, so that I can use the 8 values to insert into the 8 columns in the table in the if(linenumber==8) condition part but now it only keeps the value from the last line only.
I will really appreciate your kind help and ideas. If you have trouble understanding the problem, please let me i will explain with more details or if there is another way to do this.

It seems that you need to move the declaration and initialization of your HashTable outside the loop and clear it when you have done the eight line block read
static void Main(string[] args)
{
int counter = 0;
int linecounter = 1;
string line;
//hashtable to store the key value pairs from the text file
Hashtable myHT = new Hashtable();
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\sample.txt");
while ((line = file.ReadLine()) != null)
{
....
// Continue to add values to the hashtable until you reach the 8 row boundary
myHT.Add(values[0], values[1]);
if (linecounter == 8)
{
..... insert ...
// reset the linecounter and empty the hashtable here for the next row to insert
linecounter = 0;
myHT.Clear();
}
linecounter++;
counter++;
}
A part from this. I suggest to use a different class for your work. In your situation I would use a Dictionary<string,string> to have a strong typed approach to the problem

you have to created the table with all the destination fields first.then
LOAD DATA INFILE '../sample.txt' INTO TABLE PerformanceReport;
By default LOAD DATA INFILE uses tab delimited, one row per line, so should take it in just fine.

if the format in the TXT file is always the same why not use this..
` while ((line = file.ReadLine()) != null)
{
Console.WriteLine(line);
// splite with the : delimeter
string[] values = line.Split(':');
if (values[0]== "Company Name") company= value[1];
if ((line = file.ReadLine()) != null)
string[] values = line.Split(':');
if (values[0]== "Sector") Sector= value[1];
...
...
insert into table(company, sector, operation, license, expiry, contact, email, address)
(#company, #sector,.....
///please insure injection protection
}`

If its a big file you can store your data in a DataTable (System.Data.DataTable) and then write it quickly using SqlBulkCopy (System.Data.SqlClient.SqlBulkCopy). The code would look something like this:
System.IO.StreamReader file = new System.IO.StreamReader(#"c:\sample.txt");
string line = null;
int linecounter = 0;
//structure to hold data to be written to the database
System.Data.DataTable table = new System.Data.DataTable();
table.Columns.Add("company");
table.Columns.Add("sector");
table.Columns.Add("operation");
table.Columns.Add("license");
table.Columns.Add("expiry");
table.Columns.Add("contact");
table.Columns.Add("email");
table.Columns.Add("address");
System.Data.DataRow row = null;
while ((line = file.ReadLine()) != null)
{
//create a new table row if the line is {0,8,16,...}
if (linecounter % 8 == 0)
row = table.NewRow();
string[] values = line.Split(':');
//put the data in the appropriate column based on "linecounter % 8"
row[linecounter % 8] = values[1];
//add the row to the table if its been fully populated
if (linecounter % 8 == 7)
table.Rows.Add(row);
linecounter++;
}
file.Close();
string connectionString = "<CONNECTION STRING GOES HERE>";
using (System.Data.SqlClient.SqlBulkCopy copy = new System.Data.SqlClient.SqlBulkCopy(connectionString))
{
copy.DestinationTableName = "MyTable";
copy.WriteToServer(table);
}
You can get help creating your connection string at : Connection strings Reference
NOTE: This method assumes that you've already created a table called "MyTable" in SQL Server, and that it has the 8 varchar columns specified in the DataTable.