We Keep Coding

How to pick specific column from txt file with delimiters in c# Console application

Hello I've got a data(in txt file) which contains 1,4 million rows ++ and i need to separate the data based on ID. So if in the data has 10 differents ID, then the console application will create 10 different files which each files contains data who has same ID.
My problem is , the data that have been given to me , not all columns to be insert, so if the data has has 10 columns, i only need to take 8 columns
Here's the code that i use to write and separate data into files with differents id
string appPath = AppDomain.CurrentDomain.BaseDirectory;
string dirTxt = appPath + "VA_" + tglskrg;
string dirZip = appPath + "VA_" + tglskrg + "\\ZIP";
var writers = new Dictionary<string, TextWriter>();
string header = "COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS"; //its still 10 columns because my code still write all the columns in the old data .
string inputFile = appPath + "va_txn_success_daily_"+tglkemarin+".txt";
string outputPath = dirTxt;
string outputPathh = dirZip;
TextWriter writer;
using (var reader = File.OpenText(inputFile))
{
//skip header
reader.ReadLine();
try
{
while (!reader.EndOfStream)
{
//read one line and separate key and value
var line = reader.ReadLine();
var separatorIndex = line.IndexOf(';');
var id = line.Substring(0, separatorIndex);
var value = line.Substring(separatorIndex - 5);
//get a writer or create one
if (!writers.TryGetValue(id, out writer))
{
writer = File.CreateText(dirTxt + "\\" + "va_txn_success_" + id + "_" + tglskrg + ".txt");
writer.WriteLine(header);
writers.Add(id, writer);
}
//write the line to the correct file
writer.WriteLine(value);
}
}
finally
{
reader.Close();
//dispose all the writers
foreach (var writerr in writers.Values)
{
writerr.Dispose();
}
}
I cant show the data , because the data its restricted
if i can give example so will be
COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS;
A;01;Ricky;15-Jan;ABC01;1000;0;BCD123;Success;Trans success
B;02;John;15-Feb;ABC02;1500;1000;BCC122;Success;Trans success
A;02;Ricky;1-Jan;Abc03;2000;0;BCC;123;Success;Trans success
So it will be create 2 file , for A company Code and B company Code
And i want to take the company code, cust number, cust name, insert date, trans amount, trans ref, flag status ,and trans status only
Thankyou

Read the items of the line into an array. Easiest way: String.Split
string[] items = String.Split(";");
Then you are able to acces the columns;
string company = item[0};
string customer = item[1];
....

Read a SqlCommand column (multiple rows) into a single C# byte[] array

Given a SQL table Document with data:
ID Content ByteLength Sequence
1 Part1... 945669 1
1 Part2... 945669 2
1 Part3... 945669 3
...
2 Part1... 45234 1
2 Part2... 45234 2
Where:
Document.Content = Up to 32KB of data
Document.ByteLength = Total data in bytes for the Document ID
Document.Sequence = Order of content in the Document ID
How can I read all of Document.Content into a single byte array byte[] Content?
using (var command = new SqlCommand("SELECT Content FROM Document WHERE ID=1 ORDER BY Sequence", connection))
using (var reader = command.ExecuteReader())
{
while(reader.Read())
{
// What goes here?
// if we just want one row:
//var fileBytes = (byte[])reader.GetValue(0); // read the file data from the selected row (first column in above query)
}
}

Given the fact that this Content field contains text data, you can simply use a StringBuilder to add data while you read the content field
using (var command = new SqlCommand("SELECT Content FROM Document WHERE ID=1 ORDER BY Sequence", connection))
using (var reader = command.ExecuteReader())
{
// Set a large enough initial capacity
StringBuilder sb = new StringBuilder(32767);
while(reader.Read())
{
sb.Append(reader.GetString(0));
}
}
Now, at the loop exit, all the content is in the StringBuilder buffer and you can get it back in a byte array with
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(sb.ToString());

How to store special characters in sql table?

I am trying to import value HACKÅS in my sql table(throu SSIS Package), but it's getting inserted as HACKÃ…S.
tried with changing datatype from varchar(max) to nvarchar(max). No Success.
Please suggest.
Below is my code block from SSIS script task..
public void Main()
{
//Declare new aplication
Application importTextFile_app = new Application();
//Create package
Package ImportTextFile_pkg = new Package();
//Get the File_Path from package variable
string File_Path;
File_Path = (string)Dts.Variables["$Package::File_Path"].Value;
//Get the delimiter value from package variable
string Delimiter = (string)Dts.Variables["$Package::Delimiter"].Value;
Delimiter = Delimiter.Replace("\\t", "\t");
char[] delimiters = new char[Delimiter.Length];
delimiters = Delimiter.ToCharArray();
//Get the Oledb destination connection string from package avriable
string Oledb_Connection_String;
Oledb_Connection_String = (string)Dts.Variables["$Package::Oledb_Connection_String"].Value;
//Set the destination table name
string Destination_Table_Name;
Destination_Table_Name = (string)Dts.Variables["$Package::Table_Name"].Value;
//Assign relevant package name and description - given table name for uniqueness to avoid conccurrency issues
ImportTextFile_pkg.Name = Destination_Table_Name;
ImportTextFile_pkg.Description = "Programmatically create an SSIS 2012 package that loads a Flat File Source into OLE DB Destination Using Script Task's C# language";
//Insert the Data Flow Task with appropriate name and some buffer space for processing of file
ImportTextFile_pkg.Executables.Add("STOCK:PipelineTask");
TaskHost taskHost = ImportTextFile_pkg.Executables[0] as TaskHost;
MainPipe dataFlowTask = (MainPipe)taskHost.InnerObject;
taskHost.Name = "Dynamic Data Flow Task";
taskHost.Properties["DefaultBufferMaxRows"].SetValue(taskHost, "1000000");
//Insert the Flat File connection
ConnectionManager connectionManagerFlatFile = ImportTextFile_pkg.Connections.Add("FLATFILE");
//You can change this path depending on where you have stored the flat file
connectionManagerFlatFile.ConnectionString = File_Path;
//Assign name to the flat file connection
connectionManagerFlatFile.Name = "TXT_FlatFile";
//Indicate that the flat file is delimited
connectionManagerFlatFile.Properties["Format"].SetValue(connectionManagerFlatFile, "Delimited");
//Indicate whether the source file has column headings or not - in this case, our sample data has column headings.
connectionManagerFlatFile.Properties["ColumnNamesInFirstDataRow"].SetValue(connectionManagerFlatFile, Convert.ToBoolean(true));
//Indicate that the flat file is text qualified
connectionManagerFlatFile.Properties["TextQualifier"].SetValue(connectionManagerFlatFile, "\"");
//Get native Flat File connection
RuntimeWrapper.IDTSConnectionManagerFlatFile100 connectionFlatFile = connectionManagerFlatFile.InnerObject as RuntimeWrapper.IDTSConnectionManagerFlatFile100;
string line;
//Prepare create table script according to columns in a file
string create_table_script;
Destination_Table_Name = "[" + Destination_Table_Name + "]";
create_table_script = "create table "+Destination_Table_Name+" ( ";
//Determine the number of columns by reading the sample Flat File - line by line.
using (StreamReader file = new StreamReader(File_Path))
{
try
{
while ((line = file.ReadLine()) != null)
{
//char[] delimiters = new char[] { '|' };
string[] parts = line.Split(delimiters, StringSplitOptions.RemoveEmptyEntries);
for (int i = 0; i < parts.Length; i++)
{
RuntimeWrapper.IDTSConnectionManagerFlatFileColumn100 flatFileCol = connectionFlatFile.Columns.Add() as RuntimeWrapper.IDTSConnectionManagerFlatFileColumn100;
create_table_script = create_table_script +" ["+ parts[i] + "] nvarchar(max),";
sS_AssignColumnProperties(flatFileCol, parts[i], Delimiter);
}
//Exit file after reading the first line
break;
}
create_table_script = create_table_script.Remove(create_table_script.Length - 1);
create_table_script = create_table_script + ")";
}
catch (Exception ex)
{
throw ex;
}
finally
{
file.Close();
}
}
OleDbConnection conn = new OleDbConnection(Oledb_Connection_String);
conn.Open();
string commandText = create_table_script;
OleDbCommand cmd = new OleDbCommand(commandText, conn);
cmd.ExecuteNonQuery();
conn.Close();
//Edit the last Flat File column delimiter into NewLine instead of a Comma
connectionFlatFile.Columns[connectionFlatFile.Columns.Count - 1].ColumnDelimiter = Environment.NewLine;
//Insert Flat File source component
IDTSComponentMetaData100 componentSource = dataFlowTask.ComponentMetaDataCollection.New();
componentSource.Name = "FlatFileSource";
componentSource.ComponentClassID = "DTSAdapter.FlatFileSource";
//Insert source design-time instance and initialise component
CManagedComponentWrapper instanceSource = componentSource.Instantiate();
instanceSource.ProvideComponentProperties();
//Set source connection
componentSource.RuntimeConnectionCollection[0].ConnectionManagerID = connectionManagerFlatFile.ID;
componentSource.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(connectionManagerFlatFile);
//Reinitialize Flat File source metadata,
instanceSource.AcquireConnections(null);
instanceSource.ReinitializeMetaData();
instanceSource.ReleaseConnections();
//Insert the SQL Server 2008 OLE-DB connection
ConnectionManager connectionManagerOleDb = ImportTextFile_pkg.Connections.Add("OLEDB");
connectionManagerOleDb.ConnectionString = string.Format(Oledb_Connection_String);
connectionManagerOleDb.Name = "OLEDB";
connectionManagerOleDb.Description = "OLEDB Connection";
//Insert OLE-DB destination
IDTSComponentMetaData100 componentDestination = dataFlowTask.ComponentMetaDataCollection.New();
componentDestination.Name = "OLEDBDestination";
componentDestination.Description = "OLEDB Destination for the Flat File data load";
componentDestination.ComponentClassID = "DTSAdapter.OLEDBDestination";
//Insert destination design-time instance and initialise component
CManagedComponentWrapper instanceDestination = componentDestination.Instantiate();
instanceDestination.ProvideComponentProperties();
//Set destination connection
componentDestination.RuntimeConnectionCollection[0].ConnectionManagerID = connectionManagerOleDb.ID;
componentDestination.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(connectionManagerOleDb);
//Indicates the name of the database object used to open a rowset
instanceDestination.SetComponentProperty("OpenRowset", Destination_Table_Name);
//Specifies the mode used to open the database
instanceDestination.SetComponentProperty("AccessMode", 3);
//Specifies options to be used with fast load. Applies only if fast load is turned on
instanceDestination.SetComponentProperty("FastLoadOptions", "TABLOCK,CHECK_CONSTRAINTS");
//Indicates whether the values supplied for identity columns will be copied to the destination or not
//In this case, we have set this property to false
instanceDestination.SetComponentProperty("FastLoadKeepIdentity", false);
//Indicates whether the columns containing null willhave null inserted in the destination or not
//In this case, we have opted no to insert nulls
instanceDestination.SetComponentProperty("FastLoadKeepNulls", false);
//Specifies the column code page to use when code page information is unavailable from the data source
//In this case we used the default - 1252
instanceDestination.SetComponentProperty("DefaultCodePage", 1252);
//Specifies when commits are issued during data insertion
//In this case, we have opted for the default size which is set to 2147483647
instanceDestination.SetComponentProperty("FastLoadMaxInsertCommitSize", 2147483647);
//Indicates the number of seconds before a command times out
//In this case, we have opted for the default value of 0 which indicates an infinite time-out
instanceDestination.SetComponentProperty("CommandTimeout", 0);
//Indicates the usage of DefaultCodePage property value when describing the character data
//In this case, we have opted for the default value of false
instanceDestination.SetComponentProperty("AlwaysUseDefaultCodePage", false);
//Connect the Flat File source to the OLE DB Destination component
dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(componentSource.OutputCollection[0], componentDestination.InputCollection[0]);
//Get input and virtual input for destination to select and map columns
IDTSInput100 destinationInput = componentDestination.InputCollection[0];
IDTSVirtualInput100 destinationVirtualInput = destinationInput.GetVirtualInput();
IDTSVirtualInputColumnCollection100 destinationVirtualInputColumns = destinationVirtualInput.VirtualInputColumnCollection;
//Reinitialize the metadata, generating exernal columns from flat file columns
instanceDestination.AcquireConnections(null);
instanceDestination.ReinitializeMetaData();
instanceDestination.ReleaseConnections();
//Select and map destination columns
foreach (IDTSVirtualInputColumn100 virtualInputColumn in destinationVirtualInputColumns)
{
// Select column, and retain new input column
IDTSInputColumn100 inputColumn = instanceDestination.SetUsageType(destinationInput.ID, destinationVirtualInput, virtualInputColumn.LineageID, DTSUsageType.UT_READONLY);
// Find external column by name
IDTSExternalMetadataColumn100 externalColumn = destinationInput.ExternalMetadataColumnCollection[inputColumn.Name];
// Map input column to external column
instanceDestination.MapInputColumn(destinationInput.ID, inputColumn.ID, externalColumn.ID);
}
//Execute the package or disable the below code if you intend running the package later
ImportTextFile_pkg.Execute();
//Finally, save the package - in this case, we have opted to save the package into file system
//importTextFile_app.SaveToXml(#"D:\newArticle.dtsx", ImportTextFile_pkg, null);
Dts.TaskResult = (int)ScriptResults.Success;
}
private static void sS_AssignColumnProperties(RuntimeWrapper.IDTSConnectionManagerFlatFileColumn100 flatFileCol, string getColName, string getDelim)
{
//Assign delimiter
flatFileCol.ColumnType = "Delimited";
flatFileCol.ColumnDelimiter = getDelim;
flatFileCol.TextQualified = true;
//Indicate column data type - in this case, all the source columns will be set to String Data Type
flatFileCol.DataType = RuntimeWrapper.DataType.DT_WSTR;
//Indicate column width - in this case, width of all source columns will be set to a length of 100
flatFileCol.ColumnWidth = 4000;
flatFileCol.MaximumWidth = 4000;
//Assign column name
RuntimeWrapper.IDTSName100 columnName = flatFileCol as RuntimeWrapper.IDTSName100;
columnName.Name = getColName.ToString();
}

Got the Solution...
Specified CodePage for flat file connection as below,
connectionManagerFlatFile.Properties["CodePage"].SetValue(connectionManagerFlatFile, 65001);

try this
insert into your_table (column_name) values (N'HACKÅS')

Insert data from text file into a table SQL Server

I have a large text file, i.e. Samples.txt file, every 8 line is a one row to be inserted into a table in sql server and the data is in the following format in the mentioned text file,
Company Name:Xpress Care
Sector:Transportation and storage
Operation Type:Logistic Services
License Number:D-39277
Expiry Date:2012-07-18
Contact Numbers:0771709155 / 0789444211
Email:naikmalemail#hotmail.com
Address:House 119, Street 4, Taemany, District 4
So far I wrote the following code in an attempt to bring it into a format so that I can insert into the table like the following.
insert into table(company, sector, operation, license, expiry, contact, email, address) Values ('Xpress Care','Transportation and storage','Logistic Services','D-39277','2012-07-18', '0771709155 / 0789444211','naikmalemail#hotmail.com','House 119, Street 4, Taemany, District 4');
Here is the code I wrote:
static void Main(string[] args)
{
int counter = 0;
int linecounter = 1;
string line;
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\sample.txt");
while ((line = file.ReadLine()) != null)
{
Console.WriteLine(line);
// splite with the : delimeter
string[] values = line.Split(':');
//Console.WriteLine("column name:- {0} value:- {1}",values[0],values[1]);
//hashtable to store the key value pairs from the text file
Hashtable myHT = new Hashtable();
// I AM STUCK here!!! I want to add to and keep the values for 8 lines
myHT.Add(values[0], values[1]);
//if linecounter is 8 then I have the values for one new row to be inserted in the table
if (linecounter == 8)
{
Console.WriteLine("\n\r code to insert the values in the query example below from the hashtable\n\r");
// insert into table(company, sector, operation, license, expiry, contact, email, address) Values ('Xpress Care','Transportation and storage','Logistic Services','D-39277','2012-07-18', '0771709155 / 0789444211','naikmalemail#hotmail.com','House 119, Street 4, Taemany, District 4');
// reset the linecounter and empty the hashtable here for the next row to insert
linecounter = 0;
}
linecounter++;
counter++;
}
file.Close();
// Suspend the screen.
Console.ReadLine();
}
What I am trying to do with the code is that I want to add and keep the key value pairs into a HashTable for 8 lines, so that I can use the 8 values to insert into the 8 columns in the table in the if(linenumber==8) condition part but now it only keeps the value from the last line only.
I will really appreciate your kind help and ideas. If you have trouble understanding the problem, please let me i will explain with more details or if there is another way to do this.

It seems that you need to move the declaration and initialization of your HashTable outside the loop and clear it when you have done the eight line block read
static void Main(string[] args)
{
int counter = 0;
int linecounter = 1;
string line;
//hashtable to store the key value pairs from the text file
Hashtable myHT = new Hashtable();
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\sample.txt");
while ((line = file.ReadLine()) != null)
{
....
// Continue to add values to the hashtable until you reach the 8 row boundary
myHT.Add(values[0], values[1]);
if (linecounter == 8)
{
..... insert ...
// reset the linecounter and empty the hashtable here for the next row to insert
linecounter = 0;
myHT.Clear();
}
linecounter++;
counter++;
}
A part from this. I suggest to use a different class for your work. In your situation I would use a Dictionary<string,string> to have a strong typed approach to the problem

you have to created the table with all the destination fields first.then
LOAD DATA INFILE '../sample.txt' INTO TABLE PerformanceReport;
By default LOAD DATA INFILE uses tab delimited, one row per line, so should take it in just fine.

if the format in the TXT file is always the same why not use this..
` while ((line = file.ReadLine()) != null)
{
Console.WriteLine(line);
// splite with the : delimeter
string[] values = line.Split(':');
if (values[0]== "Company Name") company= value[1];
if ((line = file.ReadLine()) != null)
string[] values = line.Split(':');
if (values[0]== "Sector") Sector= value[1];
...
...
insert into table(company, sector, operation, license, expiry, contact, email, address)
(#company, #sector,.....
///please insure injection protection
}`

If its a big file you can store your data in a DataTable (System.Data.DataTable) and then write it quickly using SqlBulkCopy (System.Data.SqlClient.SqlBulkCopy). The code would look something like this:
System.IO.StreamReader file = new System.IO.StreamReader(#"c:\sample.txt");
string line = null;
int linecounter = 0;
//structure to hold data to be written to the database
System.Data.DataTable table = new System.Data.DataTable();
table.Columns.Add("company");
table.Columns.Add("sector");
table.Columns.Add("operation");
table.Columns.Add("license");
table.Columns.Add("expiry");
table.Columns.Add("contact");
table.Columns.Add("email");
table.Columns.Add("address");
System.Data.DataRow row = null;
while ((line = file.ReadLine()) != null)
{
//create a new table row if the line is {0,8,16,...}
if (linecounter % 8 == 0)
row = table.NewRow();
string[] values = line.Split(':');
//put the data in the appropriate column based on "linecounter % 8"
row[linecounter % 8] = values[1];
//add the row to the table if its been fully populated
if (linecounter % 8 == 7)
table.Rows.Add(row);
linecounter++;
}
file.Close();
string connectionString = "<CONNECTION STRING GOES HERE>";
using (System.Data.SqlClient.SqlBulkCopy copy = new System.Data.SqlClient.SqlBulkCopy(connectionString))
{
copy.DestinationTableName = "MyTable";
copy.WriteToServer(table);
}
You can get help creating your connection string at : Connection strings Reference
NOTE: This method assumes that you've already created a table called "MyTable" in SQL Server, and that it has the 8 varchar columns specified in the DataTable.

C# Importing CSV using OleDb

I'm trying to set up code to import .CSV files into .NET.
I've tried both Microsoft.Jet.OLEDB.4.0 and Microsoft.ACE.OLEDB.12.0 providers, including modifying the Extended Properties and even modifying corresponding registry keys for each. I have yet to come up with a solution for what I am attempting to do:
I would like to import each field as text, but leave fields longer than 255 characters un-truncated.
What I've found so far is that I can have one or the other, but not both.
If I set the ImportMixedTypes registry value to Majority Type, it leaves 255+ character text fields un-truncated, but converts other fields to unwanted types.
If I set the ImportMixedTypes registry value to Text, it truncates 255+ character text fields, but leaves the other field types as text.
How do I accomplish this using OleDb?
Additional info:
I have a "notes" column, which can contain very lengthy text. I also have a "zip code" column, which contains mixed zip-code formats (5-digit and 9-digit with a dash). Typically, the 5-digit zip-code format is more popular, so the importer thinks that the column should be integer type, leaving the 9-digit zip-codes as null values after import.

Have you considered using something as versatile as the FileHelpers library (http://filehelpers.sourceforge.net/) instead?
Or alternatively if your requirements are no more than you state (read csv file, get string fields), use something really simple such as:
public static class SimpleCsvImport
{
public static IEnumerable<List<string>> Import(string csvFileName)
{
using (var reader = File.OpenText(csvFileName))
{
while (!reader.EndOfStream)
{
var fields = reader.ReadLine().Split(new[] { ',' }, StringSplitOptions.None).Select(f => f.Trim()).ToList();
if (fields.Count > 0)
yield return fields;
}
}
}
}

i have implemented this code to read memo field (Microsoft Access):
private string GetMemoField(string TableName, string FieldName, string IdentityFieldName, string IdentityFieldValue, OleDbConnection conn)
{
string ret = "";
OleDbCommand cmd1 = new OleDbCommand("SELECT " + FieldName + " FROM “ + TableName + “ WHERE " + IdentityFieldName + "=" + IdentityFieldValue, conn);
var reader = cmd1.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); // Create the DataReader that will get the memo field one buffer at a time
if (reader.Read())
{
long numberOfChars = reader.GetChars(/*Field pos*/ 0, 0, null, 0, 0); // Total number of memo field's chars
if (numberOfChars > 0)
{
int bufferSize = 1024;
char[] totalBuffer = new char[64*bufferSize]; // Array to hold memo field content
long dataIndex = 0;
do
{
char[] buffer = new char[bufferSize]; // Buffer to hold single read
long numberOfCharsReaded = reader.GetChars(0, dataIndex, buffer, 0, bufferSize);
if (numberOfCharsReaded == 0)
{
ret = new string(totalBuffer,0, (int)numberOfChars);
break;
}
Array.Copy(buffer, 0, totalBuffer, dataIndex, numberOfCharsReaded); // Add temporary buffer to main buffer
dataIndex += numberOfCharsReaded;
} while (true);
}
}
return ret;
}