I'm trying to parse values from a text file using c#, then append it as a column to the existing data set coming from the same text file. Example file data:
As of 1/31/2015
1 data data data
2 data data data
So I want to use a script component within an ssis data flow to append the 1/31/2015 value as a 4th column. This package iterates through several files, so I would like this to take place within each dataflow. I have no issues with getting the rest of the data into the database into individual columns, but I parse most of those out using tsql after fast loading everything as one big column into the database.
edit:
here is the .NET code I used to get it started, I know this is probably far from optimal, but I actually want to parse two values from the resulting string, and that would be easy to do with regular expressions
string every = "";
string[] lines = File.ReadLines(filename).Take(7).ToArray();
string.Join(",", lines);
every = lines.ToString();
MessageBox.Show(every).ToString();
I am working on a system where i need to select millionsof records from mysql and there is no key is defined on that table as there is mass inserting and updating work simultaneously .
So I use this command to a genrate csv file from selected data and its working for me in great way .
SELECT *
INTO OUTFILE 'E:\\31october\\SP\\Export\\xyz.csv'
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n'
FROM tblspmaster;
but my problem is i also have to update the selected records and needs to show those records on aspx page . if i run select its just running and running .
So I have two questions
How can I update another fields in that table using INTO OUTFILE of mysql .
Is it possible that instead of showing records on web page from mysql response i just use this csv file to bind my gridview ? or right custom HTML ?
if you want show million of records the best way is "slickgrid", may be it will help you.
https://github.com/mleibman/SlickGrid
https://github.com/mleibman/SlickGrid/wiki/Used-by
https://github.com/mleibman/SlickGrid/wiki/Examples
I am trying to read a folder that contain a number of xls files (27) and i need to read only 3 specific columns after 21 row e.g. A21,: B21,: ... In a new column a would like to have just the sum of the previous columns. I am thinking to insert a database grid and to insert there the parsing data. My problem is that i have never try to read something from xls. Do you have any ideas. Thanks in advance! (All the data are in the same workshhet in all workbooks)
We read a tab delimited file into a DataTable with the following code.
//read the uploaded file...
var records = File.ReadAllLines(Server.MapPath("~/UploadFiles/") + Session.SessionID + "/orderEDI.txt").ToList();
//load the data into the temporary table...
records.ForEach(record => loadTable.Rows.Add(record.Split((char)9)));
This works just fine providing there are not more tabs in the file than there are columns in the DataTable.
I'm looking to find out if there's a way I can limit the number of columns it reads from the file. Or any other suggestions around this issue. It must read an absolute minimum of 10 columns (and ideally, only 10)
I build the DataTable and add columns to it before this load occurs. Would it be better to not add columns and just load the file into the DataTable and then read the table by column index rather than name?
Really not sure which way to go and would appreciate some experienced opinions.
Thanks
Since split results in an array, why don't you just use Take(10)?
//read the uploaded file...
var records = File.ReadAllLines(Server.MapPath("~/UploadFiles/") + Session.SessionID + "/orderEDI.txt").ToList();
//load the data into the temporary table...
records.ForEach(record => loadTable.Rows.Add((record.Split((char)9)).Take(10)));
I'm importing geocode data to our database from a csv file.
I've used the following library
A fast csv reader to read the csv and then using SqlBulkCopy
Here's an example of the data I'm importing
"AB10","1BH","L",0,0,,,20
"AB10","1BR","L",39320,80570,57.14214,-2.11400,21
It works ok on good data but on the top line it will throw an exception because the database is set up to not accept null values.
Is there a way to tell bulkcopy to ignore bad data? I've tried to get the csv reader to ignore bad lines by using the in built properties of the library like so but they don't appear to work.
csv.SkipEmptyLines = true;
csv.MissingFieldAction = MissingFieldAction.ParseError;
csv.DefaultParseErrorAction = ParseErrorAction.AdvanceToNextLine;
I guess another option would be to pre-parse the csv and remove all the offending rows. Perhaps there's a better csv library out there for .net?
If you could post your csv reader code then we could help more. But looking at the code on your linked page, you could do something like this:
while (csv.ReadNextRecord())
{
for (int i = 0; i < fieldCount; i++)
Console.Write(string.Format("{0} = {1};",
headers[i],
csv[i] ?? "null"));
Console.WriteLine();
}
See where I have added that null-coalescing operator? This should change your output from:
"AB10","1BH","L",0,0,,,20
to
"AB10","1BH","L",0,0,null,null,20
I used the Microsoft Text Driver to import CSV information for a project. It worked pretty well. I defined a Schema.ini file to specify the column headers, data types, number of rows to scan (MaxScanRows=0, will scan the whole file).
I haven't tried this but when you use Microsoft Text Driver you issue a select query to pull the data out of the csv file, I'm wondering if you could add criteria to filter out the null records.
How to populate IDataReader from .csv for use with SqlBulkCopy.WriteToServer(IDataReader)
http://msdn.microsoft.com/en-us/library/windows/desktop/ms709353(v=vs.85).aspx
http://www.connectionstrings.com/textfile
Hope this helps.
To deal with the null entries I ended up parsing the csv into a DataTable object 1000 entries at a time and then imported them as I went.