I've loaded a csv into a datatable but some of the columns are being read as blank when they're not blank. I initially posed this question only having an issue with the header, but I'm now also seeing this issue in my data rows so I need to reask... what is the problem with my dataset and why are some columns being read as a blank?
Currently, this setup will read data for columns 1-7 (I don't need 8-10). Data is populated for all columns correctly except column 4. Strangely, I have two files I've tested, both similar in structure, but one of them has values in column 4 and the other doesn't. The full code is setup to loop through many files, checking for start and end of data, then load to sql server.
Sample CSV:
By OrgID/Location
As of: December 6, 2017 at 10:13 AM
Date Range: summaryYM 2017M08 to 2017M08
"orgid=13778 medType=' '"
"col1","col2","col3","col4","col5","col6","col7","col8","col9","col10"
13778,140242,"2A","2017M08",0,0.058,78,".",".",
13778,140242,"2B","2017M08",0,0.014,19,".",".",
13778,140242,"2C","2017M08",0,0.083,133,".",".",
13778,140242,"2ICU","2017M08",0,0.099,114,".",".",
13778,140242,"3 ICU","2017M08",0,0.076,88,".",".",
code
//open connection to csv
string connStrCsv = string.Format(#"Provider=Microsoft.Jet.OleDb.4.0; Data Source={0};Extended Properties=""Text;HDR=NO;FMT=Delimited"""
, Path.GetDirectoryName(file));
OleDbConnection connCsv = new OleDbConnection(connStrCsv);
connCsv.Open();
//store csv data in datatable
string readCsv = "select * from [" + Path.GetFileName(file) + "]";
OleDbDataAdapter adapter = new OleDbDataAdapter(readCsv, connCsv);
DataSet ds = new DataSet();
adapter.Fill(ds, "sheet1");
DataTable table = ds.Tables["sheet1"];
connCsv.Close();
//find header to define start of data
int start = 0;
StreamReader headerSearch = null;
int incr = 0;
headerSearch = new StreamReader(file);
while (!headerSearch.EndOfStream)
{
incr++;
string line = headerSearch.ReadLine();
if (line.Contains("\"col1\",\"col2\",\"col3\",\"col4\",\"col5\",\"col6\""))
{
start = incr;
}
}
headerSearch.Close();
//load each row of excel into SQL server until first empty row
string sqlConnStr = "Data Source=mysource;Initial Catalog=mydatabase;Trusted_Connection=Yes;Integrated Security=SSPI;";
SqlConnection connSql = new SqlConnection(sqlConnStr);
connSql.Open();
int end = start;
while (table.Rows[end][0].ToString().Length != 0)
{
string sql = string.Format
(#"
delete from schema.table
where ss_col1 = {0}
and ss_col2 = '{1}'
and ss_col3 = '{2}'
and ss_col4 = '{3}';
insert into schema.table
values ({4}
,'{5}'
,'{6}'
,'{7}'
, {8}
,'{9}'
,'{10}'
,getdate()
,user_name()
,getdate()
,user_name());"
//delete statement variables
, table.Rows[end][0].ToString()
, table.Rows[end][2].ToString()
, table.Rows[end][3].ToString()
, infTypes[i]
//insert statement variables
, table.Rows[end][0].ToString()
, table.Rows[end][2].ToString()
, table.Rows[end][3].ToString()
, infTypes[i]
, table.Rows[end][4]
, table.Rows[end][5].ToString()
, table.Rows[end][6]
);
SqlCommand execSql = new SqlCommand(sql, connSql);
execSql.ExecuteNonQuery();
end++;
}
connSql.Close();
Do you have to load it into a datatable?
If "header1","header2","header3","header4","header5","header6" are unique would it not be easier just read the csv file until you find those?
Example...
StreamReader Reader = null;
string FilePath = "Your File Path";
try
{
Reader = new StreamReader(FilePath);
while(Reader.Peek() > 0)
{
string line = Reader.ReadLine();
bool HeaderFound = false;
if(line == "What ever your headers are")
{
HeaderFound = true;
}
if(HeaderFound)
{
//Here is all your data you were looking for.
//Do whatever you need to do with it now.
}
}
} catch(exception e)
{/*Deal with the issues*/}
finally
{
if(Reader != null)
{
Reader.Close();
Reader.Dispose();
}
}
I am creating a winform application where every day, a user will select a xlsx file with the day's shipping information to be merged with our invoicing data.
The challenge I am having is when the user does not download the xlsx file with the specification that the winform data requires. (I wish I could eliminate this step with an API connection but sadly I cannot)
My first step is checking to see if the xlsx file has headers to that my file path is valid
Example
string connString = "provider=Microsoft.ACE.OLEDB.12.0;Data Source='" + *path* + "';Extended Properties='Excel 12.0;HDR=YES;';";
Where path is returned from an OpenFileDialog box
If the file was chosen wasn't downloaded with headers the statement above throws an exception.
If change HDR=YES; to HDR=NO; then I have trouble identifying the columns I need and if the User bothered to include the correct ones.
My code then tries to load the data into a DataTable
private void loadRows()
{
for (int i = 0; i < deliveryTable.Rows.Count; i++)
{
DataRow dr = deliveryTable.Rows[i];
int deliveryId = 0;
bool result = int.TryParse(dr[0].ToString(), out deliveryId);
if (deliveryId > 1 && !Deliveries.ContainsKey(deliveryId))
{
var delivery = new Delivery(deliveryId)
{
SalesOrg = Convert.ToInt32(dr[8]),
SoldTo = Convert.ToInt32(dr[9]),
SoldName = dr[10].ToString(),
ShipTo = Convert.ToInt32(dr[11]),
ShipName = dr[12].ToString(),
};
Which all works only if the columns are in the right place.
If they are not in the right place my thought is to display a message to the user to get the right information
Does anyone have any suggestions?
(Sorry, first time posting a question and still learning to think through it)
I guess you're loading the spreadsheet into a Datatable? Hard to tell with one line of code. I would use the columns collection in the datatable and check to see if all the columns you want are there. Sample code to enumerate the columns below.
private void PrintValues(DataTable table)
{
foreach(DataRow row in table.Rows)
{
foreach(DataColumn column in table.Columns)
{
Console.WriteLine(row[column]);
}
}
}
private void GetExcelSheetForUpload(string PathName, string UploadExcelName)
{
string excelFile = "DateExcel/" + PathName;
OleDbConnection objConn = null;
System.Data.DataTable dt = null;
try
{
DataSet dss = new DataSet();
String connString = "Provider=Microsoft.ACE.OLEDB.12.0;Persist Security Info=True;Extended Properties=Excel 12.0 Xml;Data Source=" + PathName;
objConn = new OleDbConnection(connString);
objConn.Open();
dt = objConn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
if (dt == null)
{
return;
}
String[] excelSheets = new String[dt.Rows.Count];
int i = 0;
foreach (DataRow row in dt.Rows)
{
if (i == 0)
{
excelSheets[i] = row["TABLE_NAME"].ToString();
OleDbCommand cmd = new OleDbCommand("SELECT * FROM [" + excelSheets[i] + "]", objConn);
OleDbDataAdapter oleda = new OleDbDataAdapter();
oleda.SelectCommand = cmd;
oleda.Fill(dss, "TABLE");
}
i++;
}
grdExcel.DataSource = dss.Tables[0].DefaultView;
grdExcel.DataBind();
lblTotalRec.InnerText = Convert.ToString(grdExcel.Rows.Count);
}
catch (Exception ex)
{
ViewState["Fuletypeidlist"] = "0";
grdExcel.DataSource = null;
grdExcel.DataBind();
}
finally
{
if (objConn != null)
{
objConn.Close();
objConn.Dispose();
}
if (dt != null)
{
dt.Dispose();
}
}
}
if (grdExcel.HeaderRow.Cells[0].Text.ToString() == "CODE")
{
GetExcelSheetForEmpl(PathName);
}
else
{
divStatusMsg.Style.Add("display", "");
divStatusMsg.Attributes.Add("class", "alert alert-danger alert-dismissable");
divStatusMsg.InnerText = "ERROR !!... Upload Excel Sheet in header Defined Format ";
}
I have a huge Excel file with around 50k rows. I am reading it using the following connection string
string.Format(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties='Excel 12.0 Xml;HDR=No;IMEX=1'", MyFilePath);
In this huge Excel file, the rows are grouped in nesting pattern. Meaning, lets say first 500 rows are grouped under Group A there is a sub group in that group comprising of rows from 300-400 as Group B and then again from 350-400
in Group C. Now when I read the excel file in my program, I get all the rows, but I cannot distinguish between the row grouping I mentioned above. Is there any smart way to identify and group them accordingly?
Here's a sample of my code.
rivate List<List<string>> ReadSheetData(string _query, bool _HasHeaders = true)
{
string conn = "";
if (!_HasHeaders)
conn = string.Format(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties='Excel 12.0 Xml;HDR=No;IMEX=1'", MyFilePath);
else
conn = this.conn;
List<List<string>> ret = new List<List<string>>();
using (OleDbConnection connection = new OleDbConnection(conn))
{
connection.Open();
try
{
OleDbCommand command = new OleDbCommand(_query, connection);
using (OleDbDataReader dr = command.ExecuteReader())
{
DataTable tbl = dr.GetSchemaTable();
while (dr.Read())
{
List<string> rowVals = new List<string>();
ret.Add(rowVals);
for (int i = 0; i < dr.FieldCount; i++)
{
dynamic cell = dr[i];
string value = cell != null ? cell.ToString() : "";
rowVals.Add(value);
}
}
}
}
catch (Exception ex)
{ }
}
ret.RemoveAll(a => a.All(b => b == "") == true);
return ret;
}
I used C# for executing oracle sql queries and getting values from database after that, I need to write all the data which came from database into an excel file.I figured that out with the following code :
private static DataTable GetQueryResult(string query, Dictionary<string, string> parameterValues)
{
Oracle.ManagedDataAccess.Client.OracleConnection con = new OracleConnection(ReadConnectionString());
OracleCommand cmd = new OracleCommand(query);
cmd.Connection = con;
if (parameterValues != null && parameterValues.Count > 0) {
foreach (var item in parameterValues)
{
cmd.Parameters.Add(item.Key, item.Value);
}
}
Oracle.ManagedDataAccess.Client.OracleDataAdapter adp = new OracleDataAdapter(cmd);
DataTable tbl = new DataTable();
con.Open();
try
{
adp.Fill(tbl);
}
catch (Exception ex)
{
throw;
}
finally
{
con.Close();
}
return tbl;
}
I used a datatable for keeping the values.
DataTable queryResultTable = GetQueryResult(query, parameterValues);
if (queryResultTable != null && queryResultTable.Rows.Count > 0)
{
string strHeader = GetResultHeader(queryResultTable);
StringBuilder sb = new StringBuilder();
sb.Append(strHeader);
SetResultRows(sb, queryResultTable);
SaveFileDialog fg = new SaveFileDialog();
fg.Filter = "CSV|*.csv";
fg.Title = "Kaydedilecek dosyayi belirleyin";
DialogResult dg = fg.ShowDialog();
if (dg == DialogResult.OK)
{
File.WriteAllText(fg.FileName, sb.ToString(), Encoding.GetEncoding("windows-1254"));
}
if (dg == DialogResult.Cancel)
{
System.Environment.Exit(1);
}
and this is the code how i write values into csv files.But sometimes the values in one column looks like invalid numbers like "2,01E+13" but it should be "20130405134559" for example.
My question is how can i stop this or is there a way for me to format some specific columns.
The number is valid, it's just in exponential form. You need to look at string formatting options in/around the call to SetResultRows.
Why not use a dedicated CSV library to handle this (like https://github.com/JoshClose/CsvHelper)?
I have a xslx file with following data
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
...
Like you can see I have only 1 column used and a lot of rows.
I need to read that column from the xslx file somehow and convert it to List<string>.
Any help?
Thanks!
You can use EPPlus, it's simple, something like this :
var ep = new ExcelPackage(new FileInfo(excelFile));
var ws = ep.Workbook.Worksheets["Sheet1"];
var domains = new List<string>();
for (int rw = 1; rw <= ws.Dimension.End.Row; rw++)
{
if (ws.Cells[rw, 1].Value != null)
domains.Add(ws.Cells[rw, 1].Value.ToString());
}
the easiest way is to use OleDb, you can do something like this:
List<string> values = new List<string>();
string constr = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\your\\path\\file.xlsx;Extended Properties=\"Excel 12.0 Xml;HDR=NO;\"";
using (OleDbConnection conn = new OleDbConnection(constr))
{
conn.Open();
OleDbCommand command = new OleDbCommand("Select * from [SheetName$]", conn);
OleDbDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
// this assumes just one column, and the value is text
string value = reader[0].ToString();
values.Add(value);
}
}
}
foreach (string value in values)
Console.WriteLine(value);
Have you tried using ClosedXML? Super simple to do.
var wb = new XLWorkbook(FileName);
var ws2 = wb.Worksheet(1);
List<string> myData = new List<string>();
foreach (var r in ws2.RangeUsed().RowsUsed())
{
myData.Add(r.Cell(1).GetString());
}
You can use OOXML to read the file and this library simplify your work http://simpleooxml.codeplex.com.