We Keep Coding

C# Bulk Copy via OleDBDataReader from large excel file throwing out of memory exception

I have a large excel file(530K Rows with a lot of columns). Ends up being 247MB in .xlsb format. I am attempting to import to SQL Server using BulkCopy in C#, however I am having issues where the datareader ends up running out of memory before it even starts reading the file once I run the ExecuteReader() command.
string exlConnString = $"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={_filepath};Extended Properties=\"Excel 12.0;HDR=YES;\"";
string sqlQuery = $"SELECT * FROM [{SheetName}]";
using OleDbConnection conn = new OleDbConnection(_connstring)) {
OleDbCommand exlCmd = new OleDbCommand(sqlQuery, conn)
conn.Open();
OleDbDataReader dr = exlcmd.ExecuteReader(); <---NEVER GETS PAST THIS LINE BEFORE RUNNING OUT OF MEMORY.
SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnString);
bulkCopy.DestinationTable = TableName;
while(dr.Read()) {
bulkcopy.WriteToServer(dr);
}
dr.Close();
}
I am running in x86 mode because I was geting an error that the ACE Database was not installed on my local machine and corporate policy restrictions prevent me from downloading and installing the needed file to run it in x64 mode.
The code works perfectly fine when I test it on smaller files, but not when I test it on this bigger file, so it definitely is the filesize causing the issue. Any suggestions or help would be appreciated. Doesn't make much sense that a bulk copy runs out of memory when it is meant for handling large sets of data, which also means that the filesize is going to be large as well...
And yes, I know I SHOULD be able to import this using OPENROWSET or OPENDATASOURCE in SQL Server but THAT is ALSO Turned off and they will not enable it, so this is not an option.

So your problem is next.
When you try to ExecuteReader DataReader attempt to read all data from your excel file to memory. You could think about this, like a speciality working with excel through a OleDbProvider.
So my suggestion is to work with csv files instead of excel, because with csv file you have an ability to read and parse them line by line. For this aim i would recommend you to use CSV helper

Refer this code..
Here dtExcelData is datatable variable & da is OleDbDataAdapter variable.
string excelConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source={_filepath};Extended Properties='Excel 12.0;HDR=YES';";
// Create Connection to Excel Workbook
using (OleDbConnection connection = new OleDbConnection(excelConnectionString))
{
connection.Open();
da = new OleDbDataAdapter("Select * FROM [Sheet1$]", connection);
da.Fill(dtExcelData);
//store data in sql server database table
// below connection string "conString" is I mention in app.config file.(sql server connection string to store data in sql server database)
string str = ConfigurationManager.ConnectionStrings["conString"].ConnectionString;
using (SqlConnection con = new SqlConnection(str))
{
// Bulk Copy to SQL Server
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(con))
{
bulkCopy.DestinationTableName = "TableName";
con.Open();
bulkCopy.WriteToServer(dtExcelData);
con.Close();
}
}
connection.Close();
}
Mark it as a answer if it is useful to you. :)

IErrorInfo.GetDescription failed with E_FAIL(0x80004005) in asp.net c#?

I am trying to read an excel file and convert the contents of the file to a datatable but I keep getting this exception IErrorInfo.GetDescription failed with E_FAIL(0x80004005) pointing at a specific line POCCommand.Fill(dt);This is what I tried so far. What could I be doing wrong ?
string POCpath = #"p.xlsx";
string POCConnection = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + POCpath + ";Extended Properties=\"Excel 12.0;HDR=Yes;IMEX=1\";";
OleDbConnection POCcon = new OleDbConnection(POCConnection);
OleDbCommand POCcommand = new OleDbCommand();
DataTable dt = new DataTable();
OleDbDataAdapter POCCommand = new OleDbDataAdapter("select * from [Sheet1$] ", POCcon);
POCCommand.Fill(dt);
Console.WriteLine(dt.Rows.Count);

First, since you're using .NET, you need to wrap anything that inherits or implements from IDisposable into a using statement.
such as:
using (OleDbConnection connection = new OleDbConnection("connectionstring")){
//Do stuff here
}
Second:
Please refer to this great answer by MethodMan on this SO Answer.
It has everything you will need to know. He does both OleDb connections and a TextFieldParser which will be faster and less expensive.

xlsx file upload not working correctly - works when file is open but not when closed

I've been doing research on this problem for about a day and a half and haven't found a solution to my problem yet. This is my scenario:
I have a AsyncFileUpload and a button which is configured to accept only xls and xlsx files (Excel) file. This file that is uploaded is used to import information into a gridview. The old format xls files works fine. The problem comes when I want to upload the xlsx file. I've tested multiple scenarios and I have found that it works perfectly when the xlsx file is currectly/actively open. When it is closed it gives me errors. I have fiddled and explored solutions to this problem and the range of errors I have received is quite large. But the most prominent errors where the following:
1. External table is not in the expected format.
2. could-not-find-installable-isam
I have found an article relating to the second error (Could not find installable ISAM) but I haven't tried all the suggested solutions yet because I would rather avoid making changes to the windows registry because I do not have exstensive knowledge regarding this matter.
I'm attaching the code that I use to make the connection between the OleDb and the excel file:
protected void AsyncUpload_UploadedComplete(object sender, AjaxControlToolkit.AsyncFileUploadEventArgs e)
{
string connString = "";
string strFileType = Path.GetExtension(e.FileName).ToLower();
string path = e.FileName;
////Connection String to Excel Workbook
if (strFileType.Trim() == ".xls")
{
connString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + path + ";Extended Properties=\"Excel 8.0;HDR=Yes;IMEX=2\"";
}
else if (strFileType.Trim() == ".xlsx")
{
connString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + path + ";Extended Properties=Excel 12.0;HDR=Yes;IMEX=1";
}
string query = "SELECT * FROM [sheet1$]";
OleDbConnection conn = new OleDbConnection(connString);
if (conn.State == ConnectionState.Closed)
conn.Open();
OleDbCommand cmd = new OleDbCommand(query, conn);
OleDbDataAdapter da = new OleDbDataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds);
using (OleDbDataReader dr = cmd.ExecuteReader())
{
_InsertWrapper.clearLists();
while (dr.Read())
{
var row1Col0 = dr[0];
Console.WriteLine(row1Col0);
_InsertWrapper.GenerateList(dr);
}
}
da.Dispose();
conn.Close();
conn.Dispose();
InventoryGrid.DataBind();
ErrorsGrid.DataBind();
}
If someone has additional knowledge or an explanation that can contribute please add it so that more people that are struggling with the same type of problem can gain as much information from this question as possible.We are all still learning.
So, main idea: Need to find a solution to successfully upload the xlsx file

Try using third party tools. They work efficiently:
http://code.google.com/p/excellibrary/

Dependent Excel cells are not updated automatically

I wrote this method (almost similar in other post)
public void update(string fileName, string sheetName)
{
string connString = connString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + Server.MapPath(fileName) + ";Extended Properties='Excel 12.0;HDR=NO'";
try
{
OleDbConnection oledbConn = new OleDbConnection(connString);
oledbConn.Open();
OleDbCommand cmd = new OleDbCommand("UPDATE ["+sheetName+"$B5:B5] SET F1=17", oledbConn);
cmd.ExecuteNonQuery();
oledbConn.Close();
}
catch(Exception ex)
{
Debug.Write("Error: " + ex.Message);
}
}
and calling that method:
update("test.xls", "test");
So far, it works fine because when I open the test.xls file, B5 gets updated to 17. However, if there is a cell: B1 is dependent on B5: B1=B5*5, then B1 will not get updated automatically. I have to manually open the Excel file and save it with warning in order to get B1 updated. How can I do it programmatically?

I don't think that you can depend on Excel updating calculated columns when you use the ACE driver to interact with the Excel worksheet. When you are using OLEDB to operate on the workbook's worksheet, it is treating the worksheet as a database table like structure.
I think you may want to use OpenXML to read/write to the file. There are several threads on StackOverflow with more info on using OpenXML that are worth checking out.
This post shows your exactly how to force a cell re-calc using OpenXML.

Read Excel using c# .net Issue

I read data from excel sheet using C#.
Here is my code and it is working.
var fileName = #"C:\Users\yohan\Desktop\Brandix\y.xlsx";
var connectionString = string.Format("Provider=Microsoft.ACE.OLEDB.12.0; data source= {0}; Extended Properties=Excel 12.0;", fileName);
var adapter = new OleDbDataAdapter("SELECT * FROM [BOM$]", connectionString);
var ds = new DataSet();
adapter.Fill(ds);
DataTable data = ds.Tables[0];
But it always skip the top row of the excel sheet why is that ?
Please help ...!!
Thank You
yohan

That is the normal behaviour of the DataAdapter. The top row is considered as an Header Row or a "Column Name" Row.
To change this behaviour add to the Extended Properties of your connection string the property "HDR=NO"
Example:
var fileName = #"C:\Users\yohan\Desktop\Brandix\y.xlsx";
var connectionString = string.Format("Provider=Microsoft.ACE.OLEDB.12.0; data source={0}; Extended Properties=""Excel 12.0;HDR=NO;""", fileName);
var adapter = new OleDbDataAdapter("SELECT * FROM [BOM$]", connectionString);
var ds = new DataSet();
adapter.Fill(ds);
DataTable data = ds.Tables[0];

Try adding HDR=NO to the extended properties.
see this link for details

Try changing your connection string to ...
"Provider=Microsoft.ACE.OLEDB.12.0; data source={0}; Extended Properties=\"Excel 12.0;HDR:No\""
See this page for more possible connection strings to try. The HDR setting determines wether the provider considers the top row to be column names.

Recommended way to read an Excel file in Server side app is Open XML.
You can try installing ACE component but still it remains un-recommended and unsupported.
For using Open XML you can go through below links which will help you using Open XML in your code for reading excel files -
Link1
Link2
Link3