We Keep Coding

Slow building excel worksheet using NPOI

I am using the NPOI library to build Excel worksheets in an app that my supervisor wants to host on a Linux server. The app works fine with files under 1000 rows but occasionally I have large files and it takes a lot longer than I would expect. Does anyone have any idea how I can speed up the processing for these large files? Thanks.
Here is the code:
foreach (DataRow row in dataMalformed.Rows)
{
IWorkbook workbook = new XSSFWorkbook();
//Console.WriteLine(row[0]);
if (dataMalformed.Rows.Count > 0)
{
strFileName = strMalformed + "\\MVPSMalformedProd" + "-";
strFileName2 = strFileName + strDate2 + ".xls";
}
var fs = new FileStream(strFileName, FileMode.Create, FileAccess.ReadWrite);
var fs2 = new FileStream(strFileName2, FileMode.Create, FileAccess.ReadWrite);
strSheetName = "Malformed Messages - " + ", " + strDate2;
using (fs)
{
ISheet excelSheet = workbook.CreateSheet(strSheetName);
intRow = 2;
List<String> columns = new List<string>();
IRow row1 = excelSheet.CreateRow(0);
int columnIndex = 0;
foreach (System.Data.DataColumn column in dataMalformed.Columns)
{
columns.Add(column.ColumnName);
row1.CreateCell(columnIndex).SetCellValue(column.ColumnName);
columnIndex++;
}
foreach (DataRow row2 in dataMalformed.Rows)
{
//Console.WriteLine(row2[0]);
row1 = excelSheet.CreateRow(intRow);
int cellIndex = 0;
foreach (String col in columns)
{
row1.CreateCell(cellIndex).SetCellValue(row2[col].ToString());
excelSheet.AutoSizeColumn(cellIndex);
cellIndex++;
}
if (dictSenders.ContainsKey(row2[3].ToString()))
{
row1.CreateCell(--cellIndex).SetCellValue(dictSenders[row2[3].ToString()]);
}
intRow++;
}
}

And you are sure that NPOI is the problem? I just tried the following to create an excel file with 10000 rows and 50 columns.
var workbook = new XSSFWorkbook ();
var sheet = workbook.CreateSheet ("Sheet1");
for (var rowNum = 0; rowNum < 10000; rowNum++)
{
var rowData = sheet.CreateRow (rowNum);
for (var columnNum = 0; columnNum < 50; columnNum++)
{
rowData.CreateCell (columnNum).SetCellValue ($"Row {rowNum + 1}, Column {columnNum + 1}");
}
}
using (FileStream fileStream = File.Create ("D:\\Temp\\test.xlsx"))
{
workbook.Write (fileStream);
fileStream.Close ();
}

How to load on demand excel rows in a data table c#

I have a requirement where-in I have to fill dataTable from a sheet of Microsoft excel.
The sheet may have lots of data so the requirement is that when a foreach loop is iterated over the data table which is supposed to hold the data from Microsoft excel sheet should fill the table on demand.
Meaning if there are 1000000 records in the sheet the data table should fetch data in batches of 100 depending on the current position of the foreach current item in the loop.
Any pointer or suggestion will be appreciated.

I would suggest you to use OpenXML to parse and read your excel data from file.
This will also allow you to read out specific sections/regions from your workbook.
You will find more information and also an example at this link:
Microsoft Docs - Parse and read a large spreadsheet document (Open XML SDK)
This will be more efficiently and easier to develop than use the official microsoft office excel interop.

**I am not near a PC with Visual stuido, so this code is untested, and may have syntax errors until I can test it later.
It will still give you the main idea of what needs to be done.
private void ExcelDataPages(int firstRecord, int numberOfRecords)
{
Excel.Application dataApp = new Excel.Application();
Excel.Workbook dataWorkbook = new Excel.Workbook();
int x = 0;
dataWorkbook.DisplayAlerts = false;
dataWorkbook.Visible = false;
dataWorkbook.AutomationSecurity = Microsoft.Office.Core.MsoAutomationSecurity.msoAutomationSecurityLow;
dataWorkbook = dataApp.Open(#"C:\Test\YourWorkbook.xlsx");
try
{
Excel.Worksheet dataSheet = dataWorkbook.Sheet("Name of Sheet");
while (x < numberOfRecords)
{
Range currentRange = dataSheet.Rows[firstRecord + x]; //For all columns in row
foreach (Range r in currentRange.Cells) //currentRange represents all the columns in the row
{
// do what you need to with the Data here.
}
x++;
}
}
catch (Exception ex)
{
//Enter in Error handling
}
dataWorkbook.Close(false); //Depending on how quick you will access the next batch of data, you may not want to close the Workbook, reducing load time each time. This may also mean you need to move the open of the workbook to a higher level in your class, or if this is the main process of the app, make it static, stopping the garbage collector from destroying the connection.
dataApp.Quit();
}

Give the following a try--it uses NuGet package DocumentFormat.OpenXml The code is from Using OpenXmlReader. However, I modified it to add data to a DataTable. Since you're reading data from the same Excel file multiple times, it's faster to open the Excel file once using an instance of SpreadSheetDocument and dispose of it when finished. Since the instance of SpreedSheetDocument needs to be disposed of before your application exits, IDisposable is used.
Where it says "ToDo", you'll need to replace the code that creates the DataTable columns with your own code to create the correct columns for your project.
I tested the code below with an Excel file containing approximately 15,000 rows. When reading 100 rows at a time, the first read took approximately 500 ms - 800 ms, whereas subsequent reads took approximately 100 ms - 400 ms.
Create a class (name: HelperOpenXml)
HelperOpenXml.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using DocumentFormat.OpenXml;
using DocumentFormat.OpenXml.Packaging;
using DocumentFormat.OpenXml.Spreadsheet;
using System.Data;
using System.Diagnostics;
namespace ExcelReadSpecifiedRowsUsingOpenXml
{
public class HelperOpenXml : IDisposable
{
public string Filename { get; private set; } = string.Empty;
public int RowCount { get; private set; } = 0;
private SpreadsheetDocument spreadsheetDocument = null;
private DataTable dt = null;
public HelperOpenXml(string filename)
{
this.Filename = filename;
}
public void Dispose()
{
if (spreadsheetDocument != null)
{
try
{
spreadsheetDocument.Dispose();
dt.Clear();
}
catch(Exception ex)
{
throw ex;
}
}
}
public DataTable GetRowsSax(int startRow, int endRow, bool firstRowIsHeader = false)
{
int startIndex = startRow;
int endIndex = endRow;
if (firstRowIsHeader)
{
//if first row is header, increment by 1
startIndex = startRow + 1;
endIndex = endRow + 1;
}
if (spreadsheetDocument == null)
{
//create new instance
spreadsheetDocument = SpreadsheetDocument.Open(Filename, false);
//create new instance
dt = new DataTable();
//ToDo: replace 'dt.Columns.Add(...)' below with your code to create the DataTable columns
//add columns to DataTable
dt.Columns.Add("A");
dt.Columns.Add("B");
dt.Columns.Add("C");
dt.Columns.Add("D");
dt.Columns.Add("E");
dt.Columns.Add("F");
dt.Columns.Add("G");
dt.Columns.Add("H");
dt.Columns.Add("I");
dt.Columns.Add("J");
dt.Columns.Add("K");
}
else
{
//remove existing data from DataTable
dt.Rows.Clear();
}
WorkbookPart workbookPart = spreadsheetDocument.WorkbookPart;
int numWorkSheetParts = 0;
foreach (WorksheetPart worksheetPart in workbookPart.WorksheetParts)
{
using (OpenXmlReader reader = OpenXmlReader.Create(worksheetPart))
{
int rowIndex = 0;
//use the reader to read the XML
while (reader.Read())
{
if (reader.ElementType == typeof(Row))
{
reader.ReadFirstChild();
List<string> cValues = new List<string>();
int colIndex = 0;
do
{
//only get data from desired rows
if ((rowIndex > 0 && rowIndex >= startIndex && rowIndex <= endIndex) ||
(rowIndex == 0 && !firstRowIsHeader && rowIndex >= startIndex && rowIndex <= endIndex))
{
if (reader.ElementType == typeof(Cell))
{
Cell c = (Cell)reader.LoadCurrentElement();
string cellRef = c.CellReference; //ex: A1, B1, ..., A2, B2
string cellValue = string.Empty;
//string/text data is stored in SharedString
if (c.DataType != null && c.DataType == CellValues.SharedString)
{
SharedStringItem ssi = workbookPart.SharedStringTablePart.SharedStringTable.Elements<SharedStringItem>().ElementAt(int.Parse(c.CellValue.InnerText));
cellValue = ssi.Text.Text;
}
else
{
cellValue = c.CellValue.InnerText;
}
//Debug.WriteLine("{0}: {1} ", c.CellReference, cellValue);
//add value to List which is used to add a row to the DataTable
cValues.Add(cellValue);
}
}
colIndex += 1; //increment
} while (reader.ReadNextSibling());
if (cValues.Count > 0)
{
//if List contains data, use it to add row to DataTable
dt.Rows.Add(cValues.ToArray());
}
rowIndex += 1; //increment
if (rowIndex > endIndex)
{
break; //exit loop
}
}
}
}
numWorkSheetParts += 1; //increment
}
DisplayDataTableData(dt); //display data in DataTable
return dt;
}
private void DisplayDataTableData(DataTable dt)
{
foreach (DataColumn dc in dt.Columns)
{
Debug.WriteLine("colName: " + dc.ColumnName);
}
foreach (DataRow r in dt.Rows)
{
Debug.WriteLine(r[0].ToString() + " " + r[1].ToString());
}
}
}
}
Usage:
private string excelFilename = #"C:\Temp\Test.xlsx";
private HelperOpenXml helperOpenXml = null;
...
private void GetData(int startIndex, int endIndex, bool firstRowIsHeader)
{
helperOpenXml.GetRowsSax(startIndex, endIndex, firstRowIsHeader);
}
Note: Make sure to call Dispose() (ex: helperOpenXml.Dispose();) before your application exits.
Update:
OpenXML stores dates as the number of days since 01 Jan 1900. For dates prior to 01 Jan 1900, they are stored in SharedString. For more info see Reading a date from xlsx using open xml sdk
Here's a code snippet:
Cell c = (Cell)reader.LoadCurrentElement();
...
string cellValue = string.Empty
...
cellValue = c.CellValue.InnerText;
double dateCellValue = 0;
Double.TryParse(cellValue, out dateCellValue);
DateTime dt = DateTime.FromOADate(dateCellValue);
cellValue = dt.ToString("yyyy/MM/dd");

Another simple alternative is this: Take a look at the NUGET package ExcelDataReader, with additional information on
https://github.com/ExcelDataReader/ExcelDataReader
Usage example:
[Fact]
void Test_ExcelDataReader()
{
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
var scriptPath = Path.GetDirectoryName(Util.CurrentQueryPath); // LinqPad script path
var filePath = $#"{scriptPath}\TestExcel.xlsx";
using (var stream = File.Open(filePath, FileMode.Open, FileAccess.Read))
{
// Auto-detect format, supports:
// - Binary Excel files (2.0-2003 format; *.xls)
// - OpenXml Excel files (2007 format; *.xlsx, *.xlsb)
using (var reader = ExcelDataReader.ExcelReaderFactory.CreateReader(stream))
{
var result = reader.AsDataSet();
// The result of each spreadsheet is in result.Tables
var t0 = result.Tables[0];
Assert.True(t0.Rows[0][0].Dump("R0C0").ToString()=="Hello", "Expected 'Hello'");
Assert.True(t0.Rows[0][1].Dump("R0C1").ToString()=="World!", "Expected 'World!'");
} // using
} // using
} // fact
Before you start reading, you need to set and encoding provider as follows:
System.Text.Encoding.RegisterProvider(
System.Text.CodePagesEncodingProvider.Instance);
The cells are addressed the following way:
var t0 = result.Tables[0]; // table 0 is the first worksheet
var cell = t0.Rows[0][0]; // on table t0, read cell row 0 column 0
And you can easily loop through the rows and columns in a for loop as follows:
for (int r = 0; r < t0.Rows.Count; r++)
{
var row = t0.Rows[r];
var columns = row.ItemArray;
for (int c = 0; c < columns.Length; c++)
{
var cell = columns[c];
cell.Dump();
}
}

I use this code with EPPlus DLL, Don't forget to add reference. But should check to match with your requirement.
public DataTable ReadExcelDatatable(bool hasHeader = true)
{
using (var pck = new OfficeOpenXml.ExcelPackage())
{
using (var stream = File.OpenRead(this._fullPath))
{
pck.Load(stream);
}
var ws = pck.Workbook.Worksheets.First();
DataTable tbl = new DataTable();
int i = 1;
foreach (var firstRowCell in ws.Cells[1, 1, 1, ws.Dimension.End.Column])
{
//table head
tbl.Columns.Add(hasHeader ? firstRowCell.Text : string.Format("Column {0}", firstRowCell.Start.Column));
tbl.Columns.Add(_tableHead[i]);
i++;
}
var startRow = hasHeader ? 2 : 1;
for (int rowNum = startRow; rowNum <= ws.Dimension.End.Row; rowNum++)
{
var wsRow = ws.Cells[rowNum, 1, rowNum, ws.Dimension.End.Column];
DataRow row = tbl.Rows.Add();
foreach (var cell in wsRow)
{
row[cell.Start.Column - 1] = cell.Text;
}
}
return tbl;
}
}

I'm going to give you a different answer. If the performance is bad loading a million rows into a DataTable resort to using a Driver to load the data: How to open a huge excel file efficiently
DataSet excelDataSet = new DataSet();
string filePath = #"c:\temp\BigBook.xlsx";
// For .XLSXs we use =Microsoft.ACE.OLEDB.12.0;, for .XLS we'd use Microsoft.Jet.OLEDB.4.0; with "';Extended Properties=\"Excel 8.0;HDR=YES;\"";
string connectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source='" + filePath + "';Extended Properties=\"Excel 12.0;HDR=YES;\"";
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
conn.Open();
OleDbDataAdapter objDA = new System.Data.OleDb.OleDbDataAdapter
("select * from [Sheet1$]", conn);
objDA.Fill(excelDataSet);
//dataGridView1.DataSource = excelDataSet.Tables[0];
}
Next filter the DataSet's DataTable using a DataView. Using a DataView's RowFilter property you can specify subsets of rows based on their column values.
DataView prodView = new DataView(excelDataSet.Tables[0],
"UnitsInStock <= ReorderLevel",
"SupplierID, ProductName",
DataViewRowState.CurrentRows);
Ref: https://www.c-sharpcorner.com/article/dataview-in-C-Sharp/
Or you could use the DataTables' DefaultView RowFilter directly:
excelDataSet.Tables[0].DefaultView.RowFilter = "Amount >= 5000 and Amount <= 5999 and Name = 'StackOverflow'";

Saving DataTable to excel using EPPlus, not saving empty string over previous value

I have an .xlsx file stored on my desktop that my C# program reads from, then it loads each worksheet into datatables that my program uses. When a modification is made to a dataTable, I save to the .xlsx file by first loading the datatable back into the worksheet, and then saving the excelPackage with the modified dataTable information.
The problem is, I sometimes need to overwrite a cell with a blank string, and after saving, the previous value is still there.
It will let me update a value to a space (" "), but I want to save the cell as empty, "", or null.
This is loading my dataTables from the .xlsx
using (var pck = new OfficeOpenXml.ExcelPackage())
{
try
{
using (var stream = File.OpenRead(filePath + "/dataSet.xlsx"))
{
pck.Load(stream);
}
for (int i = 1; i < pck.Workbook.Worksheets.Count; i++)
{
var ws = pck.Workbook.Worksheets[i];
DataTable tbl = new DataTable();
foreach (var firstRowCell in ws.Cells[1, 1, 1, ws.Dimension.End.Column])
{
tbl.Columns.Add(true ? firstRowCell.Text : string.Format("Column {0}", firstRowCell.Start.Column));
}
var startRow = true ? 2 : 1;
for (int rowNum = startRow; rowNum <= ws.Dimension.End.Row; rowNum++)
{
var wsRow = ws.Cells[rowNum, 1, rowNum, ws.Dimension.End.Column];
DataRow row = tbl.Rows.Add();
foreach (var cell in wsRow)
{
row[cell.Start.Column - 1] = cell.Text;
}
}
if (ws.Name == "customerDataTable") { _customerDataTable = tbl; }
else if (ws.Name == "vehicleDataTable") { _vehicleDataTable = tbl; }
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
This is saving the dataTable information into worksheet, then to .xlsx
string tableName = "";
if (table == _customerDataTable) { tableName = "customerDataTable"; }
else if (table == _vehicleDataTable) { tableName = "vehicleDataTable"; }
FileInfo file = new FileInfo(filePath + "/dataSet.xlsx");
using (ExcelPackage excelPackage = new ExcelPackage(file))
{
ExcelWorkbook excelWorkBook = excelPackage.Workbook;
ExcelWorksheet excelWorksheet = excelWorkBook.Worksheets[tableName];
excelWorksheet.Cells.LoadFromDataTable(table, true);
excelPackage.Save();
}
I got it to work following VDWWDs post, using the following code:
for (int i = 0; i < table.Rows.Count; i++)
{
for (int j = 0; j < table.Columns.Count; j++)
{
excelWorksheet.Cells[i+2,j+1].Value = table.Rows[i][j];
}
}
//excelWorksheet.Cells.LoadFromDataTable(table, true);
I replaced the LoadFromTable() line with a nested For loop. Although it functionally works, I'm going to be using this method a lot throughout my program, and this solution seems a bit bulky compared to EPPlus LoadFromTable method. I'm thinking there has to be a better way...

EPPlus save two million rows with 200+ columns datatable to multiple excel files

I have function that saves all records from SQL table to excel worksheet using EPPlus.
If I export small amount of data everything works fine, but with 200+ columns and 500 000+ rows I get OutOfMemory exception.
I'd like to modify my code in a way to be able to save 50 000 records per file.
Here is my code that works for small data:
private Task SaveAsync(string tableName)
{
return Task.Run(() =>
{
try
{
using (var conn = new SqlConnection(_connectionString))
{
using (var cmd = new SqlCommand(string.Format(DataQuery, tableName), conn))
{
cmd.CommandType = CommandType.Text;
cmd.CommandTimeout = 360;
conn.Open();
using (SqlDataReader sdr = cmd.ExecuteReader())
{
var fileName = string.Format(TargetFile, tableName);
if (File.Exists(fileName))
{
File.Delete(fileName);
}
sdr.Read();
var numberOfRecordsInTable = sdr.GetInt32(0);
sdr.NextResult();
using (ExcelPackage pck = new ExcelPackage(new FileInfo(fileName)))
{
ExcelWorksheet ws = pck.Workbook.Worksheets.Add("Results");
int count = sdr.FieldCount;
int col = 1, row = 1;
for (int i = 0; i < count; i++)
{
ws.SetValue(row, col++, sdr.GetName(i));
}
row++;
col = 1;
while (sdr.Read())
{
for (int i = 0; i < count; i++)
{
var val = sdr.GetValue(i);
ws.SetValue(row, col++, val);
}
row++;
col = 1;
}
//autosize
ws.Cells[ws.Dimension.Address].AutoFitColumns();
//autofiltr
ws.Cells[1, 1, 1, count].AutoFilter = true;
}
}
conn.Close();
}
}
}
catch (Exception e)
{
Debug.WriteLine("Error at: " + Thread.CurrentThread.ManagedThreadId);
Debug.WriteLine(e);
}
});
}
and my modified code that splits records 50 000 per file:
private Task SaveAsync2(string tableName)
{
return Task.Run(() =>
{
try
{
using (var conn = new SqlConnection(_connectionString))
{
using (var cmd = new SqlCommand(string.Format(DataQuery, tableName), conn))
{
cmd.CommandType = CommandType.Text;
cmd.CommandTimeout = 360;
conn.Open();
using (SqlDataReader sdr = cmd.ExecuteReader())
{
var fileName = string.Format(TargetFile, tableName,"");
if (File.Exists(fileName))
{
File.Delete(fileName);
}
sdr.Read();
var max = sdr.GetInt32(0);
int filesCount = 1;
if (max > 50000)
{
fileName = string.Format(TargetFile, tableName, filesCount);
}
sdr.NextResult();
ExcelPackage pck = new ExcelPackage(new FileInfo(fileName));
ExcelWorksheet ws = pck.Workbook.Worksheets.Add("RESULTS");
int count = sdr.FieldCount;
int col = 1, row = 1;
for (int i = 0; i < count; i++)
{
ws.SetValue(row, col++, sdr.GetName(i));
}
row++;
col = 1;
while (sdr.Read())
{
for (int i = 0; i < count; i++)
{
var val = sdr.GetValue(i);
ws.SetValue(row, col++, val);
}
row++;
col = 1;
if (row > 50000)
{
pck.Save();
filesCount++;
fileName = string.Format(TargetFile, tableName, filesCount);
pck = new ExcelPackage(new FileInfo(fileName));
ws = pck.Workbook.Worksheets.Add("RESULTS");
count = sdr.FieldCount;
col = 1;
row = 1;
for (int i = 0; i < count; i++)
{
ws.SetValue(row, col++, sdr.GetName(i));
}
row++;
col = 1;
}
}
//autosize
ws.Cells[ws.Dimension.Address].AutoFitColumns();
//autofiltr
ws.Cells[1, 1, 1, count].AutoFilter = true;
pck.Save();
}
}
conn.Close();
}
}
catch (Exception e)
{
Debug.WriteLine("Error at: " + Thread.CurrentThread.ManagedThreadId);
Debug.WriteLine(e);
}
});
}
basically this works fine, but in first version of my code I was using everything inside using statement, when in second version I'm calling same code twice.
How can I fix my code to remove duplicate code and put everything inside using.
Can I add next set (50 000 records) as new worksheet instead of creating new file?
What would be EPPlus limit when saving data to file? rows x columns? I found information that EPPlus should handle more than million rows, but not so much columns as I have. I thinks that I can export million rows with single column, but for 200+ columns for me 50 000 rows is limit. I'm wondering if there is number (rows x columns) that will be limit to which my export will work fine. I want that export function to be universal, so when I pass datatable with 50 columns it will export for example 100 000 rows per file and for 2 columns it will export half million per file.

I've run up against memory limits with EPPlus in the past, and ended up generating multiple .xlsx files as a workaround (similar to your approach). Another alternative would be to change your compiler settings to target 64-bit only (if you can get by without supporting 32-bit platforms). As I recall, EPPlus is compiled for "Any CPU", so if you can change your code to target "x64" that would likely relax the memory limitations and allow you to generate a single .xlsx file. Targeting x64 likely would have worked in my case, but I didn't think of it until after the fact, so I never had an opportunity to test.
UPDATE:
I just ran a quick test using EPPlus 3.1.3 creating 500,000 rows, 70 columns each. My 32-bit application was able to generate about 119,000 rows before generating an out-of-memory exception. After switching the target to x64, it successfully generated all 500,000 rows, although it took forever. Creating the actual worksheet took just a few minutes, but ExcelPackage.SaveAs() took close to 20 minutes. RAM consumption was quite high as well (roughly 11GB of RAM). The resulting .xlsx is 220MB, which 32-bit Excel is not able to open (out of memory). The bottom line: Targeting x64 is probably not a viable solution; you'd be better off splitting the output into multiple .xlsx files.
I was tempted to delete this answer since it has turned out to be a dead end, but decided to leave it in case it helps someone else avoid this path in the future.

Unfortunately, there is no easy way to merge that much data with Epplus in a single file. Basically, the entire file is loaded into memory when open - its either all or nothing. In theory, you could generate the XML files that XLSX contains (they are zip files renamed) and manually insert it since it would have a smaller memory footprint but that is no small feat.
For you current code, you could always just call .dispose() manually if you want to avoid the using statement. But I understand you wanting to avoid duplicate code. What about something like this (but watch for memory usage when copying all the object data):
const int max = 10;
var loop = 0;
using (var sdr = cmd.ExecuteReader())
{
var fieldcount = sdr.FieldCount;
var getfi = new Func<int, FileInfo>(i =>
{
var fi = new FileInfo(String.Format(#"c:\temp\Multi_Files{0}.xlsx", i));
if (fi.Exists) fi.Delete();
return fi;
});
var savefile = new Action<FileInfo, List<Object[]>>((info, rows) =>
{
using (var pck = new ExcelPackage(info))
{
var wb = pck.Workbook;
var ws = wb.Worksheets.Add("RESULTS");
for (var row = 0; row < rows.Count; row++)
for (var col = 0; col < fieldcount; col++)
ws.SetValue(row + 1, col + 1, rows[row][col]);
pck.Save();
}
});
var rowlist = new List<Object[]>();
while (sdr.Read())
{
var rowdata = new Object[sdr.FieldCount];
sdr.GetValues(rowdata);
rowlist.Add(rowdata);
if (rowlist.Count == max)
{
savefile(getfi(++loop), rowlist);
rowlist.Clear();
}
}
if (rowlist.Count > 0)
savefile(getfi(++loop), rowlist);
}

Since you are creating a new excel file (correct me if I'm wrong), you can simply write an XML file with some specific contents. Excel supports .xml files if they contain the correct stuff.
You can simply create the contents of the XML file in-memory, and afterwards write this contents to an .XML file. You do not need the EPPlus package, therefore you are bypassing the limitations of the EPPlus package.
Of course, you have to figure out manually what you need to write in the .XML file. Of you are going to use formatting and formulas, it might be a complex.
See here:
https://www.google.nl/search?q=excel%2520xml%2520format&oq=excel%2520xml%2520format
https://technet.microsoft.com/en-gb/magazine/2006.01.blogtales.aspx
http://blogs.msdn.com/b/brian_jones/archive/2005/06/27/433152.aspx

simple solution without any tricks (not tested, but the intentions should be clear)
using (var conn = new SqlConnection(_connectionString))
{
int filesCount = 1;
int col = 1, row = 1;
string fileName = String.Empty;
int count;
ExcelPackage pck;
ExcelWorksheet ws;
using (var cmd = new SqlCommand(string.Format(DataQuery, tableName), conn))
{
cmd.CommandType = CommandType.Text;
cmd.CommandTimeout = 360;
conn.Open();
using (SqlDataReader sdr = cmd.ExecuteReader())
{
while (sdr.Read())
{
if (row == 1)
{
fileName = string.Format(TargetFile, tableName, filesCount);
if (File.Exists(fileName))
{
File.Delete(fileName);
}
pck = new ExcelPackage(new FileInfo(fileName));
ws = pck.Workbook.Worksheets.Add("RESULTS");
}
count = sdr.FieldCount;
for (int i = 0; i < count; i++)
{
var val = sdr.GetValue(i);
ws.SetValue(row, col++, val);
}
row++;
col = 1;
if (row >= 50000)
{
ws.Cells[ws.Dimension.Address].AutoFitColumns();
ws.Cells[1, 1, 1, count].AutoFilter = true;
pck.Save();
row = 1;
filesCount+
}
}
}
if (row > 1)
{
ws.Cells[ws.Dimension.Address].AutoFitColumns();
ws.Cells[1, 1, 1, count].AutoFilter = true;
pck.Save();
}
}
}
conn.Close();

Trying to read excel file with epplus and getting System.NullException error?

Edit
Based on the replies below, the error I am experiencing may or may not be causing my inability to read my excel file. That is, I am not getting data from the line worksheet.Cells[row,col].Value in my for loop given below.
Problem
I am trying to return a DataTable with information from an excel file. Specifically, it is an xlsx file from 2013 excel I believe. Please see the code below:
private DataTable ImportToDataTable(string Path)
{
DataTable dt = new DataTable();
FileInfo fi = new FileInfo(Path);
if(!fi.Exists)
{
throw new Exception("File " + Path + " Does not exist.");
}
using (ExcelPackage xlPackage = new ExcelPackage(fi))
{
//Get the worksheet in the workbook
ExcelWorksheet worksheet = xlPackage.Workbook.Worksheets.First();
//Obtain the worksheet size
ExcelCellAddress startCell = worksheet.Dimension.Start;
ExcelCellAddress endCell = worksheet.Dimension.End;
//Create the data column
for(int col = startCell.Column; col <= endCell.Column; col++)
{
dt.Columns.Add(col.ToString());
}
for(int row = startCell.Row; row <= endCell.Row; row++)
{
DataRow dr = dt.NewRow(); //Create a row
int i = 0;
for(int col = startCell.Column; col <= endCell.Column; col++)
{
dr[i++] = worksheet.Cells[row, col].Value.ToString();
}
dt.Rows.Add(dr);
}
}
return dt;
}
Error
This is where things get weird. I can see the proper value in startCell and endCell. However, when I look at worksheet I take a peek under Cells and I see something I don't understand:
worksheet.Cells.Current' threw an exception of type 'System.NullReferenceException
Attempts
Reformatting my excel with general fields.
Making sure no field in my excel was empty
RTFM'ed epplus documentation. Nothing suggestive of this error.
Looked at EPPlus errors on stackoverflow. My problem is unique.
Honestly, I am having trouble figuring out what this error is really saying? Is something wrong with my format? Is something wrong with epplus? I have read on here people had no problems with 2013 xlsx with eeplus and I am only trying to parse the excel file by row. If someone could help me shed light on what this error means and how to rectify it. I would be most grateful. I've spent quite a long time trying to figure this out.

When we give:
dr[i++] = worksheet.Cells[row, col].Value.ToString();
it search for value at that column, if the column is empty, it gives Null reference error.
Try instead:
dr[i++] = worksheet.Cells[row, col].Text;
Hope this will help

Like #Thorians said, current is really meant to use when you enumerating the cells. If you want to use it in purest form and actually be able to call current then you would need something like this:
using (var pck = new ExcelPackage(existingFile))
{
var worksheet = pck.Workbook.Worksheets.First();
//this is important to hold onto the range reference
var cells = worksheet.Cells;
//this is important to start the cellEnum object (the Enumerator)
cells.Reset();
//Can now loop the enumerator
while (cells.MoveNext())
{
//Current can now be used thanks to MoveNext
Console.WriteLine("Cell [{0}, {1}] = {2}"
, cells.Current.Start.Row
, cells.Current.Start.Column
, cells.Current.Value);
}
}
Note that you have to create a kind of local collection cells for this to work properly. Otherwise Current will be null if you tried `worksheet.cells.current'
But it would be simpler to use a ForEach and have the CLR do the work for you.
UPDATE: Based on comments. Your code should work fine as is, could it be your excel file:
[TestMethod]
public void Current_Cell_Test()
{
//http://stackoverflow.com/questions/32516676/trying-to-read-excel-file-with-epplus-and-getting-system-nullexception-error
//Throw in some data
var datatable = new DataTable("tblData");
datatable.Columns.AddRange(new[] { new DataColumn("Col1", typeof (int)), new DataColumn("Col2", typeof (int)),new DataColumn("Col3", typeof (object)) });
for (var i = 0; i < 10; i++)
{
var row = datatable.NewRow(); row[0] = i; row[1] = i * 10; row[2] = Path.GetRandomFileName(); datatable.Rows.Add(row);
}
//Create a test file
var fi = new FileInfo(#"c:\temp\test1.xlsx");
if (fi.Exists)
fi.Delete();
using (var pck = new ExcelPackage(fi))
{
var worksheet = pck.Workbook.Worksheets.Add("Sheet1");
worksheet.Cells.LoadFromDataTable(datatable, true);
pck.Save();
}
var dt = new DataTable();
using (ExcelPackage xlPackage = new ExcelPackage(fi))
{
//Get the worksheet in the workbook
ExcelWorksheet worksheet = xlPackage.Workbook.Worksheets.First();
//Obtain the worksheet size
ExcelCellAddress startCell = worksheet.Dimension.Start;
ExcelCellAddress endCell = worksheet.Dimension.End;
//Create the data column
for (int col = startCell.Column; col <= endCell.Column; col++)
{
dt.Columns.Add(col.ToString());
}
for (int row = startCell.Row; row <= endCell.Row; row++)
{
DataRow dr = dt.NewRow(); //Create a row
int i = 0;
for (int col = startCell.Column; col <= endCell.Column; col++)
{
dr[i++] = worksheet.Cells[row, col].Value.ToString();
}
dt.Rows.Add(dr);
}
}
Console.Write("{{dt Rows: {0} Columns: {1}}}", dt.Rows.Count, dt.Columns.Count);
}
Give this in the output:
{Rows: 11, Columns: 3}

Current is the current range when enumerating.
there is nothing wrong with this throwing an exception in debugging inspection when it is not being used within an enumerating scope.
code sample:
var range = ws.Cells[1,1,1,100];
foreach (var cell in range)
{
var a = range.Current.Value; // a is same as b
var b = cell.Value;
}

I am also getting same issue while reading excel file and none of the solution provided worked for me. Here is working code:
public void readXLS(string FilePath)
{
FileInfo existingFile = new FileInfo(FilePath);
using (ExcelPackage package = new ExcelPackage(existingFile))
{
//get the first worksheet in the workbook
ExcelWorksheet worksheet = package.Workbook.Worksheets[1];
int colCount = worksheet.Dimension.End.Column; //get Column Count
int rowCount = worksheet.Dimension.End.Row; //get row count
for (int row = 1; row <= rowCount; row++)
{
for (int col = 1; col <= colCount; col++)
{
Console.WriteLine(" Row:" + row + " column:" + col + " Value:" + worksheet.Cells[row, col].Value.ToString().Trim());
}
}
}
}