We Keep Coding

Copy contents of the Excel file from a stream asynchronously using SqlBulkCopy

I have a project where I need to copy the contents of the .xlsx file I received in Web API Controller (in the form of the Stream from MultipartReader) to SQL Server Database. I'm using SqlBulkCopy for copying itself (I already did a similar task for .csv files), but all of the solutions I was able to find suffer from one or more of the following problems:
Require saving the file to the disk first (not possible in my case)
Don't have any way of reading the file asynchronously
Load entire file into memory first (I'm expecting to deal with fairly large files, so this is not acceptable for me)
Are commercially licensed
Are there any ways of doing this?

Jeroen is correct, in that it is not possible to handle Excel files in a purely streaming manner. While it might require loading the entire .xlsx file in memory, the efficiency of the library can have an even larger impact on the memory usage than the file size. I say this as the author of the most efficient Excel reader for .NET: Sylvan.Data.Excel.
In benchmarks comparing it to other libraries, you can see that not only is it significantly faster than other implementations, but it also uses only a tiny fraction of the memory that other libraries consume.
With the exception of "Load entire file into memory first", it should satisfy all of your requirements. It can process data out of a MemoryStream, it doesn't need to write to disk. It implements DbDataReader which provides ReadAsync. The ReadAsync implementation defaults to the base DbDataReader implementation which defers to the synchronous Read() method, but when the file is buffered in a MemoryStream this doesn't present a problem, and allows the SqlBulkCopy.WriteToServerAsync to process it asynchronously. Finally, it is MIT licensed, so you can do whatever you want with it.
using Sylvan.Data;
using Sylvan.Data.Excel;
using System.Data.Common;
using System.Data.SqlClient;
// provide a schema that maps the columns in the Excel file to the names/types in your database.
var opts = new ExcelDataReaderOptions
{
Schema = MyDataSchemaProvider.Instance
};
var filename = "mydata.xlsx";
var ms = new MemoryStream();
// asynchronously load the file into memory
// this might be loading from an Asp.NET IFormFile instead
using(var f = File.OpenRead(filename))
{
await f.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
}
// determine the workbook type from the file-extension
var workbookType = ExcelDataReader.GetWorkbookType(filename);
var edr = ExcelDataReader.Create(ms, workbookType, opts);
// "select" the columns to load. This extension method comes from the Sylvan.Data library.
var dataToLoad = edr.Select("PartNumber", "ServiceDate");
// bulk copy the data to the server.
var conn = new SqlConnection("Data Source=.;Initial Catalog=mydb;Integrated Security=true;");
conn.Open();
var bc = new SqlBulkCopy(conn);
bc.DestinationTableName = "MyData";
bc.EnableStreaming = true;
await bc.WriteToServerAsync(dataToLoad);
// Implement an ExcelSchemaProvider that maps the columns in the excel file
sealed class MyDataSchemaProvider : ExcelSchemaProvider
{
public static ExcelSchemaProvider Instance = new MyDataSchemaProvider();
static readonly DbColumn PartNumber = new MyColumn("PartNumber", typeof(int));
static readonly DbColumn ServiceDate = new MyColumn("ServiceDate", typeof(DateTime));
// etc...
static readonly Dictionary<string, DbColumn> Mapping = new Dictionary<string, DbColumn>(StringComparer.OrdinalIgnoreCase)
{
{ "partnumber", PartNumber },
{ "number", PartNumber },
{ "prt_nmbr", PartNumber },
{ "servicedate", ServiceDate },
{ "service_date", ServiceDate },
{ "svc_dt", ServiceDate },
{ "sd", ServiceDate },
};
public override DbColumn? GetColumn(string sheetName, string? name, int ordinal)
{
if (string.IsNullOrEmpty(name))
{
// There was no name in the header row, can't map to anything.
return null;
}
if (Mapping.TryGetValue(name, out DbColumn? col))
{
return col;
}
// header name is unknown. Might be better to throw in this case.
return null;
}
class MyColumn : DbColumn
{
public MyColumn(string name, Type type, bool allowNull = false)
{
this.ColumnName = name;
this.DataType = type;
this.AllowDBNull = allowNull;
}
}
public override bool HasHeaders(string sheetName)
{
return true;
}
}
The most complicated part of this is probably the "schema provider" which is used to provide header name mappings and define the column types, which are required for SqlBulkCopy to operate correctly.
I also maintain the Sylvan.Data.Csv library, which provides very similar capabilities for CSV files, and is a fully asynchronous streaming CSV reader impelementation. The API it provides is nearly identical to the Sylvan ExcelDataReader. It is also the fastest CSV reader for .NET.
If you end up trying these libraries and have any troubles, open an issue in the github repo and I can take a look.

How to write a CSV file after reading it with CsvHelper?

I need to write a HttpPostedFileBase csv (and save it) after mapping it into a list using the CsvHelper nuget package. However, after mapping it with CsvHelper, the ContentLength is 0, and I end up saving an empty .csv file.
CsvHelper itself states that the Read method should not be used when using the GetRecords<T>() method.
// Summary:
// Gets all the records in the CSV file and converts each to System.Type T. The
// Read method should not be used when using this.
//
// Type parameters:
// T:
// The System.Type of the record.
//
// Returns:
// An System.Collections.Generic.IEnumerable`1 of records.
public virtual IEnumerable<T> GetRecords<T>();
I tried placing it into a copy variable:
HttpPostedFileBase csvCopy = csvFile;
But this didn't work. Tried some other solutions I found on stackoverflow, which didn't work either. I "solved" this problem by sending the same file twice to the controller as a parameter. Then I use the first one with CsvHelper, and I read and save the other one.
public async Task<ActionResult> ImportCSV(HttpPostedFileBase csvFile, HttpPostedFileBase csvFileCopy)
However, I think this is a bad solution. I would like to use a single file, map it, reread it and save it.
Mapping it into a list:
using(var reader = new StreamReader(csvFile.InputStream)) {
using(var csvReader = new CsvReader(reader)) {
csvReader.Configuration.RegisterClassMap(new CSVModelMap(mapDictionary));
csvReader.Configuration.BadDataFound = null;
csvReader.Configuration.HeaderValidated = null;
csvReader.Configuration.MissingFieldFound = null;
importData = csvReader.GetRecords<CSVModel>().ToList();
}
}
Saving it:
var fileName = serverPath + "\\" + hashedFileName;
CheckIfDirectoryExists(serverPath);
var reader = new StreamReader(csvFile.InputStream);
var csvContent = await reader.ReadToEndAsync();
File.WriteAllText(fileName, csvContent);

I'm not to sure how GetRecords works, but there might be the possibility that it leaves the cursor on the stream pointed towards the end.
Meaning on your saving sequence you start reading the InputStream at the end, which results in no data to read.
So you could try
csvFile.InputStream.Seek(0, SeekOrigin.Begin)
EDIT:
For your try to copy the stream you only copy the reference to the object and not the object itself.
To copy the stream data you would need to use the method CopyTo(stream) which would leave the cursor on the end of the stream, so here seek is needed for sure.

The issue was CsvHelper:
using(var csvReader = new CsvReader(reader))
At the end of the using statement, CsvReader closes the reader. Then when I try
csvFile.InputStream.Seek(0, SeekOrigin.Begin);
or
reader.BaseStream.Position = 0;
it throws a NullReferenceException. I solved this by simply overriding the CsvReader:
using (var csvReader = new CsvReader(reader, true))
true being leaveOpen:
// Summary:
// Creates a new CSV reader using the given System.IO.TextReader.
//
// Parameters:
// reader:
// The reader.
//
// leaveOpen:
// true to leave the reader open after the CsvReader object is disposed, otherwise
// false.
public CsvReader(TextReader reader, bool leaveOpen);
Then I set the position back to 0, using reader.BaseStream.Position = 0;, and then after saving the file, dispose the reader.

C# Generated Excel File: File Format or File Extension is not valid

I'm calling the action "Export" where i pass a list of viewmodels and define the format
public ActionResult DownloadTokenlist(string startDate = null, string endDate = null)
{
using (HRCTSStatisticDb db = new HRCTSStatisticDb(Setting.ClientId))
{
List<TokenExportViewModel> tokenExportViewModels = new List<TokenExportViewModel>();
Response.AddHeader("content-disposition", $"attachment;filename=Tokenlist_{DateTime.Now.ToString("dd.MM.yyyy")}.xlsx");
log.InfoFormat($"The {new HomeController().UserRole(Context.LoggedInUser)}: {Context.LoggedInUser} has used the exceldownload");
return File(new ExcelExport().Export(tokenExportViewModels), "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
}
}
The action i call (ExcelEngine is by Syncfusion):
public MemoryStream Export(List<TokenExportViewModel> list)
{
MemoryStream stream = new MemoryStream();
using (ExcelEngine excelEngine = new ExcelEngine())
{
IApplication application = excelEngine.Excel;
application.DefaultVersion = ExcelVersion.Excel2010;
IWorkbook workbook = application.Workbooks.Create(1);
IWorksheet worksheet = workbook.Worksheets.Create("Tokenlist");
IStyle defaultStyle = workbook.Styles.Add("default");
defaultStyle.Font.Size = 12;
worksheet.SetDefaultColumnStyle(1, 20, defaultStyle);
worksheet.SetDefaultRowStyle(1, 300, defaultStyle);
worksheet.UsedRange.AutofitColumns();
worksheet.Range["A1"].Text = $"Tokenlist - {DateTime.Today.ToString("dd.MM.yyyy")}";
worksheet.Range["A1"].CellStyle = h1Style;
workbook.SaveAs(stream);
workbook.Close();
}
return stream;
}
I only posted the code which has an impact on the file and (maybe) could create the error.
There is no error, until i open the file, then this exception pops up:
Excel cannot open the file 'Tokenlist_22.05.2018.xlsx' because the
file format or file extension is not valid. Verify that the file has
not been corrupted and that the file extension matches the format of
the file.
I've tried to change the file format to .xls and .vbs but neither works. With .xls I can open the document but then it has no data in it.
The .close() doesn't change much, it just closes the output stream previously opened.

As stream reached end position while returning it, the downloaded file is getting corrupted. So, it is recommended to set its current position to 0 to resolve this issue. Please refer below code to achieve the same.
Code Example:
workbook.SaveAs(stream);
workbook.Close();
stream.Position = 0;
We have also shared a simple sample for your reference which can be downloaded from following link.
Sample Link: http://www.syncfusion.com/downloads/support/directtrac/general/ze/Sample1020485770.zip
I work for Syncfusion.

Use the FileContentResult Overload where you can provide thefileDownloadName like this:
return File(excelExport.Export(tokenExportViewModels).ToArray(),"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", $"Tokenlist_{DateTime.Now.ToString("dd.MM.yyyy")}.xlsx");
And use the stream ToArray() extension to return a byte[].
(I assume your Export method is generating a valid document)

Add PDF file as attachment to TestTrack using SOAP

I have recently started using C# over the past year so I'm somewhat new to this, but can usually hack through things with some effort, but this one is eluding me. We use TestTrack for development bug/issue tracking at our company. I've created a custom windows forms app to be the front-end to TestTrack for one of our departments. It connects using SOAP. I'm not using WPF/WCF and don't want to go that route. I'm having difficulty finding any examples of how to correctly encode a file for attachment that is a PDF. The code below does actually create an attachment in TestTrack to an already-existing issue, but when you try to open it in TestTrack, it pops up an error message that says "Insufficient Data For An Image". The example below does work if you're wanting to add a text file to TestTrack using SOAP. I'm wanting to know what I need to change below so that I can get a PDF file into TestTrack and then be able to open it in the TestTrack application without the error mentioned above. Thanks in advance for any input/help.
public void getAttachments(long lSession, CDefect def)
{
ttsoapcgi cgiengine = new ttsoapcgi();
// Lock the defect for edit.
CDefect lockedDefect = cgiengine.editDefect(lSession, def.recordid, "", false);
string attachment = "c:\\TEST\\TEST_PDF.PDF";
CFileAttachment file = new CFileAttachment();
file.mstrFileName = Path.GetFileName(attachment);
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
StreamReader reader = new StreamReader(attachment);
file.mstrFileName = Path.GetFileName(attachment);
file.mpFileData = enc.GetBytes(reader.ReadToEnd());
reader.Close();
CReportedByRecord reprec = lockedDefect.reportedbylist[0];
CFileAttachment[] afile = reprec.attachmentlist;
if (afile == null)
{
lockedDefect.reportedbylist[0].attachmentlist = new CFileAttachment[1];
lockedDefect.reportedbylist[0].attachmentlist[0] = file;
}
// Save our changes.
cgiengine.saveDefect(lSession, lockedDefect);
}
}

Here is the modified method that allowed me to attach a PDF to SOAP and get it into TestTrack as an attachment to an issue:
public void getAttachments(long lSession, CDefect def)
{
ttsoapcgi cgiengine = new ttsoapcgi();
// Lock the defect for edit.
CDefect lockedDefect = cgiengine.editDefect(lSession, def.recordid, "", false);
string attachment = "c:\\TEST\\TEST_PDF.PDF";
CFileAttachment file = new CFileAttachment();
file.mpFileData = File.ReadAllBytes(attachment);
file.mstrFileName = Path.GetFileName(attachment);
CReportedByRecord reprec = lockedDefect.reportedbylist[0];
CFileAttachment[] afile = reprec.attachmentlist;
if (afile == null)
{
lockedDefect.reportedbylist[0].attachmentlist = new CFileAttachment[1];
lockedDefect.reportedbylist[0].attachmentlist[0] = file;
}
// Save our changes.
cgiengine.saveDefect(lSession, lockedDefect);
}

LINQtoCSV Stream provided to read is either null, or does not support seek

I am trying to use LINQtoCSV to parse out a CSV file into a list of objects and am receiving the error "Stream provided to read is either null, or does not support seek."
The error is happening at foreach(StockQuote sq in stockQuotesStream)
Below is the method that is throwing the error. The .CSV file is being downloaded from the internet and is never stored to disk (only stored to StreamReader).
public List<StockQuote> CreateStockQuotes(string symbol)
{
List<StockQuote> stockQuotes = new List<StockQuote>();
CsvFileDescription inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = false
};
CsvContext cc = new CsvContext();
IEnumerable<StockQuote> stockQuotesStream = cc.Read<StockQuote>(GetCsvData(symbol));
foreach (StockQuote sq in stockQuotesStream)
{
stockQuotes.Add(sq);
}
return stockQuotes;
}

The .CSV file is being downloaded from the internet and is never stored to disk (only stored to StreamReader).
Well presumably that's the problem. It's not quite clear what you mean by this, in that if you have wrapped a StreamReader around it, that's a pain in terms of the underlying stream - but you can't typically seek on a stream being downloaded from the net, and it sounds like the code you're using requires a seekable stream.
One simple option is to download the whole stream into a MemoryStream (use Stream.CopyTo if you're using .NET 4), then rewind the MemoryStream (set Position to 0) and pass that to the Read method.

Using a MemoryStream first and then a StreamReader was the answer, but I went about it a little differently than mentioned.
WebClient client = new WebClient();
using (MemoryStream download = new MemoryStream(client.DownloadData(url)))
{
using (StreamReader dataReader = new StreamReader(download, System.Text.Encoding.Default, true))
{
return dataReader;
}
}