We Keep Coding

Reading CSV of unknown number of rows/columns into Unity array

I want a 2D array generated from a CSV file with unknown number of rows/columns. The column count is fixed based on the header data. I need to be able to process it as a grid going both across rows and down columns hence needing array.
At the moment, I can split the data into rows, then split each row into components. I then add each row to a list. This all seems to work fine.
What I cant do is convert a list of string arrays into a 2d array.
It currently is failing on the line string[,] newCSV = csvFile.ToArray(); with error Cannot implicitly convert type 'string[][]' to 'string[ * , * ]' so I'm obviously not declaring something properly - I've just no idea what!
List<string[]> csvFile = new List<string[]>();
void Start()
{
// TODO: file picker
TextAsset sourceData = Resources.Load<TextAsset>("CSVData");
if (sourceData != null)
{
// Each piece of data in a CSV has a newline at the end
// Split the base data into an array each time the newline char is found
string[] data = sourceData.text.Split(new char[] {'\n'} );
for (int i = 0; i < data.Length; i ++)
{
string[] row = data[i].Split(new char[] {','} );
Debug.Log(row[0] + " " + row[1]);
csvFile.Add(row);
}
string[,] newCSV = csvFile.ToArray();
} else {
Debug.Log("Can't open source file");
}

Since your data is in the form of a table, I highly suggest using a DataTable instead of a 2d array like you're currently using to model/hold the data from your csv.
There's a ton of pre baked functionality that comes with this data structure that will make working with your data much easier.
If you take this route, you could then also use this which will copy CSV data into a DataTable using the structure of your CSV data to create the DataTable.
It's very easy to configure and use.
Just a small tip, you should always try to use data structures that best fit your task, whenever possible. Think of the data structures and algorithms you use as tools used to build a house, while you could certainly use a screw driver to pound in a nail, it's much easier and more efficient to use a hammer.

You can use this function to get 2d array.
static public string[,] SplitCsvGrid(string csvText)
{
string[] lines = csvText.Split("\n"[0]);
// finds the max width of row
int width = 0;
for (int i = 0; i < lines.Length; i++)
{
string[] row = SplitCsvLine(lines[i]);
width = Mathf.Max(width, row.Length);
}
// creates new 2D string grid to output to
string[,] outputGrid = new string[width + 1, lines.Length + 1];
for (int y = 0; y < lines.Length; y++)
{
string[] row = SplitCsvLine(lines[y]);
for (int x = 0; x < row.Length; x++)
{
outputGrid[x, y] = row[x];
// This line was to replace "" with " in my output.
// Include or edit it as you wish.
outputGrid[x, y] = outputGrid[x, y].Replace("\"\"", "\"");
}
}
return outputGrid;
}

Write a 2D array of objects in range Excel C#

I created a new user defined function in excel using c#, excelDna Add-In,what i want to get is when a user open an excel file, clic on a cell and write =myFunction() press enter, data should be displayed in the excel file. these data are retrieved from sql server database and stocked in a 2D array of object, my problem is when i try to display this array in excel range i got this exception
Exception de HRESULT : 0x800A03EC
Below is my code :
public static void LoadViewData()
{
var target = (ExcelReference)XlCall.Excel(XlCall.xlfCaller);
var sheetName = (string)XlCall.Excel(XlCall.xlSheetNm, target);
var application = (Microsoft.Office.Interop.Excel.Application)ExcelDnaUtil.Application;
var sheet = application.Sheets[Regex.Replace(sheetName, #"\[[^]]*\]", string.Empty)];
object[,] result = LoadFromDbData();
var startCell =sheet.Cells[target.RowFirst + 1, target.ColumnFirst];
var endCell =sheet.Cells[target.RowFirst+ result.GetUpperBound(0) - result.GetLowerBound(0) + 1,
target.ColumnFirst+ result.GetUpperBound(1) - result.GetLowerBound(1) + 1];
var writeRange = sheet.Range[startCell, endCell];
writeRange.Value2 = result;
}
target returns the correct value of the cell where the user has written the formula (=myFunction())
sheetName returns the correct activeSheet in which the user writes the formula
result contains the data retrieved from sql server, it is an array of object[854,8]
startcell and endcell represents the range from which cell to which cell data will be displayed
when debugging, all variables contain the correct values, the exception appears in this instruction :
writeRange.Value2 = result;
Anyone has already worked with this or can help please ?
Thanks

I think your "LoadFromDbData()" is returning the data as typed. Try converting each value to a string. Here is a sample (and I can recreate that error code if I do not convert to string):
void Main()
{
var tbl = new System.Data.DataTable();
new SqlDataAdapter(#"
WITH tally ( OrderNo, UniqueId, RandNumber )
AS (
SELECT TOP 50000
ROW_NUMBER() OVER ( ORDER BY t1.object_id ),
NEWID(),
CAST(CAST(CAST(NEWID() AS VARBINARY(4)) AS INT) AS DECIMAL) / 1000
FROM master.sys.all_columns t1
CROSS JOIN master.sys.all_columns t2
)
SELECT OrderNo,
DATEADD(DAY, -OrderNo, GETDATE()) as OrnekDate,
UniqueId, RandNumber,
abs(RandNumber)%100 / 100 as pct
FROM [tally];", #"server=.\SQLExpress;Database=master;Trusted_Connection=yes;").Fill(tbl);
object[,] arr = new object[tbl.Rows.Count + 1, tbl.Columns.Count];
for (int i = 0; i < tbl.Columns.Count; i++)
{
arr[0, i] = tbl.Columns[i].Caption;
}
for (int i = 0; i < tbl.Rows.Count; i++)
{
for (int j = 0; j < tbl.Columns.Count; j++)
{
arr[i + 1, j] = tbl.Rows[i][j].ToString(); // without .ToString() you should have the error
}
}
// Excel dosya yarat ve arrayi koy
Microsoft.Office.Interop.Excel.Application xl = new Microsoft.Office.Interop.Excel.Application();
var workbook = xl.Workbooks.Add();
xl.Visible = true;
Worksheet sht = ((Worksheet)workbook.ActiveSheet);
Range target = (Range)sht.Range[ (Range)sht.Cells[1,1], (Range)sht.Cells[arr.GetUpperBound(0)+1,arr.GetUpperBound(1)+1] ];
target.Value2 = arr;
}
Note: As a side note, why would you transfer the data as a 2D array? That is one of the ways, but beware it is limited (I don't know a good value for the upper limit - try a high number like 200K rows).
Getting data into excel is best done via QueryTables.Add or CopyFromRecordSet to my experience. Depending on your needs you might also directly use the Excel file itself as a data table and do inserts. There is also EPPlus library on Nuget but that would be a little bit slow + may not contain all the capabilities you need.

Storing Excel Rows in List<Range> for later insertion into different workbook

I am splitting one large spreadsheet into many (100's) smaller spreadsheet. My approach is to store the rows of the source spreadsheet in a list:
List<Range> ranges = new List<Range>();
Workbook book = xl.Workbooks.Add("path to book");
Worksheet sheet = book.sheets[1];
for (int r = 1; r <= sheet.UsedRange.Rows.Count; r++)
{
ranges.Add((Range)sheet.Rows[r]);
}
book.Close();
......
Workbook book2 = xl.Workbooks.Add();
Worksheet sheet2 = book2.sheets[1];
for (int r2 = 0; r2 <= ranges.Count; r2++)
{
Range row = (Range)ranges[r2]; //
sheet2.rows[r2+1].Value2 = row; //fails;
//querying in debug, the properties of row all throw an exception
//queryying sheet.rows[r2+1] expands as expected
}
If you see where my error is please advise.
Thanks.

I think after you close the first source book, you can no longer use the range references you've taken from it. So move book.Close() to the end of your snippet. Your upper bound for the for second loop should just be < not <=.
for (int r2 = 0; r2 < ranges.Count; r2++)
{
ranges[r2].Copy();
sheet2.Paste(sheet2.Rows[r2+1]);
}

Excel Work Book - Read from C# substantially slow ?

was experimenting with reading from an excel workbook and noticed it takes a long time to read a sheet with 3560 rows and 7 columns, about 1 minute and 17 seconds. All I did was loop through the whole sheet and store the values in a list.
Is this normal, or am I doing something wrong ?
static void Main(string[] args)
{
List<string> testList = new List<string>();
Excel.Application excelApp = new Excel.Application();
Excel.Workbook workbook = excelApp.Workbooks.Open(#"C:\Users\rnewell\Desktop\FxData.xlsx");
Excel.Worksheet worksheet = workbook.Sheets[1];
Excel.Range range = worksheet.UsedRange;
int rowCount = range.Rows.Count;
int colCount = range.Columns.Count;
int rowCounter = 1;
int colCounter = 1;
while (rowCounter < rowCount)
{
colCounter = 1;
while (colCounter <= colCount)
{
//Console.Write(range.Cells[rowCounter, colCounter].Value2.ToString() + " ");
testList.Add(range.Cells[rowCounter, colCounter].Value2.ToString());
colCounter++;
}
Console.WriteLine();
rowCounter++;
}
Console.ReadKey();
excelApp.Workbooks.Close();
}

#TimWilliams' comment is the correct answer. Reading a single cell takes as long as reading a range of any size. This is the overhead of talking to the COM layer, and you are incurring it thousands of times. You should write the range to an object[,], and then access that array cell by cell.
int rowCount = range.Rows.Count;
int colCount = range.Columns.Count;
object[,] values= range.Value2;
int rowCounter = 1;
int colCounter = 1;
while (rowCounter < rowCount)
{
colCounter = 1;
while (colCounter <= colCount)
{
// check for null?
testList.Add(values[rowCounter, colCounter].ToString());
}
}
Note that the array will be one-based instead of zero-based like normal C# arrays. The data will go from 1 to rowCount and from 1 to colCount, but Rows and Columns properties will return rowCount and colCount, not 1 + rowCount and 1 + colCount. If you want to write data back, you can use a zero-based array of the right size (in fact you have to AFAIK since you can't create a one-based array) and it will work fine.

Since you are loading data from the Open XML (*.xlsx) file format, I would suggest you use Open XML SDK. It doesn't start Excel in the background which is always a good thing, in particular if you need to run your code non-interactively.
I've also written a blog post on different methods of accessing data in Excel which you might find useful.

In general, it should be a matter of seconds.
But as you are creating an instance of Excel itself including its addons it may take a long time to initialize everything in your instance.
For your purpose you can use any public domain excel sheet reading library which doesn't launch Excel.

How to speed up dumping a DataTable into an Excel worksheet?

I have the following routine that dumps a DataTable into an Excel worksheet.
private void RenderDataTableOnXlSheet(DataTable dt, Excel.Worksheet xlWk,
string [] columnNames, string [] fieldNames)
{
// render the column names (e.g. headers)
for (int i = 0; i < columnNames.Length; i++)
xlWk.Cells[1, i + 1] = columnNames[i];
// render the data
for (int i = 0; i < fieldNames.Length; i++)
{
for (int j = 0; j < dt.Rows.Count; j++)
{
xlWk.Cells[j + 2, i + 1] = dt.Rows[j][fieldNames[i]].ToString();
}
}
}
For whatever reason, dumping DataTable of 25 columns and 400 rows takes about 10-15 seconds on my relatively modern PC. Takes even longer testers' machines.
Is there anything I can do to speed up this code? Or is interop just inherently slow?
SOLUTION: Based on suggestions from Helen Toomik, I've modified the method and it should now work for several common data types (int32, double, datetime, string). Feel free to extend it. The speed for processing my dataset went from 15 seconds to under 1.
private void RenderDataTableOnXlSheet(DataTable dt, Excel.Worksheet xlWk, string [] columnNames, string [] fieldNames)
{
Excel.Range rngExcel = null;
Excel.Range headerRange = null;
try
{
// render the column names (e.g. headers)
for (int i = 0; i < columnNames.Length; i++)
xlWk.Cells[1, i + 1] = columnNames[i];
// for each column, create an array and set the array
// to the excel range for that column.
for (int i = 0; i < fieldNames.Length; i++)
{
string[,] clnDataString = new string[dt.Rows.Count, 1];
int[,] clnDataInt = new int[dt.Rows.Count, 1];
double[,] clnDataDouble = new double[dt.Rows.Count, 1];
string columnLetter = char.ConvertFromUtf32("A".ToCharArray()[0] + i);
rngExcel = xlWk.get_Range(columnLetter + "2", Missing.Value);
rngExcel = rngExcel.get_Resize(dt.Rows.Count, 1);
string dataTypeName = dt.Columns[fieldNames[i]].DataType.Name;
for (int j = 0; j < dt.Rows.Count; j++)
{
if (fieldNames[i].Length > 0)
{
switch (dataTypeName)
{
case "Int32":
clnDataInt[j, 0] = Convert.ToInt32(dt.Rows[j][fieldNames[i]]);
break;
case "Double":
clnDataDouble[j, 0] = Convert.ToDouble(dt.Rows[j][fieldNames[i]]);
break;
case "DateTime":
if (fieldNames[i].ToLower().Contains("time"))
clnDataString[j, 0] = Convert.ToDateTime(dt.Rows[j][fieldNames[i]]).ToShortTimeString();
else if (fieldNames[i].ToLower().Contains("date"))
clnDataString[j, 0] = Convert.ToDateTime(dt.Rows[j][fieldNames[i]]).ToShortDateString();
else
clnDataString[j, 0] = Convert.ToDateTime(dt.Rows[j][fieldNames[i]]).ToString();
break;
default:
clnDataString[j, 0] = dt.Rows[j][fieldNames[i]].ToString();
break;
}
}
else
clnDataString[j, 0] = string.Empty;
}
// set values in the sheet wholesale.
if (dataTypeName == "Int32")
rngExcel.set_Value(Missing.Value, clnDataInt);
else if (dataTypeName == "Double")
rngExcel.set_Value(Missing.Value, clnDataDouble);
else
rngExcel.set_Value(Missing.Value, clnDataString);
}
// figure out the letter of the last column (supports 1 letter column names)
string lastColumn = char.ConvertFromUtf32("A".ToCharArray()[0] + columnNames.Length - 1);
// make the header range bold
headerRange = xlWk.get_Range("A1", lastColumn + "1");
headerRange.Font.Bold = true;
// autofit for better view
xlWk.Columns.AutoFit();
}
finally
{
ReleaseObject(headerRange);
ReleaseObject(rngExcel);
}
}
private void ReleaseObject(object obj)
{
try
{
System.Runtime.InteropServices.Marshal.ReleaseComObject(obj);
obj = null;
}
catch
{
obj = null;
}
finally
{
GC.Collect();
}
}

Instead of setting cell values one by one, do it in a batch.
Step 1. Transfer the data from your DataTable into an array with the same dimensions.
Step 2. Define an Excel Range object that spans the appropriate range.
Step 3. Set the Range.Value to the array.
This will be a lot faster because you will have a total two calls across the Interop boundary (one to get the Range object, one to set its value), instead of two per cell (get cell, set value).
There is some sample code at MSDN KB article 302096.

Interop is inherently very slow.
There is a large overhead associated with each call.
To speed it up try writing back an object array of data to a range of cells in one assignment statement.
Or if this is a serious problem try using one of the Managed Code Excel extensions that can read/write data using managed code via the XLL interface. (Addin Express, Managed XLL etc.)

If you have a recordset, the fastest way to write to Excel is CopyFromRecordset.

Do you have a specific requirement to go the COM automation route? If not, you have a few other options.
Use the OLEDB provider to create/write to an Excel file http://support.microsoft.com/kb/316934
Use a third party library to write to Excel. Depending on your licensing requirements there are a few options.
Update: A good free library is NPOI http://npoi.codeplex.com/
Write the data to a csv file, and load that into Excel
Write the data as XML which can be loaded into Excel.
Use the Open XML SDK
http://www.microsoft.com/downloads/details.aspx?familyid=C6E744E5-36E9-45F5-8D8C-331DF206E0D0&displaylang=en

Interop has the fastest method called CopyFromRecordset
but ADODB library has to be used
Definitely the fastest way/method and I have tried a few. Perhaps, not easy to use but the speed is astonishing:
https://learn.microsoft.com/en-us/office/vba/api/excel.range.copyfromrecordset
a short sample:
using ADODB;
using Microsoft.Office.Interop;
//--- datatable --- already exists
DataTable dt_data = new DataTable();
//--- or your dt code is here ..........
//--- mine has 3 columns ------
//--- code to populate ADO rs with DataTable data --- nothing special
//--- create empty rs .....
ADODB.Recordset rs = new ADODB.Recordset();
rs.CursorType = CursorTypeEnum.adOpenKeyset;
rs.CursorLocation = CursorLocationEnum.adUseClient;
rs.LockType = LockTypeEnum.adLockOptimistic;
rs.Fields.Append("employee_id",DataTypeEnum.adBSTR,255,FieldAttributeEnum.adFldIsNullable);
rs.Fields.Append("full_name", DataTypeEnum.adBSTR, 255, FieldAttributeEnum.adFldIsNullable);
rs.Fields.Append("start_date", DataTypeEnum.adBSTR, 10, FieldAttributeEnum.adFldIsNullable);
rs.Open();
//--- populate ADO rs with DataTable data ----
for (int i = 0; i < dt_data.Rows.Count; i++)
{
rs.AddNew();
rs.Fields["employee_id"].Value = dt_data.Rows[i]["employee_id"].ToString();
rs.Fields["full_name"].Value = dt_data.Rows[i]["full_name"].ToString();
//--- if date is empty......
if (dt_data.Rows[i]["start_date"].ToString().Length > 0)
{
rs.Fields["start_date"].Value = dt_data.Rows[i]["start_date"].ToString();
}
rs.Update();
}
Microsoft.Office.Interop.Excel.Application xlexcel;
Microsoft.Office.Interop.Excel.Workbook xlWorkBook;
Microsoft.Office.Interop.Excel.Worksheet xlWorkSheet;
object misValue = System.Reflection.Missing.Value;
xlexcel = new Microsoft.Office.Interop.Excel.Application();
xlexcel.Visible = true;
xlWorkBook = xlexcel.Workbooks.Add(misValue);
xlWorkSheet = (Microsoft.Office.Interop.Excel.Worksheet)xlWorkBook.Worksheets.get_Item(1);
//--- populate columns from rs --
for (int i = 0; i < rs.Fields.Count; i++)
{
xlWorkSheet.Cells[1, i + 1] = rs.Fields[i].Name.ToString();
};
//----- .CopyFromRecordset method -- (rs object, MaxRows, MaxColumns) --- in this case 3 columns but it can 1,2,3 etc ------
xlWorkSheet.Cells[2, 1].CopyFromRecordset(CloneFilteredRecordset(rs), rs.RecordCount, 3);

You could create an Excel add-in, with VBA code to do all your db heavy lifting. from .NET, all you'd need to do is instantiate Excel, add the add-in, and call the Excel VBA routine, passing any parameters to it that it needs to execute your SQL statements.

I agree with Charles. Interop is really slow. But try this:
private void RenderDataTableOnXlSheet(DataTable dt, Excel.Worksheet xlWk,
string [] columnNames, string [] fieldNames)
{
// render the column names (e.g. headers)
int columnLength = columnNames.Length;
for (int i = 0; i < columnLength; i++)
xlWk.Cells[1, i + 1] = columnNames[i];
// render the data
int fieldLength = fieldNames.Length;
int rowCount = dt.Rows.Count;
for (int j = 0; j < rowCount; j++)
{
for (int i = 0; i < fieldLength; i++)
{
xlWk.Cells[j + 2, i + 1] = dt.Rows[j][fieldNames[i]].ToString();
}
}
}
HTH