We Keep Coding

What is the fastest way to populate a C# DataTable with data stored on columns?

I have a DataTable object that I need to fill based on data stored in a stream of columns - i.e. the stream initially contains the schema of the DataTable, and subsequently, values that should go into it organised by column.
At present, I'm taking the rather naive approach of
Create enough empty rows to hold all data values.
Fill those rows per cell.
The result is a per-cell iteration, which is not especially quick to say the least.
That is:
// Create rows first...
// Then populate...
foreach (var col in table.Columns.Cast<DataColumn>)
{
List<object> values = GetValuesfromStream(theStream);
// Actual method has some DBNull checking here, but should
// be immaterial to any solution.
for (var i=0; i<values.Count; i++)
table.Rows[i][col] = values[i];
}
My guess is the backing DataStorage items for each column aren't expanding as the rows are added, but as values are added to each column, but I'm far from certain. Any tips for loading this kind of data.
NB that loading all lists first and then reading in by row is probably not sensible - this approach is being taken in the first place to mitigate potential out of memory exceptions that tend to result when serializing huge DataTable objects, so grabbing a clone of the entire data grid and reading it in would probably just move the problem elsewhere. There's definitely enough memory for the original table and another column of values, but there probably isn't for two copies of the DataTable.

Whilst I haven't found a way to avoid iterating cells, as per the comments above, I've found that writing to DataRow items that have already been added to the table turns out to be a bad idea, and was responsible for the vast majority of the slowdown I observed.
The final approach I used ended up looking something like this:
List<DataRow> rows = null;
// Start population...
var cols = table.Columns.Cast<DataColumn>.Where(c => string.IsNullOrEmpty(c.Expression));
foreach (var col in cols)
{
List<object> values = GetValuesfromStream(theStream);
// Create rows first if required.
if (rows == null)
{
rows = new List<DataRow>();
for (var i=0; i<values.Count; i++)
rows.Add(table.NewRow());
}
// Actual method has some DBNull checking here, but should
// be immaterial to any solution.
for (var i=0; i<values.Count; i++)
rows[i][col] = values[i];
}
rows.ForEach(r => table.Rows.Add(r));
This approach addresses two problems:
If you try to add an empty DataRow to a table that has null-restrictions or similar, then you'll get an error. This approach ensures all the data is there before it's added, which should address most such issues (although I haven't had need to check how it works with auto-incrementing PK columns).
Where expressions are involved, these are evaluated when row state changes for a row that has been added to a table. Consequently, where before I had re-calculation of all expressions taking place every time a value was added to a cell (expensive and pointless), now all calculation takes place just once after all base data has been added.
There may of course be other complications with writing to a table that I've not yet encountered because the tables I am making use of don't use those features of the DataTable class/model. But for simple cases, this works well.

Managed Esent - quicker way to read data?

Using the Managed Esent interface to read data from a table. I am doing this with (pseudo):
List<ColumnInfo> columns; //Three columns to be read
using (var table = Table(session,DBID,"tablename",OpenTableGrbit.Readonly))
{
while (Api.TryMoveNext(session, table))
{
foreach (ColumnInfo col in columns)
{
string data = GetFormattedColumnData(session,table,col);
}
}
}
I am interested in data from three columns only, which is around 4,000 rows. However, the table itself is 1,800,000 rows. Hence this approach is very slow to just read the data I want as I need to read all 1,800,000 rows. Is there a quicker way?

There are many things you can do. Here are a few things off the top of my head:
Set the minimum cache size SystemParameters.CacheSizeMin. The default cache sizing algorithm is a bit conservative sometimes.
Also set OpenTableGrbit.Squential when opening your table. This helps a little bit with prefetching.
Use Api.RetrieveColumns to retrieve the three values at once. This reduces the number of calls/pinvokes you'll do.
-martin

Simplifying complexity in for a table object structure

I have an object structure that is mimicking the properties of an excel table. So i have a table object containing properties such as title, header row object and body row objects. Within the header row and each body row object, i have a cell object containing info on each cell per row. I am looking for a more efficient way to store this table structure since in one of my uses for this object, i am printing its structure to screen. Currently, i am doing an O(n^2) complexity for printing each row for each cell:
foreach(var row in Table.Rows){
foreach(var cell in row.Cells){
Console.WriteLine(cell.ToString())
}
}
Is there a more efficient way of storing this structure to avoid the n^2? I ask this because this printing functionality exists in another n^2 loop. Basically i have a list of tables titles and a list of tables. I need to find those tables whose titles are in the title list. Then for each of those tables, i need to print their rows and the cells in each row. Can any part of this operation be optimized by using a different data structure for storage perhaps? Im not sure how exactly they work but i have heard of hashing and dictionary?
Thanks

Since you are looking for tables with specific titles, you could use a dictionary to store the tables by title
Dictionary<string,Table> tablesByTitle = new Dictionary<string,Table>();
tablesByTitle.Add(table.Title, table);
...
table = tablesByTitle["SomeTableTitle"];
This would make finding a table an O(1) operation. Finding n tables would be an O(n) operation.
Printing the tables then of cause depends on the number of rows and columns. There is nothing, which can change that.
UPDATE:
string tablesFromGuiElement = "Employees;Companies;Addresses";
string[] selectedTables = tablesFromGuiElement.Split(';');
foreach (string title in selectedTables) {
Table tbl = tablesByTitle[title];
PrintTable(tbl);
}

There isn't anything more efficient than an N^2 operation for outputting an NxN matrix of values. Worst-case, you will always be doing this.
Now, if instead of storing the values in a multidimensional collection that defines the graphical relationship of rows and columns, you put them in a one-dimensional collection and included the row-column information with each cell, then you would only need to iterate through the cells that had values. Worst-case is still N^2 for a table of N rows and N columns that is fully populated (the one-dimensional array, though linear to enumerate, will have N^2 items), but the best case would be that only one cell in that table is populated (or none are) which would be constant-time.

This answer applies to the, printing the table part, but the question was extended.
for the getting the table part, see the other answer.
No, there is not.
Unless perhaps your values follow some predictable distribution, then you could use a function of x and y and store no data at all, or maybe a seed and a function.
You could cache the print output in a string or StringBuider if you require it multiple times.
If there is enough data I guess you might apply some compression algorithm but I wouldn't say that was simpler or more efficient.

Is there a more efficient way to create lists that are the same?

I have a series of lists and classes that implement a table of data. The basic classes are: Columns, Rows, and Cells. The Rows contains some ID information and list of Cells which contains the row's value for each column. Currently I create the rows in a cell with code like this
void CreateRow()
{
Row newRow = new Row();
newRow.ID = idInfo;
foreach (var Column in Columns)
{
newRow.Cells.Add(new Cell(Column.ID));
}
Rows.Add(newRow);
}
The works fine, but in some cases am calling CreateRow() 20,000 times and have 200+ columns. So I am wondering if there is a more efficient way to populate the cells since the cells in a certain column in each row are identical.
Any ideas?
Thanks,
Jerry

Currently you create unique Cell object for each position in your matrix - that's a lot of cells given your use case of 20.000 + rows.
One approach to be more efficient could be to not add the cells at all when you construct the matrix, but only when you try to get or set its value (i.e using Lazy<T>).
Assuming you set the value of a cell before retrieving it, you could then have a factory method for creating a cell with a value - make the Cell object immutable and when you are "creating" a Cell for which you already have another cell with an identical value, return that cell instead. This could reduce the total number of Cell objects significantly, of course there's more overhead since you need to check whether you have a cell of the same value already and need to call the factory method again if you need to update the value of a cell.
Then again all of this could not be worth it if you do not experience any memory/performance problems with your current approach - measuring performance is key here.

Isn't Columns a collection?
var Ids = Columns.Select(c => c.Id).ToArray();
var Names = Columns.Select(c => c.Name).ToArray();
etc. Except why do that if Columns is already a collection? For you could do Columns[index].Id
Or if you must have the code you outlined:
Row newRow = new Row();
newRow.ID = idInfo;
// presuming Cells is a List<>
newRow.Cells.AddRange(Columns.Select(c => new Cell(c.Id)));
Rows.Add(newRow);

Some suggestions (depends on what you are looking for)
Consider using (strongly typed) DataSet/DataTable
If using List and you know the size, set the capacity to avoid reallocation (new List(2000))
Use struct instead of class if it makes sense
Cache objects if it makes sense (instead of duplicating the same object over and over)

You're creating the cells anyways. So I gather that the question refers to when you will fill the cells with their values, which are always in each column for all rows.
I actually think that from a correctness point of view, it makes sense to have the data duplicated, since they are in effect separate instances of the same data.
That said, if it is not really data, but you just want to show a view-column with the same value for each row, and you just want it as a data column in order to ease showing it as a view-column, then in your property-get Row.Cells(Id) you can check the ID, and if it's one of those columns where the value is always the same, return that value, bypassing looking up your _Cells collection.
If the data is mostly the same and sometimes different, you may want to use 'default values' where if the Cell object does not exist, a default value for that column will be returned. This necessitates a GetValue() method on the row, though, if you want to avoid having the Cell object altogether for places where it is default.
If you don't care about #1, you can really make a single instance of whatever the value is, and reference it in your Cell's value. This is harder to do for a Value Type than for a Reference Type (definition here) but it can be done.
Lastly, is there any reason you're not using .NET's supplied DataTable and DataRow types? I'm sure the MS geeks programmed as much efficiency as they could into those.

Why does Columns.Insert not update the DisplayIndex in DataGridView (C#)?

I am inserting a column in a DataGridView programmatically (i.e., not bound to any data tables/databases) as follows:
int lastIndex = m_DGV.Columns.Count - 1; // Count = 4 in this case
DataGridViewTextBoxColumn col = (DataGridViewTextBoxColumn)m_DGV.Columns[lastIndex];
m_DGV.Columns.RemoveAt(lastIndex);
m_DGV.Columns.Insert(insertIndex, col); // insertIndex = 2
I have found that my columns are visually out of order sometimes using this method. A workaround is to manually set the DisplayIndex property of the column afterwards. Adding this code "fixes it", but I don't understand why it behaves this way.
Console.Write(m_DGV.Columns[0].DisplayIndex); // Has value of 0
Console.Write(m_DGV.Columns[1].DisplayIndex); // Has value of 1
Console.Write(m_DGV.Columns[2].DisplayIndex); // Has value of 3
Console.Write(m_DGV.Columns[3].DisplayIndex); // Has value of 2
col.DisplayIndex = insertIndex;
Console.Write(m_DGV.Columns[0].DisplayIndex); // Has value of 0
Console.Write(m_DGV.Columns[1].DisplayIndex); // Has value of 1
Console.Write(m_DGV.Columns[2].DisplayIndex); // Has value of 2
Console.Write(m_DGV.Columns[3].DisplayIndex); // Has value of 3
As an aside, my grid can grow its column count dynamically. I wanted to grow it in chunks, so each insert didn't require a column allocation (and associated initialization). Each "new" column would then be added by grabbing an unused column from the end, inserting it into the desired position, and making it visible.

I suspect this is because the order of the columns in the DataGridView do not necessarily dictate the display order, though without explicitly being assigned by default the order of the columns dictate the DisplayIndex property values. That is why there is a DisplayIndex property, so you may add columns to the collection without performing Inserts - you just need to specify the DisplayIndex value and a cascade update occurs for everything with an equal or greater DisplayIndex. It appears from your example the inserted column is also receiving the first skipped DisplayIndex value.
From a question/answer I found:
Changing the DisplayIndex will cause
all the columns between the old
DisplayIndex and the new DisplayIndex
to be shifted.
As with nearly all collections (other than LinkedLists) its always better to add to a collection than insert into a collection. The behavior you are seeing is a reflection of that rule.

I have a couple of ideas.
How about addressing your columns by a unique name, rather than the index in the collection? They might not already have a name, but you could keep track of who's who if you gave them a name that meant something.
You can use the GetFirstColumn, GetNextColumn, GetPreviousColumn, GetLastColumn methods of the DataGridViewColumnCollection class, which work on display order, not the order in the collection. You can also just iterate through the collection using a for loop and m_DGV.Columns[i] until you find the one you want.
Create an inherited DataGridView and DataGridViewColumnCollection. The DataGridView simply is overridden to use your new collection class. Your new DataGridViewColumnCollection will include a method to address the collection by display index, presumably by iterating through the collection until you find the one you want (see #2). Or you can save a dictionary and keep it updated for very large numbers of columns.
I doubt the performance increase of keeping a dictionary, since every time a column moves, you essentially have to rewrite the entire thing. Iterating through is O(n) anyway, and unless you're talking asynchronous operations with hundreds of columns, you're probably okay.
You might be able to override the this[] operator as well, assuming it doesn't screw up the DataGridView.
Idea #1 might be the easiest to implement, but not necessarily the prettiest. Idea #2 works, and you can put it in a function DataGridViewColumn GetColumnByDisplayIndex(int Index). Idea #3 is cute, and certainly the most encapsulated approach, but isn't exactly trivial.

Thanks to cfeduke for excellent advice. I suspected Insert would be slower, but the provided link enlightened me on JUST HOW MUCH slower.
This brings up the question of how to efficiently insert and remove columns dynamically on a DataGridView. It looks like the ideal design would be to add plenty of columns using Add or AddRange, and then never really remove them. You could then simulate removal by setting the Visible property to false. And you could insert a column by grabbing an invisible column, setting its DisplayIndex and making it visible.
However, I suspect there would be landmines to avoid with this approach. Foremost being that you can no longer index your data in a straightforward manner. That is, m_DGV.Columns[i] and m_DGV.Rows[n].Cells[i] will not be mapped properly. I suppose you could create a Map/Dictionary to maintain an external intuitive mapping.
Since my application (as currently designed) requires frequent column insertion and removal it might be worth it. Anyone have any suggestions?