I have a .csv file with around 200 columns and the order of columns changes all the time. I want to read each row from the file, identify the corresponding column names in the database, and write data to the table accordingly.
For this I can use a simple switch case checking for the name of column. Since there are 200 columns, I'm wondering if there is any other way to do it.
Example:
public void ColName(string str, Type a)
{
SampleTableName obj = new SampleTableName();
obj."str" = a;
connection.AddSampleTableName(obj);
connection.savechanges();
}
/* SampleTableName has columns: [Name, Age] */
ColName("Name","XYZ");
Output:
Name Age
XYZ NULL
Any ideas please? Thanks.
If the column names are the same you can use SqlBulkCopy and add a list of Column Mappings. The order doesn't matter, as long as the DataTable name is set.
DataTable table = CreateTable(rows);
using (var bulkCopy = new SqlBulkCopy(connectionString))
{
foreach (var col in table.Columns.OfType<DataColumn>())
{
bulkCopy.ColumnMappings.Add(
new SqlBulkCopyColumnMapping(col.ColumnName, col.ColumnName));
}
bulkCopy.BulkCopyTimeout = 600; // in seconds
bulkCopy.DestinationTableName = "<tableName>";
bulkCopy.WriteToServer(table);
}
If the column names are no the same, a dictionary to lookup the different names could be used.
To keep it simple for maintenance purpose, I went with a switch case sigh. However, I wrote a small script to add all those fields values to the table object.
Related
I have this:
var productDetailsFromFile = (from row in dt.AsEnumerable()
select new ProductDetails
{
ItemNumber = row.Field<string>("Item Number"),
Cost = row.Field<string>("Cost").ToDecimal(),//custom method .ToDecimal
WHQtyList = new List<int>()
{
row.Field<string>("foo").ToInteger(),//custom method .ToInteger
row.Field<string>("bar").ToInteger(),
row.Field<string>("foo2").ToInteger(),
}
}).ToList();
It reads the info from a .csv file. What I am trying to achieve is an elegant way of checking if Fields "foo", "bar" or "foo2".
Right now the issue is that if from the CSV file I remove one of the columns, column not in datatable error pops up. I can't get this to work for 2 hours now.
What I am essentially seeking is - how to check if a column exists as I use it to initialize the list, or if the column doesn't exist the default value to be 0 for each row, where it doesn't exist.
I did it through a method. I was wondering if there was a way to do it faster without having to add additional lines of code or make the additional lines of code less than what they are now.
int ContainsColumn (string columnName, DataTable table, DataRow row)
{
DataColumnCollection columns = table.Columns;
if (columns.Contains(columnName))
{
return int.Parse(row.Field<string>(columnName));
}
else
{
return 0;
}
}
I have a dataset that looks like the image. I'm trying to filter by table and get all the columns next to it and compare them against other datasets
This dataset has tables named table 1 and table 2 and when they're selected they look like the picture below. It shows the columns and I need to compare those columns against the rows from the matching table in the first dataset
I've looked at dataview but that would be a lot of work and I'm very inexperienced. I'm trying to find a way to implement a foreach loop that'll get the name of the table in the first dataset and then compare the rows in it against the columns inside the datatable in the second dataset that matched the table name from the first dataset.
Without knowing more about these DataSets (like do they have primary keys, the data types of the columns, the number of rows in each table, etc), I can only provide limited help. The following example tries to be as general as possible and avoid some basic problems:
DataSet ds1 = <<fetch dataset1>>;
DataSet ds2 = <<fetch dataset2>>;
foreach (DataTable tbl1 in ds1.Tables)
{
if (ds2.Tables.Contains(tbl1.TableName))
{
DataTable tbl2 = ds2.Tables[tbl1.TableName];
List<string> commonColumnNames = new List<string>(tbl1.Columns.Cast<DataColumn>().Select(c => c.ColumnName).Intersect(tbl2.Columns.Cast<DataColumn>().Select(c => c.ColumnName)));
int maxRows = Math.Min(tbl1.Rows.Count, tbl2.Rows.Count);
for (int r = 0; r <= maxRows; r++)
{
foreach (string colName in commonColumnNames)
{
if (tbl1.Rows[r][colName] != tbl2.Rows[r][colName])
{
// Different value
}
}
}
}
}
Update 1: I've added comments to the following example to explain step-by-step what this code is doing. As I try to say before, since I didn't know much about your data, I had to put in extra code. This extra code is for things like: 'Does the table ABC exist in both DataSets?', 'Do the two tables have the same columns in them?', 'Do the tables have the same number of rows in them?'. Your original question did not have this information, so I made this code a little more robust to handle those unknowns.
DataSet ds1 = <<fetch dataset1>>;
DataSet ds2 = <<fetch dataset2>>;
// Loop through all of the tables in the 1st DataSet
foreach (DataTable tbl1 in ds1.Tables)
{
// If the 2nd DataSet has a table with same name as the one from the 1st DataSet
if (ds2.Tables.Contains(tbl1.TableName))
{
DataTable tbl2 = ds2.Tables[tbl1.TableName];
// Create a list of column names that the two tables have in common.
// We will only compare the values in these two tables, in this set of matching column names.
List<string> commonColumnNames = new List<string>(tbl1.Columns.Cast<DataColumn>().Select(c => c.ColumnName).Intersect(tbl2.Columns.Cast<DataColumn>().Select(c => c.ColumnName)));
// Before we start comparing the rows in the two tables, find out which one has the fewer number of rows in it.
int maxRows = Math.Min(tbl1.Rows.Count, tbl2.Rows.Count);
// If the tables have a different number of rows, then we will only compare the set of rows numbered 0-to-MinRowCount
for (int r = 0; r <= maxRows; r++)
{
// For each row, compare the values of common columns
foreach (string colName in commonColumnNames)
{
if (tbl1.Rows[r][colName] != tbl2.Rows[r][colName])
{
// Different value
}
}
}
}
}
I wrote the following code in order to copy a DataTable content into a MS Access table.
The problem is that the data set is very huge, it takes a long time (more than 10mns), and stops when the file reaches 2GB. I know entire set of data is about 785Mo in RAM for about 820000 rows.
public static bool InsertmyDataTableDAO(string filePathName, DataTable myDataTable)
{
string connectionString = string.Format(ConnectionParameters.MsAccessConnectionStringOledb, filePathName);
DBEngine dbEngine = new DBEngine();
Database db = dbEngine.OpenDatabase(filePathName);
db.Execute("DELETE FROM " + myDataTable.TableName);
Recordset rs = db.OpenRecordset(myDataTable.TableName);
Field[] tableFields = new Field[myDataTable.Columns.Count];
foreach(DataColumn column in myDataTable.Columns)
{
tableFields[column.Ordinal] = rs.Fields[column.ColumnName];
}
foreach(DataRow row in myDataTable.Rows)
{
rs.AddNew();
foreach(DataColumn col in row.Table.Columns)
{
tableFields[col.Ordinal].Value = row[col.Ordinal];
}
rs.Update();
}
rs.Close();
db.Close();
return true;
}
Is there a faster way to copy data set from datatable to MS Access DB?
The max db size for access is 2GB, you can't bypass this limit :
https://support.office.com/en-us/article/access-specifications-0cf3c66f-9cf2-4e32-9568-98c1025bb47c?ui=en-US&rs=en-US&ad=US
I see you're using a DELETE statement to remove the rows beforehand. DELETE doesn't necessarily recover free space. Here's what I'd do...
Use your existing code to delete the data in the table.
Next, use Microsoft.Interop.Access to compact/repair the database
Finally, run your above code to insert the DataTable.
I'd also add that you could probably use Microsoft.Interop.Access to import the datatable too... Perhaps save it to a CSV file first... then import it that way rather than using INSERT statements.
I have a DataSet and read data from two sources in it. One table from a XML file and another table from a Firebird SQL database. What I try to get is only one table that has all columns from the XML file AND a few fields from the SQL data in a single table. Both tables have the same unique key field in it so it could be merged easily. I would like to bind all fields of this table to fields on a form.
Is it possible like described or do I not see that there is a simpler solution to my problem?
Edit:
To show what I try to do a bit of extra code.
DataSet dataSet = new DataSet();
DataTable table1 = new DataTable("test1", "test1");
table1.Columns.Add("id");
table1.Columns.Add("name");
table1.Columns[0].Unique = true;
table1.Rows.Add(new object[2] { 1, "name1" });
table1.Rows.Add(new object[2] { 2, "name2" });
DataTable table2 = new DataTable("test2", "test2");
table2.Columns.Add("id");
table2.Columns.Add("thing");
table2.Columns[0].Unique = true;
table2.Rows.Add(new object[2] { 1, "thing1" });
table2.Rows.Add(new object[2] { 2, "thing2" });
dataSet.Tables.Add(table1);
dataSet.Tables[0].Merge(table2, false);
When I run this code I get a ConstraintException. When I remove the unique on the id fields it fills the list with all the needed columns but one row with data from table1 and another one with table2 data. How can I merge them?
Edit 2:
I tried to use the PrimaryKey solution as follows in live data.
xmlData.Tables[0].PrimaryKey = new[] { xmlData.Tables[0].Columns["usnr"] };
dbData.PrimaryKey = new[] { dbData.Columns["usid"] };
xmlData.Tables[0].Merge(dbData, true, MissingSchemaAction.Add);
xmlData is a DataSet which comes from a XML file. It has id, usnr and a few other fields in it. dbData is a DataTable with data from db it has id, usid, name and a few other fields. The id fields are not relevant to my data. Both fields usnr and usid are strings in the table as I tested with GetType().
When I now add xmlData.Tables[0].Merge(dbData, true, MissingSchemaAction.Add); it throws a DataException
<target>.ID and <source>.ID have conflicting properties: DataType property mismatch.
While writing this I realized that the id fields where different in both tables but I dont need them anyways so did remove the column before changing the primaryKey entries and merging. Now I get a NullReferenceException with no further information in it. The tables are all fine and have data in them, where could the Exception come frome now?
Instead of ...
table1.Columns[0].Unique = true;
table2.Columns[0].Unique = true;
... add these lines:
table1.PrimaryKey = new[] { table1.Columns[0] };
table2.PrimaryKey = new[] { table2.Columns[0] };
Because the merge command needs tho know the primary keys of the data tables. Just indicating which columns are unique is not enough. With the primary keys the output will be:
id name thing
== ===== ======
1 name1 thing1
2 name2 thing2
Note that for this to work properly, the primary key fields must have matching data types and names. If the data types don't match, you get a decent error message. However, if the names don't match, a nondescript null reference exception is thrown. Microsoft could have done a better job there.
That means that in your case, I'd recommend to rename either usnr or usid before merging the data tables.
You can use Linq for this purpose and join your two DataTables like this:
.........
.........
dataSet.Tables.Add(table1);
//dataSet.Tables[0].Merge(table2, false);
var collection = from t1 in dataSet.Tables[0].AsEnumerable()
join t2 in table2.AsEnumerable()
on t1["id"] equals t2["id"]
select new
{
ID = t1["id"],
Name = t1["name"],
Thing = t2["thing"]
};
DataTable result = new DataTable("Result");
result.Columns.Add("ID", typeof(string));
result.Columns.Add("Name", typeof(string));
result.Columns.Add("Thing", typeof(string));
foreach (var item in collection)
{
result.Rows.Add(item.ID, item.Name, item.Thing);
}
The result in a DataGridView will be what you want as shown below:
dataGridView1.DataSource = result;
Here you cannot merge those two data tables together. you need to merge data in those two tables iterating each.
Create new data table with containing all the columns(Id,Name,Thing). Then populate that table reading other two.
I am iterating a DataTable in my C# code. I try to get the contents using of a column named "columnName" of row named "row" using -
object value = row["ColumnName"];
I get this error -
Error: System.Reflection.TargetInvocationException: Exception has been
thrown by the target of an invocation.
---> System.ArgumentException: Column 'FULL_COUNT' does not belong to table . at System.Data.DataRow.GetDataColumn(String columnName)
How is this possible ? My SQL query/result set has a column by that name and the query even runs in management studio.
How do I fix this error ?
I am guessing your code is iteration supposed to be something like this
DataTable table = new DataTable();
foreach (DataRow row in table.Rows) {
foreach (DataColumn col in table.Columns) {
object value = row[col.ColumnName];
}
}
If this is the case, row["ColumnName"] in each iteration looks for the same column with name ColumnName which obviously does not exists in your table.
The correct way is row[ColumnName] or row[col.ColumnName] in iteration above
I had a similar issue on my c# code, using a dataset which I had successfully initialized and populated with data from the DB.
So my return set was:
data = new Byte[0];
data = (Byte[])(dataset.Tables[0].Rows[0]["systemLogo_img"]);
Of course the error was in t finding the column 'systemLogo_img'.
I noted that you simply do NOT have to call /qualify the column name. So the correction is:
data = new Byte[0];
data = (Byte[])(dataset.Tables[0].Rows[0].ItemArray[0]);
Simply put: use "ItemArray" at position.
Thanks
Do not write Gridview column names instead of Database column names.
dataGridViewEmployeeClass.Rows[n].Cells[0].Value = item["write the Database Column names"].ToString();
Try this, make sure your column name is same
DataTable dt = new DataTable();
dt.Columns.Add("abc", typeof(string));
DataRow dr = dt.NewRow();
dr["abc"]="";
I had a similar issue which was very basic to fix.
I was querying with a specific column name rather than Select * (i.e. Select Title). Beginner's error but happens to everyone.
If you want to check if the column exists in the DataRow before accessing the value the following block can help...
if (dataRow.Table.Columns.Contains("theColumnName"))
{
// do work
string text = string.Empty;
if (dataRow["theColumnName"] != System.DBNull.Value)
{
text = Convert.ToString(dataRow["theColumnName"]);
}
}
If it doesn't exist and it needs to be added to the data table you can add the column using #Karthick Ganesan's example
// add a column
dataTable.Columns.Add("theColumnName", typeof(string));
I had same issue was trying to pass two different keys for same product.
item.Product = SqlHelper.GetSafeString(dr, "ProductName");
item.Product = SqlHelper.GetSafeString(dr, "Product");