We Keep Coding

Copy large Datatable into MS Access table C#

I wrote the following code in order to copy a DataTable content into a MS Access table.
The problem is that the data set is very huge, it takes a long time (more than 10mns), and stops when the file reaches 2GB. I know entire set of data is about 785Mo in RAM for about 820000 rows.
public static bool InsertmyDataTableDAO(string filePathName, DataTable myDataTable)
{
string connectionString = string.Format(ConnectionParameters.MsAccessConnectionStringOledb, filePathName);
DBEngine dbEngine = new DBEngine();
Database db = dbEngine.OpenDatabase(filePathName);
db.Execute("DELETE FROM " + myDataTable.TableName);
Recordset rs = db.OpenRecordset(myDataTable.TableName);
Field[] tableFields = new Field[myDataTable.Columns.Count];
foreach(DataColumn column in myDataTable.Columns)
{
tableFields[column.Ordinal] = rs.Fields[column.ColumnName];
}
foreach(DataRow row in myDataTable.Rows)
{
rs.AddNew();
foreach(DataColumn col in row.Table.Columns)
{
tableFields[col.Ordinal].Value = row[col.Ordinal];
}
rs.Update();
}
rs.Close();
db.Close();
return true;
}
Is there a faster way to copy data set from datatable to MS Access DB?

The max db size for access is 2GB, you can't bypass this limit :
https://support.office.com/en-us/article/access-specifications-0cf3c66f-9cf2-4e32-9568-98c1025bb47c?ui=en-US&rs=en-US&ad=US

I see you're using a DELETE statement to remove the rows beforehand. DELETE doesn't necessarily recover free space. Here's what I'd do...
Use your existing code to delete the data in the table.
Next, use Microsoft.Interop.Access to compact/repair the database
Finally, run your above code to insert the DataTable.
I'd also add that you could probably use Microsoft.Interop.Access to import the datatable too... Perhaps save it to a CSV file first... then import it that way rather than using INSERT statements.

DataTable.Update Fails and Throws DBConcurrencyException

I'm trying to take advantage of the DataTable.Update method to update a SQL data source. Below is the code for the method that performs the update.
string connString = "SQL connection string...this works.";
string sqlSelect = "SELECT Payment, Amount, Date, Month, StartDate, EndDate, FROM Payment";
private void updateDataSource(DataTable dt) {
SqlDataAdapter da = new SqlDataAdapter(sqlSelect, connString);
SqlCommandBuilder cb = new SqlCommandBuilder(da);
int result = 0; // store the result of dt.Update
// copy over any rows from dt that have a payment date
DataTable temp = dt.Clone();
foreach (DataRow dr in dt.Rows) {
if (dr.ItemArray[5].ToString() != "") // if payment date is not empty
temp.ImportRow(dr);
}
da.ContinueUpdateOnError = true; // this forces an insert but does not update any other records
try {
result = da.Update(temp);
} catch (DBConcurrencyException dbce) {
alertUser(
#"There was an error updating the database.\n" +
dbce.Message + #"\n" +
#"The payment type id for the row was: " + dbce.Row.ItemArray[1] + #"\n" +
#"There were " + temp.Rows.Count + #" rows in the table to be updated.\n");
}
if (result == temp.Rows.Count) {
alertUser("Successful update."); // alert the user
btnSearchCancel_Click(null, null);
}
// store updated data in session variable to store data between posts to server
Session["gv"] = dt;
}
The above method is called when the user clicks an 'Update Table' button.
What is happening is before I included the da.ContinueUpdateOnError = true the try catch would throw the DBConcurrencyException giving Concurrency violation: the UpdateCommand affected 0 of the expected 1 records. And no records would be updated/inserted in the table.
After I added da.ContinueUpdateOnError = true the da.Update() would continue without error but, the first row of DataTable dt would still not be updated, however, the second row of dt would be inserted.
Even more strange is that when I am calling the update passing a table of ~20 rows the update executes perfectly, updating 2, or 3 rows and inserting 2 or three rows. If I call the update passing a table of 2 rows the exception is thrown. The two different tables have the same structure.

This error only occurs when, to quote MSDN
An attempt to execute an INSERT, UPDATE, or DELETE statement resulted
in zero records affected.
To get this error means that the database has changed since the DataTable was created.
The error tells you that
the UpdateCommand affected 0 of the expected 1 records
One of the records that was trying to be updated was not there anymore or had changed and no longer matches the expected signature.
For reference: DBConcurrencyException and DbDataAdapter.Update Method, a little more explanation.
It seems that there may be some other code that is changing the database after the DataTable is created, or you are running on a production DB and other users are making changes.

SSIS - Object variable converted to DataTable becomes empty

I'm trying to create an object variable that will hold a collection from an Execute SQL Task. This collection will be used in multiple Script Task throughout the ETL package.
The problem is, after the first Fill of the first Script Task, the object variable becomes empty. Here's a code on how I used the variable to a DataTable:
try
{
DataTable dt = new DataTable();
OleDbDataAdapter da = new OleDbDataAdapter();
da.Fill(dt, Dts.Variables["reportMetrics"].Value);
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception Ex)
{
MessageBox.Show(Ex.Message);
Dts.TaskResult = (int)ScriptResults.Failure;
}
Throughout the ETL package, Script Task components will have this piece of code. Since the variable becomes empty after the first Fill, I can't reuse the object variable.
I'm guessing that the Fill method has something to do with this.
Thanks!

It looks like your Dts.Variables["reportMetrics"].Value object holds DataReader object. This object allows forward-only read-only access to the data. You cannot fill DataTable twice using DataReader. To accomplish your task you need to create another script task that performs exactly what you described here: it reads the Reader to DataTable object and stores this DataTable object in another Dts.Variable with type Object.
Dts.Variables["reportMetricsTable"].Value = dt
After that all your subsequequent script tasks shall either create a copy of this table if they modify the data, or use it directly if they do not modify it.
DataTable dtCopy = (Dts.Variables["reportMetricsTable"].Value as DataTable).Copy()

I had a similar situation. While I think you can do a SQL Task with a SELECT COUNT(*) query and assign the result to an SSIS variable, what I did was create an int SSIS variable called totalCount with an original value of 0. I expect the total count to be > 0 (otherwise, I won't have anything to iterate on) so I created an if statement within my Script Task. If the value is zero, I assume totalCount has not been initialized, so I use the same code you are using (with the Fill method). Otherwise (i.e, in further iterations), I skip that part and continue to use totalCount variable. Here's the block of code. Hope it helps:
if ((int)Dts.Variables["User::totalCount"].Value == 0) // if the total count variable has not been initialized...
{
System.Data.OleDb.OleDbDataAdapter da = new System.Data.OleDb.OleDbDataAdapter();
DataTable stagingTablesQryResult = new DataTable();
da.Fill(stagingTablesQryResult, Dts.Variables["User::stagingTablesQryResultSet"].Value); // to be used for logging how many files are we iterating. It may be more efficient to do a count(*) outside this script and save the total number of rows for the query but I made this as proof of concept for future developments.
Dts.Variables["User::totalCount"].Value = stagingTablesQryResult.Rows.Count;
}
Console.WriteLine("{0}. Looking for data file {0} of {1} using search string '{2}'.", counter, Dts.Variables["User::totalCount"].Value, fileNameSearchString);

Excellent
This has helped me around an issue in building myt ETL platform.
Essentially I execute a SQL task to build a dataset of tasks, there is some in line transformations and rules which pull the relevant tasks to the fore, which for obvious reasons I only want to execute the once per execution.
I then need to get the unique ProcessIDs from the data set (to use in a For Each Loop)
Within the FEL, I want to then fetch the relevant records from the original dataset to then push through a further FEL process.
I was facing the same "empty data set" for the 2nd execution against the dataset.
I thought I'd try to share my solution to assist others
You'll need to add the Namespaces
using System.Data.OleDb;
into the Scripts
Screen shot of solution
Get dataset
Execute SQL - Get your data and pass into a Variable Object
Pull Ds
Declare the Variable Objects
public void Main()
{
DataTable dt = new DataTable();
OleDbDataAdapter da = new OleDbDataAdapter();
//Read the original table
da.Fill(dt, Dts.Variables["Tbl"].Value);
//Push to a replica
Dts.Variables["TblClone"].Value = dt;
Dts.TaskResult = (int)ScriptResults.Success;
}
Build Proc List
This gets a list of ProcessIDs (and Names) by filtering on a Rank field in the dataset
Declare the Variable Objects
public void Main()
{ //Take a copy of the Cloned Dataset
DataTable dtRead = (Dts.Variables["TblClone"].Value as DataTable).Copy();
//Lock the output object variable
Dts.VariableDispenser.LockForWrite("User::ProcTbl");
//Create a data table to place the results into which we can write to the output object once finished
DataTable dtWrite = new DataTable();
//Create elements to the Datatable programtically
//dtWrite.Clear();
dtWrite.Columns.Add("ID", typeof(Int64));
dtWrite.Columns.Add("Nm");
//Start reading input rows
foreach (DataRow dr in dtRead.Rows)
{
//If 1st col from Read object = ID var
if (Int64.Parse(dr[9].ToString()) == 1) //P_Rnk = 1
{
DataRow newDR = dtWrite.NewRow();
newDR[0] = Int64.Parse(dr[0].ToString());
newDR[1] = dr[4].ToString();
//Write the row
dtWrite.Rows.Add(newDR);
}
}
//Write the dataset back to the object variable
Dts.Variables["User::ProcTbl"].Value = dtWrite;
Dts.Variables.Unlock();
Dts.TaskResult = (int)ScriptResults.Success;
}
Build TaskList from ProcList
Cycle round ProcessID in a For Each Loop
Build TL Collection
..and map Vars
Build TL Var Mappings
Build TL Script
This will dynamically build the output for you (NB this works for me although havent extensively tested it, so if it doesnt work....have a fiddle with it).
You'll see I've commented out some Debug stuff
public void Main()
{
//Clone the copied table
DataTable dtRead = (Dts.Variables["TblClone"].Value as DataTable).Copy();
//Read the var to filter the records by
var ID = Int64.Parse(Dts.Variables["User::ProcID"].Value.ToString());
//Lock the output object variable
Dts.VariableDispenser.LockForWrite("User::SubTbl");
//Debug Test the ProcID being passed
//MessageBox.Show(#"Start ProcID = " + ID.ToString());
//MessageBox.Show(#"TblCols = " + dtRead.Columns.Count);
//Create a data table to place the results into which we can write to the output object once finished
DataTable dtWrite = new DataTable();
//Create elements to the Datatable programtically
//dtWrite.Clear();
foreach (DataColumn dc in dtRead.Columns)
{
dtWrite.Columns.Add(dc.ColumnName, dc.DataType);
}
MessageBox.Show(#"TblRows = " + dtRead.Rows.Count);
//Start reading input rows
foreach (DataRow dr in dtRead.Rows)
{
//If 1st col from Read object = ID var
if (ID == Int64.Parse(dr[0].ToString()))
{
DataRow newDR = dtWrite.NewRow();
//Dynamically create data for each column
foreach (DataColumn dc in dtRead.Columns)
{
newDR[dc.ColumnName] = dr[dc.ColumnName];
}
//Write the row
dtWrite.Rows.Add(newDR);
//Debug
//MessageBox.Show(#"ProcID = " + newDR[0].ToString() + #"TaskID = " + newDR[1].ToString() + #"Name = " + newDR[4].ToString());
}
}
//Write the dataset back to the object variable
Dts.Variables["User::SubTbl"].Value = dtWrite;
Dts.Variables.Unlock();
Dts.TaskResult = (int)ScriptResults.Success;
}
For Each Loop Container
FEL Cont Collection
N.B. Dont forget to map the items in the Variable Mappings
Now you can consume the records and do stuff with that data
I included the Msg Loop script as an easy data check...in reality this will go off and trigger other processes but just to aid you in data checks I though Id include it
Msg Loop
Msg Loop Script
public void Main()
{
// TODO: Add your code here
MessageBox.Show("ID = " + Dts.Variables["User::ProcID"].Value + ", and val = " + Dts.Variables["User::TaskID"].Value, "Name = Result");
Dts.TaskResult = (int)ScriptResults.Success;
}
Hope that helps somebody solve their issue (Ive been tring to resolve this for a working day or so :/

DataGridview duplication of data or not saving new rows to database

I have DataGridView in windows Form using C#.
The data is showing successfully and update also being done successfully.
But the problem is with insertion. When I try to insert multiple data into database using Iteration then the it stores 13 times into database because the datagrid initially show 13 records from database. I meant that it stores new rows multiple times(datagridview rows count) into database.
Lets suppose I want to save two rows into databse but it stores 1st row into database 13 times(total number of datagridview rows).
Please check where I am doing mistake
Note: I want to use single datagridview for displaying, update, insert and delete data from database. I have button through which I want to start insertion all new rows to database.
My code is below
string connection = System.Configuration.ConfigurationManager.ConnectionStrings["AuzineConnection"].ConnectionString;
using (SqlConnection sqlconn = new SqlConnection(connection))
{
sqlconn.Open();
foreach (DataGridViewRow row in dataGridView1.Rows)
{
if (row.IsNewRow)
{
using (SqlCommand sqlcomm = sqlconn.CreateCommand())
{
//sqlcomm.CommandText = "Insert into QF (profileID, UserNameLogInName, UserFullName,Email, forumtitle, subtitle, subjecttitle ,noreply, noview , qtags ,Question ,Questiondetails ,questionstatus, qdate, Status ,todate) values(#status)";
sqlcomm.CommandText = "Insert into QF (UserNameLogInName,Status ) values(#UserNameLogInName,#status)";
try
{
sqlcomm.Parameters.AddWithValue("UserNameLogInName", dataGridView1.CurrentRow.Cells["UserNameLogInName"].Value.ToString());
sqlcomm.Parameters.AddWithValue("Status", dataGridView1.CurrentRow.Cells["Status"].Value.ToString());
sqlcomm.ExecuteNonQuery();
//MessageBox.Show(dataGridView1.Rows[Convert.ToInt32(RecordIndexNumber.Text)].Cells[Convert.ToInt32(ColInexNum.Text)].Value.ToString());
}
catch (Exception ex)
{
btnInsert.Text = ex.Message.ToString();
}
}
}
}
}
I have tried it if (!row.IsNewRow) and it stores only one row. Without it stores multiple unwanted times into database.
I also used for (int i = 0; i < dataGridView1.Rows.Count - 1; i++), but the same issue either it stores one or multiple times.
I just wanted to insert the new rows generated by the user on button click.

Try this:
sqlcomm.Parameters.AddWithValue("UserNameLogInName", row.Cells["UserNameLogInName"].Value.ToString());
sqlcomm.Parameters.AddWithValue("Status", row.Cells["Status"].Value.ToString());

Add row while assigning datasource for datagridview

I have a datagridview assigned a datasource to it. now how to add a new row to that grid and remove a row from it?

One way to do this is as follows:
Step #1 Setup the Data Adapter, Data Grid etc:
// the data grid
DataGridView dataGrid;
// create a new data table
DataTable table = new DataTable();
// create the data adapter
SqlDataAdapter dataAdapter = new SqlDataAdapter(strSQL, strDSN);
// populate the table using the SQL adapter
dataAdapter.Fill(table);
// bind the table to a data source
BindingSource dbSource = new BindingSource();
dbSource.DataSource = table;
// finally bind the data source to the grid
dataGrid.DataSource = dbSource;
Step #2 Setup the Data Adapter SQL Commands:
These SQL commands define how to move the data between the grid and the database via the adapter.
dataAdapter.DeleteCommand = new SqlCommand(...);
dataAdapter.InsertCommand = new SqlCommand(...);
dataAdapter.UpdateCommand = new SqlCommand(...);
Step #3 Code to Remove Select lines from the Data Grid:
public int DeleteSelectedItems()
{
int itemsDeleted = 0;
int count = dataGrid.RowCount;
for (int i = count - 1; i >=0; --i)
{
DataGridViewRow row = dataGrid.Rows[i];
if (row.Selected == true)
{
dataGrid.Rows.Remove(row);
// count the item deleted
++itemsDeleted;
}
}
// commit the deletes made
if (itemsDeleted > 0) Commit();
}
Step #4 Handling Row Inserts and Row Changes:
These types of changes are relatively easy to implement as you can let the grid manage the cell changes and new row inserts.
The only thing you will have to decide is when do you commit these changes.
I would recomment putting the commit in the RowValidated event handler of the DataGridView as at that point you should have a full row of data.
Step #5 Commit Method to Save the Changes back to the Database:
This function will handle all the pending updates, insert and deletes and move these changes from the grid back into the database.
public void Commit()
{
SqlConnection cn = new SqlConnection();
cn.ConnectionString = "Do the connection using a DSN";
// open the connection
cn.Open();
// commit any data changes
dataAdapter.DeleteCommand.Connection = cn;
dataAdapter.InsertCommand.Connection = cn;
dataAdapter.UpdateCommand.Connection = cn;
dataAdapter.Update(table);
dataAdapter.DeleteCommand.Connection = null;
dataAdapter.InsertCommand.Connection = null;
dataAdapter.UpdateCommand.Connection = null;
// clean up
cn.Close();
}

I believe you'll have to get the Table collection item and retrieve the Row collection item from that. Then you can loop through the rows or however you want to remove the row.
You do this after binding it, of course.

Property "Rows" in GridView has not a Delete method, than you can't delete a row directly. You must delete item from your datasource, than remake DataBind. You can also set Visibile = false to that row, so it will appear "deleted" to the user.