We Keep Coding

Copying data from one database to another - Out of Memory Exception

I have a task where i want to copy all data from one database to another database & skipping 2 tables. There are more than 200 tables.
I have table structure ready for my 2nd databas.
So as a solution i created a page & on a button click i have below code :-
DataSet ds = new DataSet();
string connectionString = "Data Source=COMP112\\MSSQLSERVER2014;Initial Catalog=HCMBL;Integrated Security=True;Persist Security Info=True";
SqlConnection con = new SqlConnection(connectionString);
//render table name from database
string sqlTable = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' and TABLE_Schema='" + Session["SchemaName"].ToString() + "' and TABLE_NAME!='ENTRY' and TABLE_NAME!='OT' and TABLE_NAME!='BL_ENTRY' and TABLE_NAME!='BL_OT'";
con.Open();
SqlDataAdapter da = new SqlDataAdapter();
SqlCommand cmd = new SqlCommand(sqlTable, con);
cmd.CommandType = CommandType.Text;
da.SelectCommand = cmd;
da.Fill(ds);
con.Close();
//render connection string from WebConfig file
string strcon = ConfigurationManager.ConnectionStrings["SPSchema"].ConnectionString;
for (int i = 0; i < ds.Tables[0].Rows.Count; i++)
{
if (!(ds.Tables[0].Rows[i]["TABLE_NAME"].ToString().Contains("Asp")))
{
string deleteQuery = "Truncate table " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
con.Open();
SqlCommand cmdDelete = new SqlCommand(deleteQuery, con);
cmdDelete.ExecuteNonQuery();
con.Close();
DataSet dataSet = new DataSet();
SqlConnection conn = new SqlConnection(strcon);
conn.Open();
string selectData = "select * from " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
SqlCommand command = new SqlCommand(selectData, conn);
DataTable dataTable = new DataTable();
SqlDataAdapter dataAdapter = new SqlDataAdapter(selectData, conn);
dataAdapter.FillSchema(dataSet, SchemaType.Mapped);
dataAdapter.Fill(dataSet);
dataTable = dataSet.Tables[0];
conn.Close();
if (dataSet.Tables[0].Rows.Count > 0)
{
//Connect to second Database and Insert row/rows.
SqlConnection conn2 = new SqlConnection(connectionString);
conn2.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(conn2);
bulkCopy.DestinationTableName = Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"].ToString();
bulkCopy.WriteToServer(dataTable);
conn2.Close();
}
}
}
As i run the above code after inserting data in less than 10 tables, it gives out of memory exception & program crashes.
How to handle this? I tried increasing the memory capacity of SQL Server but still same error.
Is there any other way to achieve the task?

What you are doing is very far from the best solution. You are using an ASP.NET MVC process to get all data of your entire database into memory, and then outputting it to another database. If your database is anything more than small and trivial, that will most definitely fill your process's alotted memory.
This type of task should never be done through the memory of a process, but rather using some form of Backup/Restore pattern.
You should look into SSIS projects and create an extract, transfer, and load (ETL) solution, which can be triggered from your ASP.NET MVC solution asynchronously.
An SSIS solution can be triggered from C# code in this way:
var app = new Application();
var package = app.LoadPackage("compiled-package.dtsx", null);
var results = package.Execute();
See this question for a little more information (not specifically about duplicating databases, but has information about triggering SSIS packages from code): How to execute an SSIS package from .NET?
Alternatively
You also have the option of running a query against both databases at once, however this requires some additional plumbing to be done. The user account of your ASP.NET MVC solution needs to have access to both databases. If your databases are hosted on different servers, you also need to link one server to the other: Create linked servers
To perform an insert directly from the output of a select, consider this:
string source = "NAME_OF_SOURCE_DATABASE";
string target = "NAME_OF_TARGET_DATABASE";
string schema = Session["SchemaName"].ToString();
string table = ds.Tables[0].Rows[i]["TABLE_NAME"];
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery = $"SET IDENTITY_INSERT {target}.{schema}.{table} ON";
var idInsCommand = new SqlCommand(idInsQuery, conn);
idInsCommand.ExecuteNonQuery();*/
string insQuery = $"INSERT INTO {target}.{schema}.{table} SELECT * FROM {source}.{schema}.{table}";
var insCommand = new SqlCommand(insQuery, conn);
insCommand.ExecuteNonQuery();
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery2 = $"SET IDENTITY_INSERT {target}.{schema}.{table} OFF";
var idInsCommand2 = new SqlCommand(idInsQuery2, conn);
idInsCommand2.ExecuteNonQuery();*/
This will only work if the table structures are identical. There might be problems with autoincrement ids or columns with default values, too.

This will copy data from a table in database 1 to a table in database 2
Insert into db2.dbo.table2 (col1,col2)
Select col1,col2 from db1.dbo.table1
Run this sql statement and the data will be copied without a round trip to your app.

Let me know if you find my approach is useful.
First of all, why you want to write down one whole application to do this job while SQL Server have inherited property to do it.
My approach would be configure an Linked Server and configure it which tables you want to copy and which one not.
https://learn.microsoft.com/en-us/sql/relational-databases/linked-servers/create-linked-servers-sql-server-database-engine
Secondly, You can just write down simple stored procedure and schedule that in your sql server to push into another server database as per your schedule. In this way you can control it in N number of ways. I mean about controlling any dependencies(Table level or Business level).
To do this in t-sql, you can use the following system stored procedures to schedule a daily job. This example schedules daily at 1:00 AM. See Microsoft help for details on syntax of the individual stored procedures and valid range of parameters.
DECLARE #job_name NVARCHAR(128), #description NVARCHAR(512), #owner_login_name NVARCHAR(128), #database_name NVARCHAR(128);
SET #job_name = N'Some Title';
SET #description = N'Periodically do something';
SET #owner_login_name = N'login';
SET #database_name = N'Database_Name';
-- Delete job if it already exists:
IF EXISTS(SELECT job_id FROM msdb.dbo.sysjobs WHERE (name = #job_name))
BEGIN
EXEC msdb.dbo.sp_delete_job
#job_name = #job_name;
END
-- Create the job:
EXEC msdb.dbo.sp_add_job
#job_name=#job_name,
#enabled=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name;
-- Add server:
EXEC msdb.dbo.sp_add_jobserver #job_name=#job_name;
-- Add step to execute SQL:
EXEC msdb.dbo.sp_add_jobstep
#job_name=#job_name,
#step_name=N'Execute SQL',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=1,
#on_fail_action=2,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0,
#subsystem=N'TSQL',
#command=N'EXEC my_stored_procedure; -- OR ANY SQL STATEMENT',
#database_name=#database_name,
#flags=0;
-- Update job to set start step:
EXEC msdb.dbo.sp_update_job
#job_name=#job_name,
#enabled=1,
#start_step_id=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name,
#notify_email_operator_name=N'',
#notify_netsend_operator_name=N'',
#notify_page_operator_name=N'';
-- Schedule job:
EXEC msdb.dbo.sp_add_jobschedule
#job_name=#job_name,
#name=N'Daily',
#enabled=1,
#freq_type=4,
#freq_interval=1,
#freq_subday_type=1,
#freq_subday_interval=0,
#freq_relative_interval=0,
#freq_recurrence_factor=1,
#active_start_date=20170101, --YYYYMMDD
#active_end_date=99991231, --YYYYMMDD (this represents no end date)
#active_start_time=010000, --HHMMSS
#active_end_time=235959; --HHMMSS
Let me know in case you need more details on this.
Thanks,
Ayan

Update using MySqlDataAdapter doesn't work

I am trying to use MySqlDatAdapter to update a MySql table. But, the table never updates!!! I did this before but with SQL server. Is there anything else that is specific to MySql that I am missing in my code?
DataTable myTable = new DataTable("testtable");
MySqlConnection mySqlCon = new MySqlConnection(ConfigurationManager.ConnectionStrings["DBConStr"].ConnectionString);
MySqlCommand mySqlCmd = new MySqlCommand("SELECT * FROM testtable WHERE Name = 'Tom'");
mySqlCmd.Connection = mySqlCon;
MySqlDataAdapter adapter = new MySqlDataAdapter(mySqlCmd);
MySqlCommandBuilder myCB = new MySqlCommandBuilder(adapter);
adapter.UpdateCommand = myCB.GetUpdateCommand();
mySqlCon.Open();
adapter.Fill(myTable);
myTable.Rows[0]["Name"] = "Was Tom";
myTable.AcceptChanges();
adapter.Update(myTable);
mySqlCon.Close();
Thanks

Remove myTable.AcceptChanges() before the update. Othwerwise that will set all rows RowState to Unchanged, hence the DataAdapter will not know that something was changed.
adapter.Update(myTable) will call AcceptChanges itself after the update is finished.
So...
myTable.Rows[0]["Name"] = "Was Tom";
//myTable.AcceptChanges();
adapter.Update(myTable);

My some one need to look into the following solution; In other scenario people may need different solution. Even Don't do any manipulation with Datatable when you Debug at Run-time like this,
myTable.GetChanges(); // Return Any of Chnages Made without applying myTable.Accepchanges()
myTable.GetChanges(DataRowState.Added); // Return added rows without applying myTable.Accepchanges()
myTable.GetChanges(DataRowState.Deleted);
myTable.GetChanges(DataRowState.Detached);
myTable.GetChanges(DataRowState.Modified);
myTable.GetChanges(DataRowState.Unchanged);
You may get Data According to the above commands. So better try to debug before you pass the datatable to update or insert or delete command.
If myTable.GetChanges() return null then you can SetAdded() or SetModified() back to your DataTable;
foreach(DataRow row in myTable.Rows)
{
row.SetAdded(); // For Insert Command
row.SetModified(); // For Update Command
}

local database won't update and doesn't show errors

Hey i'm new to this and from what i managed to pick up this should be working but it doesn't update my local database.
I have a TelemarketingDatabaseDataSet that was auto generated when i created my local database, which then i dragged the table onto the dataset and i guess they're linked.
Now i have this code :
SqlCeConnection connection = new SqlCeConnection();
connection.ConnectionString = TelemarketingTracker.Properties.Settings.Default.TelemarketingDatabaseConnectionString;
TelemarketingDatabaseDataSet ds = new TelemarketingDatabaseDataSet();
// DataTable tbl = new DataTable();
SqlCeDataAdapter adapter = new SqlCeDataAdapter("select * from Calls", connection);
//adapter.InsertCommand = new SqlCeCommand("InsertQuery", connection);
adapter.Fill(ds,"Calls");
DataTable tbl = ds.Tables["Calls"];
//tbl.Columns.Add("caller");
//tbl.Columns.Add("called");
//tbl.Columns.Add("duration");
//tbl.Columns.Add("time");
var row = tbl.NewRow();
row[1] = Convert.ToString(caller);
row[2] = Convert.ToString(called);
row[3] = Convert.ToString(duration);
row[4] = Convert.ToDateTime(time);
tbl.Rows.Add(row);
adapter.Update(ds, "Calls");
connection.Close();
MessageBox.Show("Database should be updated!");
And please, i'm not intrested in using an SqlCommand as i prefer using DataSet.
Could the problem be related to datatypes of my table? it doesn't show errors to suggest that but i guess this could be the problem. my Table consists of :
ID - int,key
caller - varchar
called - varchar
duration - varchar
time - datetime
EDIT:
Now if i uncomment the insertQuery row i get an unhandled error occured in Syste.Data dll.
Now even if i try to use a regular insert command i get no errors but the database won't update.
if this makes any diffrence after i close the debugging window i see an X next to the local database but it doesn't show any errors.
This is the command i've tried :
using (SqlCeCommand com = new SqlCeCommand("INSERT INTO Calls (caller, called, duration, time) Values(#Caller,#Called,#Duration,#Time)", connection))
{
com.Parameters.AddWithValue("#Caller", row[1]);
com.Parameters.AddWithValue("#Called", row[2]);
com.Parameters.AddWithValue("#Duration", row[3]);
com.Parameters.AddWithValue("#Time", row[4]);
com.ExecuteNonQuery();
}

The Fill() method "Adds or refreshes rows in the DataSet to match those in the data source." The key part of this sentence being "to match those in the data source". The row you're adding gets wiped out when you call Fill() because it's not already in the source.
I'm not positive, but I don't think that you need to even call Fill() if you're only adding new records and not worried about modifying/removing existing ones. If you do need to call it though, it would obviously need to be moved before any new record insertions you make.

Try something similar to this..
string dbfile = new System.IO.FileInfo(System.Reflection.Assembly.GetExecutingAssembly().Location).DirectoryName + "\\TelemarketingDatabase.sdf";
SqlCeConnection connection = new SqlCeConnection("datasource=" + dbfile);
TelemarketingDatabaseDataSet ds = new TelemarketingDatabaseDataSet();
SqlCeDataAdapter adapter = new SqlCeDataAdapter();
string qry = #"select * from Calls";
da.SelectCommand = new SqlCommand(qry, connection);
SqlCommandBuilder cb = new SqlCommandBuilder(adapter);
adapter.Fill(ds,"Calls");
DataTable tbl = ds.Tables["Calls"];
var row = tbl.NewRow();
row[0] = caller;
row[1] = called;
row[2] = duration;
row[3] = Convert.ToDateTime(time);
tbl.Rows.Add(row);
adapter.Update(ds,"Calls");
See the example here http://www.java2s.com/Code/CSharp/Database-ADO.net/UseDataTabletoupdatetableinDatabase.htm

Well in the end i didn't manage to solve this, instead i used a remote database and regular sql commands.
Thanks for those who helped!

just want to share this even the if the question is old
using System;
using System.IO; //needed for path.getdirectoryname() and directory.getcurrentdirectory()
string path = Path.GetDirectoryName(Path.GetDirectoryName(Directory.GetCurrentDirectory()));
AppDomain.CurrentDomain.SetData("DataDirectory", path);
Directory.GetcurrentDirectory() will output "C:/..projectname/bin/debug" which is where the temporary database.mdf is located
by using Path.GetDirectoryName(Directory.GetcurrentDirectory()) it will give the directory of the current directory thus moving one location back
"C:/..projectname/bin"
then use it again
Path.GetDirectoryName(Path.GetDirectoryName(Directory.GetCurrentDirectory())) will give you the location of the root database in your project folder
"C:/..projectname"
then just use AppDomain.CurrentDomain.SetData()

Problem with ADO.NET UPDATE code

Could somebody take a quick peek at my ado.net code? I am trying to update the row from a dataset, but it just isn't working. I am missing some elemental piece of the code, and it is just eluding me. I have verified that the DataRow actually has the correct data in it, so the row itself is accurate.
Many thanks in advance.
try
{
//basic ado.net objects
SqlDataAdapter dbAdapter = null;
DataSet returnDS2 = new DataSet();
//a new sql connection
SqlConnection myConn = new SqlConnection();
myConn.ConnectionString = "Server=myserver.mydomain.com;"
+ "Database=mydatabase;"
+ "User ID=myuserid;"
+ "Password=mypassword;"
+ "Trusted_Connection=True;";
//the sqlQuery
string sqlQuery = "select * from AVLUpdateMessages WHERE ID = 21";
//another ado.net object for the command
SqlCommand cmd = new SqlCommand();
cmd.Connection = myConn;
cmd.CommandText = sqlQuery;
//open the connection, execute the SQL statement and then close the connection.
myConn.Open();
//instantiate and fill the sqldataadapter
dbAdapter = new SqlDataAdapter(cmd);
dbAdapter.Fill(returnDS2, #"AVLUpdateMessages");
//loop through all of the rows; I have verified that the rows are correct and returns the correct data from the db
for (int i = 0; i <= returnDS2.Tables[0].Rows.Count - 1; i++)
{
DataRow row = returnDS2.Tables[0].Rows[i];
row.BeginEdit();
row["UpdatedText"] = #"This is a test...";
row.EndEdit();
}
//let's accept the changes
dbAdapter.Update(returnDS2, "AVLUpdateMessages");
returnDS2.AcceptChanges();
myConn.Close();
}

I think you need an update query in your data adapter. I know, this sucks... Alternatively you can use CommandBuilder class to automatically generate queries for CRUD operations.
example at: http://www.programmersheaven.com/2/FAQ-ADONET-CommandBuilder-Prepare-Dataset

You might be able to use SqlCommandBuilder to help out. After the Fill call, add the following statement. That will associate a command builder with the data adapter and (if there is a primary key available) it should generate the update statement for you. Note that there is some expense behind the command builder. It may not be much relative to everything else, but it does involve looking at schema information (to get primary key information, field names, field types, etc.) for the table and generating INSERT, DELETE, and UPDATE statements involving all fields in the table.
SqlCommandBuilder cb = new SqlCommandBuilder(dbAdapter);

Wait, why not something like
update AVLUpdateMessages set UpdatedText = 'This is a test...' where id = 21
If you're picking through all the rows of a table to update one at a time, you're probably doing it wrong. SQL is your friend.

Help with DataAdapter - DB doesn't update?

I have a data adapter. When I check the function at runtime I see the changes, but nothing happens in the database. How do I debug this? What went wrong?
OleDbDataAdapter adapter = new OleDbDataAdapter();
string queryString = "SELECT * FROM tasks";
OleDbConnection connection = new OleDbConnection(cn.ConnectionString);
adapter.SelectCommand = new OleDbCommand(queryString, connection);
OleDbCommandBuilder builder = new OleDbCommandBuilder(adapter);
connection.Open();
adapter.Fill(ds.Tables["Tasks"]);
adapter.Update(ds);
return ds;

Nothing is happening in the database because you're running a SELECT query. What are you expecting to happen?

Well unless you snipped a lot of code, you are not changing anything in the dataset per se.
What are you trying to achieve?
Here, you are selecting some data, filling the dataset with it, then putting back the unchanged dataset in the DB.
You should first change something in the dataset itself before calling adapter.Update(ds)
Cheers,
Florian

You are selecting data via the SelectCommand. If you want to update data, you need to run the UpdateCommand.

I assume you didn't post the full source code, as it looks as though you're not modifying the dataset. However, I just tweaked your code a bit (see my comments).
Hopefully this will help...
string queryString = "SELECT * FROM tasks";
OleDbConnection connection = new OleDbConnection(cn.ConnectionString);
connection.Open(); // open connection first
SqlCommand cmd = new SqlCommand(queryString, connection);
OleDbDataAdapter adapter = new OleDbDataAdapter(cmd); // use the cmd above when instantiating the adapter
OleDbCommandBuilder builder = new OleDbCommandBuilder(adapter);
adapter.Fill(ds.Tables["Tasks"]);
// Modify your dataset here.
// Don't call AcceptChanges() as this will prevent the update from working.
adapter.Update(ds);
connection.Close(); // Close connection before ending the function
return ds;
This should allow OleDbCommandBuilder to do its thing by automatically scripting the database updates.

As far as I can see you've not actually changed any of the contents of the tasks table. When you call adapter.Fill you are populated the empty dataset with the records from the database. You are then calling adapter.Update without changing any records within the dataset. The update command only performs changes when the dataset > datatable > datarows are edited and marked as dirty.

you are only specifying the select command. you also need to specify the insert command... i.e:
DataAdapter.InsertCommand = new OleDbCommand....