We Keep Coding

How to get newly inserted Ids from identity primary key column when using SqlBulkCopy

I'm using C# and .NET Core 6. I want to bulk insert about 100 rows at once into database and get back their Ids from BigInt identity column.
I have tried lot of different variants, but still do not have the working solution. When I preview table variable, the Id column has DbNull value and not the newly inserted Id.
How to get Ids of newly inserted rows?
What I have:
SQL Server table:
CREATE TABLE dbo.ExamResult
(
Id bigint IDENTITY(1, 1) NOT NULL,
Caption nvarchar(1024) NULL,
SortOrder int NOT NULL,
CONSTRAINT PK_ExmRslt PRIMARY KEY CLUSTERED (Id)
) ON [PRIMARY]
GO
C# code:
private static void BulkCopy(IEnumerable<ExamResultDb> examResults, SqlConnection connection)
{
using (var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.Default, null))
{
var table = CreateDataTable(examResults);
bulkCopy.DestinationTableName = "ExamResult";
bulkCopy.EnableStreaming = true;
foreach (var item in table.Columns.Cast<DataColumn>())
{
if (item.ColumnName != "Id")
{
bulkCopy.ColumnMappings.Add(item.ColumnName, item.ColumnName);
}
}
using (var reader = new DataTableReader(table))
{
bulkCopy.WriteToServer(reader);
}
}
}
private static DataTable CreateDataTable(IEnumerable<ExamResultDb> examResults)
{
var table = new DataTable("ExamResult");
table.Columns.AddRange(new[]
{
new DataColumn("Id", typeof(long)),
new DataColumn("Caption", typeof(string)),
new DataColumn("SortOrder", typeof(int))
});
////table.Columns[0].AutoIncrement = true;
////table.PrimaryKey = new[] { table.Columns[0] };
foreach (var examResult in examResults)
{
var row = table.NewRow();
row["Caption"] = examResult.Caption;
row["SortOrder"] = examResult.SortOrder;
table.Rows.Add(row);
}
return table;
}

You need an OUTPUT clause on your insert, but Bulk Copy does not allow this kind of customization.
While nowhere near as neat, you could do this by using a Table Valued Parameter and a custom INSERT statement. TVPs use the bulk copy mechanism, so this should still be pretty fast, although there will be two inserts: one to the TVP and one to the real table.
First create a table type
CREATE TYPE dbo.Type_ExamResult AS TABLE
(
Caption nvarchar(1024) NULL,
SortOrder int NOT NULL
);
This function will iterate the rows as SqlDatRecord
private static IEnumerable<SqlDataRecord> AsExamResultTVP(this IEnumerable<ExamResultDb> examResults)
{
// fine to reuse object, see https://stackoverflow.com/a/47640131/14868997
var record = new SqlDataRecord(
new SqlMetaData("Caption", SqlDbType.NVarChar, 1024),
new SqlMetaData("SortOrder", SqlDbType.Int)
);
foreach (var examResult in examResults)
{
record.SetString(0, examResult.Caption);
record.SetInt32(0, examResult.SortOrder);
yield return record; // looks weird, see above link
}
}
Finally insert using OUTPUT
private static void BulkCopy(IEnumerable<ExamResultDb> examResults, SqlConnection connection)
{
const string query = #"
INSERT dbo.ExamResult (Caption, SortOrder)
OUTPUT Id, SortOrder
SELECT t.Caption, t.SortOrder
FROM #tvp t;
";
var dict = examResults.ToDictionary(er => er.SortOrder);
using (var comm = new SqlCommand(query, connection))
{
comm.Parameters.Add(new SqlParameter("#tmp", SqlDbType.Structured)
{
TypeName = "dbo.Type_ExamResult",
Value = examResults.AsExamResultTVP(),
});
using (var reader = comm.ExecuteReader())
{
while(reader.Read())
dict[(int)reader["SortOrder"]].Id = (int)reader["Id"];
}
}
}
Note that the above code assumes that SortOrder is a natural key within the dataset to be inserted. If it is not then you will need to add one, and if you are not inserting that column then you need a rather more complex MERGE statement to be able to access that column in OUTPUT, something like this:
MERGE dbo.ExamResult er
USING #tvp t
ON 1 = 0 -- never match
WHEN NOT MATCHED THEN
INSERT (Caption, SortOrder)
VALUES (t.Caption, t.SortOrder)
OUTPUT er.Id, t.NewIdColumn;

C# how to transact Entity Framework and ADO.NET at a time

I have a parent-child data windows form. My parent data is fetched by an entity class created with Entity Framework.
But my child datagridview datasource is bound with a datatable with multiply data table query from my database.
So I use Entity Framework to insert or update the ‘parent’ data, and I use ADO.NET to create and update the ‘child’ data records.
When I insert child data I have to get the Id of parent data as foreign key.
In this setup, how can I have a transaction over both?
Just simply use TransactionScope?

//Simpy use TransactionScope can solve this problem.
using (TransactionScope scope = new TransactionScope())
{
tblOrder tblOrder = new tblOrder() { FOrderNum = "12345", FOrderType = "OrderTYpe", FClientID = 1086, FOrderStatus = "ReadyToProduction" };
EFContext context = new EFContext();
context.tblOrders.Add(tblOrder);
context.SaveChanges();
scope.Dispose();
int orderID = tblOrder.FOrderID;
SqlConnection conn = new SqlConnection("****");
string sql = $"INSERT INTO tblOrderSub (FOrderID,FMaterialID,FQty,FQtyShip,FFinishSKU,FModelCust,FColorCust)" +
$" VALUES(#FOrderID,#FMaterialID,#FQty,#FQtyShip,#FFinishSKU,#FModelCust,#FColorCust)";
SqlParameter[] sqlParameters =
{
new SqlParameter("#FOrderID",orderID),
new SqlParameter("#FMaterialID",1),
new SqlParameter("#FQty",1),
new SqlParameter("#FQtyShip",1),
new SqlParameter("#FFinishSKU",false),
new SqlParameter("#FModelCust", "TEst"),
new SqlParameter("#FColorCust", "C1")
};
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.Parameters.AddRange(sqlParameters);
conn.Open();
cmd.ExecuteNonQuery();
scope.Complete();
}

FluentMigrator - Check if Foreign Key exists before deleting it

I am using FluentMigrator to migrate one database schema to another. I have a case in which I want to check if a foreign key exists before deleting it.
Previously, I just delete the foreign key by doing:
Delete.ForeignKey("FK_TableName_FieldName").OnTable("TableName");
How do I check that the foreign key exists first?

This is how to delete a foreign key if it exists using FluentMigrator:
if (Schema.Table("TableName").Constraint("FK_TableName_FieldName").Exists())
{
Delete.ForeignKey("FK_TableName_FieldName").OnTable("TableName");
}

Based on this https://stackoverflow.com/a/17501870/10460456 you can use Execute.WithConnection function to test if foreign key exist before delete it.
Execute.WithConnection((connection, transaction) =>
{
DeleteForeignKeyIfExist(connection, transaction, "yourReferencedTable", "yourTable", "foreignColumnName", "foreignKeyName");
});
public bool DeleteForeignKeyIfExist(IDbConnection connection, IDbTransaction transaction, string referenceTable, string table, string foreignKeyColumn, string foreignKeyConstrainName)
{
using (var cmd = transaction.Connection.CreateCommand())
{
cmd.Transaction = transaction;
cmd.CommandType = CommandType.Text;
cmd.CommandText = ForeignKeyExistCommand(referenceTable, foreignKeyColumn);
bool foreignKeyExist = false;
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
// If this code is reached, the foreign key exist
foreignKeyExist = true;
break;
}
}
if (foreignKeyExist)
{
cmd.CommandText = $"ALTER TABLE [{table}] DROP CONSTRAINT [{foreignKeyConstrainName}];";
cmd.ExecuteNonQuery();
return true;
}
}
return false;
}
private string ForeignKeyExistCommand(string foreignTable, string innerColumn)
{
return $"SELECT OBJECT_NAME(f.parent_object_id) TableName, " +
"COL_NAME(fc.parent_object_id, fc.parent_column_id) ColName " +
"FROM sys.foreign_keys AS f INNER JOIN sys.foreign_key_columns AS fc " +
"ON f.OBJECT_ID = fc.constraint_object_id INNER JOIN sys.tables t " +
$"ON t.OBJECT_ID = fc.referenced_object_id WHERE OBJECT_NAME(f.referenced_object_id) = '{foreignTable}' " +
$"and COL_NAME(fc.parent_object_id,fc.parent_column_id) = '{innerColumn}'";
}

Is it possible to get a list of DataTables from SQL Server?

I am trying to get a list of tables from my SQL Server database. I would like the DataTable object not just the name.
So far I have got:
using (SqlConnection connection = new SqlConnection(Settings.Default.DatabaseString))
{
List<DataTable> tables = new List<DataTable>();
connection.Open();
}
I can get the names of the tables using:
DataTable schema = connection.GetSchema("Tables");
foreach (DataRow row in schema.Rows)
{
var name = row[2].ToString();
}
But I would like more information than just the name of the table I would like to return primary and foreign keys etc. Is this possible?

You can get infromations about the database, tables, columns and indexes with the Conenction.GetSchema overloads:
using (var con = new SqlConnection(Settings.Default.DatabaseString))
{
con.Open();
DataTable tables = con.GetSchema("Tables");
foreach (DataRow tableRow in tables.Rows)
{
String database = tableRow.Field<String>("TABLE_CATALOG");
String schema = tableRow.Field<String>("TABLE_SCHEMA");
String tableName = tableRow.Field<String>("TABLE_NAME");
String tableType = tableRow.Field<String>("TABLE_TYPE");
DataTable columns = con.GetSchema("Columns", new[] { database, null, tableName });
foreach (DataRow col in columns.Rows)
{
Console.WriteLine(string.Join(",",col.ItemArray));
}
DataTable indexes = con.GetSchema("Indexes", new[] { database, null, tableName });
foreach (DataRow index in indexes.Rows)
{
Console.WriteLine(string.Join(",", index.ItemArray));
}
DataTable indexColumns = con.GetSchema("IndexColumns", new[] { database, null, tableName });
foreach (DataRow indexCol in indexColumns.Rows)
{
Console.WriteLine(string.Join(",", indexCol.ItemArray));
}
}
}
Here's a list of possible values for GetSchema:
https://msdn.microsoft.com/en-us/library/cc716722(v=vs.110).aspx

SQL Server is very open about querying the catalog. You can easily create the query that will return the data you require.
Multiple tables exists to help you:
SELECT * FROM sys.tables
SELECT * FROM sys.columns
SELECT * FROM sys.indexes
SELECT * FROM sys.objects o WHERE o.type IN ('PK', 'F')
Querying the name of the objects from any of these tables can be done with the system method OBJECT_NAME.

copy all rows of a table to another table

I have two databases in MySQL and SQL Server, and I want to create tables in SQL Server and copy all rows from the table in MySQL into the new table in SQL Server.
I can create table in SQL Server same as MySQL, with this code:
List<String> TableNames = new List<string>();
{
IDataReader reader=
ExecuteReader("SELECT Table_Name FROM information_schema.tables WHERE table_name LIKE 'mavara%'",MySql);
while (reader.Read()) {
TableNames.Add(reader[0].ToString());
}
reader.Close();
}
foreach (string TableName in TableNames) {
IDataReader reader =
ExecuteReader("SELECT Column_Name,IS_NULLABLE,DATA_TYPE FROM information_schema.columns where TABLE_Name='" + TableName + "'",MySql);
List<string[]> Columns = new List<string[]>();
while (reader.Read()) {
string[] column = new string[3];
column[0] = reader[0].ToString();
column[1] = reader[1].ToString();
column[2] = reader[2].ToString();
Columns.Add(column);
}
reader.Close();
// create table
string queryCreatTables= "CREATE TABLE [dbo].[" + TableName + "](\n";
foreach(string[] cols in Columns)
{
queryCreatTables +="["+ cols[0] + "] " + cols[2] + " ";
if (cols[1] == "NO")
queryCreatTables += "NOT NULL";
// else
// queryCreatTables += "NULL";
queryCreatTables += " ,\n ";
}
queryCreatTables += ")";
System.Data.SqlClient.SqlCommand smd =
new System.Data.SqlClient.SqlCommand(queryCreatTables, MsSql);
System.Data.SqlClient.SqlDataReader sreader = smd.ExecuteReader();
sreader.Close();
but I have problem to copy rows from one table into another table.
for select query, I use Idatareader, but I don't know how insert rows to another table.

For inserting rows from one table into another table please refer the below sample query
INSERT INTO Store_Information (store_name, Sales, Date)
SELECT store_name, sum(Sales), Date
FROM Sales_Information

The algorithm is as follows:
1. For each table in source database
2. Get a list of columns for that table
3. Create table in destination database
4. SELECT * FROM the table in source
5. For each row in data
6. Generate INSERT statement and execute on destination database
The information you need for a column is Name, Type, Length, etc.
Then you generate the insert statement by iterating on the columns
var insertStatement = "INSERT INTO " + tableName + " VALUES( ";
foreach( var column in columns )
insertStatement += "#" + column.Name + ",";
insertStatement[insertStatement.Length-1] = ')';
var command = new SqlCommand( insertStatement, MsSql );
// iterate over the columns again, but this time set values to the parameters
foreach( var column in columns )
command.Parameters.AddWithValue( "#"+column.Name, currentRow[column.Name] );

But I have a problem to copy rows from one table into another table.
You can use the SqlDataAdapter.UpdateBatchSize to perform Batch Updates/Inserts with a DataAdapter against the database to copy the data from one table to the other. After you get all records from the first MYSQL table using something like:
//Get the rows from the first mysql table as you did in your question
DataTable mysqlfirstTableRowsTobeCopied = GetDataTableFromMySQLTable();
Then you have to create a commend text that do the INSERT something like:
cmd.CommandText = "INSERT INTO TableName Column_Name, ... VALUES(...)";
Or you can use a stored procedure:
CREATE PROCEDURE sp_BatchInsert ( #ColumnName VARCHAR(20), ... )
AS
BEGIN
INSERT INTO TableName VALUES ( #ColumnNamne, ...);
END
Then:
DataTable mysqlfirstTableRowsTobeCopied = GetDataTableFromMySQLTable();
SqlConnection conn = new SqlConnection("Your connection String");
SqlCommand cmd = new SqlCommand("sp_BatchInsert", conn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.UpdatedRowSource = UpdateRowSource.None;
// Set the Parameter with appropriate Source Column Name
cmd.Parameters.Add("#ColumnName", SqlDbType.Varchar, 50,
mysqlfirstTableRowsTobeCopied.Columns[0].ColumnName);
...
SqlDataAdapter adpt = new SqlDataAdapter();
adpt.InsertCommand = cmd;
// Specify the number of records to be Inserted/Updated in one go. Default is 1.
adpt.UpdateBatchSize = 20;
conn.Open();
int recordsInserted = adpt.Update(mysqlfirstTableRowsTobeCopied);
conn.Close();
This code actually is quoted from a full tutorial about this subject in the codeproject, you can refer to it for more information:
Multiple Ways to do Multiple Inserts

Assuming that you already have a datatable with rows that you want to merge with another table...
There are two ways to do this...again, assuming you've already selected the data and put it into a new table.
oldTable.Load(newTable.CreateDataReader());
oldTable.Merge(newTable);

Usually that's sth. you can do in SQL directly: INSERT INTO table FROM SELECT * FROM othertable;

For insertion of all record into new table(If the second table is not exist)
Select * into New_Table_Name from Old_Table_Name;
For insertion of all records into Second Table(If second table is exist But the table structure should be same)
Insert into Second_Table_Name from(Select * from First_Table_Name);

Just in case it can help someone, I've found an easy way to do it in C# using SqlDataAdapter. It automatically detects the structure of the table so you don't have to enumerate all the columns or worry about table structure changing. Then bulk inserts the data in the destination table.
As long as the two tables have the same structure, this will work. (For example, copying from a production table to a dev empty table.)
using(SqlConnection sqlConn = new SqlConnection(connectionStringFromDatabase))
{
sqlConn.Open();
using(var adapter = new SqlDataAdapter(String.Format("SELECT * FROM [{0}].[{1}]", schemaName, tableName), sqlConn))
{
adapter.Fill(table);
};
}
using(SqlConnection sqlConn = new SqlConnection(connectionStringDestination))
{
sqlConn.Open();
// perform bulk insert
using(var bulk = new SqlBulkCopy(sqlConn, SqlBulkCopyOptions.KeepIdentity|SqlBulkCopyOptions.KeepNulls, null))
{
foreach(DataColumn column in table.Columns)
{
bulk.ColumnMappings.Add(column.ColumnName, column.ColumnName);
}
bulk.DestinationTableName = String.Format("[{0}].[{1}]", schemaName, tableName);
bulk.WriteToServer(table);
}
}