We have a big list around 100000 records and want to insert it into a sql table.
What are we doing is; converting that list into data table and passing datatable to SqlBulkcopy method.
This conversion from list to Datatable taking more time. Tried using Parallel but as Datatable is not thread safe so avoided that.
Adding sample poc code which generates integer list and insert it into temp table
static void Main(string[] args)
{
List<int> valueList = GenerateList(100000);
Console.WriteLine("Starting with Bulk Insert ");
DateTime startTime = DateTime.Now;
int recordCount = BulkInsert(valueList);
TimeSpan ts = DateTime.Now.Subtract(startTime);
Console.WriteLine("Bulk insert for {0} records in {1} miliseconds.-> ", recordCount, ts.Milliseconds);
Console.WriteLine("Done.");
Console.ReadLine();
}
private static int BulkInsert(List<int> valueList)
{
SqlBulkHelper sqlBulkHelper = new SqlBulkHelper();
var eventIdDataTable = CreateIdentityDataTable(valueList, "SqlTable", "Id");
return FillBulkPoundTable(eventIdDataTable, "#SqlTable");
}
private static List<int> GenerateList(int size)
{
return Enumerable.Range(0, size).ToList();
}
private static DataTable CreateIdentityDataTable(List<int> ids, string dataTableName, string propertyName)
{
if (ids == null) return null;
using (var dataTable = new DataTable(dataTableName))
{
dataTable.Locale = CultureInfo.CurrentCulture;
var dtColumn = new DataColumn(propertyName, Type.GetType("System.Int32"));
dataTable.Columns.Add(dtColumn);
foreach (int id in ids)
{
DataRow row = dataTable.NewRow();
row[propertyName] = id;
dataTable.Rows.Add(row);
}
return dataTable;
}
}
private static int FillBulkPoundTable(DataTable dataTable, string destinationTableName)
{
int totalInsertedRecordCount = 0;
using (SqlConnection _connection = new SqlConnection(CongifUtil.sqlConnString))
{
string sql =
#"If object_Id('tempdb..#EventIds') is not null drop table #EventIds
CREATE TABLE #EventIds(EvId int) ";
_connection.Open();
using (var command = new SqlCommand(sql, _connection))
{
command.ExecuteNonQuery();
}
using (var sqlBulkCopy = new SqlBulkCopy(_connection))
{
sqlBulkCopy.BulkCopyTimeout = 0;
sqlBulkCopy.DestinationTableName = destinationTableName;
sqlBulkCopy.WriteToServer(dataTable);
}
using (var command = new SqlCommand(sql, _connection))
{
command.CommandText = "Select Count(1) as RecordCount from #EventIds";
SqlDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
totalInsertedRecordCount = Convert.ToInt32(reader["RecordCount"]);
}
}
}
}
return totalInsertedRecordCount;
}
Currently it is taking around 8 seconds but we need to make it more faster. Reason is our target is to insert 900,000 records which will be devided into 100,000 batch each.
Can you give us any hint how can we make it perfect and faster?
PS. Tried with Dapper insert too but it is not faster than BulkCopy.
First Covert your list into XML something like
List<int> Branches = new List<int>();
Branches.Add(1);
Branches.Add(2);
Branches.Add(3);
XElement xmlElements = new XElement("Branches", Branches.Select(i => new
XElement("branch", i)));
Then pass the xml to a SP as parameter and insert it directly to your table, Example :
DECLARE #XML XML
SET #XML = '<Branches>
<branch>1</branch>
<branch>2</branch>
<branch>3</branch>
</Branches>'
DECLARE #handle INT
DECLARE #PrepareXmlStatus INT
EXEC #PrepareXmlStatus= sp_xml_preparedocument #handle OUTPUT, #XML
SELECT * FROM OPENXML(#handle, '/Branches/branch', 2)
WITH (
branch varchar
)
EXEC sp_xml_removedocument #handle
Bach Size
From what I understand, you try to insert with a BatchSize of 100000. Higher is not always better.
Try to lower this amount to 5,000 instead and check for the performance difference.
You increase the amount of database round-trip but it may also go faster (Too much factor such as the row size are involved here)
TableLock
Using the SqlBulkCopyOptions.TableLock will improve your insert performance.
using (var sqlBulkCopy = new SqlBulkCopy(_connection, SqlBulkCopyOptions.KeepIdentity))
Related
Currently, I have a database called "testDB.db" that has 100 rows and 3 columns. Is there some library in C# that allows me to easily check how many columns and rows there are in table "test001" in "testDB.db"?
namespace SQLiteExtractor
{
class Program
{
static void Main()
{
string liteString = #"Data Source = .\testDB.db";
connectToSQLite(liteString);
}
public static void connectToSQLite(string liteConString)
{
using SQLiteConnection liteCon = new SQLiteConnection(liteConString);
liteCon.Open();
string query = "SELECT * FROM test001";
int sizeOfDR = 0;
List<string> liteEntries = new List<string>();
using var cmd = new SQLiteCommand(query, liteCon);
using SQLiteDataReader SQLiteDR = cmd.ExecuteReader();
while (SQLiteDR.Read())
{
liteEntries.Add(SQLiteDR.GetString(1));
}
foreach (string entry in liteEntries)
Console.WriteLine(entry);
}
}
}
For rows, you can do a SELECT COUNT() query against the table.
For cols, you can inspect the table metadata via system table queries. Use PRAGMA table_info(table_name); to get the col info.
Specific to your table:
SELECT COUNT(*) FROM test001
and
PRAGMA table_info('test001')
I am really clueless here.
I import ~2 million rows into my Azure SQL database. I create a temp table where I put my values, when I use merge technique to insert only rows that do not have duplicates. My code and scripts are shown below.
public async Task BulkImportWithoutDuplicates(DataTable reader)
{
var tableName = "##tempImport";
using (var connection = new SqlConnection(sqlCOnn.ConnectionString))
{
using (SqlCommand command = new SqlCommand("", sqlCOnn))
{
try
{
sqlCOnn.Open();
// Creating temp table on database
command.CommandText = Scripts.GetTempTableScript();
command.ExecuteNonQuery();
// Bulk insert into temp table
using (SqlBulkCopy b = new SqlBulkCopy(conString, SqlBulkCopyOptions.TableLock))
{
b.BulkCopyTimeout = 0;
b.BatchSize = reader.Rows.Count;
b.DestinationTableName = tableName;
//dataTable
await b.WriteToServerAsync(reader);
b.Close();
}
// Updating destination table, and dropping temp table
command.CommandText = Scripts.GetMergeScript();
var rows = command.ExecuteNonQuery();
}
catch (Exception ex)
{
// Handle exception properly
}
finally
{
connection.Close();
}
}
}
}
public static string GetTempTableScript()
{
return $#"
IF OBJECT_ID('tempdb.dbo.##tempImport', 'U') IS NOT NULL
BEGIN
DROP TABLE ##tempImport;
END
CREATE TABLE ##tempImport ( ... all the columns);";
}
public static string GetMergeScript()
{
return $#"MERGE INTO dbo.Data AS target
USING ##tempImport AS source
ON (source.TransactionId = target.TransactionId AND source.UserId = target.UserId)
WHEN NOT MATCHED THEN
INSERT (Start, Spend, UserId, Product, Shop, ClientId, UploadDataId, UniqueId, TransactionId, q, cq, c2)
VALUES (source.Start, source.Spend, source.UserId, source.Product, source.Shop,
source.ClientId, source.UploadDataId, source.UniqueId, source.TransactionId, source.q, source.c1, source.c2);
";
}
I really do not get why it takes ages until it finishes. I waited for 24 minutes until it was added to temporary table alone.
I was reading this article and it seems that it shouldn't take long. https://www.adathedev.co.uk/2011/01/sqlbulkcopy-to-sql-server-in-parallel.html?m=1
What I am doing wrong here? How can I improve the speed?
I tried using both IDataReader and DataTable but both of them do not work well for me...
I am trying to figure out if there is a way to perform a multiple values insert in Sql Server while using parameters, to be precise, having a command like this:
com = new SqlCommand("insert into myTable values (#recID,#tagID)", con);
com.Parameters.Add("#recID", SqlDbType.Int).Value = recID;
com.Parameters.Add("#tagID", SqlDbType.Int).Value = tagID;
com.ExecuteNonQuery();
Is there a way to perform a multiple values single insert with parameters taking into account that parameters may be different for each value? (Example: tagID may be always different)
I have been searching in Internet but no luck so far, thanks in advance, greetings.
You can use a table valued parameters : How to pass table value parameters to stored procedure from .net code
First, create the type, in SQL Server :
CREATE TYPE [dbo].[myTvpType] AS TABLE
(
[RecordID] int,
[TagID] int
)
And the C# code to insert your data :
internal void InsertData(SqlConnection connection, Dictionary<int, int> valuesToInsert)
{
using (DataTable myTvpTable = CreateDataTable(valuesToInsert))
using (SqlCommand cmd = connection.CreateCommand())
{
cmd.CommandText = "INSERT INTO myTable SELECT RecordID, TagID FROM #myValues";
cmd.CommandType = CommandType.Text;
SqlParameter parameter = cmd.Parameters.AddWithValue("#myValues", myTvpTable);
parameter.SqlDbType = SqlDbType.Structured;
cmd.ExecuteNonQuery();
}
}
private DataTable CreateDataTable(Dictionary<int, int> valuesToInsert)
{
// Initialize the DataTable
DataTable myTvpTable = new DataTable();
myTvpTable.Columns.Add("RecordID", typeof(int));
myTvpTable.Columns.Add("TagID", typeof(int));
// Populate DataTable with data
foreach(key in valuesToInsert.Key)
{
DataRow row = myTvpTable.NewRow();
row["RecordID"] = valuesToInsert[key];
row["TagID"] = key;
}
}
You can do this by sending your data as an xml string and convert in into table in a stored procedure in sql. For example:
suppose I am sending multiple rows to add/update in an sql table then here are the steps:
Convert your class or list of class into an xml string using following method:
public static string SerializeObjectToXmlString(object value)
{
var emptyNamepsaces = new XmlSerializerNamespaces(new[] {
XmlQualifiedName.Empty });
var serializer = new XmlSerializer(value.GetType());
var settings = new XmlWriterSettings();
settings.Indent = true;
settings.OmitXmlDeclaration = true;
using (var stream = new StringWriter())
using (var writer = XmlWriter.Create(stream, settings))
{
serializer.Serialize(writer, value, emptyNamepsaces);
return stream.ToString();
}
}
Now while sending data to the database convert your class object into xml
string (Here I am using entity framework in my code, you can do this without using it as well):
bool AddUpdateData(List<MyClass> data)
{
bool returnResult = false;
string datatXml = Helper.SerializeObjectToXmlString(data);
var sqlparam = new List<SqlParameter>()
{
new SqlParameter() { ParameterName = "dataXml", Value = datatXml}
};
var result = this.myEntity.Repository<SQL_StoredProc_ComplexType>().ExecuteStoredProc("SQL_StoredProc", sqlparam);
if (result != null && result.Count() > 0)
{
returnResult = result[0].Status == 1 ? true : false;
}
return returnResult;
}
Now your SQL Code:
3.1 Declare a table variable:
DECLARE #tableVariableName TABLE
(
ID INT, Name VARCHAR(20)
)
3.2 Insert Your xml string into Table variable
INSERT INTO #tableVariableName
SELECT
Finaldata.R.value ('(ID/text())[1]', 'INT') AS ID,
Finaldata.R.value ('(Name/text())[1]', 'VARCHAR(20)') AS Name
FROM #MyInputXmlString.nodes ('//ArrayMyClass/MyClass') AS Finaldata (R)
3.3 Finally insert this table value into your sql table
INSERT INTO MyTable (ID, Name)
SELECT ID, Name
FROM #tableVariableName
This will save your effort of hitting database again and again using a for loop.
Hope it will help you
I have a SQL Server 2005 database. In a few procedures I have table parameters that I pass to a stored proc as an nvarchar (separated by commas) and internally divide into single values. I add it to the SQL command parameters list like this:
cmd.Parameters.Add("#Logins", SqlDbType.NVarchar).Value = "jim18,jenny1975,cosmo";
I have to migrate the database to SQL Server 2008. I know that there are table value parameters, and I know how to use them in stored procedures. But I don't know how to pass one to the parameters list in an SQL command.
Does anyone know correct syntax of the Parameters.Add procedure? Or is there another way to pass this parameter?
DataTable, DbDataReader, or IEnumerable<SqlDataRecord> objects can be used to populate a table-valued parameter per the MSDN article Table-Valued Parameters in SQL Server 2008 (ADO.NET).
The following example illustrates using either a DataTable or an IEnumerable<SqlDataRecord>:
SQL Code:
CREATE TABLE dbo.PageView
(
PageViewID BIGINT NOT NULL CONSTRAINT pkPageView PRIMARY KEY CLUSTERED,
PageViewCount BIGINT NOT NULL
);
CREATE TYPE dbo.PageViewTableType AS TABLE
(
PageViewID BIGINT NOT NULL
);
CREATE PROCEDURE dbo.procMergePageView
#Display dbo.PageViewTableType READONLY
AS
BEGIN
MERGE INTO dbo.PageView AS T
USING #Display AS S
ON T.PageViewID = S.PageViewID
WHEN MATCHED THEN UPDATE SET T.PageViewCount = T.PageViewCount + 1
WHEN NOT MATCHED THEN INSERT VALUES(S.PageViewID, 1);
END
C# Code:
private static void ExecuteProcedure(bool useDataTable,
string connectionString,
IEnumerable<long> ids)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "dbo.procMergePageView";
command.CommandType = CommandType.StoredProcedure;
SqlParameter parameter;
if (useDataTable) {
parameter = command.Parameters
.AddWithValue("#Display", CreateDataTable(ids));
}
else
{
parameter = command.Parameters
.AddWithValue("#Display", CreateSqlDataRecords(ids));
}
parameter.SqlDbType = SqlDbType.Structured;
parameter.TypeName = "dbo.PageViewTableType";
command.ExecuteNonQuery();
}
}
}
private static DataTable CreateDataTable(IEnumerable<long> ids)
{
DataTable table = new DataTable();
table.Columns.Add("ID", typeof(long));
foreach (long id in ids)
{
table.Rows.Add(id);
}
return table;
}
private static IEnumerable<SqlDataRecord> CreateSqlDataRecords(IEnumerable<long> ids)
{
SqlMetaData[] metaData = new SqlMetaData[1];
metaData[0] = new SqlMetaData("ID", SqlDbType.BigInt);
SqlDataRecord record = new SqlDataRecord(metaData);
foreach (long id in ids)
{
record.SetInt64(0, id);
yield return record;
}
}
Further to Ryan's answer you will also need to set the DataColumn's Ordinal property if you are dealing with a table-valued parameter with multiple columns whose ordinals are not in alphabetical order.
As an example, if you have the following table value that is used as a parameter in SQL:
CREATE TYPE NodeFilter AS TABLE (
ID int not null
Code nvarchar(10) not null,
);
You would need to order your columns as such in C#:
table.Columns["ID"].SetOrdinal(0);
// this also bumps Code to ordinal of 1
// if you have more than 2 cols then you would need to set more ordinals
If you fail to do this you will get a parse error, failed to convert nvarchar to int.
Generic
public static DataTable ToTableValuedParameter<T, TProperty>(this IEnumerable<T> list, Func<T, TProperty> selector)
{
var tbl = new DataTable();
tbl.Columns.Add("Id", typeof(T));
foreach (var item in list)
{
tbl.Rows.Add(selector.Invoke(item));
}
return tbl;
}
The cleanest way to work with it. Assuming your table is a list of integers called "dbo.tvp_Int" (Customize for your own table type)
Create this extension method...
public static void AddWithValue_Tvp_Int(this SqlParameterCollection paramCollection, string parameterName, List<int> data)
{
if(paramCollection != null)
{
var p = paramCollection.Add(parameterName, SqlDbType.Structured);
p.TypeName = "dbo.tvp_Int";
DataTable _dt = new DataTable() {Columns = {"Value"}};
data.ForEach(value => _dt.Rows.Add(value));
p.Value = _dt;
}
}
Now you can add a table valued parameter in one line anywhere simply by doing this:
cmd.Parameters.AddWithValueFor_Tvp_Int("#IDValues", listOfIds);
Use this code to create suitable parameter from your type:
private SqlParameter GenerateTypedParameter(string name, object typedParameter)
{
DataTable dt = new DataTable();
var properties = typedParameter.GetType().GetProperties().ToList();
properties.ForEach(p =>
{
dt.Columns.Add(p.Name, Nullable.GetUnderlyingType(p.PropertyType) ?? p.PropertyType);
});
var row = dt.NewRow();
properties.ForEach(p => { row[p.Name] = (p.GetValue(typedParameter) ?? DBNull.Value); });
dt.Rows.Add(row);
return new SqlParameter
{
Direction = ParameterDirection.Input,
ParameterName = name,
Value = dt,
SqlDbType = SqlDbType.Structured
};
}
If you have a table-valued function with parameters, for example of this type:
CREATE FUNCTION [dbo].[MyFunc](#PRM1 int, #PRM2 int)
RETURNS TABLE
AS
RETURN
(
SELECT * FROM MyTable t
where t.column1 = #PRM1
and t.column2 = #PRM2
)
And you call it this way:
select * from MyFunc(1,1).
Then you can call it from C# like this:
public async Task<ActionResult> MethodAsync(string connectionString, int? prm1, int? prm2)
{
List<MyModel> lst = new List<MyModel>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.OpenAsync();
using (var command = connection.CreateCommand())
{
command.CommandText = $"select * from MyFunc({prm1},{prm2})";
using (var reader = await command.ExecuteReaderAsync())
{
if (reader.HasRows)
{
while (await reader.ReadAsync())
{
MyModel myModel = new MyModel();
myModel.Column1 = int.Parse(reader["column1"].ToString());
myModel.Column2 = int.Parse(reader["column2"].ToString());
lst.Add(myModel);
}
}
}
}
}
View(lst);
}
The follow code will insert some values in my database. It gets 6 random values, puts the stuff in an array and then inserts it in the database.
public void LottoTest(object sender, EventArgs e)
{
Dictionary<int, int> numbers = new Dictionary<int, int>();
Random generator = new Random();
while (numbers.Count < 6)
{
numbers[generator.Next(1, 49)] = 1;
}
string[] lotto = numbers.Keys.OrderBy(n => n).Select(s => s.ToString()).ToArray();
foreach (String _str in lotto)
{
Response.Write(_str);
Response.Write(",");
}
var connectionstring = "Server=C;Database=lotto;User Id=lottoadmin;Password=password;";
using (var con = new SqlConnection(connectionstring)) // Create connection with automatic disposal
{
con.Open();
using (var tran = con.BeginTransaction()) // Open a transaction
{
// Create command with parameters (DO NOT PUT VALUES IN LINE!!!!!)
string sql =
"insert into CustomerSelections(val1,val2,val3,val4,val5,val6) values (#val1,#val2,#val3,#val4,#val5,#val6)";
var cmd = new SqlCommand(sql, con);
cmd.Parameters.AddWithValue("val1", lotto[0]);
cmd.Parameters.AddWithValue("val2", lotto[1]);
cmd.Parameters.AddWithValue("val3", lotto[2]);
cmd.Parameters.AddWithValue("val4", lotto[3]);
cmd.Parameters.AddWithValue("val5", lotto[4]);
cmd.Parameters.AddWithValue("val6", lotto[5]);
cmd.Transaction = tran;
cmd.ExecuteNonQuery(); // Insert Record
tran.Commit(); // commit transaction
Response.Write("<br />");
Response.Write("<br />");
Response.Write("Ticket has been registered!");
}
}
}
What is the best way to loop and insert MASS entries into the database. Lets say, 100,000 records via C#? I want to be able to generate the random numbers by my method and utilize the insert which i have too..
For true large scale inserts, SqlBulkCopy is your friend. The easy but inefficient way to do this is just to fill a DataTable with the data, and throw that at SqlBulkCopy, but it can be done twice as fast (trust me, I've timed it) by spoofing an IDataReader. I recently moved this code into FastMember for convenience, so you can just do something like:
class YourDataType {
public int val1 {get;set;}
public string val2 {get;set;}
... etc
public DateTime val6 {get;set;}
}
then create an iterator block (i.e. a non-buffered forwards only reader):
public IEnumerable<YourDataType> InventSomeData(int count) {
for(int i = 0 ; i < count ; i++) {
var obj = new YourDataType {
... initialize your random per row values here...
}
yield return obj;
}
}
then:
var data = InventSomeData(1000000);
using(var bcp = new SqlBulkCopy(connection))
using(var reader = ObjectReader.Create(data))
{ // note that you can be more selective with the column map
bcp.DestinationTableName = "CustomerSelections";
bcp.WriteToServer(reader);
}
You need Sql bulk insert. There is a nice tutorial on msdn http://blogs.msdn.com/b/nikhilsi/archive/2008/06/11/bulk-insert-into-sql-from-c-app.aspx
MSDN Table Value Parameters
Basically, you fill a datatable with the data you want to put into SqlServer.
DataTable tvp = new DataTable("LottoNumbers");
forach(var numberSet in numbers)
// add the data to the dataset
Then you pass the data through ADO using code similar to this...
command.Parameters.Add("#CustomerLottoNumbers", SqlDbType.Structured);
command.Parameters["CustomerLottoNumbers"].Value = tvp;
Then you could use sql similar to this...
INSERT CustomerSelections
SELECT * from #CustomerLottoNumbers