We Keep Coding

SqlBulkCopy Ignore Duplicate Records of Datatable From DataBase

Can i ignore the duplicate records of a data already present in sql database from a datatable which i am passing to SqlBulkCopy. If Yes then How and also explain me if No and other option.

No, that's not built-in. You need to clean the data on the client or insert into a staging table first.

As previous poster said, this is not built in. I achieve similar using the following:
SQL Stored Procedure that accepts a TableValuedParameter with the data you require.
In the stored proc, I then INSERT all records into a temp table. Once you have it there, you can use SQLs MERGE statement in your stored proc to insert data where it doesn't already exist.
So, let us assume that our data is simply people's names stored in a table people. We hold only an ID and a name. I also assume this table is called 'people'.
Here's how I create my Table Valued Parameter type (created in SQL Server)
CREATE TYPE udt_person AS TABLE(
[id] [INT] NOT NULL,
[name] [nvarchar(50)] NULL
)
GO
I now create the stored procedure:
CREATE PROCEDURE SaveNewPeople #pPeople udt_Person
AS
BEGIN
-- Create Temp table
CREATE TABLE #tmpPeople (id INT, name VARCHAR 50)
-- We will stage all data passed in into temp table
INSERT INTO #tmpPeople
SELECT id, name FROM #pPeople
-- NB: you will need to think about locking strategy a bit here
MERGE people AS p
USING #tmpPeople AS t
ON p.id = t.id
WHEN NOT MATCHED BY TARGET THEN
-- We want to insert new person
INSERT (id, name) VALUES (t.id, t.name)
WHEN MATCHED THEN
-- you may not need this, assume updating name for example
UPDATE SET p.name = t.name
END
Now we have the SQL in place.
Let us create the bulk of data in C#:
DataTable ppl = new DataTable();
ppl.Columns.Add("id", typeof(int));
ppl.Columns.Add("name", typeof(string));
// table is created, let's add some people
var bob = ppl.NewRow();
bob["id"] = 1;
bob["name"] = "Bob";
ppl.Rows.Add(bob);
var jim = ppl.NewRow();
jim["id"] = 2;
jim["name"] = "Jim";
ppl.Rows.Add(jim);
// that's enough people for now, let's call the stored procedure
using(var conn = new SqlConnection("YouConnStringHere"))
{
using(var cmd = new SqlCommand("SaveNewPeople", conn))
{
cmd.CommandType = CommandType.StoredProcedure;
var tvp = new SqlParameter
{
ParameterName = "#pPeople",
SqlDbType = SqlDbType.Structured,
Value = ppl,
TypeName = "udt_person"
}
cmd.Parameters.Add(tvp);
conn.Open();
cmd.ExecuteNonQuery();
}
}
Hopefully this gives you the idea. If you then modified the C# datatable, you should see rows inserted, updated or ignored.
Good luck.

Another way to do it is to create a database trigger to replace the inserts initiated by SqlBulkCopy. The performance will be impeded, depending on, among other things, the size of the batch, but it works nonetheless.
CREATE TABLE [dbo].[TempTable] (
[Id] INT IDENTITY PRIMARY KEY,
[Val] NVARCHAR(20)
)
GO
CREATE OR ALTER TRIGGER [IgnoreDuplicates] ON [dbo].[TempTable]
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON
INSERT INTO [dbo].[TempTable]([Val])
SELECT [Val] FROM [INSERTED] WHERE [Val] NOT IN (
SELECT [Val] FROM [dbo].[TempTable]
)
END
GO

Quote default values in create table SQL Server command

I want to write a C# method that creates a table and adds a column. The default values for the column is given as argument to the method. What is the safest way to quote the default value in the SQL command?
The code would be something like this:
var defaultValueThatOriginatesFromAnEvilSource = "'; DROP TABLE #Entities; SELECT '"; // the argument
database.RawExecute(string.Format("create table #Entities (somecolumn int default {0})", defaultValueThatOriginatesFromAnEvilSource));
Would a simple replace of the single quotes in the argument take care of all types of injection? Or is there a special method in C# or SQL I should use for this?
One thing to note, is that I create a temp table, and I need that to be accessible to the the next SQL statement (using the same connection).

As what shA.t said you can do this by sqlcommand in C# though it's quite hardwork if you'll be having multiple columns
but here's how to do it using stored procedure in SQL and sqlcommand in C#
in SQL
CREATE PROCEDURE sp_CreateTable
#TableName VARCHAR(50),
#ColumnName VARCHAR(50)
AS
BEGIN
DECLARE #SQLCreate VARCHAR(MAX)
SET #SQLCreate ='CREATE TABLE ' + #TableName
+ '('+#ColumnName+')'
EXEC (#SQLCreate)
END
GO
and in C#
add this using directives
using System.Data;
using System.Data.Client;
SqlConnection con = new SqlConnection("your connection string here");
con.Open();
SqlCommand cmd = new SqlCommand("sp_CreateTable");
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#TableName", SqlDbType.VarChar).Value = "someName";
cmd.Parameters.Add("#ColumnName", SqlDbType.VarChar).Value = "someColumnName";
cmd.ExecuteNonQuery();
con.Close();

Dynamically print number of rows updated

The company that I work for has large databases, millions of records in a single table. I have written a C# program that migrates tables between remote servers.
I first create all the tables using SMO without copying data and then the data insertion is done after all the tables have been created.
During the record insertion since there are so many records the console window remains blank until all the rows have been inserted. Due to the sheer volumes of data this takes a long time.
What I want now is a way to print n rows updated like in MSSQL import export data wizard.
The insert part is just a simple insert into select * query.

It sounds like you might be using SqlCommands, if so here is a sample
using (SqlConnection connection = new SqlConnection(Connection.ConnectionString) )
{
using(SqlCommand command = new SqlCommand("insert into OldCustomers select * from customers",connection))
{
connection.Open();
var numRows = command.ExecuteNonQuery();
Console.WriteLine("Affected Rows: {0}",numRows);
}
}

You definitely need to look on OUTPUT clause. There are useful examples on MSDN.
using (SqlConnection conn = new SqlConnection(connectionStr) )
{
var sqlCmd = "
CREATE TABLE #tmp (
InsertedId BIGINT
);
INSERT INTO TestTable
OUTPUT Inserted.Id INTO #tmp
VALUES ....
SELECT COUNT(*) FROM #tmp";
using(SqlCommand cmd = new SqlCommand(sqlCmd,conn))
{
conn .Open();
var numRows = command.ExecuteNonQuery();
Console.WriteLine("Affected Rows: {0}",numRows);
}
}
Also I suggest to use stored procedure for such purposes.

A better way to achieve INSERT without hitting the database multiple times

I have the following, I could make it work as I want to but I think i'm doing it the wrong way, could you please explain how this could be done in a more efficient way ? While also looping on Categories and doing the same as with Districts within the same Insert() Method.
Thanks in advance.
#region Methods
public int Insert(List<District> Districts, List<Category> Categories)
{
StringBuilder sqlString = new StringBuilder("INSERT INTO Stores (name, image) VALUES (#Name, #Image);");
using (SqlConnection sqlConnection = new
SqlConnection(ConfigurationManager.ConnectionStrings["OahuDB"].ConnectionString))
{
SqlCommand sqlCommand = new SqlCommand(sqlString.ToString(), sqlConnection);
sqlCommand.Parameters.AddWithValue("#Name", this.Name);
sqlCommand.Parameters.AddWithValue("#Image", this.Image);
sqlConnection.Open();
int x = (int)sqlCommand.ExecuteScalar();
sqlString.Clear();
sqlCommand.Parameters.Clear();
foreach (District item in Districts)
{
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
return x;
}
}
EDIT
Is is wrong to achieve the above by doing the following ?
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (District item in Districts)
{
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO categories_has_stores (category_id, store_id) VALUES (#CategoryID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (Category item in Categories)
{
sqlCommand.Parameters.AddWithValue("#CategoryID", item.ID);
sqlCommand.ExecuteNonQuery();
}

The first obvious thing is to move the invariant part of the sqlCommand out of the loop
sqlCommand.Parameters.Clear();
sqlString.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", 0); // as dummy value
sqlCommand.Parameters.AddWithValue("#StoreID", x); // invariant
foreach (District item in Districts)
{
sqlCommand.Parameters["#DistrictID"].Value = item.ID;
sqlCommand.ExecuteNonQuery();
}
But this doesn't answer your fundamental problem. How to avoid hitting the database multiple times.
You could build a query with multiple inserts like this
sqlString.Clear();
sqlString.Append("INSERT INTO districts_has_stores (district_id, store_id) VALUES (");
foreach(District item in Districts)
{
sqlString.Append(item.ID.ToString);
sqlString.Append(", ")
sqlString.Append(x.ToString());
sqlString.Append("),");
}
sqlString.Length--;
sqlCommand.CommandText = sqlString.ToString()
But string concatenation is really a bad practice and I present this solution just as an example and I don't want to suggest this kind of approach.
The last possibility are Table-Valued Parameters (Only from SqlServer 2008).
First you need to create a Sql Type for the table you will pass in
CREATE TYPE dbo.DistrictsType AS TABLE
( DistrictID int, StoreID int )
and a StoredProcedure that will insert the data from the datatable passed in
CREATE PROCEDURE usp_InsertDistricts
(#tvpNewDistricts dbo.DistrictsType READONLY)
AS
BEGIN
INSERT INTO dbo.Districts (DistrictID, StoreID)
SELECT dt.DistrictID, dt.StoreID FROM #tvpNewDistricts AS dt;
END
then, back to your code you pass the district into the storedprocedure
(Probably you need to convert your List in a DataTable)
DataTable dtDistricts = ConvertListToDataTable(Districts);
SqlCommand insertCommand = new SqlCommand("usp_InsertDistricts", sqlConnection);
SqlParameter p1 = insertCommand.Parameters.AddWithValue("#tvpNewDistricts", dtDistricts);
p1.SqlDbType = SqlDbType.Structured;
p1.TypeName = "dbo.DistrictsType";
insertCommand.ExecuteNonQuery();
Well, if you look back at the link above, you will find other ways to pass your data in a single step to the database backend.... (Scroll to the end and you will find also a method that doesn't require a stored procedure on the database)

Assuming Stores has an identity column, in SQL Server, create a table type and a table-valued parameter to take advantage of it:
CREATE TYPE dbo.DistrictsTVP AS TABLE
(
DistrictID INT -- PRIMARY KEY? I hope so.
);
GO
CREATE PROCEDURE dbo.InsertStoreAndDistricts
#Name NVARCHAR(255),
#Image <some data type???>,
#Districts dbo.DistrictsTVP READONLY
AS
BEGIN
SET NOCOUNT ON;
DECLARE #StoreID INT;
INSERT dbo.Stores(name, [image]) SELECT #Name, #Image;
SET #StoreID = SCOPE_IDENTITY();
INSERT dbo.district_has_stores(district_id, store_id)
SELECT DistrictID, #StoreID
FROM #Districts;
END
GO
Then in C#, you can pass your List in directly without any looping:
using (...)
{
SqlCommand cmd = new SqlCommand("dbo.InsertStoreAndDistricts", sqlConnection);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#Districts", Districts);
tvparam.SqlDbType    = SqlDbType.Structured;
// other params here - name and image
cmd.ExecuteNonQuery();
}

Recently in my project i used XML as a data type in my stored proc and did insert update and delete in just one shot instead of hitting the database multiple times .
Sample Stored proc
ALTER PROCEDURE [dbo].[insertStore]
#XMLDATA xml,
#name varchar(50),
#image datatype
AS
Begin
INSERT INTO Store
(name
,image
)
Select XMLDATA.item.value('#name[1]', 'varchar(10)') AS Name,
XMLDATA.item.value('#image[1]', 'yourData type') AS Image
FROM #XMLDATA.nodes('//Stores/InsertList/Store') AS XMLDATA(item)
END
Similarly you can write for update and delete .In C# u need to create the xml
public string GenerateXML(List<District> Districts)
var xml = new StringBuilder();
var insertxml = new StringBuilder();
xml.Append("<Stores>");
for (var i = 0; i < Districts.Count; i++)
{ var obj = Districts[i];
insertxml.Append("<Store");
insertxml.Append(" Name=\"" + obj.Name + "\" ");
insertxml.Append(" Image=\"" + obj.Image + "\" ");
insertxml.Append(" />");
}
xml.Append("<InsertList>");
xml.Append(insertxml.ToString());
xml.Append("</InsertList>");
SqlCommand cmd= new SqlCommand("insertStore",connectionString);
cmd.CommandType=CommandType.StoredProcedure;
SqlParameter param = new SqlParameter ();
param.ParameterName ="#XMLData";
param.value=xml;
paramter.Add(param);
cmd.ExecuteNonQuery();

Personally, I would create a stored procedure for the insert and pass in a Table-Valued param, which would allow you to do
INSERT tbl (f1, f2, ... fN)
SELECT * FROM #TVP
http://msdn.microsoft.com/en-us/library/bb510489.aspx
Unless you're using SQL 2005, then I would use an XML param in my stored proc and Serialize a collection to be inserted.

Think about your system design. Where is the data that you need to insert coming from? If it's already in the database, or another database, or some other kind of data store, you should be able to achieve a more bulk kind of transfer, simply inserting from one database to the other in a loop in stored procedure.
If the data is coming from a user, or some incompatible data store, like say an export from some third party program, then you basically have to realize that to get it into the database will involve quite of few round-trips to the database. You can use some tables, or XML or such , but those are actually closer to doing a bulk insert using other methods.
The bottom line is that SQL databases are designed to do inserts one at a time. This is 99% of the time OK because you are never asking users using the UI to type in thousands of things at one time.

C# SQL Server - Passing a list to a stored procedure

I am calling a SQL Server stored procedure from my C# code:
using (SqlConnection conn = new SqlConnection(connstring))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand("InsertQuerySPROC", conn))
{
cmd.CommandType = CommandType.StoredProcedure;
var STableParameter = cmd.Parameters.AddWithValue("#QueryTable", QueryTable);
var NDistanceParameter = cmd.Parameters.AddWithValue("#NDistanceThreshold", NDistanceThreshold);
var RDistanceParameter = cmd.Parameters.AddWithValue(#"RDistanceThreshold", RDistanceThreshold);
STableParameter .SqlDbType = SqlDbType.Structured;
NDistanceParameter.SqlDbType = SqlDbType.Int;
RDistanceParameter.SqlDbType = SqlDbType.Int;
// Execute the query
SqlDataReader QueryReader = cmd.ExecuteReader();
My stored proc is fairly standard but does a join with QueryTable (hence the need for for using a stored proc).
Now: I want to add a list of strings, List<string>, to the parameter set. For example, my stored proc query goes like this:
SELECT feature
FROM table1 t1
INNER JOIN #QueryTable t2 ON t1.fid = t2.fid
WHERE title IN <LIST_OF_STRINGS_GOES_HERE>
However, the list of strings is dynamic and a few hundred long.
Is there a way to pass a list of strings List<string> to the stored proc??? Or is there a better way to do this?
Many thanks,
Brett

If you're using SQL Server 2008, there's a new featured called a User Defined Table Type. Here is an example of how to use it:
Create your User Defined Table Type:
CREATE TYPE [dbo].[StringList] AS TABLE(
[Item] [NVARCHAR](MAX) NULL
);
Next you need to use it properly in your stored procedure:
CREATE PROCEDURE [dbo].[sp_UseStringList]
#list StringList READONLY
AS
BEGIN
-- Just return the items we passed in
SELECT l.Item FROM #list l;
END
Finally here's some sql to use it in c#:
using (var con = new SqlConnection(connstring))
{
con.Open();
using (SqlCommand cmd = new SqlCommand("exec sp_UseStringList #list", con))
{
using (var table = new DataTable()) {
table.Columns.Add("Item", typeof(string));
for (int i = 0; i < 10; i++)
table.Rows.Add("Item " + i.ToString());
var pList = new SqlParameter("#list", SqlDbType.Structured);
pList.TypeName = "dbo.StringList";
pList.Value = table;
cmd.Parameters.Add(pList);
using (var dr = cmd.ExecuteReader())
{
while (dr.Read())
Console.WriteLine(dr["Item"].ToString());
}
}
}
}
To execute this from SSMS
DECLARE #list AS StringList
INSERT INTO #list VALUES ('Apple')
INSERT INTO #list VALUES ('Banana')
INSERT INTO #list VALUES ('Orange')
-- Alternatively, you can populate #list with an INSERT-SELECT
INSERT INTO #list
SELECT Name FROM Fruits
EXEC sp_UseStringList #list

The typical pattern in this situation is to pass the elements in a comma delimited list, and then in SQL split that out into a table you can use. Most people usually create a specified function for doing this like:
INSERT INTO <SomeTempTable>
SELECT item FROM dbo.SplitCommaString(#myParameter)
And then you can use it in other queries.

No, arrays/lists can't be passed to SQL Server directly.
The following options are available:
Passing a comma-delimited list and then having a function in SQL split the list. The comma delimited list will most likely be passed as an Nvarchar()
Pass xml and have a function in SQL Server parse the XML for each value in the list
Use the new defined User Defined table type (SQL 2008)
Dynamically build the SQL and pass in the raw list as "1,2,3,4" and build the SQL statement. This is prone to SQL injection attacks, but it will work.

Yep, make Stored proc parameter as VARCHAR(...)
And then pass comma separated values to a stored procedure.
If you are using Sql Server 2008 you can leverage TVP (Table Value Parameters): SQL 2008 TVP and LINQ if structure of QueryTable more complex than array of strings otherwise it would be an overkill because requires table type to be created within SQl Server

Make a datatable with one column instead of List and add strings to the table. You can pass this datatable as structured type and perform another join with title field of your table.

If you prefer splitting a CSV list in SQL, there's a different way to do it using Common Table Expressions (CTEs). See Efficient way to string split using CTE.

The only way I'm aware of is building CSV list and then passing it as string. Then, on SP side, just split it and do whatever you need.

CREATE TYPE [dbo].[StringList1] AS TABLE(
[Item] [NVARCHAR](MAX) NULL,
[counts][nvarchar](20) NULL);
create a TYPE as table and name it as"StringList1"
create PROCEDURE [dbo].[sp_UseStringList1]
#list StringList1 READONLY
AS
BEGIN
-- Just return the items we passed in
SELECT l.item,l.counts FROM #list l;
SELECT l.item,l.counts into tempTable FROM #list l;
End
The create a procedure as above and name it as "UserStringList1"
s
String strConnection = ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString.ToString();
SqlConnection con = new SqlConnection(strConnection);
con.Open();
var table = new DataTable();
table.Columns.Add("Item", typeof(string));
table.Columns.Add("count", typeof(string));
for (int i = 0; i < 10; i++)
{
table.Rows.Add(i.ToString(), (i+i).ToString());
}
SqlCommand cmd = new SqlCommand("exec sp_UseStringList1 #list", con);
var pList = new SqlParameter("#list", SqlDbType.Structured);
pList.TypeName = "dbo.StringList1";
pList.Value = table;
cmd.Parameters.Add(pList);
string result = string.Empty;
string counts = string.Empty;
var dr = cmd.ExecuteReader();
while (dr.Read())
{
result += dr["Item"].ToString();
counts += dr["counts"].ToString();
}
in the c#,Try this