We Keep Coding

Parameterize insert of multiple rows

Is there any way to parameterize an SQL INSERT statement (in C#), which inserts multiple rows? Currently I can think of only one way, to generate a statement for inserting mulitple rows, but that is quite open to SQL injection:
string sql = " INSERT INTO my_table"
+ " (a, b, c)"
+ " VALUES";
// Add each row of values to the statement
foreach (var item in collection) {
sql = sql
+ String.Format(" ({0}, {1}, {2}),",
aVal, bVal, cVal);
}
// Remove the excessive comma
sql = sql.Remove(sql.Length - 1);
What is the smarter/safer way to do this?

You could add paramaters inside the loop, like:
using (var comm = new SqlCommand()) {
var counter = 0;
foreach (var item in collection) {
sql = sql + String.Format(" (#a{0}, #b{0}, #c{0})," counter);
comm.Parameters.AddWithValue("#a" + counter, aVal);
comm.Parameters.AddWithValue("#b" + counter, bVal);
comm.Parameters.AddWithValue("#c" + counter, cVal);
counter++;
}
}
But I really wouldn't do a multi-row insert like this. IIRC the maximum amount of parameters in a query is about 2100, and this could get very big very fast. As you're looping through a collection anyway, you could just send it to the database in your loop, something like:
using (var con = new SqlConnection("connectionString here"))
{
con.Open();
var sql = "INSERT INTO my_table (a, b, c) VALUES (#a,#b,#c);"
using (var comm = new SqlCommand(sql, con))
{
comm.Parameters.Add("#a", SqlDbType.Int);
comm.Parameters.Add("#b", SqlDbType.NVarChar);
comm.Parameters.Add("#c", SqlDbType.Int);
foreach (var item in collection) {
{
comm.Parameters["#a"].Value = aVal;
comm.Parameters["#b"].Value = bVal;
comm.Parameters["#b"].Size = bVal.Length;
comm.Parameters["#c"].Value = cVal;
comm.ExecuteNonQuery();
}
}
}
The statement is prepared only once (and faster than a huge statement with 100's of parameters), and it doesn't fail all records when one record fails (add some exception handling for that). If you want to fail all when one record fails, you could wrap the thing up in a transaction.
Edit:
Ofcourse, when you regularly have to input 1000's of rows, this approach isn't the most efficient either, and your DBA might start to complain.
There are other approaches to this problem to remove the strain from the database: for example, create a stored procedure in your database that will insert the data from an xml document, or use Table Valued Parameters.
NYCdotNet wrote 2 nice blogs about these options, which I won't recreate here, but they're worth exploring (I'll paste some code below from the blog, as per guidelines, but credit where it's due: NYCdotNet)
XML document approach
Table Valued Parameters
The "meat" from the blog about TVP (in VB.NET but that shouldn't matter):
So I created this "generic" table-valued type:
CREATE TYPE dbo.UniqueIntegerList AS TABLE
(
TheInteger INT NOT NULL
PRIMARY KEY (TheInteger)
);
Creating the Save Stored Procedure
Next, I created a new stored procedure which would accept my new
Table-Valued Type as a parameter.
CREATE PROC DoTableValuedParameterInsert(#ProductIDs
dbo.UniqueIntegerList READONLY)
AS BEGIN
INSERT INTO ProductsAccess(ProductID)
SELECT TheInteger AS [ProductID]
FROM #ProductIDs;
END
In this procedure, I am passing in a parameter called #ProductIDs.
This is of type "dbo.UniqueIntegerList" which I just created in the
previous step. SQL Server looks at this and says "oh I know what this
is - this type is actually a table". Since it knows that the
UniqueIntegerList type is a table, I can select from it just like I
could select from any other table-valued variable. You have to mark
the parameter as READONLY because SQL 2008 doesn't support updating
and returning a passed table-valued parameter.
Creating the Save Routine
Then I had to create a new save routine on my business object that
would call the new stored procedure. The way you prepare the
Table-Valued parameter is to create a DataTable object with the same
column signature as the Table-Valued type, populate it, and then pass
it inside a SqlParameter object as SqlDbType.Structured.
Public Sub SaveViaTableValuedParameter()
'Prepare the Table-valued Parameter'
Dim objUniqueIntegerList As New DataTable
Dim objColumn As DataColumn =
objUniqueIntegerList.Columns.Add("TheInteger", _
System.Type.GetType("System.Int32"))
objColumn.Unique = True
'Populate the Table-valued Parameter with the data to save'
For Each Item As Product In Me.Values
objUniqueIntegerList.Rows.Add(Item.ProductID)
Next
'Connect to the DB and save it.'
Using objConn As New SqlConnection(DBConnectionString())
objConn.Open()
Using objCmd As New SqlCommand("dbo.DoTableValuedParameterInsert")
objCmd.CommandType = CommandType.StoredProcedure
objCmd.Connection = objConn
objCmd.Parameters.Add("ProductIDs", SqlDbType.Structured)
objCmd.Parameters(0).Value = objUniqueIntegerList
objCmd.ExecuteNonQuery()
End Using
objConn.Close()
End Using
End Sub

Get many rows from a list of IDs

I'm using C# and SQL Server. I have a list of IDs for documents which corresponds to the primary key for a table in SQL Server that has a row for each document and the row contains (among other things) the ID and the document for that ID. I want to get the document in the row for each of the IDs. Currently, I execute a query for each ID, but since there are 10,000s of them, this runs a ton of queries and takes a very long time. It ends up being faster to simply load everything from the table into memory and then filter by the ids I have, but that seems inefficient and won't scale over time. If that doesn't make sense, hopefully the following code that takes a long time to run shows what I'm trying to do.
private static Dictionary<Guid, string> foo(IEnumerable<Guid> guids, SqlConnection conn)
{
using (SqlCommand command = new SqlCommand(null, conn))
{
command.CommandText = "select document from Documents where id = #id";
SqlParameter idParam = new SqlParameter("#id", SqlDbType.UniqueIdentifier);
command.Parameters.Add(idParam);
command.Prepare();
var documents = new Dictionary<Guid, string>();
foreach (var guid in guids)
{
idParam.Value = guid;
object obj = command.ExecuteScalar();
if (obj != null)
{
documents[guid] = (string)obj;
}
}
return documents;
}
}
I could programmatically construct query strings to use where clause like this: ".... where id in (ID1, ID2, ID3, ..., ID100)" to get 100 documents at a time or something like that, but this feels janky and it seems to me like there's got to be a better way.
I'm sure I'm not the only one to run into this. Is there an accepted way to go about this?

You can use Table-Valued Parameters with no limits in amount of guids
In the code you will create SqlParameter with all Id's you need to
First you need create type of parameter in the sql server
CREATE TYPE IdTableType AS TABLE
(
Id uniqueidentifier
);
Then in the code
private static Dictionary<Guid, string> foo(IEnumerable<Guid> guids, SqlConnection conn)
{
using (SqlCommand command = new SqlCommand(null, conn))
{
// use parameter as normal table in the query
command.CommandText =
"select document from Documents d inner join #AllIds a ON d.id = a.Id";
// DataTable is used for Table-Valued parameter as value
DataTable allIds = new DataTable();
allIds.Columns.Add("Id"); // Name of column need to be same as in created Type
foreach(var id in guids)
allids.Rows.Add(id);
SqlParameter idParam = new SqlParameter
{
ParameterName = "#AllIds",
SqlDbType=SqlDbType.Structured // Important for table-valued parameters
TypeName = "IdTableType", // Important! Name of the type must be provided
Value = allIds
};
command.Parameters.Add(idParam);
var documents = new Dictionary<Guid, string>();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
documents[guid] = reader[0].ToString();
}
}
return documents;
}
}
You don't need to prepare the command any more. Besides after first execution next queries will use same compiled query plan, because query text remain same.

You can bunch them into sets of ids and pass a table valued parameter into the query. With Dapper this looks a bit like:
connection.Query("select document from Documents where id in #ids", new { ids = guids});
BEWARE though theres an 8000 parameter limit in sql so you will need to batch up your reads.
btw.. I'd highly recommend looking at Dapper or another micro orm for this type of data access.

handling large amounts of data to include in an oracle select statement

Recent bug report states that a method being called is crashing the service causing it to restart. After troubleshooting, the cause was found to be an obnoxious Oracle SQL call with thousands of strings passed. There is a collection of strings being passed to a method from an external service which often is more than 10,000 records. The original code used a where clause on the passed collection using the LIKE keyword, which I think is really, really bad.
public IList<ContainerState> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Join("OR", containerNumbers
.Select(item => string.Concat(" cntr_no LIKE '", item.SliceLeft(10), "%' ")))
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
Clarification of in house methods used which may be confusing:
DataBase.SelectQuery is an internal library method using generics which gets passed the sql string, a function to map the records to .NET objects, and the parameters being passed and returns an IEnumerable of Objects of type retuned by the Mapping function.
SliceLeft is an extension method from another internal helper library that just returns the first part of a string up to the number of characters specified by the parameter.
The reason that the LIKE statement was apparently used, is that the strings being passed and the strings in the database only are guaranteed to match the first 10 characters. Example ("XXXX000000-1" in the strings being passed should match a database record like "XXXX000000-8").
I believed that the IN clause using the SUBSTR would be more efficent than using multiple LIKE clauses and replaced the code with:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Format("SUBSTR(CNTR_NO, 1, 10) IN ({0}) ",
string.Join(",", containerNumbers.Select(item => string.Format("\'{0}\'", item.SliceLeft(10) ) ) )
)
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
This helped slightly, and there were fewer issues in my tests, but when there are huge amounts of records passed, there is still an exception thrown and core dumps occur, as the SQL is longer than the server can parse during these times. The DBA suggests saving all the strings being passed to a temporary table, and then joining against that temp table.
Given that advice, I changed the function to:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
#"
CREATE TABLE T1(cntr_num VARCHAR2(10));
DECLARE GLOBAL TEMPORARY TABLE SESSION.T1 NOT LOGGED;
INSERT INTO SESSION.T1 VALUES (:containerNumbers);
SELECT
DISTINCT cntr_no,
'_IT' cntr_state
FROM
tb_master
WHERE
cntr_seq = 0
AND cntr_state IN ({0})
AND adjustment <> :adjustment
AND SUBSTR(CTNR_NO, 1, 10) IN (SELECT CNTR_NUM FROM SESSION.T1);
";
var parameters = new
{
#containerNumbers = containerNumbers.Select( item => item.SliceLeft(10)).ToList()
};
return DataBase.SelectQuery(sql, MapRecordToContainerState, parameters).ToList();
}
Now I'm getting a "ORA-00900: invalid SQL statement". This is really frustrating, how can I properly write a SQL Statement that will put this list of strings into a temporary table and then use it in a SELECT Statement to return the list I need?

There are couple possible places could cause this error, it seams that the "DECLARE GLOBAL TEMPORARY" is a JAVA API, I don't think .net has this function. Please try "Create global temporary table" instead. And, I don't know whether your internal API could handle multiple SQLs in one select sql. As far as I know, ODP.net Command class can only execute one sql per call. Moreover, "create table" is a DDL, it therefore has its own transaction. I can't see any reason we should put them in the same sql to execute. Following is a sample code for ODP.net,
using (OracleConnection conn = new OracleConnection(BD_CONN_STRING))
{
conn.Open();
using (OracleCommand cmd = new OracleCommand("create global temporary table t1(id number(9))", conn))
{
// actually this should execute once only
cmd.ExecuteNonQuery();
}
using (OracleCommand cmd = new OracleCommand("insert into t1 values (1)", conn)) {
cmd.ExecuteNonQuery();
}
// customer table is a permenant table
using (OracleCommand cmd = new OracleCommand("select c.id from customer c, t1 tmp1 where c.id=tmp1.id", conn)) {
cmd.ExecuteNonQuery();
}
}

Convert C# SQL Loop to Linq

I have a list Called ListTypes that holds 10 types of products. Below the store procedure loops and gets every record with the product that is looping and it stores it in the list ListIds. This is killing my sql box since I have over 200 users executing this constantly all day.
I know is not a good architecture to loop a sql statement, but this the only way I made it work. Any ideas how I can make this without looping? Maybe a Linq statement, I never used Linq with this magnitude. Thank you.
protected void GetIds(string Type, string Sub)
{
LinkedIds.Clear();
using (SqlConnection cs = new SqlConnection(connstr))
{
for (int x = 0; x < ListTypes.Count; x++)
{
cs.Open();
SqlCommand select = new SqlCommand("spUI_LinkedIds", cs);
select.CommandType = System.Data.CommandType.StoredProcedure;
select.Parameters.AddWithValue("#Type", Type);
select.Parameters.AddWithValue("#Sub", Sub);
select.Parameters.AddWithValue("#TransId", ListTypes[x]);
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
ListIds.Add(Convert.ToInt32(dr["LinkedId"]));
}
cs.Close();
}
}
}

Not a full answer, but this wouldn't fit in a comment. You can at least update your existing code to be more efficient like this:
protected List<int> GetIds(string Type, string Sub, IEnumerable<int> types)
{
var result = new List<int>();
using (SqlConnection cs = new SqlConnection(connstr))
using (SqlCommand select = new SqlCommand("spUI_LinkedIds", cs))
{
select.CommandType = System.Data.CommandType.StoredProcedure;
//Don't use AddWithValue! Be explicit about your DB types
// I had to guess here. Replace with the actual types from your database
select.Parameters.Add("#Type", SqlDBType.VarChar, 10).Value = Type;
select.Parameters.Add("#Sub", SqlDbType.VarChar, 10).Value = Sub;
var TransID = select.Parameters.Add("#TransId", SqlDbType.Int);
cs.Open();
foreach(int type in types)
{
TransID.Value = type;
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
result.Add((int)dr["LinkedId"]);
}
}
}
return result;
}
Note that this way you only open and close the connection once. Normally in ADO.Net it's better to use a new connection and re-open it for each query. The exception is in a tight loop like this. Also, the only thing that changes inside the loop this way is the one parameter value. Finally, it's better to design methods that don't rely on other class state. This method no longer needs to know about the ListTypes and ListIds class variables, which makes it possible to (among other things) do better unit testing on the method.
Again, this isn't a full answer; it's just an incremental improvement. What you really need to do is write another stored procedure that accepts a table valued parameter, and build on the query from your existing stored procedure to JOIN with the table valued parameter, so that all of this will fit into a single SQL statement. But until you share your stored procedure code, this is about as much help as I can give you.

Besides the improvements others wrote.
You could insert your ID's into a temp table and then make one
SELECT * from WhatEverTable WHERE transid in (select transid from #tempTable)
On a MSSQL this works really fast.
When you're not using a MSSQL it could be possible that one great SQL-Select with joins is faster than a SELECT IN. You have to test these cases by your own on your DBMS.

According to your comment:
The idea is lets say I have a table and I have to get all records from the table that has this 10 types of products. How can I get all of this products? But this number is dynamic.
So... why use a stored procedure at all? Why not query the table?
//If [Type] and [Sub] arguments are external inputs - as in, they come from a user request or something - they should be sanitized. (remove or escape '\' and apostrophe signs)
//create connection
string queryTmpl = "SELECT LinkedId FROM [yourTable] WHERE [TYPE] = '{0}' AND [SUB] = '{1}' AND [TRANSID] IN ({2})";
string query = string.Format(queryTmpl, Type, Sub, string.Join(", ", ListTypes);
SqlCommand select = new SqlCommand(query, cs);
//and so forth
To use Linq-to-SQL you would need to map the table to a class. This would make the query simpler to perform.

How to pass sqlparameter to IN()? [duplicate]

This question already has answers here:
Pass Array Parameter in SqlCommand
(11 answers)
Closed 6 years ago.
For some reason the Sqlparameter for my IN() clause is not working. The code compiles fine, and the query works if I substitute the parameter with the actual values
StringBuilder sb = new StringBuilder();
foreach (User user in UserList)
{
sb.Append(user.UserId + ",");
}
string userIds = sb.ToString();
userIds = userIds.TrimEnd(new char[] { ',' });
SELECT userId, username
FROM Users
WHERE userId IN (#UserIds)

You have to create one parameter for each value that you want in the IN clause.
The SQL needs to look like this:
SELECT userId, username
FROM Users
WHERE userId IN (#UserId1, #UserId2, #UserId3, ...)
So you need to create the parameters and the IN clause in the foreach loop.
Something like this (out of my head, untested):
StringBuilder sb = new StringBuilder();
int i = 1;
foreach (User user in UserList)
{
// IN clause
sb.Append("#UserId" + i.ToString() + ",");
// parameter
YourCommand.Parameters.AddWithValue("#UserId" + i.ToString(), user.UserId);
i++;
}

Possible "cleaner" version:
StringBuilder B = new StringBuilder();
for (int i = 0; i < UserList.Count; i++)
YourCommand.Parameters.AddWithValue($"#UserId{i}", UserList[i].UserId);
B.Append(String.Join(",", YourCommand.Parameters.Select(x => x.Name)));

If you are using SQL 2008, you can create a stored procedure which accepts a Table Valued Parameter (TVP) and use ADO.net to execute the stored procedure and pass a datatable to it:
First, you need to create the Type in SQL server:
CREATE TYPE [dbo].[udt_UserId] AS TABLE(
[UserId] [int] NULL
)
Then, you need to write a stored procedure which accepts this type as a parameter:
CREATE PROCEDURE [dbo].[usp_DoSomethingWithTableTypedParameter]
(
#UserIdList udt_UserId READONLY
)
AS
BEGIN
SELECT userId, username
FROM Users
WHERE userId IN (SELECT UserId FROM #UserIDList)
END
Now from .net, you cannot use LINQ since it does not support Table Valued Parameters yet; so you have to write a function which does plain old ADO.net, takes a DataTable, and passes it to the stored procedure: I've written a generic function I use which can do this for any stored procedure as long as it takes just the one table-typed parameter, regardless of what it is;
public static int ExecStoredProcWithTVP(DbConnection connection, string storedProcedureName, string tableName, string tableTypeName, DataTable dt)
{
using (SqlConnection conn = new SqlConnection(connection.ConnectionString))
{
SqlCommand cmd = new SqlCommand(storedProcedureName, conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter p = cmd.Parameters.AddWithValue(tableName, dt);
p.SqlDbType = SqlDbType.Structured;
p.TypeName = tableTypeName;
conn.Open();
int rowsAffected = cmd.ExecuteNonQuery(); // or could execute reader and pass a Func<T> to perform action on the datareader;
conn.Close();
return rowsAffected;
}
}
Then you can write DAL functions which use this utility function with actual names of stored procedures; to build on the example in your question, here is what the code would look like:
public int usp_DoSomethingWithTableTypedParameter(List<UserID> userIdList)
{
DataTable dt = new DataTable();
dt.Columns.Add("UserId", typeof(int));
foreach (var userId in updateList)
{
dt.Rows.Add(new object[] { userId });
}
int rowsAffected = ExecStoredProcWithTVP(Connection, "usp_DoSomethingWithTableTypedParameter", "#UserIdList", "udt_UserId", dt);
return rowsAffected;
}
Note the "connection" parameter above - I actually use this type of function in a partial DataContext class to extend LINQ DataContext with my TVP functionality, and still use the (using var context = new MyDataContext()) syntax with these methods.
This will only work if you are using SQL Server 2008 - hopefully you are and if not, this could be a great reason to upgrade! Of course in most cases and large production environments this is not that easy, but FWIW I think this is the best way of doing this if you have the technology available.

SQL Server sees your IN clause as:
IN ('a,b,c')
What it needs to look like is:
IN ('a','b','c')
There is a better way to do what you're trying to do.
If the user id's are in the DB, then the IN clause should be changed to a subquery, like so:
IN (SELECT UserID FROM someTable WHERE someConditions)
This is a hack -- it doesn't work well with indexes, and you have to be careful it works right with your data, but I've used it successfully in the past:
#UserIDs LIKE '%,' + UserID + ',%' -- also requires #UserID to begin and end with a comma