We Keep Coding

Sql/Dapper: How do perform LIKE in WHERE clause for array of values?

I have a table with a lot of employees in it, every person has a Name column with their full name.
I then want to do a query similar to this when searching for people:
SELECT * FROM Employees WHERE Name LIKE '%' + #value1 + '%' AND Name LIKE '%' + #value2 +'%' AND so forth...
for an arbitrary array of values.
My Dapper code would look something like this:
public IEnumerable<Employee> Search(string[] words)
{
using var connection = CreateConnection();
connection.Query<Employee>("SELECT * etc.", words);
}
Is there ANY way to do this with SQL without resorting to string concatenation, and the risk of SQL Injection attacks that follows?
Caveat: I don't know how Dapper actually passes an array to the query, which limits my creative ideas for working around this :-D
And also: Changing the Table structure is, unfortunately, out of the question. And I'd rather avoid fetching every single person into .Net memory and doing the filtering there.

Is there ANY way to do this with SQL without resorting to string concatenation, and the risk of SQL Injection attacks that follows?
Because the set of where conditions is not fixed you will need to build the query dynamically. But that does not mean you cannot parameterise the query, you just build the parameter list alongside building the query. Each time a word from the list add to the condition and add a parameter.
As Dapper doesn't directly include anything that takes a collection of DbParameter, consider using ADO.NET to get an IDataReader and then Dappter's
IEnumerable<T> Parse<T>(this IDataReader reader)
for the mapping.
Such a builder would be very roughly
var n = 0;
for (criterion in cirteria) {
var cond = $"{crition.column} like #p{n}";
var p = new SqlPatameter($"#p{n}", $"%{crition.value}%";
conditions.Add(cond);
cmd.Parameters.Add(p);
}
var sql = "select whetever from table where " + String.Join(" and ", conditions);
cmd.CommandText = sql;
var reader = await cmd.ExecuteReaderAsync();
var res = reader.Parse<TResult>();

For performance reasons, it's much better to do this as a set-based operation.
You can pass through a datatable as a Table-Value Parameter, then join on that with LIKE as the condition. In this case you want all values to match, so you need a little bit of relational division.
First create your table type:
CREATE TYPE dbo.StringList AS TABLE (str varchar(100) NOT NULL);
Your SQL is as follows:
SELECT *
FROM Employees e
WHERE NOT EXISTS (SELECT 1
FROM #words w
WHERE e.Name NOT LIKE '%' + w.str + '%' ESCAPE '/' -- if you want to escape wildcards you need to add ESCAPE
);
Then you pass through the list as follows:
public IEnumerable<Employee> Search(string[] words)
{
var table = new DataTable{ Columns = {
{"str", typeof(string)},
} };
foreach (var word in words)
table.Rows.Add(SqlLikeEscape(word)); // make a function that escapes wildcards
using var connection = CreateConnection();
return connection.Query<Employee>(yourQueryHere, new
{
words = table.AsTableValuedParameter("dbo.StringList"),
});
}

Parameterize insert of multiple rows

Is there any way to parameterize an SQL INSERT statement (in C#), which inserts multiple rows? Currently I can think of only one way, to generate a statement for inserting mulitple rows, but that is quite open to SQL injection:
string sql = " INSERT INTO my_table"
+ " (a, b, c)"
+ " VALUES";
// Add each row of values to the statement
foreach (var item in collection) {
sql = sql
+ String.Format(" ({0}, {1}, {2}),",
aVal, bVal, cVal);
}
// Remove the excessive comma
sql = sql.Remove(sql.Length - 1);
What is the smarter/safer way to do this?

You could add paramaters inside the loop, like:
using (var comm = new SqlCommand()) {
var counter = 0;
foreach (var item in collection) {
sql = sql + String.Format(" (#a{0}, #b{0}, #c{0})," counter);
comm.Parameters.AddWithValue("#a" + counter, aVal);
comm.Parameters.AddWithValue("#b" + counter, bVal);
comm.Parameters.AddWithValue("#c" + counter, cVal);
counter++;
}
}
But I really wouldn't do a multi-row insert like this. IIRC the maximum amount of parameters in a query is about 2100, and this could get very big very fast. As you're looping through a collection anyway, you could just send it to the database in your loop, something like:
using (var con = new SqlConnection("connectionString here"))
{
con.Open();
var sql = "INSERT INTO my_table (a, b, c) VALUES (#a,#b,#c);"
using (var comm = new SqlCommand(sql, con))
{
comm.Parameters.Add("#a", SqlDbType.Int);
comm.Parameters.Add("#b", SqlDbType.NVarChar);
comm.Parameters.Add("#c", SqlDbType.Int);
foreach (var item in collection) {
{
comm.Parameters["#a"].Value = aVal;
comm.Parameters["#b"].Value = bVal;
comm.Parameters["#b"].Size = bVal.Length;
comm.Parameters["#c"].Value = cVal;
comm.ExecuteNonQuery();
}
}
}
The statement is prepared only once (and faster than a huge statement with 100's of parameters), and it doesn't fail all records when one record fails (add some exception handling for that). If you want to fail all when one record fails, you could wrap the thing up in a transaction.
Edit:
Ofcourse, when you regularly have to input 1000's of rows, this approach isn't the most efficient either, and your DBA might start to complain.
There are other approaches to this problem to remove the strain from the database: for example, create a stored procedure in your database that will insert the data from an xml document, or use Table Valued Parameters.
NYCdotNet wrote 2 nice blogs about these options, which I won't recreate here, but they're worth exploring (I'll paste some code below from the blog, as per guidelines, but credit where it's due: NYCdotNet)
XML document approach
Table Valued Parameters
The "meat" from the blog about TVP (in VB.NET but that shouldn't matter):
So I created this "generic" table-valued type:
CREATE TYPE dbo.UniqueIntegerList AS TABLE
(
TheInteger INT NOT NULL
PRIMARY KEY (TheInteger)
);
Creating the Save Stored Procedure
Next, I created a new stored procedure which would accept my new
Table-Valued Type as a parameter.
CREATE PROC DoTableValuedParameterInsert(#ProductIDs
dbo.UniqueIntegerList READONLY)
AS BEGIN
INSERT INTO ProductsAccess(ProductID)
SELECT TheInteger AS [ProductID]
FROM #ProductIDs;
END
In this procedure, I am passing in a parameter called #ProductIDs.
This is of type "dbo.UniqueIntegerList" which I just created in the
previous step. SQL Server looks at this and says "oh I know what this
is - this type is actually a table". Since it knows that the
UniqueIntegerList type is a table, I can select from it just like I
could select from any other table-valued variable. You have to mark
the parameter as READONLY because SQL 2008 doesn't support updating
and returning a passed table-valued parameter.
Creating the Save Routine
Then I had to create a new save routine on my business object that
would call the new stored procedure. The way you prepare the
Table-Valued parameter is to create a DataTable object with the same
column signature as the Table-Valued type, populate it, and then pass
it inside a SqlParameter object as SqlDbType.Structured.
Public Sub SaveViaTableValuedParameter()
'Prepare the Table-valued Parameter'
Dim objUniqueIntegerList As New DataTable
Dim objColumn As DataColumn =
objUniqueIntegerList.Columns.Add("TheInteger", _
System.Type.GetType("System.Int32"))
objColumn.Unique = True
'Populate the Table-valued Parameter with the data to save'
For Each Item As Product In Me.Values
objUniqueIntegerList.Rows.Add(Item.ProductID)
Next
'Connect to the DB and save it.'
Using objConn As New SqlConnection(DBConnectionString())
objConn.Open()
Using objCmd As New SqlCommand("dbo.DoTableValuedParameterInsert")
objCmd.CommandType = CommandType.StoredProcedure
objCmd.Connection = objConn
objCmd.Parameters.Add("ProductIDs", SqlDbType.Structured)
objCmd.Parameters(0).Value = objUniqueIntegerList
objCmd.ExecuteNonQuery()
End Using
objConn.Close()
End Using
End Sub

handling large amounts of data to include in an oracle select statement

Recent bug report states that a method being called is crashing the service causing it to restart. After troubleshooting, the cause was found to be an obnoxious Oracle SQL call with thousands of strings passed. There is a collection of strings being passed to a method from an external service which often is more than 10,000 records. The original code used a where clause on the passed collection using the LIKE keyword, which I think is really, really bad.
public IList<ContainerState> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Join("OR", containerNumbers
.Select(item => string.Concat(" cntr_no LIKE '", item.SliceLeft(10), "%' ")))
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
Clarification of in house methods used which may be confusing:
DataBase.SelectQuery is an internal library method using generics which gets passed the sql string, a function to map the records to .NET objects, and the parameters being passed and returns an IEnumerable of Objects of type retuned by the Mapping function.
SliceLeft is an extension method from another internal helper library that just returns the first part of a string up to the number of characters specified by the parameter.
The reason that the LIKE statement was apparently used, is that the strings being passed and the strings in the database only are guaranteed to match the first 10 characters. Example ("XXXX000000-1" in the strings being passed should match a database record like "XXXX000000-8").
I believed that the IN clause using the SUBSTR would be more efficent than using multiple LIKE clauses and replaced the code with:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Format("SUBSTR(CNTR_NO, 1, 10) IN ({0}) ",
string.Join(",", containerNumbers.Select(item => string.Format("\'{0}\'", item.SliceLeft(10) ) ) )
)
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
This helped slightly, and there were fewer issues in my tests, but when there are huge amounts of records passed, there is still an exception thrown and core dumps occur, as the SQL is longer than the server can parse during these times. The DBA suggests saving all the strings being passed to a temporary table, and then joining against that temp table.
Given that advice, I changed the function to:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
#"
CREATE TABLE T1(cntr_num VARCHAR2(10));
DECLARE GLOBAL TEMPORARY TABLE SESSION.T1 NOT LOGGED;
INSERT INTO SESSION.T1 VALUES (:containerNumbers);
SELECT
DISTINCT cntr_no,
'_IT' cntr_state
FROM
tb_master
WHERE
cntr_seq = 0
AND cntr_state IN ({0})
AND adjustment <> :adjustment
AND SUBSTR(CTNR_NO, 1, 10) IN (SELECT CNTR_NUM FROM SESSION.T1);
";
var parameters = new
{
#containerNumbers = containerNumbers.Select( item => item.SliceLeft(10)).ToList()
};
return DataBase.SelectQuery(sql, MapRecordToContainerState, parameters).ToList();
}
Now I'm getting a "ORA-00900: invalid SQL statement". This is really frustrating, how can I properly write a SQL Statement that will put this list of strings into a temporary table and then use it in a SELECT Statement to return the list I need?

There are couple possible places could cause this error, it seams that the "DECLARE GLOBAL TEMPORARY" is a JAVA API, I don't think .net has this function. Please try "Create global temporary table" instead. And, I don't know whether your internal API could handle multiple SQLs in one select sql. As far as I know, ODP.net Command class can only execute one sql per call. Moreover, "create table" is a DDL, it therefore has its own transaction. I can't see any reason we should put them in the same sql to execute. Following is a sample code for ODP.net,
using (OracleConnection conn = new OracleConnection(BD_CONN_STRING))
{
conn.Open();
using (OracleCommand cmd = new OracleCommand("create global temporary table t1(id number(9))", conn))
{
// actually this should execute once only
cmd.ExecuteNonQuery();
}
using (OracleCommand cmd = new OracleCommand("insert into t1 values (1)", conn)) {
cmd.ExecuteNonQuery();
}
// customer table is a permenant table
using (OracleCommand cmd = new OracleCommand("select c.id from customer c, t1 tmp1 where c.id=tmp1.id", conn)) {
cmd.ExecuteNonQuery();
}
}

How to build a parameterised query with IN sql keyword?

That's what I tried & failed:
string sql = "... WHERE [personID] IN (#sqlIn) ...";
string sqlIn = "1,2,3,4,5";
SqlCeCommand cmd.Parameters.Add("#sqlIn", SqlDbType.NText).Value = sqlIn;
SqlCeDataAdapter da = new SqlCeDataAdapter(cmd);
da.Fill(ds); // > Error
Error details:
The ntext and image data types cannot be used in WHERE, HAVING, GROUP BY, ON, or IN clauses, except when these data types are used with the LIKE or IS NULL predicates.
Can't I pass all the IDs as one parameter? Should I add one by one all IDs?
P.S: Notice SqlCE

You can't parameterise that as a single parameter. Your query is doing an "in" on a single value, so is essentially:
... Where personId = '1,2,3,4,5'
(give or take a parameter). This is usually also an invalid or sub-optimal equality test, and certainly isn't what you were trying to query.
Options;
use raw concatenation: often involves a SQL injection risk, and allows poor query plan re-use
on full SQL server: use a UDF to split a single param
on full SQL server, use a TVP
add parameters dynamically, and add the various #param3 etc to the TSQL
The last is the most reliable, and "dapper-dot-net" has a feature built in to do this for you (since it is commonly needed):
int[] ids = ...
var rows = conn.Query<SomeType>(
#"... Where Id in #ids",
new { ids }).ToList();
This, when run via dapper-dot-net, will add a parameter per item in "ids", giving it the right value etc, and fixing the SQL so it executes, for example:
"... Where Id in (#ids0, #ids1, #ids2)"
(if there were 3 items in "ids")

You'll need to split the sqlIn string by comma, convert each to an integer, and build the IN statement manually.
string sqlIn = "1,2,3,4,5";
string inParams = sqlIn.Split(',');
List<string> paramNames = new List<string>();
for(var i = 0; i < inParams.Length; ++i){
string paramName = "#param" + i.ToString();
SqlCeCommand cmd.Parameters.Add(paramName, SqlDbType.Int).Value = int.Parse(inParams[i]);
paramNames.Add(paramName);
}
string sql = "... WHERE [personID] IN (" +
string.Join(",", paramNames) +
") ...";

String list in SqlCommand through Parameters in C#

Working with a SqlCommand in C# I've created a query that contains a IN (list...) part in the where clause. Instead of looping through my string list generating the list I need for the query (dangerous if you think in sqlInjection). I thought that I could create a parameter like:
SELECT blahblahblah WHERE blahblahblah IN #LISTOFWORDS
Then in the code I try to add a parameter like this:
DataTable dt = new DataTable();
dt.Columns.Add("word", typeof(string));
foreach (String word in listOfWords)
{
dt.Rows.Add(word);
}
comm.Parameters.Add("LISTOFWORDS", System.Data.SqlDbType.Structured).Value = dt;
But this doesn't work.
Questions:
Am I trying something impossible?
Did I took the wrong approach?
Do I have mistakes in this approach?
Thanks for your time :)

What you are trying to do is possible but not using your current approach. This is a very common problem with all possible solutions prior to SQL Server 2008 having trade offs related to performance, security and memory usage.
This link shows some approaches for SQL Server 2000/2005
SQL Server 2008 supports passing a table value parameter.
I hope this helps.

You want to think about where that list comes from. Generally that information is in the database somewhere. For example, instead of this:
SELECT * FROM [Table] WHERE ID IN (1,2,3)
You could use a subquery like this:
SELECT * FROM [Table] WHERE ID IN ( SELECT TableID FROM [OtherTable] WHERE OtherTableID= #OtherTableID )

If I understand right, you're trying to pass a list as a SQL parameter.
Some folks have attempted this before with limited success:
Passing Arrays to Stored Procedures
Arrays and Lists in SQL 2005
Passing Array of Values to SQL Server without String Manipulation
Using MS SQL 2005's XML capabilities to pass a list of values to a command

Am I trying something impossible?
No, it isn't impossible.
Did I took the wrong approach?
Your approach is not working (at least in .net 2)
Do I have mistakes in this approach?
I would try "Joel Coehoorn" solution (2nd answers) if it is possible.
Otherwise, another option is to send a "string" parameter with all values delimited by an separator. Write a dynamic query (build it based on values from string) and execute it using "exec".
Another solution will be o build the query directly from code. Somthing like this:
StringBuilder sb = new StringBuilder();
for (int i=0; i< listOfWords.Count; i++)
{
sb.AppendFormat("p{0},",i);
comm.Parameters.AddWithValue("p"+i.ToString(), listOfWords[i]);
}
comm.CommandText = string.Format(""SELECT blahblahblah WHERE blahblahblah IN ({0})",
sb.ToString().TrimEnd(','));
The command should look like:
SELECT blah WHERE blah IN (p0,p1,p2,p3...)...p0='aaa',p1='bbb'
In MsSql2005, "IN" is working only with 256 values.

I would recommend setting the parameter as a comma delimited string of values and use a Split function in SQL to turn that into a single column table of values and then you can use the IN feature.
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=50648 - Split Functions

If you want to pass the list as a string in a parameter, you could just build the query dynamically.
DECLARE #query varchar(500)
SET #query = 'SELECT blah blah WHERE blahblah in (' + #list + ')'
EXECUTE(#query)

I used to have the same problem, I think there is now way to do this directly over the ADO.NET API.
You might consider inserting the words into a temptable (plus a queryid or something) and then refering to that temptable from the query. Or dynamically creating the query string and avoid sql injection by other measures (e.g. regex checks).

This is an old question but I've come up with an elegant solution for this that I love to reuse and I think everyone else will find it useful.
First of all you need to create a FUNCTION in SqlServer that takes a delimited input and returns a table with the items split into records.
Here is the following code for this:
ALTER FUNCTION [dbo].[Split]
(
#RowData nvarchar(max),
#SplitOn nvarchar(5) = ','
)
RETURNS #RtnValue table
(
Id int identity(1,1),
Data nvarchar(100)
)
AS
BEGIN
Declare #Cnt int
Set #Cnt = 1
While (Charindex(#SplitOn,#RowData)>0)
Begin
Insert Into #RtnValue (data)
Select
Data = ltrim(rtrim(Substring(#RowData,1,Charindex(#SplitOn,#RowData)-1)))
Set #RowData = Substring(#RowData,Charindex(#SplitOn,#RowData)+1,len(#RowData))
Set #Cnt = #Cnt + 1
End
Insert Into #RtnValue (data)
Select Data = ltrim(rtrim(#RowData))
Return
END
You can now do something like this:
Select Id, Data from dbo.Split('123,234,345,456',',')
And fear not, this can't be susceptible to Sql injection attacks.
Next write a stored procedure that takes your comma delimited data and then you can write a sql statement that uses this Split function:
CREATE PROCEDURE [dbo].[findDuplicates]
#ids nvarchar(max)
as
begin
select ID
from SomeTable with (nolock)
where ID in (select Data from dbo.Split(#ids,','))
end
Now you can write a C# wrapper around it:
public void SomeFunction(List<int> ids)
{
var idsAsDelimitedString = string.Join(",", ids.Select(id => id.ToString()).ToArray());
// ... or however you make your connection
var con = GetConnection();
try
{
con.Open();
var cmd = new SqlCommand("findDuplicates", con);
cmd.Parameters.Add(new SqlParameter("#ids", idsAsDelimitedString));
var reader = cmd.ExecuteReader();
// .... do something here.
}
catch (Exception)
{
// catch an exception?
}
finally
{
con.Close();
}
}