We Keep Coding

Get many rows from a list of IDs

I'm using C# and SQL Server. I have a list of IDs for documents which corresponds to the primary key for a table in SQL Server that has a row for each document and the row contains (among other things) the ID and the document for that ID. I want to get the document in the row for each of the IDs. Currently, I execute a query for each ID, but since there are 10,000s of them, this runs a ton of queries and takes a very long time. It ends up being faster to simply load everything from the table into memory and then filter by the ids I have, but that seems inefficient and won't scale over time. If that doesn't make sense, hopefully the following code that takes a long time to run shows what I'm trying to do.
private static Dictionary<Guid, string> foo(IEnumerable<Guid> guids, SqlConnection conn)
{
using (SqlCommand command = new SqlCommand(null, conn))
{
command.CommandText = "select document from Documents where id = #id";
SqlParameter idParam = new SqlParameter("#id", SqlDbType.UniqueIdentifier);
command.Parameters.Add(idParam);
command.Prepare();
var documents = new Dictionary<Guid, string>();
foreach (var guid in guids)
{
idParam.Value = guid;
object obj = command.ExecuteScalar();
if (obj != null)
{
documents[guid] = (string)obj;
}
}
return documents;
}
}
I could programmatically construct query strings to use where clause like this: ".... where id in (ID1, ID2, ID3, ..., ID100)" to get 100 documents at a time or something like that, but this feels janky and it seems to me like there's got to be a better way.
I'm sure I'm not the only one to run into this. Is there an accepted way to go about this?

You can use Table-Valued Parameters with no limits in amount of guids
In the code you will create SqlParameter with all Id's you need to
First you need create type of parameter in the sql server
CREATE TYPE IdTableType AS TABLE
(
Id uniqueidentifier
);
Then in the code
private static Dictionary<Guid, string> foo(IEnumerable<Guid> guids, SqlConnection conn)
{
using (SqlCommand command = new SqlCommand(null, conn))
{
// use parameter as normal table in the query
command.CommandText =
"select document from Documents d inner join #AllIds a ON d.id = a.Id";
// DataTable is used for Table-Valued parameter as value
DataTable allIds = new DataTable();
allIds.Columns.Add("Id"); // Name of column need to be same as in created Type
foreach(var id in guids)
allids.Rows.Add(id);
SqlParameter idParam = new SqlParameter
{
ParameterName = "#AllIds",
SqlDbType=SqlDbType.Structured // Important for table-valued parameters
TypeName = "IdTableType", // Important! Name of the type must be provided
Value = allIds
};
command.Parameters.Add(idParam);
var documents = new Dictionary<Guid, string>();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
documents[guid] = reader[0].ToString();
}
}
return documents;
}
}
You don't need to prepare the command any more. Besides after first execution next queries will use same compiled query plan, because query text remain same.

You can bunch them into sets of ids and pass a table valued parameter into the query. With Dapper this looks a bit like:
connection.Query("select document from Documents where id in #ids", new { ids = guids});
BEWARE though theres an 8000 parameter limit in sql so you will need to batch up your reads.
btw.. I'd highly recommend looking at Dapper or another micro orm for this type of data access.

Convert C# SQL Loop to Linq

I have a list Called ListTypes that holds 10 types of products. Below the store procedure loops and gets every record with the product that is looping and it stores it in the list ListIds. This is killing my sql box since I have over 200 users executing this constantly all day.
I know is not a good architecture to loop a sql statement, but this the only way I made it work. Any ideas how I can make this without looping? Maybe a Linq statement, I never used Linq with this magnitude. Thank you.
protected void GetIds(string Type, string Sub)
{
LinkedIds.Clear();
using (SqlConnection cs = new SqlConnection(connstr))
{
for (int x = 0; x < ListTypes.Count; x++)
{
cs.Open();
SqlCommand select = new SqlCommand("spUI_LinkedIds", cs);
select.CommandType = System.Data.CommandType.StoredProcedure;
select.Parameters.AddWithValue("#Type", Type);
select.Parameters.AddWithValue("#Sub", Sub);
select.Parameters.AddWithValue("#TransId", ListTypes[x]);
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
ListIds.Add(Convert.ToInt32(dr["LinkedId"]));
}
cs.Close();
}
}
}

Not a full answer, but this wouldn't fit in a comment. You can at least update your existing code to be more efficient like this:
protected List<int> GetIds(string Type, string Sub, IEnumerable<int> types)
{
var result = new List<int>();
using (SqlConnection cs = new SqlConnection(connstr))
using (SqlCommand select = new SqlCommand("spUI_LinkedIds", cs))
{
select.CommandType = System.Data.CommandType.StoredProcedure;
//Don't use AddWithValue! Be explicit about your DB types
// I had to guess here. Replace with the actual types from your database
select.Parameters.Add("#Type", SqlDBType.VarChar, 10).Value = Type;
select.Parameters.Add("#Sub", SqlDbType.VarChar, 10).Value = Sub;
var TransID = select.Parameters.Add("#TransId", SqlDbType.Int);
cs.Open();
foreach(int type in types)
{
TransID.Value = type;
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
result.Add((int)dr["LinkedId"]);
}
}
}
return result;
}
Note that this way you only open and close the connection once. Normally in ADO.Net it's better to use a new connection and re-open it for each query. The exception is in a tight loop like this. Also, the only thing that changes inside the loop this way is the one parameter value. Finally, it's better to design methods that don't rely on other class state. This method no longer needs to know about the ListTypes and ListIds class variables, which makes it possible to (among other things) do better unit testing on the method.
Again, this isn't a full answer; it's just an incremental improvement. What you really need to do is write another stored procedure that accepts a table valued parameter, and build on the query from your existing stored procedure to JOIN with the table valued parameter, so that all of this will fit into a single SQL statement. But until you share your stored procedure code, this is about as much help as I can give you.

Besides the improvements others wrote.
You could insert your ID's into a temp table and then make one
SELECT * from WhatEverTable WHERE transid in (select transid from #tempTable)
On a MSSQL this works really fast.
When you're not using a MSSQL it could be possible that one great SQL-Select with joins is faster than a SELECT IN. You have to test these cases by your own on your DBMS.

According to your comment:
The idea is lets say I have a table and I have to get all records from the table that has this 10 types of products. How can I get all of this products? But this number is dynamic.
So... why use a stored procedure at all? Why not query the table?
//If [Type] and [Sub] arguments are external inputs - as in, they come from a user request or something - they should be sanitized. (remove or escape '\' and apostrophe signs)
//create connection
string queryTmpl = "SELECT LinkedId FROM [yourTable] WHERE [TYPE] = '{0}' AND [SUB] = '{1}' AND [TRANSID] IN ({2})";
string query = string.Format(queryTmpl, Type, Sub, string.Join(", ", ListTypes);
SqlCommand select = new SqlCommand(query, cs);
//and so forth
To use Linq-to-SQL you would need to map the table to a class. This would make the query simpler to perform.

SqlCommand AddWithValue not working properly

I have the following block of code... fieldCount always returns 0.
I suspect AddWithValue isn't forming the SELECT statement properly... Any ideas why? As you can see from the Watch, they (field, fieldId) have valid values.
public void deleteRows(string table, string field, string fieldId)
{
int fieldCount;
using (SqlCommand command = new SqlCommand(String.Format("SELECT COUNT(*) FROM {0} WHERE #field IN (#fieldId)", table), connection))
{
command.Parameters.AddWithValue("#field", field);
command.Parameters.AddWithValue("#fieldId", fieldId);
fieldCount = (int)command.ExecuteScalar();
}
if (fieldCount > 0)
{

There are two errors in your code:
First, you cannot use a parameter to express a table name or a column name. So your field parameter is not valid in this context.
Second, you cannot use a parameter to express the whole set of values for the IN clause. In your example the parameter #fieldID will be translated in
WHERE ..... IN ('1,2,3,4')
It will be treated as a string not as the individual values to be included in your where context-
For the field part, if you are absolutely sure that the string field parameter passed to your method is not typed directly by your user then you could use a string concatenation expression (well you are already doing this for the table so the warning is good also for that value)
String.Format("SELECT COUNT(*) FROM {0} WHERE {1} IN (....)", table, field);
For the IN part, I suggest, instead of passing the string, to build, in the calling function, a list of parameters to be added at the query.
public void deleteRows(string table, string field, List<SqlParameter> inParameters)
{
StringBuilder sb new StringBuilder();
sb.AppendFormat("SELECT COUNT(*) FROM {0} WHERE {1} IN (", table, field));
using(SqlCommand cmd = new SqlCommand())
{
cmd.Connection = connection;
// Loop over the parameter list, adding the parameter name to the
// IN clause and the parameter to the SqlCommand collection
foreach(SqlParameter p in inParameters)
{
sb.Append(p.Name + ",");
cmd.Parameters.Add(p);
}
// Trim out the last comma
sb.Length--;
// Close the IN clause
sn.Append(")";
cmd.CommandText = sb.ToString();
fieldCount = (int)command.ExecuteScalar();
}
}

How to pass sqlparameter to IN()? [duplicate]

This question already has answers here:
Pass Array Parameter in SqlCommand
(11 answers)
Closed 6 years ago.
For some reason the Sqlparameter for my IN() clause is not working. The code compiles fine, and the query works if I substitute the parameter with the actual values
StringBuilder sb = new StringBuilder();
foreach (User user in UserList)
{
sb.Append(user.UserId + ",");
}
string userIds = sb.ToString();
userIds = userIds.TrimEnd(new char[] { ',' });
SELECT userId, username
FROM Users
WHERE userId IN (#UserIds)

You have to create one parameter for each value that you want in the IN clause.
The SQL needs to look like this:
SELECT userId, username
FROM Users
WHERE userId IN (#UserId1, #UserId2, #UserId3, ...)
So you need to create the parameters and the IN clause in the foreach loop.
Something like this (out of my head, untested):
StringBuilder sb = new StringBuilder();
int i = 1;
foreach (User user in UserList)
{
// IN clause
sb.Append("#UserId" + i.ToString() + ",");
// parameter
YourCommand.Parameters.AddWithValue("#UserId" + i.ToString(), user.UserId);
i++;
}

Possible "cleaner" version:
StringBuilder B = new StringBuilder();
for (int i = 0; i < UserList.Count; i++)
YourCommand.Parameters.AddWithValue($"#UserId{i}", UserList[i].UserId);
B.Append(String.Join(",", YourCommand.Parameters.Select(x => x.Name)));

If you are using SQL 2008, you can create a stored procedure which accepts a Table Valued Parameter (TVP) and use ADO.net to execute the stored procedure and pass a datatable to it:
First, you need to create the Type in SQL server:
CREATE TYPE [dbo].[udt_UserId] AS TABLE(
[UserId] [int] NULL
)
Then, you need to write a stored procedure which accepts this type as a parameter:
CREATE PROCEDURE [dbo].[usp_DoSomethingWithTableTypedParameter]
(
#UserIdList udt_UserId READONLY
)
AS
BEGIN
SELECT userId, username
FROM Users
WHERE userId IN (SELECT UserId FROM #UserIDList)
END
Now from .net, you cannot use LINQ since it does not support Table Valued Parameters yet; so you have to write a function which does plain old ADO.net, takes a DataTable, and passes it to the stored procedure: I've written a generic function I use which can do this for any stored procedure as long as it takes just the one table-typed parameter, regardless of what it is;
public static int ExecStoredProcWithTVP(DbConnection connection, string storedProcedureName, string tableName, string tableTypeName, DataTable dt)
{
using (SqlConnection conn = new SqlConnection(connection.ConnectionString))
{
SqlCommand cmd = new SqlCommand(storedProcedureName, conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter p = cmd.Parameters.AddWithValue(tableName, dt);
p.SqlDbType = SqlDbType.Structured;
p.TypeName = tableTypeName;
conn.Open();
int rowsAffected = cmd.ExecuteNonQuery(); // or could execute reader and pass a Func<T> to perform action on the datareader;
conn.Close();
return rowsAffected;
}
}
Then you can write DAL functions which use this utility function with actual names of stored procedures; to build on the example in your question, here is what the code would look like:
public int usp_DoSomethingWithTableTypedParameter(List<UserID> userIdList)
{
DataTable dt = new DataTable();
dt.Columns.Add("UserId", typeof(int));
foreach (var userId in updateList)
{
dt.Rows.Add(new object[] { userId });
}
int rowsAffected = ExecStoredProcWithTVP(Connection, "usp_DoSomethingWithTableTypedParameter", "#UserIdList", "udt_UserId", dt);
return rowsAffected;
}
Note the "connection" parameter above - I actually use this type of function in a partial DataContext class to extend LINQ DataContext with my TVP functionality, and still use the (using var context = new MyDataContext()) syntax with these methods.
This will only work if you are using SQL Server 2008 - hopefully you are and if not, this could be a great reason to upgrade! Of course in most cases and large production environments this is not that easy, but FWIW I think this is the best way of doing this if you have the technology available.

SQL Server sees your IN clause as:
IN ('a,b,c')
What it needs to look like is:
IN ('a','b','c')
There is a better way to do what you're trying to do.
If the user id's are in the DB, then the IN clause should be changed to a subquery, like so:
IN (SELECT UserID FROM someTable WHERE someConditions)
This is a hack -- it doesn't work well with indexes, and you have to be careful it works right with your data, but I've used it successfully in the past:
#UserIDs LIKE '%,' + UserID + ',%' -- also requires #UserID to begin and end with a comma

OleDbParameters and Parameter Names

I have an SQL statement that I'm executing through OleDb, the statement is something like this:
INSERT INTO mytable (name, dept) VALUES (#name, #dept);
I'm adding parameters to the OleDbCommand like this:
OleDbCommand Command = new OleDbCommand();
Command.Connection = Connection;
OleDbParameter Parameter1 = new OleDbParameter();
Parameter1.OleDbType = OleDbType.VarChar;
Parameter1.ParamterName = "#name";
Parameter1.Value = "Bob";
OleDbParameter Parameter2 = new OleDbParameter();
Parameter2.OleDbType = OleDbType.VarChar;
Parameter2.ParamterName = "#dept";
Parameter2.Value = "ADept";
Command.Parameters.Add(Parameter1);
Command.Parameters.Add(Parameter2);
The problem I've got is, if I add the parameters to command the other way round, then the columns are populated with the wrong values (i.e. name is in the dept column and vice versa)
Command.Parameters.Add(Parameter2);
Command.Parameters.Add(Parameter1);
My question is, what is the point of the parameter names if parameters values are just inserted into the table in the order they are added command? The parameter names seems redundant?

The Problem is that OleDb (and Odbc too) does not support named parameters.
It only supports what's called positional parameters.
In other words: The name you give a parameter when adding it to the commands parameters list does not matter. It's only used internally by the OleDbCommand class so it can distinguish and reference the parameters.
What matters is the order in which you add the parameters to the list. It must be the same order as the parameters are referenced in the SQL statement via the question mark character (?).
But here is a solution that allows you to use named parameters in the SQL statement. It basically replaces all parameter references in the SQL statement with question marks and reorders the parameters list accordingly.
It works the same way for the OdbcCommand class, you just need to replace "OleDb" with "Odbc" in the code.
Use the code like this:
command.CommandText = "SELECT * FROM Contact WHERE FirstName = #FirstName";
command.Parameters.AddWithValue("#FirstName", "Mike");
command.ConvertNamedParametersToPositionalParameters();
And here is the code
public static class OleDbCommandExtensions
{
public static void ConvertNamedParametersToPositionalParameters(this OleDbCommand command)
{
//1. Find all occurrences of parameter references in the SQL statement (such as #MyParameter).
//2. Find the corresponding parameter in the commands parameters list.
//3. Add the found parameter to the newParameters list and replace the parameter reference in the SQL with a question mark (?).
//4. Replace the commands parameters list with the newParameters list.
var newParameters = new List<OleDbParameter>();
command.CommandText = Regex.Replace(command.CommandText, "(#\\w*)", match =>
{
var parameter = command.Parameters.OfType<OleDbParameter>().FirstOrDefault(a => a.ParameterName == match.Groups[1].Value);
if (parameter != null)
{
var parameterIndex = newParameters.Count;
var newParameter = command.CreateParameter();
newParameter.OleDbType = parameter.OleDbType;
newParameter.ParameterName = "#parameter" + parameterIndex.ToString();
newParameter.Value = parameter.Value;
newParameters.Add(newParameter);
}
return "?";
});
command.Parameters.Clear();
command.Parameters.AddRange(newParameters.ToArray());
}
}

Parameter NAMES are generic in the SQL support system (i.e. not OleDb specific). Pretty much ONLY OleDb / Odbc do NOT use them. They are there because OleDb is a specific implementation of the generic base classes.