We Keep Coding

SQL query on ADO.net limitation with 2100+ parameters

I am trying to implement an ADO.NET code which executes the SQL query with multiple parameters. Looks like SQL parameter limit is 2100 and does not accept more than this limit. How do I achieve with my below code to have this accept more than the limitation.
I am finding it difficult to understand the implementations when validating online articles related how to send the queries in subsets or chunks to fulfill my request.
This is my code:
using (Connection = new SqlConnection(CS))
{
Connection.Open();
string query = "SELECT FamilyID, FullName, Alias FROM TABLE (nolock) WHERE FamilyID IN ({0})";
var stringBuiler = new StringBuilder();
var familyIds = new List<string>();
string line;
while ((line = TextFileReader.ReadLine()) != null)
{
line = line.Trim();
if (!familyIds.Contains(line) & !string.IsNullOrEmpty(line))
{
familyIds.Add(line);
}
}
var sqlCommand = new SqlCommand
{
Connection = Connection,
CommandType = CommandType.Text
};
var index = 0; // Reset the index
var idParameterList = new List<string>();
foreach (var familyId in familyIds)
{
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
}
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
var dt = new DataTable();
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
{
dt.Load(sqlReader);
}
try
{
if (dt.Rows.Count > 0)
{
OutputdataGridView.DataSource = lstDownloadOwnerOutput;
OutputdataGridView.ColumnHeadersDefaultCellStyle.Font = new Font(DataGridView.DefaultFont, FontStyle.Bold);
OutputdataGridView.Columns[0].AutoSizeMode = DataGridViewAutoSizeColumnMode.AllCells;
Gridviewdisplaylabel.Text = "Total no of rows: " + this.OutputdataGridView.Rows.Count.ToString();
}
else if (dt.Rows.Count == 0)
{
MessageBox.Show("Data returned blank!!!");
}
}
catch (Exception Ex)
{
if (Connection != null)
{
Connection.Close();
}
MessageBox.Show(Ex.Message);
}
}

Having a WHERE IN clause with 2100, or even 100, parameters is generally not good coding practice. You might want to consider putting those values into a separate bona fide table, e.g.
families (ID int PK, ...)
Then, you may rewrite your query as:
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock)
WHERE FamilyID IN (SELECT ID FROM families);
You could also express the above using an EXISTS clause or a join, but all three approaches might just optimize to a very similar query plan anyway.

You can just add a table load call every 2000 parameters in your code:
var index = 0; // Reset the index
var idParameterList = new List<string>();
var dt = new DataTable();
foreach (var familyId in familyIds) {
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
if (index > 2000) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
sqlCommand.Parameters.Clear();
idParameterList.Clear();
index = 0;
}
}
if (index > 0) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
}

For dynamic sql like this, I generally recommend using a Table-Valued Parameter.
It does require a bit of setup: you have to create a user-defined Type in the DB to hold the values, but that is a fairly trivial operation:
CREATE TYPE PrimaryKeyType AS TABLE ( VALUE INT NOT NULL );
We generally use these in conjunction with stored procedures:
CREATE PROCEDURE dbo.getFamily(#PrimaryKeys PrimaryKeyType READONLY)
AS
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock) INNER JOIN #PrimaryKeys ON TABLE.FamilyID = #PrimaryKeys.Value
GO
However, you can also use inline SQL if you prefer.
Assigning the values to the stored proc or inline parameter is fairly straightforward, but there is one gotcha (more later):
public static void AssignValuesToPKTableTypeParameter(DbParameter parameter, ICollection<int> primaryKeys)
{
// Exceptions are handled by the caller
var sqlParameter = parameter as SqlParameter;
if (sqlParameter != null && sqlParameter.SqlDbType == SqlDbType.Structured)
{
// The type name may look like DatabaseName.dbo.PrimaryKeyType,
// so remove the database name if it is present
var parts = sqlParameter.TypeName.Split('.');
if (parts.Length == 3)
{
sqlParameter.TypeName = parts[1] + "." + parts[2];
}
}
if (primaryKeys == null)
{
primaryKeys = new List<int>();
}
var table = new DataTable();
table.Columns.Add("Value", typeof(int));
foreach (var wPrimaryKey in primaryKeys)
{
table.Rows.Add(wPrimaryKey);
}
parameter.Value = table;
}
The thing to watch out for here is the naming of the parameter. See the code in the method above that removes the database name to resolve this issue.
If you have dynamic SQL, you can generate a correct parameter using the following method:
public static SqlParameter CreateTableValuedParameter(string typeName, string parameterName)
{
// Exceptions are handled by the caller
var oParameter = new SqlParameter();
oParameter.ParameterName = parameterName;
oParameter.SqlDbType = SqlDbType.Structured;
oParameter.TypeName = typeName;
return oParameter;
}
Where typeName is the name of your type in the DB.

SQL query return List

I can't seem to get this working:
My table column headers are 'genre' 'artist' 'album'
and the params I'm passing in are (type, filter, value) ("artist", "genre", "Rock") where there are two rows in the db with "Rock" for the genre.
When I follow the debugger, the 'while (reader.Read())' must return false because the loop is never entered and thus nothing written to the List.
public static List<String> getWithFilter(String type, String filter, String value)
{
List<String> columnData = new List<String>();
string query = "SELECT #type FROM Music WHERE" +
" #filter = '#value'";
SqlConnection connection = Database.GetConnection();
SqlCommand getData = new SqlCommand(query, connection);
getData.Parameters.AddWithValue("#type", type);
getData.Parameters.AddWithValue("#filter", filter);
getData.Parameters.AddWithValue("#value", value);
connection.Open();
using (connection)
{
using (getData)
{
using (SqlDataReader reader = getData.ExecuteReader())
{
while (reader.Read())
{
columnData.Add(reader.GetString(0));
}
}
}
}
return columnData;
}

You cannot use parameters for the names of columns and you don't put quotes around them when using them. Right now your query is the equivalent of
SELECT 'artist' FROM Music WHERE 'genre' = '#value'
You can do the following instead.
string query = "SELECT " + type + " FROM Music WHERE " + filter + " = #value";
And just remove the lines that create the #type and #fitler parameters.

You're looking either for formatting or string interpolation (requires C# 6.0):
string query =
$#"SELECT {type}
FROM Music
WHERE {filter} = #value";
...
getData.Parameters.AddWithValue("#value", value);
Formatting is a bit more wordy:
string query = String.Format(
#"SELECT {0}
FROM Music
WHERE {1} = #value", type, filter);

I assuming that you're using .net 2
DateTime current = DateTime.Now;
Console.WriteLine(current);
SqlConnection conn = new SqlConnection();
string q = "SELECT #field FROM student";
SqlDataAdapter da = new SqlDataAdapter(q, conn);
da.SelectCommand.Parameters.AddWithValue("#field", "snName");
DataTable dt = new System.Data.DataTable();
conn.Open();
da.Fill(dt);
conn.Close();
List<string> names = new List<string>();
foreach (DataRow dr in dt.Rows)
{
names.Add(dr[0].ToString());
}
Console.WriteLine("Fetching {0} data for {1}", names.Count, DateTime.Now - current);
Console.ReadKey();
You can use lambda expression to mapping the datatable in .net >4

Must declare the scalar variable in Select command

I am passing a long list of employeeIds to employeeIdlist and I split them into a List. Using this list I am adding parameters to my query.
I am getting the following error
{"Must declare the scalar variable \"#EmployeeId\"."}
public List<versionInfo> GetVersion(string employeeIdlist)
{
DbHelper helper = new DbHelper();
List<versionInfo> empVerInfo = new List<versionInfo>();
using (SqlConnection conn = new SqlConnection(connString))
{
conn.Open();
using (SqlCommand getVersion = new SqlCommand())
{
getVersion.Connection = conn;
getVersion.CommandText = #"SELECT EmployeeId,Version
FROM [dbo].[EmployeeVersion]
WHERE EmployeeId in (#EmployeeId)";
getVersion.CommandType = CommandType.Text;
List<int> empIds = employeeIdlist.Split(',').Select(int.Parse).ToList();
StringBuilder sb = new StringBuilder();
int i = 0;
foreach (var emp in empIds)
{
// IN clause
sb.Append("#EmployeeId" + i.ToString() + ",");
// parameter
getVersion.Parameters.AddWithValue("#EmployeeId" + i.ToString(), emp);
i++;
}
// getVersion.Parameters.AddWithValue("#EmployeeId", employeeIdlist);
SqlDataReader rdr = getVersion.ExecuteReader();
while (rdr.Read())
{
versionInfo vi = new versionInfo();
vi.employeeId = helper.GetDb<int>(rdr, "EmployeeId");
vi.version = helper.GetDb<decimal>(rdr, "Version");
empVerInfo.Add(vi);
}
rdr.Close();
}
conn.Close();
}
return empVerInfo;
}

Remove the text after the IN
getVersion.CommandText = #"SELECT EmployeeId,Version
FROM [dbo].[EmployeeVersion]
WHERE EmployeeId in (";
then the foreach could build the full list of parameters and texts
foreach (var emp in empIds)
{
sb.Append("#EmployeeId" + i.ToString() + ",");
getVersion.Parameters.AddWithValue("#EmployeeId" + i.ToString(), emp);
i++;
}
after exiting the loop remove the last comma from the StringBuilder
sb.Length--;
finally, complete the command text appending the content of the StringBuilder and do not forget the closing parenthesys for the IN clause.
getVersion.CommandText += sb.ToString() + ")";
Now you can run the command with the correct IN clause and a matching list of parameters

If fails because your string query has one parameter named #EmployeeId and your Command object has many parameters with different names ("#EmployeeId1" is not equal to "#EmployeeId")
It seems like you are trying to apply this approach, which is a good idea.
You are two lines away of getting it to work:
Add this lines:
sb.Lenght--;
getVersion.CommandText = getVersion.CommandText.Replace("#EmployeeId",sb.ToString())
just before:
SqlDataReader rdr = getVersion.ExecuteReader();
After doing that your added parameters will match those #parameters existing in the sql string.

This is just another option. You can achieve the same result in 3 lines of code using Dapper ORM used in Stack Overflow.
You can download via NuGet.
public class VersionInfo
{
public int EmployeeId { get; set; }
public decimal Version { get; set; }
}
class Program
{
public static string connString = "...";
static void Main(string[] args)
{
var result = GetVersion(new List<int> {1, 2});
Console.ReadLine();
}
public static List<VersionInfo> GetVersion(IList<int> employeeIds)
{
using (IDbConnection conn = new SqlConnection(connString))
{
conn.Open();
var entities = conn.Query<VersionInfo>(
#"SELECT EmployeeId, Version from EmployeeVersion WHERE EmployeeId IN #EmployeeIds",
new {EmployeeIds = employeeIds});
return entities.ToList();
}
}
}

On your select statement you have to declare a value for your variable. I have made it an Integer. If it is a text value, then you can use varchar(25).
#"DECLARE #EmployeeId INT
SELECT EmployeeId,Version
FROM [dbo].[EmployeeVersion]
WHERE EmployeeId in (#EmployeeId)";

Get all data from an Oracle DB request

I want to create a general function in c# for sql requests at my oracle db.
I want to save all columns to strings. I tried to use the reader.GetString()method but an exception is thrown, saying it is unconvertable...
How can I make a function which does that ?
string [] request (OracleConnection con, String Table, String condition, String columns, String sort){
//The magic code
return "A string-array with all information";
}

Can you use the Item property on DBReader? Also, this method is a serious security risk -- using a condition string instead of params invites sql injection.
public string[] Request(IDBConnection conn, string table, string condition, string columns, string sort)
{
List<string> output = new List<string>();
string[] cols = columns.Split(',');
string sql = string.Format("select * from {0} etc", table);
using (IDBCommand cmd = new OracleCommand(conn, sql))
{
conn.Open();
IDBReader reader = cmd.ExecuteReader();
while (reader.Read())
{
foreach (string col in cols)
{
object field = reader.Item[col];
output.Add(field.ToString());
}
}
}
return output.ToArray();
}

How to generate SQL statement from List?

If I have a list of Strings, ie. List<String>, how can I generate a SQL statement such as:
SELECT Column1 FROM Table1 WHERE Column1 IN ('String1','String2','String3')
where 'String1','String2','String3' are the contents of List<String>?
No LINQ etc. as I am using VS2005.

Take a look on following version
[Test]
public void Test()
{
var list = new List<string> {"String1", "String2", "String3"};
string values = ArrayToString(list);
string sql = string.Format("SELECT Column1 FROM Table1 WHERE Column1 IN ( {0} )", values);
}
private static string ArrayToString(IEnumerable<string> array)
{
var result = new StringBuilder();
foreach (string element in array)
{
if (result.Length > 0)
{
result.Append(", ");
}
result.Append("'");
result.Append(element);
result.Append("'");
}
return result.ToString();
}
result statement SELECT Column1 FROM Table1 WHERE Column1 IN ( 'String1', 'String2', 'String3' )

List<string> lst=new List<string>();lst.Add("Hello");lst.Add("Hello World");
string s="";
foreach(string l in lst)s+="\""+l+"\"";
s=Regex.Replace(s,"\"\"","\",\"");
string output="SELECT Column1 FROM Table1 WHERE Column1 ("+s+")";

try :
List<String> strlist = new List<string>();
strlist.Add("st1");
strlist.Add("st2");
strlist.Add("st3");
string query = "SELECT Column1 FROM Table1 WHERE Column1 IN (";
for (int i = 0; i < strlist.Count; i++)
{
query += "\'" + strlist[i] + "\'" + (i == strlist.Count - 1 ? "" : ",");
}
query += ")";

List<string> items = new List<string>();
items.Add("string1");
items.Add("string2");
items.Add("string3");
string AllItems = "";
foreach (string item in items)
{
AllItems += string.Format("\"{0}\",",item);
}
AllItems = AllItems.TrimEnd(',');
string YourSQLQuery = string.Format("SELECT Column1 FROM Table1 WHERE Column1 IN ({0})", AllItems);
MessageBox.Show(YourSQLQuery);

Don't for get to guard against SQL Injection.
string sql_list = "";
foreach (string s in lst)
sql_list+=string.Format("{0},",s.Replace("'","''"));
sql_list = string.Format("({0})",sql_list.substring(0,sql_list.length-2));
that might help some, and use string builder, or not.

Please don't use the other answers that have been submitted so far. They contain SQL injection for no obvious reason.
List<String> strlist = new List<string>();
strlist.Add("st1");
strlist.Add("st2");
strlist.Add("st3");
var dynamicPart = string.Join(", ",
Enumerable.Range(0, strlist.Count).Select(i => "#" + i).ToArray());
for(i = 0 to strlist.Count)
{ /* add parameter to SqlCommand here with name ("#" + i) */ }
string query = "SELECT Column1 FROM Table1 WHERE Column1 IN (" +
dynamicPart + ")";
Use parameters instead of literals for multiple reasons (research them!).
And instead of a clumsy concatenation loop use string.Join which does all of that for us.

To properly handle sql injection, a better answer may be to make the query of the form...
select results.* from (
select pk from table where column=value1 union
select pk from table where column=value2 union
select pk from table where column=value3 union
select pk from table where column=value4 union
select pk from table where column=value5
) filtered join table as results on filtered.pk = results.pk
and then make it more c# friendly
string items_filter = "";
int item_index=0;
OracleParameterCollection parameters = new OracleParameterCollection(); // Not sure what class to use here exactly, but just collect a bunch of stored procedure parameters
foreach (string item in list_of_items) {
string item_name = string.Format("i_item{0}",item_index);
string item_sql = string.Format("select pk from table where column=:{0} union",item_name);
parameters.Add(new Parameter("item_name",item));
item_index+=1;
}
if (items_filter.IsNullOrEmpty())
return;
string sql = String.Format("select results.* from ({0}) filtered join table as results on filtered.pk = results.pk",items_filter);
OracleCommand c = new OracleCommand();
c.command = sql;
c.parameters = parameters;
c.execute();
More or less.

Since you said its an internal operation and hence there is no need to be worried about SQL Injection, then you can achieve what you want by this.
string str = "";
foreach(string s in list)
str += "'" + s.Replace("'", "''") + "',";
str = str.SubString(0, str.Length - 1);
str = "SELECT Column1 FROM Table1 WHERE Column1 IN (" + str + ")";
//str will have your command ready.
I have tested it. It works perfectly.

// Assume your list (List<string>) is named "myList"
// Please put the next line in an external string resource...
string selectStatement = "SELECT Column1 FROM Table1 WHERE Column1 IN ({0})";
StringBuilder stringBuilder = new StringBuilder("(");
foreach(string colName in myList)
stringBuilder.Append(String.Format("'{0}',", colName));
stringBuilder.Append(")");
return String.Format(selectStatement, stringBuilder.ToString().Replace(",)", ")");