We Keep Coding

SQL query on ADO.net limitation with 2100+ parameters

I am trying to implement an ADO.NET code which executes the SQL query with multiple parameters. Looks like SQL parameter limit is 2100 and does not accept more than this limit. How do I achieve with my below code to have this accept more than the limitation.
I am finding it difficult to understand the implementations when validating online articles related how to send the queries in subsets or chunks to fulfill my request.
This is my code:
using (Connection = new SqlConnection(CS))
{
Connection.Open();
string query = "SELECT FamilyID, FullName, Alias FROM TABLE (nolock) WHERE FamilyID IN ({0})";
var stringBuiler = new StringBuilder();
var familyIds = new List<string>();
string line;
while ((line = TextFileReader.ReadLine()) != null)
{
line = line.Trim();
if (!familyIds.Contains(line) & !string.IsNullOrEmpty(line))
{
familyIds.Add(line);
}
}
var sqlCommand = new SqlCommand
{
Connection = Connection,
CommandType = CommandType.Text
};
var index = 0; // Reset the index
var idParameterList = new List<string>();
foreach (var familyId in familyIds)
{
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
}
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
var dt = new DataTable();
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
{
dt.Load(sqlReader);
}
try
{
if (dt.Rows.Count > 0)
{
OutputdataGridView.DataSource = lstDownloadOwnerOutput;
OutputdataGridView.ColumnHeadersDefaultCellStyle.Font = new Font(DataGridView.DefaultFont, FontStyle.Bold);
OutputdataGridView.Columns[0].AutoSizeMode = DataGridViewAutoSizeColumnMode.AllCells;
Gridviewdisplaylabel.Text = "Total no of rows: " + this.OutputdataGridView.Rows.Count.ToString();
}
else if (dt.Rows.Count == 0)
{
MessageBox.Show("Data returned blank!!!");
}
}
catch (Exception Ex)
{
if (Connection != null)
{
Connection.Close();
}
MessageBox.Show(Ex.Message);
}
}

Having a WHERE IN clause with 2100, or even 100, parameters is generally not good coding practice. You might want to consider putting those values into a separate bona fide table, e.g.
families (ID int PK, ...)
Then, you may rewrite your query as:
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock)
WHERE FamilyID IN (SELECT ID FROM families);
You could also express the above using an EXISTS clause or a join, but all three approaches might just optimize to a very similar query plan anyway.

You can just add a table load call every 2000 parameters in your code:
var index = 0; // Reset the index
var idParameterList = new List<string>();
var dt = new DataTable();
foreach (var familyId in familyIds) {
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
if (index > 2000) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
sqlCommand.Parameters.Clear();
idParameterList.Clear();
index = 0;
}
}
if (index > 0) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
}

For dynamic sql like this, I generally recommend using a Table-Valued Parameter.
It does require a bit of setup: you have to create a user-defined Type in the DB to hold the values, but that is a fairly trivial operation:
CREATE TYPE PrimaryKeyType AS TABLE ( VALUE INT NOT NULL );
We generally use these in conjunction with stored procedures:
CREATE PROCEDURE dbo.getFamily(#PrimaryKeys PrimaryKeyType READONLY)
AS
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock) INNER JOIN #PrimaryKeys ON TABLE.FamilyID = #PrimaryKeys.Value
GO
However, you can also use inline SQL if you prefer.
Assigning the values to the stored proc or inline parameter is fairly straightforward, but there is one gotcha (more later):
public static void AssignValuesToPKTableTypeParameter(DbParameter parameter, ICollection<int> primaryKeys)
{
// Exceptions are handled by the caller
var sqlParameter = parameter as SqlParameter;
if (sqlParameter != null && sqlParameter.SqlDbType == SqlDbType.Structured)
{
// The type name may look like DatabaseName.dbo.PrimaryKeyType,
// so remove the database name if it is present
var parts = sqlParameter.TypeName.Split('.');
if (parts.Length == 3)
{
sqlParameter.TypeName = parts[1] + "." + parts[2];
}
}
if (primaryKeys == null)
{
primaryKeys = new List<int>();
}
var table = new DataTable();
table.Columns.Add("Value", typeof(int));
foreach (var wPrimaryKey in primaryKeys)
{
table.Rows.Add(wPrimaryKey);
}
parameter.Value = table;
}
The thing to watch out for here is the naming of the parameter. See the code in the method above that removes the database name to resolve this issue.
If you have dynamic SQL, you can generate a correct parameter using the following method:
public static SqlParameter CreateTableValuedParameter(string typeName, string parameterName)
{
// Exceptions are handled by the caller
var oParameter = new SqlParameter();
oParameter.ParameterName = parameterName;
oParameter.SqlDbType = SqlDbType.Structured;
oParameter.TypeName = typeName;
return oParameter;
}
Where typeName is the name of your type in the DB.

Understanding 'The SqlParameter is already contained by another SqlParameterCollection'

I have the following piece of code:
SqlParameter invidParam = new SqlParameter("#invid",obj.INVID);
SqlParameter prodParam = new SqlParameter("#prod", obj.PROD);
SqlParameter descrParam = new SqlParameter("#descr", obj.DESCR);
SqlParameter qtyParam = new SqlParameter("#qty", 1);
if(SessionHelper.TryGetSession("curUser",out User user))
{
SqlParameter useridParamCheck = new SqlParameter("#userid", user.Username);
var queryResult = db.Database.SqlQuery<int>("EXEC CheckIfAlreadyInPR#invid#userid #invid, #userid", invidParam, useridParamCheck);
var qty = queryResult.FirstOrDefault();
if (qty == 0)
{
SqlParameter useridParamInsert = new SqlParameter("#userid", user.Username);
var insertResult = db.Database.ExecuteSqlCommand("InsertIntoPR #invid, #prod, #descr, #userid, #qty", invidParam, prodParam, descrParam, useridParamInsert, qtyParam);
}
else if (qty > 0)
{
SqlParameter qtyParameter = new SqlParameter("#newqty", (qty + 1));
SqlParameter userIdParamUpdate = new SqlParameter("#userid", user.Username);
var updateResult = db.Database.ExecuteSqlCommand("UpdatePRQty#invid#userid#newqty #invid,#userid,#newqty",invidParam,userIdParamUpdate,qtyParameter);
}
}
My program runs and hits an exception at the line var insertResult = db...
I am trying to understand which SqlParameter is already contained in another collection. Is it because the parameter #userid is already used in the previous stored procedure call ?
I declared the SqlParameter("#userid") 3 times, with different variable names:
useridParamCheck, useridParamInsert, userIdParamUpdate
Did this cause the error because the name #userid is used? If I use #userid1, #userid2, #userid3 will the problem be resolved? Or the SqlParameter that is already contained is some other SQL parameters that I declared above?
Thank you

I'm going to suggest that invidparam is your problem parameter as it is the only parameter I can see that is used twice. Once in db.Database.SqlQuery<int>("EXEC CheckIfAlreadyInPR#invid#userid #invid, #userid", invidParam, useridParamCheck); and again in both instances of db.Database.ExecuteCommand().
The error itself relates to an SqlParameter object being used multiple times rather than the same parameter name being used multiple times.

I was getting this error with the following code
var p = new System.Data.SqlClient.SqlParameter{ParameterName = "#p0", SqlDbType
SqlDbType.DateTime ,Value = ut.ReadyByDate, };
var nextTasksResults = db.Database.SqlQuery<nextTask>(sql2, p);
var isAny = nextTaskResults.Any() // error
The work around was to send to an array and use that
var nextTasks = nextTasksResults.ToArray();
var isAny = nextTasks.Any();

parameterized with session and query string

I am facing an error message and i didnt know the reason with me sql statement as at line:
DACatPgeVIPLIST.Fill(dsCatPgeVIPLIST);
It's showing this message can you help me with it:
The parameterized query (#Country nvarchar(7),#Category nvarchar(4000))SELECT a.[AdsID], expects the parameter #Category, which was not supplied.
Code:
if (Session["location"] != null)
{
using (SqlConnection CatPgeVIPLISTsqlCON = new SqlConnection(cs))
{
CatPgeVIPLISTsqlCON.Open();
SqlDataAdapter DACatPgeVIPLIST = new SqlDataAdapter("SELECT a.[AdsID], a.[Country], a.[State], a.[City], a.[AdsTit], SUBSTRING(a.[AdsDesc], 1, 70) as AdsDesc, a.[AdsPrice], a.[Img1] FROM [ads] as a INNER JOIN [UserInfo] as u on u.UID = a.UID WHERE a.[Country] = #Country and a.[Category] = #Category and u.VIP = 'Yes'", cs);
string location = Convert.ToString(Session["location"]);
string category = Request.QueryString["category"];
DACatPgeVIPLIST.SelectCommand.Parameters.AddWithValue("#Country", location);
DACatPgeVIPLIST.SelectCommand.Parameters.AddWithValue("#Category", category);
DataSet dsCatPgeVIPLIST = new DataSet();
DACatPgeVIPLIST.Fill(dsCatPgeVIPLIST);
CatPgeVIPLIST.DataSource = dsCatPgeVIPLIST.Tables[0];
CatPgeVIPLIST.DataBind();
}
}

It's possible for the following line of code to assign null to category:
string category = Request.QueryString["category"];
You could possibly get around it like this, which converts null to an empty string:
string category = Convert.ToString(Request.QueryString["category"]);
Or you could try passing DBNull.Value instead of null (untested):
DACatPgeVIPLIST.SelectCommand.Parameters
.AddWithValue("#Category", (object)category ?? DBNull.Value);

The specified argument value for the function is not valid. [ Argument # = 1,Name of function(if known) = isnull ]

I want to select from a table where a column matches a given parameter. If the parameter is null, I want to select all records from the table. The relevant code below is what throws this error.
private static string _dbJobCreate =
"CREATE TABLE Job (jID int primary key identity(1,1), jAddress nvarchar(64) not null);";
private static string _dbJobSelect =
"SELECT jID FROM Job WHERE jAddress = #jAddress OR #jAddress IS NULL";
public static DataTable GetJobs(string jAddress)
{
SqlCeParameter pjAddress = new SqlCeParameter();
pjAddress.ParameterName = "#jAddress";
if (!string.IsNullOrEmpty(jAddress))
{
pjAddress.Value = jAddress;
}
else
{
pjAddress.Value = DBNull.Value;
}
return ExecuteDataTable(_dbJobSelect, pjAddress);
}
Exception: The specified argument value for the function is not valid. [ Argument # = 1,Name of function(if known) = isnull ]
How can I efficiently accomplish this without error in SQLCE?

You can avoid this error by specifying the types of parameters which are passed to the query. So all you need to do is:
pjAddress.DbType = DbType.String;

My solution is to select all rows from the database, and filter down the rows in .NET if a parameter was passed in. This could become troublesome if there were a large number of jobs, although I suppose I'd move to a real database if that ever happens.
private static string _dbJobSelect = "SELECT jID, jAddress FROM Job";
public static DataTable GetJobs(string jAddress)
{
DataTable dt = ExecuteDataTable(_dbJobSelect);
if (!string.IsNullOrEmpty(jAddress))
{
DataView dv = dt.DefaultView;
dv.RowFilter = string.Format("jAddress = '{0}'", jAddress);
dt = dv.ToTable();
}
return dt;
}

You can cast it to varchar
SELECT jID FROM Job WHERE jAddress = #jAddress
OR cast(#jAddress AS varchar(4000)) IS NULL

I ran into this problem recently and discovered that I hadn't actually added the query parameter to the IDbCommand. I'm not sure what your ExecuteDataTable method looks like, but my code was similar to the following (where db is an IDbConnection instance):
var sql = "SELECT jID FROM Job WHERE jAddress = #jAddress OR #jAddress IS NULL";
var cmd = db.CreateCommand();
cmd.CommandText = sql;
var param = cmd.CreateParameter();
param.DbType = DbType.String;
param.ParameterName = "#jAddress";
param.Value = string.IsNullOrEmpty(pjAddress) ? DBNull.Value : (object)pjAddress;
cmd.Parameters.Add(param); // THIS WAS THE STEP I WAS MISSING!!!
// ... rest of the code to execute the query and load the results ...
After adding the cmd.Parameters.Add(param), the exception disappeared.

update sql statement with unknown name/amount of params

I have a classic ASP site, that I am slowly upgrading. I would like to create a function to securely update a SQL database without specifying parameters manually. Something just a tad more dynamic.
(I do not want to use entity framework or Linq)
Here is the code so far:
string updateSql = "UPDATE sometable" + "SET test1= #testData1 " + "WHERE a = #aData1";
SqlCommand UpdateCmd = new SqlCommand(updateSql, conn);
UpdateCmd.Parameters.Add("#testData1 ", SqlDbType.NVarChar, 10, "testData1 ");
UpdateCmd.Parameters.Add("#aData1", SqlDbType.NVarChar, 20, "aData1");
UpdateCmd.Parameters["#testData1 "].Value = "21515";
UpdateCmd.Parameters["#aData1"].Value = "32t3t";
UpdateCmd.ExecuteNonQuery();
pseudo-code (what I would like to achieve)
Create an Ilist covering all variables {get; set:} [validate type/length here]
For every variable that contains a value (without validation issues) create sql update string.
Execute it.
Possible problem:
The only problem I can foresee, is that the list may have 500 variables, but each SQL update may only have only 2 or 3 columns being updated. Is this not efficient?

you need to do something like this....needs more coding obviously....
static void Main(string[] args)
{
var values = new Dictionary<string, object>( );
values.Add( "name", "timmerz" );
values.Add( "dob", DateTime.Now );
values.Add( "sex", "m" );
SqlUpdate( "sometable", values );
}
public static void SqlUpdate( string table, Dictionary<string,object> values, string where )
{
var equals = new List<string>( );
var parameters = new List<SqlParameter>( );
var i = 0;
foreach( var item in values )
{
var pn = "#sp" + i.ToString( );
equals.Add( string.Format( "{0}={1}", item.Key, pn ) );
parameters.Add( new SqlParameter( pn, item.Value ) );
i++;
}
string command = string.Format( "update {0} set {1} where {2}", table, string.Join( ", ", equals.ToArray( ) ), where );
var sqlcommand = new SqlCommand(command);
sqlcommand.Parameters.AddRange(parameters.ToArray( ) );
sqlcommand.ExecuteNonQuery( );
}

I'm not sure I fully understand what you're trying to do, but this might be close to what you're looking for. You can create an arbitrarily long list of parameters and respective values, then build the corresponding UPDATE dynamically from that list.
//set up SqlCommand
SqlCommand UpdateCmd = new SqlCommand();
UpdateCmd.Connection = conn;
//build your dictionary (probably happens elsewhere in your code)
Dictionary<string, object> parameters = new Dictionary<string, object>();
parameters.Add("col1", "col1 value");
parameters.Add("col2", 42);
parameters.Add("col3", DateTime.Now);
//build a command string and add parameter values to your SqlCommand
StringBuilder builder = new StringBuilder("UPDATE sometable SET ");
foreach(KeyValuePair<string, object> parameter in parameters) {
builder.Append(parameter.Key).Append(" = #").Append(parameter.Key).Append(",");
UpdateCmd.Parameters.AddWithValue("#" + parameter.Key, parameter.Value);
}
builder.Remove(builder.Length - 1,1);
//set the command text and execute the command
UpdateCmd.CommandText = builder.ToString();
UpdateCmd.ExecuteNonQuery();

If you are using SQL Server 2008 you have the option of passing in the parameters and their values as a table to a Stored Procedure.
Inside the Stored Procedure you can join the table to be updated with the table passed in. That would probably be more efficient than creating hundreds of sep update statements.
Here is a link that may help http://msdn.microsoft.com/en-us/library/bb675163.aspx
And here is some sample code based on the code you posted in your question
First Create a table to play with and populate it with some data
CREATE TABLE [dbo].[sometable](
[Test1] [nvarchar](100) NULL,
[a] [nvarchar](100) NULL
) ON [PRIMARY]
Insert sometable Select 'rerere', '122342'
Insert sometable Select 'sfsfw', '343'
Insert sometable Select 'sfdrgss', '434545'
Insert sometable Select 'srgegrgeg', '3939932'
Then Create the Type in SQL Server
Create TYPE dbo.ParamsType AS TABLE
( Test1 nvarchar(100), a nvarchar(100) )
Then Create the Stored Procedure that accepts the type as a parameter
CREATE PROCEDURE usp_UpdateSomeTable
#Parameters dbo.ParamsType READONLY
AS
BEGIN
SET NOCOUNT ON;
UPDATE sometable
SET sometable.Test1 = p.Test1
FROM sometable INNER JOIN #Parameters as p
ON sometable.a = p.a;
END
GO
To test from SQL Server Management Studio you can run
Declare #t as ParamsType
Insert #t Select 'newValue1', '122342'
Insert #t Select 'morenew ', '343'
Insert #t Select 'again', '434545'
Insert #t Select 'OnceMore', '3939932'
exec usp_UpdateSomeTable #Parameters=#t
To Test from C# Try
static void Main(string[] args)
{
System.Data.DataTable YourData = new DataTable("Parameters");
DataColumn column;
DataRow row;
column = new DataColumn();
column.DataType = System.Type.GetType("System.String");
column.ColumnName = "Test1";
YourData.Columns.Add(column);
column = new DataColumn();
column.DataType = System.Type.GetType("System.String");
column.ColumnName = "a";
YourData.Columns.Add(column);
row = YourData.NewRow();
row["Test1"] = "newValue1";
row["a"] = "122342";
YourData.Rows.Add(row);
row = YourData.NewRow();
row["Test1"] = "morenew";
row["a"] = "343";
YourData.Rows.Add(row);
row = YourData.NewRow();
row["Test1"] = "again";
row["a"] = "434545";
YourData.Rows.Add(row);
SqlConnectionStringBuilder connString = new SqlConnectionStringBuilder();
connString.DataSource = "127.0.0.1";
connString.InitialCatalog = "SO";
connString.IntegratedSecurity = true;
using (SqlConnection conn = new SqlConnection())
using (SqlCommand cmd = new SqlCommand())
{
cmd.CommandText = "usp_UpdateSomeTable";
cmd.CommandType = System.Data.CommandType.StoredProcedure;
SqlParameter p = cmd.Parameters.AddWithValue("#Parameters", YourData);
p.SqlDbType = SqlDbType.Structured;
p.TypeName = "dbo.ParamsType";
cmd.Connection = conn;
conn.ConnectionString = connString.ConnectionString;
conn.Open();
cmd.ExecuteNonQuery();
}
}