I am trying to implement an ADO.NET code which executes the SQL query with multiple parameters. Looks like SQL parameter limit is 2100 and does not accept more than this limit. How do I achieve with my below code to have this accept more than the limitation.
I am finding it difficult to understand the implementations when validating online articles related how to send the queries in subsets or chunks to fulfill my request.
This is my code:
using (Connection = new SqlConnection(CS))
{
Connection.Open();
string query = "SELECT FamilyID, FullName, Alias FROM TABLE (nolock) WHERE FamilyID IN ({0})";
var stringBuiler = new StringBuilder();
var familyIds = new List<string>();
string line;
while ((line = TextFileReader.ReadLine()) != null)
{
line = line.Trim();
if (!familyIds.Contains(line) & !string.IsNullOrEmpty(line))
{
familyIds.Add(line);
}
}
var sqlCommand = new SqlCommand
{
Connection = Connection,
CommandType = CommandType.Text
};
var index = 0; // Reset the index
var idParameterList = new List<string>();
foreach (var familyId in familyIds)
{
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
}
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
var dt = new DataTable();
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
{
dt.Load(sqlReader);
}
try
{
if (dt.Rows.Count > 0)
{
OutputdataGridView.DataSource = lstDownloadOwnerOutput;
OutputdataGridView.ColumnHeadersDefaultCellStyle.Font = new Font(DataGridView.DefaultFont, FontStyle.Bold);
OutputdataGridView.Columns[0].AutoSizeMode = DataGridViewAutoSizeColumnMode.AllCells;
Gridviewdisplaylabel.Text = "Total no of rows: " + this.OutputdataGridView.Rows.Count.ToString();
}
else if (dt.Rows.Count == 0)
{
MessageBox.Show("Data returned blank!!!");
}
}
catch (Exception Ex)
{
if (Connection != null)
{
Connection.Close();
}
MessageBox.Show(Ex.Message);
}
}
Having a WHERE IN clause with 2100, or even 100, parameters is generally not good coding practice. You might want to consider putting those values into a separate bona fide table, e.g.
families (ID int PK, ...)
Then, you may rewrite your query as:
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock)
WHERE FamilyID IN (SELECT ID FROM families);
You could also express the above using an EXISTS clause or a join, but all three approaches might just optimize to a very similar query plan anyway.
You can just add a table load call every 2000 parameters in your code:
var index = 0; // Reset the index
var idParameterList = new List<string>();
var dt = new DataTable();
foreach (var familyId in familyIds) {
var paramName = "#familyId" + index;
sqlCommand.Parameters.AddWithValue(paramName, familyId);
idParameterList.Add(paramName);
index++;
if (index > 2000) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
sqlCommand.Parameters.Clear();
idParameterList.Clear();
index = 0;
}
}
if (index > 0) {
sqlCommand.CommandText = String.Format(query, string.Join(",", idParameterList));
using (SqlDataReader sqlReader = sqlCommand.ExecuteReader())
dt.Load(sqlReader);
}
For dynamic sql like this, I generally recommend using a Table-Valued Parameter.
It does require a bit of setup: you have to create a user-defined Type in the DB to hold the values, but that is a fairly trivial operation:
CREATE TYPE PrimaryKeyType AS TABLE ( VALUE INT NOT NULL );
We generally use these in conjunction with stored procedures:
CREATE PROCEDURE dbo.getFamily(#PrimaryKeys PrimaryKeyType READONLY)
AS
SELECT FamilyID, FullName, Alias
FROM TABLE (nolock) INNER JOIN #PrimaryKeys ON TABLE.FamilyID = #PrimaryKeys.Value
GO
However, you can also use inline SQL if you prefer.
Assigning the values to the stored proc or inline parameter is fairly straightforward, but there is one gotcha (more later):
public static void AssignValuesToPKTableTypeParameter(DbParameter parameter, ICollection<int> primaryKeys)
{
// Exceptions are handled by the caller
var sqlParameter = parameter as SqlParameter;
if (sqlParameter != null && sqlParameter.SqlDbType == SqlDbType.Structured)
{
// The type name may look like DatabaseName.dbo.PrimaryKeyType,
// so remove the database name if it is present
var parts = sqlParameter.TypeName.Split('.');
if (parts.Length == 3)
{
sqlParameter.TypeName = parts[1] + "." + parts[2];
}
}
if (primaryKeys == null)
{
primaryKeys = new List<int>();
}
var table = new DataTable();
table.Columns.Add("Value", typeof(int));
foreach (var wPrimaryKey in primaryKeys)
{
table.Rows.Add(wPrimaryKey);
}
parameter.Value = table;
}
The thing to watch out for here is the naming of the parameter. See the code in the method above that removes the database name to resolve this issue.
If you have dynamic SQL, you can generate a correct parameter using the following method:
public static SqlParameter CreateTableValuedParameter(string typeName, string parameterName)
{
// Exceptions are handled by the caller
var oParameter = new SqlParameter();
oParameter.ParameterName = parameterName;
oParameter.SqlDbType = SqlDbType.Structured;
oParameter.TypeName = typeName;
return oParameter;
}
Where typeName is the name of your type in the DB.
Got this detail page about bug that might lead to sql injection
URL encoded GET input classid was set to 1 AND 3*2*1=6 AND 608=608
Tests performed:
1*1*1*1 => TRUE
1*608*603*0 => FALSE
11*5*2*999 => FALSE
1*1*1 => TRUE
1*1*1*1*1*1 => TRUE
11*1*1*0*1*1*608 => FALSE
1 AND 5*4=20 AND 608=608 => TRUE
1 AND 5*4=21 AND 608=608 => FALSE ... (line truncated)
And this is the source code that might cause the matter:
if (!string.IsNullOrEmpty(Request.QueryString["classid"]))
{
string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID={0} ";
DataTable data = DbSession.Default.FromSql(string.Format(tSql, Request.QueryString["classid"])).ToDataTable();
if (data.Rows.Count > 0)
{
rptList.DataSource = data;
rptList.DataBind();
}
}
else
{
string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award ";
DataTable data = DbSession.Default.FromSql(tSql).ToDataTable();
if (data.Rows.Count > 0)
{
rptList.DataSource = data;
rptList.DataBind();
}
}
Can anyone tell me how to deal with this...thanks a lot!
Now i have modifeid my code to
if (!string.IsNullOrEmpty(Request.QueryString["classid"]))
{
//string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID={0} ";
string tSql = "SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID = #ClassID";
//DataTable data = DbSession.Default.FromSql(string.Format(tSql, Request.QueryString["classid"])).ToDataTable();
SqlConnection connection = new SqlConnection("Server=(local);Integrated Security=SSPI;database=DaysQP");
connection.Open();
SqlCommand command = new SqlCommand(tSql, connection);
command.Parameters.Add(new SqlParameter("#ClassId", System.Data.SqlDbType.Int));
command.Parameters["#ClassID"].Value = 1;
using (SqlDataReader dr = command.ExecuteReader())
{
var data = new DataTable();
data.Load(dr);
if (data.Rows.Count > 0)
{
rptList.DataSource = data;
rptList.DataBind();
}
}
connection.Close();
}
else
{
string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award ";
DataTable data = DbSession.Default.FromSql(tSql).ToDataTable();
if (data.Rows.Count > 0)
{
rptList.DataSource = data;
rptList.DataBind();
}
}
But the problem still exists..
Finally sovled the problem by using parameterized queries!
if (!string.IsNullOrEmpty(Request.QueryString["classid"]))
{
int number;
bool result = Int32.TryParse(Request.QueryString["classid"], out number);
if (result == false)
{
return;
}
//string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID={0} ";
string tSql = "SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID = #ClassID";
//DataTable data = DbSession.Default.FromSql(string.Format(tSql, Request.QueryString["classid"])).ToDataTable();
SqlConnection connection = (SqlConnection)DbSession.Default.CreateConnection();
//SqlConnection("Server=(local);Integrated Security=SSPI;database=DaysQP");
connection.Open();
SqlCommand command = new SqlCommand(tSql, connection);
command.Parameters.Add(new SqlParameter("#ClassId", System.Data.SqlDbType.Int));
command.Parameters["#ClassID"].Value = number;
using (SqlDataReader dr = command.ExecuteReader())
{
var data = new DataTable();
data.Load(dr);
if (data.Rows.Count > 0)
{
rptList.DataSource = data;
rptList.DataBind();
}
}
connection.Close();
}
The potential for injection would be here:
string tSql = #" SELECT [Award_ID],[Award_Name],[Award_Info],[Award_Pic],[Award_Num],[Award_MoneyCost],[Award_MoneyGet],[Award_Type],[Award_AddDate],[Award_Hot],[Award_OnLineTime],[AwardProP],[PrizeSlidePic],[PrizeDetailPic],[PrizeBigSlidePic],[IsTop],[ClassID] FROM dbo.Web_Award WHERE ClassID={0} ";
DataTable data = DbSession.Default.FromSql(string.Format(tSql, Request.QueryString["classid"])).ToDataTable();
You're expecting the query to return Web_Award table records whose classId matches Request.QueryString["classid"]
What happens if the value of Request.QueryString["classid"] is something like:
1 or 1=1
then the query becomes:
select award_id,..... from web_awards where classId=1 or 1=1
and you end up returning data that you never meant to.
This, in essence, is sql injection which you probably read up a bit more about. Using stored procedures or parameterized queries prevents this sort of attack.
First, I did search for this first and found this question answered: Previous Answer
The problem I have with this answer is that using this method causes a NullReferenceException. Obviously the code will still work with a try/catch block, but I had always understood that intentionally using an Exception as a means of controlling flow was poor design. Is there a better method of doing this?
To be clear as to the parameters, I am using OleDbConnection and OleDbCommand to read/write an Access 2010 database (.accdb format).
Below is the code I have currently using the above answer's approach:
public bool ReadAccessDb(string filePath, string queryString, bool hasRecords)
{
string connectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;" +
#"Data Source=" + filePath + ";" +
#"User Id=;Password=;";
using (OleDbConnection connection = new OleDbConnection(connectionString))
using (OleDbCommand command = new OleDbCommand(queryString, connection))
{
try
{
connection.Open();
int count = (int)command.ExecuteScalar();
//This works, but if no records exist it uses the Exception system to halt further action. Look for better approach.
if(count > 0)
{
hasRecords = true;
}
}
catch (System.Exception ex)
{
}
}
return hasRecords;
}
You can use :
int count = command.ExecuteScalar() is DBNull ? 0 : Convert.ToInt32(command.ExecuteScalar());
or use :
object obj = command.ExecuteScalar();
int count = obj is DBNull ? 0 : Convert.ToInt32(obj);
I'm posting this answer because the question is a bit ambiguous and the (currently) accepted answer can conceivably be "fooled" if a row exists but the first column returned by the SELECT statement happens to contain a NULL. Consider the test table
CREATE TABLE MyTable (ID COUNTER PRIMARY KEY, NullableColumn INTEGER NULL)
INSERT INTO MyTable (NullableColumn) VALUES (NULL)
which would look like
ID NullableColumn
-- --------------
1 <NULL>
If the SELECT statement used for testing was
string sql = "SELECT NullableColumn FROM MyTable WHERE ID=1";
then the method
static bool RecordExists(string queryString, OleDbConnection conn)
{
bool rtn;
using (var command = new OleDbCommand(queryString, conn))
{
object obj = command.ExecuteScalar();
int count = obj is DBNull ? 0 : Convert.ToInt32(obj);
rtn = (count != 0);
}
return rtn;
}
would return False, whereas this method
static bool RecordExists(string queryString, OleDbConnection conn)
{
bool rtn;
string rowCountString = String.Format("SELECT COUNT(*) AS n FROM ({0})", queryString);
using (var command = new OleDbCommand(rowCountString, conn))
{
int count = (int)command.ExecuteScalar();
rtn = (count != 0);
}
return rtn;
}
would return True.
I am a hobbyist programmer and writing a reporting tool to export values from an SQLite database to Excel.
The Excel part is written and working, the data I am retrieving from SQLite is creating a block in the program and taking several minutes to process.
I have rewritten the code out using generic values to help illustrate the processes. The initial populateList module is taking a negligible amount of time, but I have included it below as that is the providing the data for the doStuff module. The populateList currently retrieves approximately 500 distinct records.
I need the program to iterate through all the values retrieved by the populateList and do several counts. It is then populating another list valuesCount with the counted values.
I tried to improve the speed by looping through the list without closing the SQLite connection, but the improvement wasn't enough. Is there a more efficient way to retrieve this information from the databse?
public list<string>populateList()
{
List<string>values = new list<string>();
using (SQLiteConnection con = new SQLiteConnection(Passer.connstr))
{
con.Open();
string distinctValues = "SELECT DISTINCT \"value list\" FROM valueTable order by \"value list\" ";
using (SQLiteCommand cmd = new SQLiteCommand(distinctValues, con))
{
SQLiteDataReader sqReader;
sqReader = cmd.ExecuteReader();
while (sqReader.Read())
{
values.Add(sqReader["value list"].ToString());
}
}
}
return values;
}
public void doStuff()
{
bool blanks = false;
string singleValue = string.Empty
string query = string.Empty;
List<string> getInitialValues = populateList();
list<string> valuesCount = new list<string>();
using (SQLiteConnection con = new SQLiteConnection(Passer.connstr))
{
con.Open();
for(int i = 0; i < getInitialValues.Count; i++)
{
blanks = false;
singleValue = getInitialValues[i];
if(singlevalue == "")
{
singleValue = \"\";
blanks = true;
}
for (int x = 0; x < 6; x++)
{
string statement = string.Empty;
switch(x)
{
case 0:
statement = "SELECT COUNT(*) from valueTable where \"column1\" = ";
break;
case 1:
statement = "SELECT COUNT(*) from valueTable where \"column2\" = \"condition 1\" and \"column1\" = ";
break;
case 2:
statement = "SELECT COUNT(*) from valueTable where \"column3\" = \"condition 3\" and \"column1\" = ";
break;
case 3:
statement = "SELECT COUNT(*) from valueTable where \"column4\" = \"condition 4\" and \"column1\" = ";
break;
case 4:
statement = "SELECT COUNT(*) from valueTable where \"column5\" = \"condition 5\" and \"column1\" = ";
break;
case 5:
statement = "SELECT COUNT(*) from valueTable where \"column6\" = \"condition 6\" and \"column1\" = ";
break;
}
if (blanks == true)
{
query = System.String.Format("{0}{1}", statement, singleValue);
}
else
{
query = System.string.format("{0}\"{1}\"", statement, singleValue);
}
using (SQLiteCommand cmd = new SQLiteCommand(query, con))
{
string countValues = cmd.ExecuteScalar().ToString();
valuesCount.Add(countValues);
}
}
}
}
}
Consider writing it as a single SQL query.
You are performing many queries when it very much looks like you simply need to perform a 'conditional count' on the columns. The SQL would be along the lines of
select
val,
col1 = sum(case when col1 = 'cond1' then 1 end)
from valtbl
group by val
you wouldn't even need the first method to get the list of distinct values.
Alternatively as the table seems reasonably small select what you need into a list of 'rows' and use Linq to Objects to do the counts.
You are querying the database multiple times to get the same information.
I would suggest not to do any db calls on dostuff method, rather use a single query for fetch the records form valuetable.
then you can do the count operations on the list itself.
For example get
"SELECT 'valueCol' , 'col1', 'col2' from valueTable"
would be your only query and you would store it in a list(say valueList).
Then on C# side you can use
//not actual code just a sample idea
var distinctValues = valueList.select(v => w.valueCol).Distinct()
var count = 0;
switch(case):
case 0:
count += valueList.where(v => v.col1 == condition).Count();
break; //and so on...
Assuming that the cardDetailsID is 5.
Looking at record number 5 in the cardDetails table of the database, one can see that its other fields include "bgcolour" and "borderstyle". Therefore, for this particular record I've got cardDetailsID = 5, bgcolour = blue, bordersytle= solid.
I want to be able to get the bgcolour and bordersyle settings (blue and solid) from the cardDetailsID.
This is the code so far. The value in the querystring is working (number "5" is being passed) but now how do I get the rest of the setting of the row?
cardDetailsIDrv.Text = Request.QueryString["cardDetailsID"];
cardDetailsIDrv.Visible = false;
//create Connection
SqlConnection myConnection = new SqlConnection(ConfigurationManager.AppSettings["ConnString"]);
//create Command
SqlCommand getCardDetailsCMD = new SqlCommand("SELECT * FROM cardDetails WHERE cardDetailsID =" + Convert.ToInt32(cardDetailsIDrv.Text), myConnection);
myConnection.Open();
//create datareader
SqlDataReader myReader = getCardDetailsCMD.ExecuteReader();
//code works properly up till here
try
{
//Using DataReader to retrieve info from the database and display it on the panel
while (myReader.Read())
{
//I'm guessing here is where I'm messing things up
pnlCardps.BackColor = Color.FromName(myReader["bgcolour"].ToString());
pnlCardps.BorderStyle = (BorderStyle)Enum.Parse(typeof(BorderStyle), myReader["borderstyle"].ToString());
}
}
finally
{
if (myReader != null)
{
myReader.Close();
}
if (myConnection != null)
{
myConnection.Close();
}
}
PROBLEM SOLVED!!
All I had to do is tweak the code in the while loop to:
string bgColour = myReader["bgColour"].ToString();
pnlCardrv.BackColor = Color.FromName(bgColour);
string borderColour = myReader["borderColour"].ToString();
pnlCardrv.BorderColor = Color.FromName(borderColour);
firstly, you have to get the detailsID which is passed via the query string and then perform the query.
int id;
if(int.TryParse(Request.QueryString["detailsID"],out id))
{
string sql= string.Format("select .. from .. where id={0}",id);// using SqlParameter is much better and secure
// and execute your query and fetch the data reader
while(reader.Read())
{
// bla bla bla
}
}
Request.QueryString["detailsID"] will get you the value of the query string variable.
if (Request.QueryString("detailsID") == null || Request.QueryString("detailsID").Length <= 0 || !int.TryParse(Request.QueryString("detailsID"), out detailsID) || MessageThreadID <= 0)
{
//This is my standard exception, but you can handle it how you want.
throw new Exception("Error: unable to load detailsID. Request.QueryString.count: " + Request.QueryString.Count + " Request.QueryString(\"detailsID\"): " + Request.QueryString("detailsID"));
}