We Keep Coding

handling large amounts of data to include in an oracle select statement

Recent bug report states that a method being called is crashing the service causing it to restart. After troubleshooting, the cause was found to be an obnoxious Oracle SQL call with thousands of strings passed. There is a collection of strings being passed to a method from an external service which often is more than 10,000 records. The original code used a where clause on the passed collection using the LIKE keyword, which I think is really, really bad.
public IList<ContainerState> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Join("OR", containerNumbers
.Select(item => string.Concat(" cntr_no LIKE '", item.SliceLeft(10), "%' ")))
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
Clarification of in house methods used which may be confusing:
DataBase.SelectQuery is an internal library method using generics which gets passed the sql string, a function to map the records to .NET objects, and the parameters being passed and returns an IEnumerable of Objects of type retuned by the Mapping function.
SliceLeft is an extension method from another internal helper library that just returns the first part of a string up to the number of characters specified by the parameter.
The reason that the LIKE statement was apparently used, is that the strings being passed and the strings in the database only are guaranteed to match the first 10 characters. Example ("XXXX000000-1" in the strings being passed should match a database record like "XXXX000000-8").
I believed that the IN clause using the SUBSTR would be more efficent than using multiple LIKE clauses and replaced the code with:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Format("SUBSTR(CNTR_NO, 1, 10) IN ({0}) ",
string.Join(",", containerNumbers.Select(item => string.Format("\'{0}\'", item.SliceLeft(10) ) ) )
)
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
This helped slightly, and there were fewer issues in my tests, but when there are huge amounts of records passed, there is still an exception thrown and core dumps occur, as the SQL is longer than the server can parse during these times. The DBA suggests saving all the strings being passed to a temporary table, and then joining against that temp table.
Given that advice, I changed the function to:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
#"
CREATE TABLE T1(cntr_num VARCHAR2(10));
DECLARE GLOBAL TEMPORARY TABLE SESSION.T1 NOT LOGGED;
INSERT INTO SESSION.T1 VALUES (:containerNumbers);
SELECT
DISTINCT cntr_no,
'_IT' cntr_state
FROM
tb_master
WHERE
cntr_seq = 0
AND cntr_state IN ({0})
AND adjustment <> :adjustment
AND SUBSTR(CTNR_NO, 1, 10) IN (SELECT CNTR_NUM FROM SESSION.T1);
";
var parameters = new
{
#containerNumbers = containerNumbers.Select( item => item.SliceLeft(10)).ToList()
};
return DataBase.SelectQuery(sql, MapRecordToContainerState, parameters).ToList();
}
Now I'm getting a "ORA-00900: invalid SQL statement". This is really frustrating, how can I properly write a SQL Statement that will put this list of strings into a temporary table and then use it in a SELECT Statement to return the list I need?

There are couple possible places could cause this error, it seams that the "DECLARE GLOBAL TEMPORARY" is a JAVA API, I don't think .net has this function. Please try "Create global temporary table" instead. And, I don't know whether your internal API could handle multiple SQLs in one select sql. As far as I know, ODP.net Command class can only execute one sql per call. Moreover, "create table" is a DDL, it therefore has its own transaction. I can't see any reason we should put them in the same sql to execute. Following is a sample code for ODP.net,
using (OracleConnection conn = new OracleConnection(BD_CONN_STRING))
{
conn.Open();
using (OracleCommand cmd = new OracleCommand("create global temporary table t1(id number(9))", conn))
{
// actually this should execute once only
cmd.ExecuteNonQuery();
}
using (OracleCommand cmd = new OracleCommand("insert into t1 values (1)", conn)) {
cmd.ExecuteNonQuery();
}
// customer table is a permenant table
using (OracleCommand cmd = new OracleCommand("select c.id from customer c, t1 tmp1 where c.id=tmp1.id", conn)) {
cmd.ExecuteNonQuery();
}
}

Why does only the last item of comma-delimited list get processed by stored proc when called from C# web form?

I am using C# in Visual Studio 2013 and SQL Server 2012.
When the user of my ASP.NET web form enters a newline-separated list of codes and clicks submit, the code behind should read the values, concatenate them into a comma-delimited string, and pass the string to a method that calls a stored proc. The stored proc parses out the values and sets the active field to 1 on each record with a matching code.
The Product table is defined as:
id (PK, int, not null),
name (varchar(20), not null),
code (varchar(20), null),
active (bit, not null)
The Product table contains the following five records:
id name code active
-- ---- ---- ------
1 Product 1 AAA 0
2 Product 2 BBB 0
3 Product 3 CCC 0
4 Product 4 DDD 0
5 Product 5 EEE 0
I created the following stored proc:
CREATE PROCEDURE [dbo].[MarkListAsActive]
#codeList varchar(MAX)
AS
UPDATE
dbo.Product
SET
active = 1
WHERE
code IN (SELECT val FROM dbo.f_split(#codeList, ','))
dbo.f_split handles parsing the comma-delimited string. I copied it from this post: https://stackoverflow.com/a/17481595/2677169
If I execute the stored proc in SQL Server Management Studio, all five records get updated (as expected).
DECLARE #return_value int
EXEC #return_value = [dbo].[MarkListAsActive]
#codeList = N'AAA,BBB,CCC,DDD,EEE'
SELECT 'Return Value' = #return_value
GO
However, if I call the stored proc from the code behind of my .aspx page, only the last item in the list gets marked as active.
protected void SubmitButton_Click(object sender, EventArgs e)
{
string[] codeArray;
char separator = '\n';
OutputLabel.Text = "";
codeArray = ProductCodeTextBox.Text.Split(separator);
OutputLabel.Text += "The products with the following codes were marked as active:<br />";
string codes = "";
// TODO: Replace with regex that changes newlines to commas
for (int i = 0; i < codeArray.Length; i++)
{
codes += codeArray[i] + ",";
OutputLabel.Text += codeArray[i] + "<br />";
}
codes = codes.Substring(0, codes.Length - 1);
Utilities.Log(codes);
DataAccess.MarkListAsActive(codes);
}
public static void MarkListAsActive(string codeList)
{
try
{
string connectionString = ConfigurationManager.ConnectionStrings["ApplicationServices"].ConnectionString;
using (SqlConnection conn = new SqlConnection(connectionString))
using (SqlCommand command = new SqlCommand("[dbo].[MarkListAsActive]", conn)
{
CommandType = CommandType.StoredProcedure
})
{
conn.Open();
command.Parameters.Add("#codeList", codeList);
command.ExecuteNonQuery();
conn.Close();
}
}
catch (Exception ex)
{
Utilities.Log(String.Format("Error in MarkListAsActive: {0}\n{1}", ex.Message, ex.StackTrace));
}
return;
}
Note that I verified that the string being passed to MarkListAsActive() is correct.
Another approach: I tried looping through the codeArray and calling MarkListAsActive() for each item. Even this brute force (and inefficient) approach only updated the last item in the array.
Yet another aproach: I also tried a table valued parameter, but it too only updated the record corresponding to the last item in the input.
Thank you in advance for your help.

Change the IN clause to (SELECT val FROM dbo.f_split(#codeList, ',') where val is not null).
If that doesn't work follow these steps:
(You should know how to debug something like this anyway.)
First, use SQL Profiler to see what exactly is submitted to the database. See here for how to do that - its not as bad as it looks and is invaluable. If not what is expected, you have a problem on the front end.
Second, if the call to the database looks good then take the parameter you see in SQL Profiler and execute dbo.f_split() using it to see if you get back what you think you should.
Last, change your UPDATE statement into a SELECT statement and run it with what you did in the second step and see if you get back something that looks correct.
One of those steps will not return what is expected and that will lead you to the problem.

Your where clause is incorrect. You cannot do
... code in (select val ...
You need to join the result from the f_split function to the table.
For example:
UPDATE p
SET p.active = 1
FROM dbo.Product p inner join dbo.f_split(#codeList, ',') f
WHERE p.code = f.val;

PetaPoco not returning expected data

I have a BIT data type on one of my columns.
I have written a query that does SELECT * FROM TABLE WHERE BITCOLUMN <> #0
It works fine if I pass in 1 or 0 but if I pass in 3 PetaPoco doesn't return the results I expect.
Executing the SQL in a query window does return all records when I use 3 as the parameter value.
Any ideas?
UPDATE: If I use string SQL = "SELECT * FROM TABLE WHERE BITCOLUMN <> " + MethodParam; This returns data as expected.

Could you tell me what result you expecting? According to MSDN, A bit column can either be 1, 0 or null. It does not make senses to me when you pass 3 to it, since it will select all rows.
And my quick test shows that Petapoco behaves as expected.
using (var database = new Database("sql"))
{
string sql = "SELECT COUNT(*) FROM TBLTESTBIT WHERE BITCOLUMN <> " + "3";
var test = database.ExecuteScalar<long>("SELECT COUNT(*) FROM TBLTESTBIT WHERE BITCOLUMN <> #0", 3);
var test2 = database.ExecuteScalar<long>(sql);
Console.WriteLine(test == test2); // this output true
Console.Read();
}

I think it will be because when you use a parametrised sql string, you are passing in int with a value of 3 in as one of the args. PetaPoco will then create an IDataParameter for that argument and the DbType will be set to DbType.Int32 by default since PetaPoco has no idea what the underlying table column type is.

Use new Sql() not Append.
You should use Query = new PetaPoco.Sql( over Query = PetaPoco.Sql.Append

Oracle/c#: How do i use bind variables with select statements to return multiple records?

I have a question regarding Oracle bind variables and select statements.
What I would like to achieve is do a select on a number different values for the primary key. I would like to pass these values via an array using bind values.
select * from tb_customers where cust_id = :1
int[] cust_id = { 11, 23, 31, 44 , 51 };
I then bind a DataReader to get the values into a table.
The problem is that the resulting table only contains a single record (for cust_id=51). Thus it seems that each statement is executed independently (as it should), but I would like the results to be available as a collective (single table).
A workaround is to create a temporary table, insert all the values of cust_id and then do a join against tb_customers. The problem with this approach is that I would require temporary tables for every different type of primary key, as I would like to use this against a number of tables (some even have combined primary keys).
Is there anything I am missing?

Not asking the question as to why you would want to do this to begin with. Shouldn't the sql statement be something like
select * from tb_customers where cust_id = 11 or 23 or ...
Edit:
I am limited in Oracle but when I look at the documentation I think that you might have to do something like this:
variable i number
exec :i := 11
select * from tb_customers where cust_id = :i
This would allow you to take advantage of binding. You will have to add each record return to your own collection since it will still only return one at a time.

I know this was asked a while ago but not a brilliant answer.
I would do something like this - please excuse the crude psudo code
string bindList = "";
for(int ii=0;ii<cust_id.count;++ii)
{
if(ii == 0)
{
bindList += ":" + ii;
}
else
{
bindList += ",:" + ii;
}
OracleParameter param = new OracleParameter();
param.dbType = types.int;
param.value = cust_id[ii];
command.Parameters.Add(param);
}
query = "select * from tb_customers where cust_id in(" + bindList + ")";
So then query ends up having in(:1,:2,:3,etc) and each of these are bound separately.
There is also a similar question here: OracleParameter and IN Clause

String list in SqlCommand through Parameters in C#

Working with a SqlCommand in C# I've created a query that contains a IN (list...) part in the where clause. Instead of looping through my string list generating the list I need for the query (dangerous if you think in sqlInjection). I thought that I could create a parameter like:
SELECT blahblahblah WHERE blahblahblah IN #LISTOFWORDS
Then in the code I try to add a parameter like this:
DataTable dt = new DataTable();
dt.Columns.Add("word", typeof(string));
foreach (String word in listOfWords)
{
dt.Rows.Add(word);
}
comm.Parameters.Add("LISTOFWORDS", System.Data.SqlDbType.Structured).Value = dt;
But this doesn't work.
Questions:
Am I trying something impossible?
Did I took the wrong approach?
Do I have mistakes in this approach?
Thanks for your time :)

What you are trying to do is possible but not using your current approach. This is a very common problem with all possible solutions prior to SQL Server 2008 having trade offs related to performance, security and memory usage.
This link shows some approaches for SQL Server 2000/2005
SQL Server 2008 supports passing a table value parameter.
I hope this helps.

You want to think about where that list comes from. Generally that information is in the database somewhere. For example, instead of this:
SELECT * FROM [Table] WHERE ID IN (1,2,3)
You could use a subquery like this:
SELECT * FROM [Table] WHERE ID IN ( SELECT TableID FROM [OtherTable] WHERE OtherTableID= #OtherTableID )

If I understand right, you're trying to pass a list as a SQL parameter.
Some folks have attempted this before with limited success:
Passing Arrays to Stored Procedures
Arrays and Lists in SQL 2005
Passing Array of Values to SQL Server without String Manipulation
Using MS SQL 2005's XML capabilities to pass a list of values to a command

Am I trying something impossible?
No, it isn't impossible.
Did I took the wrong approach?
Your approach is not working (at least in .net 2)
Do I have mistakes in this approach?
I would try "Joel Coehoorn" solution (2nd answers) if it is possible.
Otherwise, another option is to send a "string" parameter with all values delimited by an separator. Write a dynamic query (build it based on values from string) and execute it using "exec".
Another solution will be o build the query directly from code. Somthing like this:
StringBuilder sb = new StringBuilder();
for (int i=0; i< listOfWords.Count; i++)
{
sb.AppendFormat("p{0},",i);
comm.Parameters.AddWithValue("p"+i.ToString(), listOfWords[i]);
}
comm.CommandText = string.Format(""SELECT blahblahblah WHERE blahblahblah IN ({0})",
sb.ToString().TrimEnd(','));
The command should look like:
SELECT blah WHERE blah IN (p0,p1,p2,p3...)...p0='aaa',p1='bbb'
In MsSql2005, "IN" is working only with 256 values.

I would recommend setting the parameter as a comma delimited string of values and use a Split function in SQL to turn that into a single column table of values and then you can use the IN feature.
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=50648 - Split Functions

If you want to pass the list as a string in a parameter, you could just build the query dynamically.
DECLARE #query varchar(500)
SET #query = 'SELECT blah blah WHERE blahblah in (' + #list + ')'
EXECUTE(#query)

I used to have the same problem, I think there is now way to do this directly over the ADO.NET API.
You might consider inserting the words into a temptable (plus a queryid or something) and then refering to that temptable from the query. Or dynamically creating the query string and avoid sql injection by other measures (e.g. regex checks).

This is an old question but I've come up with an elegant solution for this that I love to reuse and I think everyone else will find it useful.
First of all you need to create a FUNCTION in SqlServer that takes a delimited input and returns a table with the items split into records.
Here is the following code for this:
ALTER FUNCTION [dbo].[Split]
(
#RowData nvarchar(max),
#SplitOn nvarchar(5) = ','
)
RETURNS #RtnValue table
(
Id int identity(1,1),
Data nvarchar(100)
)
AS
BEGIN
Declare #Cnt int
Set #Cnt = 1
While (Charindex(#SplitOn,#RowData)>0)
Begin
Insert Into #RtnValue (data)
Select
Data = ltrim(rtrim(Substring(#RowData,1,Charindex(#SplitOn,#RowData)-1)))
Set #RowData = Substring(#RowData,Charindex(#SplitOn,#RowData)+1,len(#RowData))
Set #Cnt = #Cnt + 1
End
Insert Into #RtnValue (data)
Select Data = ltrim(rtrim(#RowData))
Return
END
You can now do something like this:
Select Id, Data from dbo.Split('123,234,345,456',',')
And fear not, this can't be susceptible to Sql injection attacks.
Next write a stored procedure that takes your comma delimited data and then you can write a sql statement that uses this Split function:
CREATE PROCEDURE [dbo].[findDuplicates]
#ids nvarchar(max)
as
begin
select ID
from SomeTable with (nolock)
where ID in (select Data from dbo.Split(#ids,','))
end
Now you can write a C# wrapper around it:
public void SomeFunction(List<int> ids)
{
var idsAsDelimitedString = string.Join(",", ids.Select(id => id.ToString()).ToArray());
// ... or however you make your connection
var con = GetConnection();
try
{
con.Open();
var cmd = new SqlCommand("findDuplicates", con);
cmd.Parameters.Add(new SqlParameter("#ids", idsAsDelimitedString));
var reader = cmd.ExecuteReader();
// .... do something here.
}
catch (Exception)
{
// catch an exception?
}
finally
{
con.Close();
}
}