We Keep Coding

handling large amounts of data to include in an oracle select statement

Recent bug report states that a method being called is crashing the service causing it to restart. After troubleshooting, the cause was found to be an obnoxious Oracle SQL call with thousands of strings passed. There is a collection of strings being passed to a method from an external service which often is more than 10,000 records. The original code used a where clause on the passed collection using the LIKE keyword, which I think is really, really bad.
public IList<ContainerState> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Join("OR", containerNumbers
.Select(item => string.Concat(" cntr_no LIKE '", item.SliceLeft(10), "%' ")))
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
Clarification of in house methods used which may be confusing:
DataBase.SelectQuery is an internal library method using generics which gets passed the sql string, a function to map the records to .NET objects, and the parameters being passed and returns an IEnumerable of Objects of type retuned by the Mapping function.
SliceLeft is an extension method from another internal helper library that just returns the first part of a string up to the number of characters specified by the parameter.
The reason that the LIKE statement was apparently used, is that the strings being passed and the strings in the database only are guaranteed to match the first 10 characters. Example ("XXXX000000-1" in the strings being passed should match a database record like "XXXX000000-8").
I believed that the IN clause using the SUBSTR would be more efficent than using multiple LIKE clauses and replaced the code with:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Format("SUBSTR(CNTR_NO, 1, 10) IN ({0}) ",
string.Join(",", containerNumbers.Select(item => string.Format("\'{0}\'", item.SliceLeft(10) ) ) )
)
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
This helped slightly, and there were fewer issues in my tests, but when there are huge amounts of records passed, there is still an exception thrown and core dumps occur, as the SQL is longer than the server can parse during these times. The DBA suggests saving all the strings being passed to a temporary table, and then joining against that temp table.
Given that advice, I changed the function to:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
#"
CREATE TABLE T1(cntr_num VARCHAR2(10));
DECLARE GLOBAL TEMPORARY TABLE SESSION.T1 NOT LOGGED;
INSERT INTO SESSION.T1 VALUES (:containerNumbers);
SELECT
DISTINCT cntr_no,
'_IT' cntr_state
FROM
tb_master
WHERE
cntr_seq = 0
AND cntr_state IN ({0})
AND adjustment <> :adjustment
AND SUBSTR(CTNR_NO, 1, 10) IN (SELECT CNTR_NUM FROM SESSION.T1);
";
var parameters = new
{
#containerNumbers = containerNumbers.Select( item => item.SliceLeft(10)).ToList()
};
return DataBase.SelectQuery(sql, MapRecordToContainerState, parameters).ToList();
}
Now I'm getting a "ORA-00900: invalid SQL statement". This is really frustrating, how can I properly write a SQL Statement that will put this list of strings into a temporary table and then use it in a SELECT Statement to return the list I need?

There are couple possible places could cause this error, it seams that the "DECLARE GLOBAL TEMPORARY" is a JAVA API, I don't think .net has this function. Please try "Create global temporary table" instead. And, I don't know whether your internal API could handle multiple SQLs in one select sql. As far as I know, ODP.net Command class can only execute one sql per call. Moreover, "create table" is a DDL, it therefore has its own transaction. I can't see any reason we should put them in the same sql to execute. Following is a sample code for ODP.net,
using (OracleConnection conn = new OracleConnection(BD_CONN_STRING))
{
conn.Open();
using (OracleCommand cmd = new OracleCommand("create global temporary table t1(id number(9))", conn))
{
// actually this should execute once only
cmd.ExecuteNonQuery();
}
using (OracleCommand cmd = new OracleCommand("insert into t1 values (1)", conn)) {
cmd.ExecuteNonQuery();
}
// customer table is a permenant table
using (OracleCommand cmd = new OracleCommand("select c.id from customer c, t1 tmp1 where c.id=tmp1.id", conn)) {
cmd.ExecuteNonQuery();
}
}

Convert C# SQL Loop to Linq

I have a list Called ListTypes that holds 10 types of products. Below the store procedure loops and gets every record with the product that is looping and it stores it in the list ListIds. This is killing my sql box since I have over 200 users executing this constantly all day.
I know is not a good architecture to loop a sql statement, but this the only way I made it work. Any ideas how I can make this without looping? Maybe a Linq statement, I never used Linq with this magnitude. Thank you.
protected void GetIds(string Type, string Sub)
{
LinkedIds.Clear();
using (SqlConnection cs = new SqlConnection(connstr))
{
for (int x = 0; x < ListTypes.Count; x++)
{
cs.Open();
SqlCommand select = new SqlCommand("spUI_LinkedIds", cs);
select.CommandType = System.Data.CommandType.StoredProcedure;
select.Parameters.AddWithValue("#Type", Type);
select.Parameters.AddWithValue("#Sub", Sub);
select.Parameters.AddWithValue("#TransId", ListTypes[x]);
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
ListIds.Add(Convert.ToInt32(dr["LinkedId"]));
}
cs.Close();
}
}
}

Not a full answer, but this wouldn't fit in a comment. You can at least update your existing code to be more efficient like this:
protected List<int> GetIds(string Type, string Sub, IEnumerable<int> types)
{
var result = new List<int>();
using (SqlConnection cs = new SqlConnection(connstr))
using (SqlCommand select = new SqlCommand("spUI_LinkedIds", cs))
{
select.CommandType = System.Data.CommandType.StoredProcedure;
//Don't use AddWithValue! Be explicit about your DB types
// I had to guess here. Replace with the actual types from your database
select.Parameters.Add("#Type", SqlDBType.VarChar, 10).Value = Type;
select.Parameters.Add("#Sub", SqlDbType.VarChar, 10).Value = Sub;
var TransID = select.Parameters.Add("#TransId", SqlDbType.Int);
cs.Open();
foreach(int type in types)
{
TransID.Value = type;
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
result.Add((int)dr["LinkedId"]);
}
}
}
return result;
}
Note that this way you only open and close the connection once. Normally in ADO.Net it's better to use a new connection and re-open it for each query. The exception is in a tight loop like this. Also, the only thing that changes inside the loop this way is the one parameter value. Finally, it's better to design methods that don't rely on other class state. This method no longer needs to know about the ListTypes and ListIds class variables, which makes it possible to (among other things) do better unit testing on the method.
Again, this isn't a full answer; it's just an incremental improvement. What you really need to do is write another stored procedure that accepts a table valued parameter, and build on the query from your existing stored procedure to JOIN with the table valued parameter, so that all of this will fit into a single SQL statement. But until you share your stored procedure code, this is about as much help as I can give you.

Besides the improvements others wrote.
You could insert your ID's into a temp table and then make one
SELECT * from WhatEverTable WHERE transid in (select transid from #tempTable)
On a MSSQL this works really fast.
When you're not using a MSSQL it could be possible that one great SQL-Select with joins is faster than a SELECT IN. You have to test these cases by your own on your DBMS.

According to your comment:
The idea is lets say I have a table and I have to get all records from the table that has this 10 types of products. How can I get all of this products? But this number is dynamic.
So... why use a stored procedure at all? Why not query the table?
//If [Type] and [Sub] arguments are external inputs - as in, they come from a user request or something - they should be sanitized. (remove or escape '\' and apostrophe signs)
//create connection
string queryTmpl = "SELECT LinkedId FROM [yourTable] WHERE [TYPE] = '{0}' AND [SUB] = '{1}' AND [TRANSID] IN ({2})";
string query = string.Format(queryTmpl, Type, Sub, string.Join(", ", ListTypes);
SqlCommand select = new SqlCommand(query, cs);
//and so forth
To use Linq-to-SQL you would need to map the table to a class. This would make the query simpler to perform.

A better way to achieve INSERT without hitting the database multiple times

I have the following, I could make it work as I want to but I think i'm doing it the wrong way, could you please explain how this could be done in a more efficient way ? While also looping on Categories and doing the same as with Districts within the same Insert() Method.
Thanks in advance.
#region Methods
public int Insert(List<District> Districts, List<Category> Categories)
{
StringBuilder sqlString = new StringBuilder("INSERT INTO Stores (name, image) VALUES (#Name, #Image);");
using (SqlConnection sqlConnection = new
SqlConnection(ConfigurationManager.ConnectionStrings["OahuDB"].ConnectionString))
{
SqlCommand sqlCommand = new SqlCommand(sqlString.ToString(), sqlConnection);
sqlCommand.Parameters.AddWithValue("#Name", this.Name);
sqlCommand.Parameters.AddWithValue("#Image", this.Image);
sqlConnection.Open();
int x = (int)sqlCommand.ExecuteScalar();
sqlString.Clear();
sqlCommand.Parameters.Clear();
foreach (District item in Districts)
{
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
return x;
}
}
EDIT
Is is wrong to achieve the above by doing the following ?
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (District item in Districts)
{
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO categories_has_stores (category_id, store_id) VALUES (#CategoryID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (Category item in Categories)
{
sqlCommand.Parameters.AddWithValue("#CategoryID", item.ID);
sqlCommand.ExecuteNonQuery();
}

The first obvious thing is to move the invariant part of the sqlCommand out of the loop
sqlCommand.Parameters.Clear();
sqlString.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", 0); // as dummy value
sqlCommand.Parameters.AddWithValue("#StoreID", x); // invariant
foreach (District item in Districts)
{
sqlCommand.Parameters["#DistrictID"].Value = item.ID;
sqlCommand.ExecuteNonQuery();
}
But this doesn't answer your fundamental problem. How to avoid hitting the database multiple times.
You could build a query with multiple inserts like this
sqlString.Clear();
sqlString.Append("INSERT INTO districts_has_stores (district_id, store_id) VALUES (");
foreach(District item in Districts)
{
sqlString.Append(item.ID.ToString);
sqlString.Append(", ")
sqlString.Append(x.ToString());
sqlString.Append("),");
}
sqlString.Length--;
sqlCommand.CommandText = sqlString.ToString()
But string concatenation is really a bad practice and I present this solution just as an example and I don't want to suggest this kind of approach.
The last possibility are Table-Valued Parameters (Only from SqlServer 2008).
First you need to create a Sql Type for the table you will pass in
CREATE TYPE dbo.DistrictsType AS TABLE
( DistrictID int, StoreID int )
and a StoredProcedure that will insert the data from the datatable passed in
CREATE PROCEDURE usp_InsertDistricts
(#tvpNewDistricts dbo.DistrictsType READONLY)
AS
BEGIN
INSERT INTO dbo.Districts (DistrictID, StoreID)
SELECT dt.DistrictID, dt.StoreID FROM #tvpNewDistricts AS dt;
END
then, back to your code you pass the district into the storedprocedure
(Probably you need to convert your List in a DataTable)
DataTable dtDistricts = ConvertListToDataTable(Districts);
SqlCommand insertCommand = new SqlCommand("usp_InsertDistricts", sqlConnection);
SqlParameter p1 = insertCommand.Parameters.AddWithValue("#tvpNewDistricts", dtDistricts);
p1.SqlDbType = SqlDbType.Structured;
p1.TypeName = "dbo.DistrictsType";
insertCommand.ExecuteNonQuery();
Well, if you look back at the link above, you will find other ways to pass your data in a single step to the database backend.... (Scroll to the end and you will find also a method that doesn't require a stored procedure on the database)

Assuming Stores has an identity column, in SQL Server, create a table type and a table-valued parameter to take advantage of it:
CREATE TYPE dbo.DistrictsTVP AS TABLE
(
DistrictID INT -- PRIMARY KEY? I hope so.
);
GO
CREATE PROCEDURE dbo.InsertStoreAndDistricts
#Name NVARCHAR(255),
#Image <some data type???>,
#Districts dbo.DistrictsTVP READONLY
AS
BEGIN
SET NOCOUNT ON;
DECLARE #StoreID INT;
INSERT dbo.Stores(name, [image]) SELECT #Name, #Image;
SET #StoreID = SCOPE_IDENTITY();
INSERT dbo.district_has_stores(district_id, store_id)
SELECT DistrictID, #StoreID
FROM #Districts;
END
GO
Then in C#, you can pass your List in directly without any looping:
using (...)
{
SqlCommand cmd = new SqlCommand("dbo.InsertStoreAndDistricts", sqlConnection);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#Districts", Districts);
tvparam.SqlDbType    = SqlDbType.Structured;
// other params here - name and image
cmd.ExecuteNonQuery();
}

Recently in my project i used XML as a data type in my stored proc and did insert update and delete in just one shot instead of hitting the database multiple times .
Sample Stored proc
ALTER PROCEDURE [dbo].[insertStore]
#XMLDATA xml,
#name varchar(50),
#image datatype
AS
Begin
INSERT INTO Store
(name
,image
)
Select XMLDATA.item.value('#name[1]', 'varchar(10)') AS Name,
XMLDATA.item.value('#image[1]', 'yourData type') AS Image
FROM #XMLDATA.nodes('//Stores/InsertList/Store') AS XMLDATA(item)
END
Similarly you can write for update and delete .In C# u need to create the xml
public string GenerateXML(List<District> Districts)
var xml = new StringBuilder();
var insertxml = new StringBuilder();
xml.Append("<Stores>");
for (var i = 0; i < Districts.Count; i++)
{ var obj = Districts[i];
insertxml.Append("<Store");
insertxml.Append(" Name=\"" + obj.Name + "\" ");
insertxml.Append(" Image=\"" + obj.Image + "\" ");
insertxml.Append(" />");
}
xml.Append("<InsertList>");
xml.Append(insertxml.ToString());
xml.Append("</InsertList>");
SqlCommand cmd= new SqlCommand("insertStore",connectionString);
cmd.CommandType=CommandType.StoredProcedure;
SqlParameter param = new SqlParameter ();
param.ParameterName ="#XMLData";
param.value=xml;
paramter.Add(param);
cmd.ExecuteNonQuery();

Personally, I would create a stored procedure for the insert and pass in a Table-Valued param, which would allow you to do
INSERT tbl (f1, f2, ... fN)
SELECT * FROM #TVP
http://msdn.microsoft.com/en-us/library/bb510489.aspx
Unless you're using SQL 2005, then I would use an XML param in my stored proc and Serialize a collection to be inserted.

Think about your system design. Where is the data that you need to insert coming from? If it's already in the database, or another database, or some other kind of data store, you should be able to achieve a more bulk kind of transfer, simply inserting from one database to the other in a loop in stored procedure.
If the data is coming from a user, or some incompatible data store, like say an export from some third party program, then you basically have to realize that to get it into the database will involve quite of few round-trips to the database. You can use some tables, or XML or such , but those are actually closer to doing a bulk insert using other methods.
The bottom line is that SQL databases are designed to do inserts one at a time. This is 99% of the time OK because you are never asking users using the UI to type in thousands of things at one time.

How to pass sqlparameter to IN()? [duplicate]

This question already has answers here:
Pass Array Parameter in SqlCommand
(11 answers)
Closed 6 years ago.
For some reason the Sqlparameter for my IN() clause is not working. The code compiles fine, and the query works if I substitute the parameter with the actual values
StringBuilder sb = new StringBuilder();
foreach (User user in UserList)
{
sb.Append(user.UserId + ",");
}
string userIds = sb.ToString();
userIds = userIds.TrimEnd(new char[] { ',' });
SELECT userId, username
FROM Users
WHERE userId IN (#UserIds)

You have to create one parameter for each value that you want in the IN clause.
The SQL needs to look like this:
SELECT userId, username
FROM Users
WHERE userId IN (#UserId1, #UserId2, #UserId3, ...)
So you need to create the parameters and the IN clause in the foreach loop.
Something like this (out of my head, untested):
StringBuilder sb = new StringBuilder();
int i = 1;
foreach (User user in UserList)
{
// IN clause
sb.Append("#UserId" + i.ToString() + ",");
// parameter
YourCommand.Parameters.AddWithValue("#UserId" + i.ToString(), user.UserId);
i++;
}

Possible "cleaner" version:
StringBuilder B = new StringBuilder();
for (int i = 0; i < UserList.Count; i++)
YourCommand.Parameters.AddWithValue($"#UserId{i}", UserList[i].UserId);
B.Append(String.Join(",", YourCommand.Parameters.Select(x => x.Name)));

If you are using SQL 2008, you can create a stored procedure which accepts a Table Valued Parameter (TVP) and use ADO.net to execute the stored procedure and pass a datatable to it:
First, you need to create the Type in SQL server:
CREATE TYPE [dbo].[udt_UserId] AS TABLE(
[UserId] [int] NULL
)
Then, you need to write a stored procedure which accepts this type as a parameter:
CREATE PROCEDURE [dbo].[usp_DoSomethingWithTableTypedParameter]
(
#UserIdList udt_UserId READONLY
)
AS
BEGIN
SELECT userId, username
FROM Users
WHERE userId IN (SELECT UserId FROM #UserIDList)
END
Now from .net, you cannot use LINQ since it does not support Table Valued Parameters yet; so you have to write a function which does plain old ADO.net, takes a DataTable, and passes it to the stored procedure: I've written a generic function I use which can do this for any stored procedure as long as it takes just the one table-typed parameter, regardless of what it is;
public static int ExecStoredProcWithTVP(DbConnection connection, string storedProcedureName, string tableName, string tableTypeName, DataTable dt)
{
using (SqlConnection conn = new SqlConnection(connection.ConnectionString))
{
SqlCommand cmd = new SqlCommand(storedProcedureName, conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter p = cmd.Parameters.AddWithValue(tableName, dt);
p.SqlDbType = SqlDbType.Structured;
p.TypeName = tableTypeName;
conn.Open();
int rowsAffected = cmd.ExecuteNonQuery(); // or could execute reader and pass a Func<T> to perform action on the datareader;
conn.Close();
return rowsAffected;
}
}
Then you can write DAL functions which use this utility function with actual names of stored procedures; to build on the example in your question, here is what the code would look like:
public int usp_DoSomethingWithTableTypedParameter(List<UserID> userIdList)
{
DataTable dt = new DataTable();
dt.Columns.Add("UserId", typeof(int));
foreach (var userId in updateList)
{
dt.Rows.Add(new object[] { userId });
}
int rowsAffected = ExecStoredProcWithTVP(Connection, "usp_DoSomethingWithTableTypedParameter", "#UserIdList", "udt_UserId", dt);
return rowsAffected;
}
Note the "connection" parameter above - I actually use this type of function in a partial DataContext class to extend LINQ DataContext with my TVP functionality, and still use the (using var context = new MyDataContext()) syntax with these methods.
This will only work if you are using SQL Server 2008 - hopefully you are and if not, this could be a great reason to upgrade! Of course in most cases and large production environments this is not that easy, but FWIW I think this is the best way of doing this if you have the technology available.

SQL Server sees your IN clause as:
IN ('a,b,c')
What it needs to look like is:
IN ('a','b','c')
There is a better way to do what you're trying to do.
If the user id's are in the DB, then the IN clause should be changed to a subquery, like so:
IN (SELECT UserID FROM someTable WHERE someConditions)
This is a hack -- it doesn't work well with indexes, and you have to be careful it works right with your data, but I've used it successfully in the past:
#UserIDs LIKE '%,' + UserID + ',%' -- also requires #UserID to begin and end with a comma

C# with SQL Server SELECT WHERE IN with data list

I have a SQL Server database with person_id and name fields.
In my application I have a array of person_id, I need to get from my database, person records with ids from my array.

If you just want the SQL string, this should work:
var sql = "select Person_Name from tbl_Person_Info where Person_Id in ("
+ string.Join( ",", PersonIdList )
+ ")";
Note that there is a limit (2000, I think) on the number of items allowed in the IN clause. Also, depending on which version of .NET you're using, string.Join might have different argument types that don't allow a list of integers (are they integers?) to be used. You might have to convert them to strings first before you can join them.
EDIT: Please be aware that if the PersonIdList items come from user input (and are strings), this is very dangerous. I'd suggest using a newer .NET technology in any event that would allow you to handle this much more safely -- such as LINQ or EF.

Now behold the awful power of the table-valued parameter! (provided that you're using SQL Server 2008)
Essentially, this is the means to pass your array of integers, properly typed, to a stored procedure... i.e.: no string concatenation / sql injection. Mainly this all centers around creating a SQL table-type having a single integer column... then you just pass a .NET DataTable (of the same structure) to a stored procedure expecting said type.
Step #1: Create a table-type (on SQL Server) for passing a series of integers. You only need to do this once so don't go placing it in your stored procedure.
create type IntegerValues as table (IntegerValue int)
Step #2: Create your stored procedure (on SQL Server).
create procedure dbo.GetPersonsByID
(
#PersonIDs IntegerValues readonly -- must be readonly
)
as begin
select
p.*
from [YourPersonTable] as p
join #PersonIDs as pi
on pi.[IntegerValue] = p.[Person_ID];
end
Step #3: Call your stored procedure from C#
// Written from my laptop straight into the textarea... so, it's untested.
public DataTable GetPersonsByIDs(int[] personIDs)
{
var dtResults = new DataTable();
var dtPersonIDs = new DataTable();
dtPersonIDs.Columns.Add("IntegerValue", typeof(int));
foreach(int id in personIDs)
{
dtPersonIDs.Rows.Add(id);
}
using(dtPersonIDs)
using(var cnx = new SqlConnection("YourConnectionString"))
using(var cmd = new SqlCommand {
Connection = cnx,
CommandText = "dbo.GetPersonsByIDs",
CommandType = CommandType.StoredProcedure,
Parameters = {
new SqlParameter {
ParameterName = "PersonIDs",
SqlDbType = SqlDbType.Structured, // must be structured
Value = dtPersonIDs,
}
}
})
{
try
{
cnx.Open();
using (var reader = cmd.ExecuteReader())
{
dt.Load(reader);
}
return dtResults;
}
catch(Exception ex)
{
throw new Exception("Error executing GetPersonsByIDs.", ex);
}
}
}