I have a list Called ListTypes that holds 10 types of products. Below the store procedure loops and gets every record with the product that is looping and it stores it in the list ListIds. This is killing my sql box since I have over 200 users executing this constantly all day.
I know is not a good architecture to loop a sql statement, but this the only way I made it work. Any ideas how I can make this without looping? Maybe a Linq statement, I never used Linq with this magnitude. Thank you.
protected void GetIds(string Type, string Sub)
{
LinkedIds.Clear();
using (SqlConnection cs = new SqlConnection(connstr))
{
for (int x = 0; x < ListTypes.Count; x++)
{
cs.Open();
SqlCommand select = new SqlCommand("spUI_LinkedIds", cs);
select.CommandType = System.Data.CommandType.StoredProcedure;
select.Parameters.AddWithValue("#Type", Type);
select.Parameters.AddWithValue("#Sub", Sub);
select.Parameters.AddWithValue("#TransId", ListTypes[x]);
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
ListIds.Add(Convert.ToInt32(dr["LinkedId"]));
}
cs.Close();
}
}
}
Not a full answer, but this wouldn't fit in a comment. You can at least update your existing code to be more efficient like this:
protected List<int> GetIds(string Type, string Sub, IEnumerable<int> types)
{
var result = new List<int>();
using (SqlConnection cs = new SqlConnection(connstr))
using (SqlCommand select = new SqlCommand("spUI_LinkedIds", cs))
{
select.CommandType = System.Data.CommandType.StoredProcedure;
//Don't use AddWithValue! Be explicit about your DB types
// I had to guess here. Replace with the actual types from your database
select.Parameters.Add("#Type", SqlDBType.VarChar, 10).Value = Type;
select.Parameters.Add("#Sub", SqlDbType.VarChar, 10).Value = Sub;
var TransID = select.Parameters.Add("#TransId", SqlDbType.Int);
cs.Open();
foreach(int type in types)
{
TransID.Value = type;
SqlDataReader dr = select.ExecuteReader();
while (dr.Read())
{
result.Add((int)dr["LinkedId"]);
}
}
}
return result;
}
Note that this way you only open and close the connection once. Normally in ADO.Net it's better to use a new connection and re-open it for each query. The exception is in a tight loop like this. Also, the only thing that changes inside the loop this way is the one parameter value. Finally, it's better to design methods that don't rely on other class state. This method no longer needs to know about the ListTypes and ListIds class variables, which makes it possible to (among other things) do better unit testing on the method.
Again, this isn't a full answer; it's just an incremental improvement. What you really need to do is write another stored procedure that accepts a table valued parameter, and build on the query from your existing stored procedure to JOIN with the table valued parameter, so that all of this will fit into a single SQL statement. But until you share your stored procedure code, this is about as much help as I can give you.
Besides the improvements others wrote.
You could insert your ID's into a temp table and then make one
SELECT * from WhatEverTable WHERE transid in (select transid from #tempTable)
On a MSSQL this works really fast.
When you're not using a MSSQL it could be possible that one great SQL-Select with joins is faster than a SELECT IN. You have to test these cases by your own on your DBMS.
According to your comment:
The idea is lets say I have a table and I have to get all records from the table that has this 10 types of products. How can I get all of this products? But this number is dynamic.
So... why use a stored procedure at all? Why not query the table?
//If [Type] and [Sub] arguments are external inputs - as in, they come from a user request or something - they should be sanitized. (remove or escape '\' and apostrophe signs)
//create connection
string queryTmpl = "SELECT LinkedId FROM [yourTable] WHERE [TYPE] = '{0}' AND [SUB] = '{1}' AND [TRANSID] IN ({2})";
string query = string.Format(queryTmpl, Type, Sub, string.Join(", ", ListTypes);
SqlCommand select = new SqlCommand(query, cs);
//and so forth
To use Linq-to-SQL you would need to map the table to a class. This would make the query simpler to perform.
Related
my Code is working, the function gives me the correct Select count (*) value but anyway, it throws an ORA-25191 Exception - Cannot reference overflow table of an index-organized table tips,
at retVal = Convert.ToInt32(cmd.ExecuteScalar());
Since I use the function very often, the exceptions slow down my program tremendously.
private int getSelectCountQueryOracle(string Sqlquery)
{
try
{
int retVal = 0;
using (DataTable dataCount = new DataTable())
{
using (OracleCommand cmd = new OracleCommand(Sqlquery))
{
cmd.CommandType = CommandType.Text;
cmd.Connection = oraCon;
using (OracleDataAdapter dataAdapter = new OracleDataAdapter())
{
retVal = Convert.ToInt32(cmd.ExecuteScalar());
}
}
}
return retVal;
}
catch (Exception ex)
{
exceptionProtocol("Count Function", ex.ToString());
return 1;
}
}
This function is called in a foreach loop
// function call in foreach loop which goes through tablenames
foreach (DataRow row in dataTbl.Rows)
{...
tableNameFromRow = row["TABLE_NAME"].ToString();
tableRows=getSelectCountQueryOracle("select count(*) as 'count' from " +tableNameFromRow);
tableColumns = getSelectCountQueryOracle("SELECT COUNT(*) as 'count' FROM INFORMATION_SCHEMA.COLUMNS WHERE table_name='" + tableNameFromRow + "'");
...}
dataTbl.rows in this outer loop, in turn, comes from the query
SELECT * FROM USER_TABLES ORDER BY TABLE_NAME
If you're using a database-agnostic API like ADO.Net, you would almost always want to use the API's framework to fetch metadata rather than writing custom queries against each database's metadata tables. The various ADO.Net providers are much more likely to write data dictionary queries that handle all the various corner cases and are much more likely to be optimized than the queries you're likely to write. So rather than writing your own query to populate the dataTbl data table, you'd want to use the GetSchema method
DataTable dataTbl = connection.GetSchema("Tables");
If you want to keep your custom-coded data dictionary query for some reason, you'd need to filter out the IOT overflow tables since you can't query those directly.
select *
from user_tables
where iot_type IS NULL
or iot_type != 'IOT_OVERFLOW'
Be aware, however, that there are likely to be other tables that you don't want to try to get a count from. For example, the dropped column indicates whether a table has been dropped-- presumably, you don't want to count the number of rows in an object in the recycle bin. So you'd want a dropped = 'NO' predicate as well. And you can't do a count(*) on a nested table so you'd want to have a nested = 'NO' predicate as well if your schema happens to contain nested tables. There are probably other corner cases depending on the exact set of features your particular schema makes use of that the developers of the provider have added code for that you'd have to deal with.
So I'd start with
select *
from user_tables
where ( iot_type IS NULL
or iot_type != 'IOT_OVERFLOW')
and dropped = 'NO'
and nested = 'NO'
but know that you'll probably need/ want to add some additional filters depending on the specific features users make use of. I'd certainly much rather let the fine folks that develop the ADO.Net provider worry about all those corner cases than to deal with finding all of them myself.
Taking a step back, though, I'd question why you're regularly doing a count(*) on every table in a schema and why you need an exact answer. In most cases where you're doing counts, you're either doing a one-off where you don't much care how long it takes (i.e. a validation step after a migration) or approximate counts would be sufficient (i.e. getting a list of the biggest tables in the system in order to triage some effort or to track growth over time for projections) in which case you could just use the counts that are already stored in the data dictionary- user_tables.num_rows- from the last time that statistics were run.
This article helped me to solve my problem.
I've changed my query to this:
SELECT * FROM user_tables
WHERE iot_type IS NULL OR iot_type != 'IOT_OVERFLOW'
ORDER BY TABLE_NAME
How would I delete a row from a sql database, either with stored procedures or without, right now I have tried without, using a button press.
This is what I have so far, _memberid has been sent over from a differnt form from the database(For context).
private void btnDelete_Click(object sender, EventArgs e)
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = Lib.SqlConnection;
cmd.CommandType = CommandType.Text;
cmd.CommandText = "Delete * From Members where MemberId = " + _memberId;
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.DeleteCommand = cmd;
adapter.Fill(MembersDataTable); // Im fairly sure this is incorrect but i used it from old code
DialogResult = DialogResult.OK;
}
If you're trying to do a simple ADO.Net-based delete, then it would be somehting like his:
private void DeleteById(int memberId)
{
// or pull the connString from config somewhere
const string connectionString = "[your connection string]";
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var command = new SqlCommand("DELETE FROM Members WHERE MemberId = #memberId", connection))
{
command.Parameters.AddWithValue("#memberId", memberId);
command.ExecuteNonQuery();
}
}
Use parameter to prevent SQL injection.
There are essentially three main things I'm seeing...
One
You don't need the * in the query. DELETE affects the whole row, so there's no need to specify columns. So just something like:
DELETE FROM SomeTable WHERE SomeColumn = 123
Two
There's no need for a SqlDataAdapter here, all you need to do is execute the query. For example:
cmd.ExecuteNonQuery();
The "non query" is basically a SQL command which doesn't query data for results. Inserts, updates, and deletes are generally "non queries" in this context. What it would return is simply the number of rows affected, which you can use to double-check that it matches what you expect if necessary.
Three
Don't do this:
cmd.CommandText = "Delete From Members where MemberId = " + _memberId;
This kind of string concatenation leads to SQL injection. While it looks intuitively like you're using _memberId as a query value, technically you're using it as executable code. It's less likely (though not impossible) to be a problem for numeric values, but it's a huge problem for string values because it means the user can send you any string and you'll execute it as code.
Instead, use query parameters. For example, you might do something like this:
cmd.CommandText = "Delete From Members where MemberId = #memberId";
cmd.Parameters.Add("#memberId", SqlDbType.Int);
cmd.Parameters["#memberId"].Value = _memberId;
This tells the database engine itself that the value is a value and not part of the executing query, and the database engine knows how to safely handle values.
You could use a DataAdapter, but since you aren't using a datatable, it's just easier to do it without like this:
var sql = "DELETE FROM Members WHERE MemberId=#MemberId";
using(var cmd = new SqlCommand(sql, Lib.SqlConnection))
{
cmd.Connection.Open();
cmd.Parameters.Add("#MemberId",SqlDbType.Int).Value = _memberId;
cmd.ExecuteNonQuery();
}
And if you are using Dapper, you can do this:
Lib.SqlConnection.Execute("DELETE FROM Members WHERE MemberId=#MemberId", new {MemberId=_memberId});
If you are still using DataTables, I would highly recommend you look into using this (or something like this) to simplify your database accesses. It'll make CRUD logic on a database a breeze, and your code will me a lot more maintainable because you can get rid of all the odd needs to do casting, boxing/unboxing, and reduce the chances of runtime bugs because of the use of magic strings that happens so often with DataTables (column names). Once you start working with POCO classes, you'll hate having to use DataTables. That said, there are a few places where DataTables are a better solution (unknown data structures, etc), but those are usually pretty rare.
I have a C# based api and I send queries to a mysql server. I wonder how can i read the id from a select to a table on C# Note that I am using MySql.Data.MySqlClient;
My code until the execute is this one below. But in this step I wonder how can I retrieve the desired id. I used ExecuteNotQuery but it seems it does not fit on what I need.
string connectionString = #"server=x.x.x.x;userid=xxxx;password=xxxxxx;database=testdatabase";
string getLastStoryIdQuery = "SELECT MAX(ID) FROM testdatabase.test";
MySqlCommand getLastTestIdCommand = new MySqlCommand(getLastStoryIdQuery, mySqlConnection);
int lastId = getLastStoryIdCommand.ExecuteNonQuery();
How can I retrieve the result as an Integer or in worst case as a string response? Thank you in advance. :)
int lastId = Convert.ToInt32(getLastStoryIdCommand.ExecuteScalar());
You can find the documentation on MySqlCommand here: https://dev.mysql.com/doc/dev/connector-net/8.0/html/T_MySql_Data_MySqlClient_MySqlCommand.htm
The method ExecuteNonQuery returns the number affected by the query, while ExecuteScalar returns the first column of the first row. You can also use ExecuteReader to get a datareader so that you can read a resultset the database produces.
In practice, I rarely use DbCommand/DbReader anymore and prefer to just use Dapper for database access in most cases where performance isn't absolutely critical. It simplifies parameter creation, and object filling which serves the vast majority of my use cases.
Dapper would look like this:
string connectionString = #"server=x.x.x.x;userid=xxxx;password=xxxxxx;database=testdatabase";
string getLastStoryIdQuery = "SELECT MAX(ID) FROM testdatabase.test";
int lastId;
using(var conn = new MySqlConnection(connectionString))
{
lastId=conn.Query<int>(getLastStoryIdQuery).First();
// you can also do the following in this instance, but you will use the
// above for results that return multiple rows or multiple columns
//lastId=conn.ExecuteScalar<int>(getLastStoryIdQuery);
// Here is how you use parameters:
// var something = conn.ExecuteScalar<int>("SELECT id FROM testdatabase.test WHERE id=#param",new {param = 10});
// This gets multiple columns and rows into a List<person> (assuming you have a person class with fname,lname,dob properties):
// var people = conn.Query<person>("SELECT fname,lname,dob FROM persons WHERE dob>#start", new {start=new DateTime(2000,1,1)}).ToList();
}
Is there any way to parameterize an SQL INSERT statement (in C#), which inserts multiple rows? Currently I can think of only one way, to generate a statement for inserting mulitple rows, but that is quite open to SQL injection:
string sql = " INSERT INTO my_table"
+ " (a, b, c)"
+ " VALUES";
// Add each row of values to the statement
foreach (var item in collection) {
sql = sql
+ String.Format(" ({0}, {1}, {2}),",
aVal, bVal, cVal);
}
// Remove the excessive comma
sql = sql.Remove(sql.Length - 1);
What is the smarter/safer way to do this?
You could add paramaters inside the loop, like:
using (var comm = new SqlCommand()) {
var counter = 0;
foreach (var item in collection) {
sql = sql + String.Format(" (#a{0}, #b{0}, #c{0})," counter);
comm.Parameters.AddWithValue("#a" + counter, aVal);
comm.Parameters.AddWithValue("#b" + counter, bVal);
comm.Parameters.AddWithValue("#c" + counter, cVal);
counter++;
}
}
But I really wouldn't do a multi-row insert like this. IIRC the maximum amount of parameters in a query is about 2100, and this could get very big very fast. As you're looping through a collection anyway, you could just send it to the database in your loop, something like:
using (var con = new SqlConnection("connectionString here"))
{
con.Open();
var sql = "INSERT INTO my_table (a, b, c) VALUES (#a,#b,#c);"
using (var comm = new SqlCommand(sql, con))
{
comm.Parameters.Add("#a", SqlDbType.Int);
comm.Parameters.Add("#b", SqlDbType.NVarChar);
comm.Parameters.Add("#c", SqlDbType.Int);
foreach (var item in collection) {
{
comm.Parameters["#a"].Value = aVal;
comm.Parameters["#b"].Value = bVal;
comm.Parameters["#b"].Size = bVal.Length;
comm.Parameters["#c"].Value = cVal;
comm.ExecuteNonQuery();
}
}
}
The statement is prepared only once (and faster than a huge statement with 100's of parameters), and it doesn't fail all records when one record fails (add some exception handling for that). If you want to fail all when one record fails, you could wrap the thing up in a transaction.
Edit:
Ofcourse, when you regularly have to input 1000's of rows, this approach isn't the most efficient either, and your DBA might start to complain.
There are other approaches to this problem to remove the strain from the database: for example, create a stored procedure in your database that will insert the data from an xml document, or use Table Valued Parameters.
NYCdotNet wrote 2 nice blogs about these options, which I won't recreate here, but they're worth exploring (I'll paste some code below from the blog, as per guidelines, but credit where it's due: NYCdotNet)
XML document approach
Table Valued Parameters
The "meat" from the blog about TVP (in VB.NET but that shouldn't matter):
So I created this "generic" table-valued type:
CREATE TYPE dbo.UniqueIntegerList AS TABLE
(
TheInteger INT NOT NULL
PRIMARY KEY (TheInteger)
);
Creating the Save Stored Procedure
Next, I created a new stored procedure which would accept my new
Table-Valued Type as a parameter.
CREATE PROC DoTableValuedParameterInsert(#ProductIDs
dbo.UniqueIntegerList READONLY)
AS BEGIN
INSERT INTO ProductsAccess(ProductID)
SELECT TheInteger AS [ProductID]
FROM #ProductIDs;
END
In this procedure, I am passing in a parameter called #ProductIDs.
This is of type "dbo.UniqueIntegerList" which I just created in the
previous step. SQL Server looks at this and says "oh I know what this
is - this type is actually a table". Since it knows that the
UniqueIntegerList type is a table, I can select from it just like I
could select from any other table-valued variable. You have to mark
the parameter as READONLY because SQL 2008 doesn't support updating
and returning a passed table-valued parameter.
Creating the Save Routine
Then I had to create a new save routine on my business object that
would call the new stored procedure. The way you prepare the
Table-Valued parameter is to create a DataTable object with the same
column signature as the Table-Valued type, populate it, and then pass
it inside a SqlParameter object as SqlDbType.Structured.
Public Sub SaveViaTableValuedParameter()
'Prepare the Table-valued Parameter'
Dim objUniqueIntegerList As New DataTable
Dim objColumn As DataColumn =
objUniqueIntegerList.Columns.Add("TheInteger", _
System.Type.GetType("System.Int32"))
objColumn.Unique = True
'Populate the Table-valued Parameter with the data to save'
For Each Item As Product In Me.Values
objUniqueIntegerList.Rows.Add(Item.ProductID)
Next
'Connect to the DB and save it.'
Using objConn As New SqlConnection(DBConnectionString())
objConn.Open()
Using objCmd As New SqlCommand("dbo.DoTableValuedParameterInsert")
objCmd.CommandType = CommandType.StoredProcedure
objCmd.Connection = objConn
objCmd.Parameters.Add("ProductIDs", SqlDbType.Structured)
objCmd.Parameters(0).Value = objUniqueIntegerList
objCmd.ExecuteNonQuery()
End Using
objConn.Close()
End Using
End Sub
Recent bug report states that a method being called is crashing the service causing it to restart. After troubleshooting, the cause was found to be an obnoxious Oracle SQL call with thousands of strings passed. There is a collection of strings being passed to a method from an external service which often is more than 10,000 records. The original code used a where clause on the passed collection using the LIKE keyword, which I think is really, really bad.
public IList<ContainerState> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Join("OR", containerNumbers
.Select(item => string.Concat(" cntr_no LIKE '", item.SliceLeft(10), "%' ")))
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
Clarification of in house methods used which may be confusing:
DataBase.SelectQuery is an internal library method using generics which gets passed the sql string, a function to map the records to .NET objects, and the parameters being passed and returns an IEnumerable of Objects of type retuned by the Mapping function.
SliceLeft is an extension method from another internal helper library that just returns the first part of a string up to the number of characters specified by the parameter.
The reason that the LIKE statement was apparently used, is that the strings being passed and the strings in the database only are guaranteed to match the first 10 characters. Example ("XXXX000000-1" in the strings being passed should match a database record like "XXXX000000-8").
I believed that the IN clause using the SUBSTR would be more efficent than using multiple LIKE clauses and replaced the code with:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
String.Format(#"Select CTNR_NO, CNTR_STATE FROM CONTAINERS WHERE CTRN_SEQ = 0 AND ({0})",
string.Format("SUBSTR(CNTR_NO, 1, 10) IN ({0}) ",
string.Join(",", containerNumbers.Select(item => string.Format("\'{0}\'", item.SliceLeft(10) ) ) )
)
);
return DataBase.SelectQuery(sql, MapRecordToContainerState, new { }).ToList();
}
This helped slightly, and there were fewer issues in my tests, but when there are huge amounts of records passed, there is still an exception thrown and core dumps occur, as the SQL is longer than the server can parse during these times. The DBA suggests saving all the strings being passed to a temporary table, and then joining against that temp table.
Given that advice, I changed the function to:
public IList<ContainerRecord> GetContainerStates(IList<string> containerNumbers)
{
string sql =
#"
CREATE TABLE T1(cntr_num VARCHAR2(10));
DECLARE GLOBAL TEMPORARY TABLE SESSION.T1 NOT LOGGED;
INSERT INTO SESSION.T1 VALUES (:containerNumbers);
SELECT
DISTINCT cntr_no,
'_IT' cntr_state
FROM
tb_master
WHERE
cntr_seq = 0
AND cntr_state IN ({0})
AND adjustment <> :adjustment
AND SUBSTR(CTNR_NO, 1, 10) IN (SELECT CNTR_NUM FROM SESSION.T1);
";
var parameters = new
{
#containerNumbers = containerNumbers.Select( item => item.SliceLeft(10)).ToList()
};
return DataBase.SelectQuery(sql, MapRecordToContainerState, parameters).ToList();
}
Now I'm getting a "ORA-00900: invalid SQL statement". This is really frustrating, how can I properly write a SQL Statement that will put this list of strings into a temporary table and then use it in a SELECT Statement to return the list I need?
There are couple possible places could cause this error, it seams that the "DECLARE GLOBAL TEMPORARY" is a JAVA API, I don't think .net has this function. Please try "Create global temporary table" instead. And, I don't know whether your internal API could handle multiple SQLs in one select sql. As far as I know, ODP.net Command class can only execute one sql per call. Moreover, "create table" is a DDL, it therefore has its own transaction. I can't see any reason we should put them in the same sql to execute. Following is a sample code for ODP.net,
using (OracleConnection conn = new OracleConnection(BD_CONN_STRING))
{
conn.Open();
using (OracleCommand cmd = new OracleCommand("create global temporary table t1(id number(9))", conn))
{
// actually this should execute once only
cmd.ExecuteNonQuery();
}
using (OracleCommand cmd = new OracleCommand("insert into t1 values (1)", conn)) {
cmd.ExecuteNonQuery();
}
// customer table is a permenant table
using (OracleCommand cmd = new OracleCommand("select c.id from customer c, t1 tmp1 where c.id=tmp1.id", conn)) {
cmd.ExecuteNonQuery();
}
}