We Keep Coding

How to execute DataAdapter.Fill() simultaneously

I have been working on an ASP.Net application for a long time and there are above 10 clients using the application. But now I found a problem in the application, that is, I have a stored procedure call which takes about 30 seconds to execute. It is not a problem, because the SQL code highly complicated and looping many times. The problem is :
Whenever that stored procedure call is executing, I am not able to using any other functions or stored procedure call.
When I tried debugging, the problem is that 'DataAdapter.Fill()' function is waiting for the first stored procedure call to finish.
My code that executes stored procedure call and returning data is :
public static DataSet ExecuteQuery_SP(string ProcedureName, object[,] ParamArray)
{
SqlDataAdapter DataAdapter = new SqlDataAdapter();
DataSet DS = new DataSet();
try
{
if (CON.State != ConnectionState.Open)
OpenConnection();
SqlCommand cmd = new SqlCommand();
cmd.CommandTimeout = 0;
cmd.CommandText = ProcedureName;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection = CON;
cmd.Transaction = SqlTrans;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
cmd.Parameters.AddWithValue(ParamName, ParamValue);
}
DataAdapter = new SqlDataAdapter(cmd);
DataAdapter.Fill(DS);
cmd.CommandText = "";
}
catch (Exception ea)
{
}
return DS;
}
All stored procedure calls are working through this function. Hence when my first stored procedure call 'A' is running, stored procedure call 'B' will not execute until 'A' is finished.
This reduces overall performance of the application and causes problem in data retrieval.
I surfed google and found that 'Threading' can be helpful but I am not able to execute threading properly. I am not so familiar with these kind of things. It will helpful if you can rectify the problem.
My first stored procedure call is:
ds = DB.ExecuteQuery_SP("SelectOutstandingReportDetailed", parArray);
Where ds is the DataSet object.
Second stored procedure call is :
ds = DB.ExecuteQuery_SP("[SelectAccLedgersDetailsByID]", ParamArray);
My current DB connection open function is :
public static bool OpenConnection()
{
try
{
Server = (String)HttpContext.GetGlobalResourceObject("Resource", "Server");
DBName = (String)HttpContext.GetGlobalResourceObject("Resource", "DBName");
UserName = (String)HttpContext.GetGlobalResourceObject("Resource", "UserName");
PassWord = (String)HttpContext.GetGlobalResourceObject("Resource", "PassWord");
string ConnectionString;
ConnectionString = "server=" + Server + "; database=" + DBName + "; uid=" + UserName + "; pwd=" + PassWord + "; Pooling='true';Max Pool Size=100;MultipleActiveResultSets=true;Asynchronous Processing=true";
CON.ConnectionString = ConnectionString;
if (CON.State != ConnectionState.Open)
{
CON.Close();
CON.Open();
}
}
catch (Exception ea)
{
}
return false;
}
Where 'CON' is a public SqlConnection variable
static SqlConnection CON = new SqlConnection();
I found the problem, that is, all stored procedure calls are performed through this 'CON' object. If there is seperate SqlConnection object for each stored procedure call, then no problem is there.
So is it possible to make separate SqlConnection for every ExecuteQuery_SP calls.
If any doubt in question, kindly comment.
Thank you

SQL Server will allow thousands of connections simultaneously, by default. This is NOT the source of your problem. You have forced every call to a stored procedure to be funneled through a single method. Factor out your calls to the stored procedures - in other words, lose the ExecuteQuery_SP method which is a bottleneck. Then test again.
Here's is an introduction to data layers.

Heres the simplest version I can create for you.
Important: to understand you should read about async-await.
You can start at the Microsoft C# Async-Await Docs
// TODO set up your connection string
private string connectionString = "<your connection string>";
// Gets data assyncronously
public static async Task<DataTable> GetDataAsync(string procedureName, object[,] ParamArray)
{
try
{
var asyncConnectionString = new SqlConnectionStringBuilder(connectionString)
{
AsynchronousProcessing = true
}.ToString();
using (var conn = new SqlConnection(asyncConnectionString))
{
using (var SqlCommand = new SqlCommand())
{
SqlCommand.Connection = conn;
SqlCommand.CommandText = procedureName;
SqlCommand.CommandType = CommandType.StoredProcedure;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
SqlCommand.Parameters.AddWithValue(ParamName, ParamValue);
}
conn.Open();
var data = new DataTable();
data.BeginLoadData();
using (var reader = await SqlCommand.ExecuteReaderAsync().ConfigureAwait(true))
{
if (reader.HasRows)
data.Load(reader);
}
data.EndLoadData();
return data;
}
}
}
catch (Exception Ex)
{
// Log error or something else
throw;
}
}
public static async Task<DataTable> GetData(object General, object Type, string FromDate, string ToDate)
{
object[,] parArray = new object[,]{
{"#BranchID",General.BranchID},
{"#FinancialYearID",General.FinancialYearID},
{"#Type",Type},
{"#FromDate",DateTime.ParseExact(FromDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)},
{"#ToDate",DateTime.ParseExact(ToDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)}
};
return await DataBaseHelper.GetDataAsync("SelectOutstandingReportDetailed", parArray);
}
// Calls database assyncronously
private async Task ConsumeData()
{
DataTable dt = null;
try
{
// TODO configure your parameters here
object general = null;
object type = null;
string fromDate = "";
string toDate = "";
dt = await GetData(general, type, fromDate, toDate);
}
catch (Exception Ex)
{
// do something if an error occurs
System.Diagnostics.Debug.WriteLine("Error occurred: " + Ex.ToString());
return;
}
foreach (DataRow dr in dt.Rows)
{
System.Diagnostics.Debug.WriteLine(dr.ToString());
}
}
// Fired when some button is clicked. Get and use the data assyncronously, i.e. without blocking the UI.
private async void button1_Click(object sender, EventArgs e)
{
await ConsumeData();
}

Connect to database in sepaerate method of my SqlCommand

I have a form that checks whether values are in a database before adding them. Each field is in a different table, and to keep everything clean, I have a checkExists method for each field. Is there a way to have a separate method that connects to the database, so that I don't have to connect in every field method?
I'd like to do something like this so that my code is less messy:
public void SetConnection()
{
SqlConnection myConnection =
new SqlConnection("user id=[username];" +
"password=[password];" +
"server=[server];" +
"database=[db_name];");
try
{
myConnection.Open();
}
catch(Exception e)
{
Console.WriteLine("Unable to Connect");
}
}
public Boolean CheckData_Company(string[] items)
{
Class_DB set_conn = new Class_DB();
try
{
set_conn.SetConnection();
}
catch(Exception e)
{
Console.WriteLine(e.ToString());
}
//check that item does not already exist
string query_string = "SELECT * FROM CR_Company WHERE ([CompanyName] = #companyName";
SqlCommand check_Company = new SqlCommand(query_string, set_conn);
check_Company.Parameters.AddWithValue("#CompanyName", items[0]);
int CompanyExist = (int)check_Company.ExecuteScalar();
if(CompanyExist > 0)
{
return true;
}
else
{
return false;
}
}
But I get a
local variable set_conn
Argument 2: Cannot Convert from Class_DB to System.Data.SqlClient.SqlConnection
I understand the error, so what can I do to return the correct value, or do I have to establish a connection within my CheckData_Comany() method?

Your method SetConnection should be returning SqlConnection back like:
public SqlConnection SetConnection()
{
SqlConnection myConnection = new SqlConnection("user id=[username];" +
"password=[password];" +
"server=[server];" +
"database=[db_name];");
try
{
myConnection.Open();
}
catch(Exception e)
{
Console.WriteLine("Unable to Connect");
}
return myConnection;
}
and then you can have something like:
SqlConnection connection = set_conn.SetConnection();
and then pass it in SqlCommand constructor as parameter :
SqlCommand check_Company = new SqlCommand(query_string, connection);
Your complete method implementation would become :
public Boolean CheckData_Company(string[] items)
{
bool Exists = false;
Class_DB set_conn = new Class_DB();
SqlConnection connection = null;
try
{
connection = set_conn.SetConnection();
//check that item does not already exist
string query_string = "SELECT * FROM CR_Company WHERE ([CompanyName] = #companyName";
SqlCommand check_Company = new SqlCommand(query_string, set_conn);
check_Company.Parameters.AddWithValue("#CompanyName", items[0]);
int CompanyExist = (int)check_Company.ExecuteScalar();
if(CompanyExist > 0)
Exists = true;
}
catch(Exception e)
{
Console.WriteLine(e.ToString());
}
finally
{
connection.Close();
}
return Exists;
}
and important thing to note is do not forget the close the connection finally by calling connection.Close(), otherwise it might cause eating up the resources that shouldn't happen when we are done with querying the database and we should release the resources occupied.

ORA-00604, ORA-01000 errors in C# with Oracle; max open cursors exceeded

Hi all: I have a program that is running 4 threads that talk to an Oracle database. I have the Oracle connections local to each thread, and I'm employing the USING statement as well as manually closing the recordset and closing the connection. As I understand it, the ORA-01000 error arises when there are more open recordsets than configured cursors on the database. I do not understand why my recordsets are staying open or why I'm getting this error. Here's the code:
static void CheckPaths()
{
int pathcount = paths.Count; //paths is a typed list
Parallel.ForEach(paths, new ParallelOptions { MaxDegreeOfParallelism = 4 }, (p) =>
{
try
{
CheckSinglePathAllHours(p);
}
catch (Exception ex)
{
//there is logging here, this is where the exception hits
}
});
}
static void CheckSinglePathAllHours(Path p)
{
string sqlBase = #"Select * from table ";//this is actually a big SQL statement
using (DBManager localdbm = new DBManager())
{
string sql = sqlBase;
OracleDataReader reader = localdbm.GetData(sql);
while (reader.Read())
{
//process the path, query always returns 24 or less rows
}
reader.Close();
reader = null; //is this even necessary?
localdbm.Close(); //is this necessary in conjunction with the USING statement?
}
}
class DBManager : IDisposable
{
OracleConnection conn;
OracleCommand cmd;
public DBManager()
{
string connStr = "blah blah blah";
conn = new OracleConnection(connStr);
conn.Open();
cmd = conn.CreateCommand();
}
public OracleDataReader GetData(string sql)
{
cmd.CommandText = sql;
cmd.CommandTimeout = 900;
return cmd.ExecuteReader();
}
public void RunSQL(string sql)
{
cmd.CommandText = sql;
cmd.CommandTimeout = 900;
cmd.ExecuteNonQuery();
}
public void Close()
{
conn.Close();
}
public void Dispose()
{
try
{
conn.Close();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
The code will usually run for about a minute or two before the exception. The exception message is two-fold: ORA-00604: error occured at recursive SQL level 1; and ORA-01000: maximum open cursors exceeded. Any ideas what I'm doing wrong?

Changed the code to call .Dispose() on OracleDataReader and OracleConnection as suggested by Paul Abbott. Also increased the number of cursors per session from 50 to 150 on the database.

Filling SqlDataAdapter

I am running a SQLQuery that takes roughly 45 seconds to run and display results. I am using Task<DataSet> to populate two drop downs on my page. Well the 1st drop down populates fine (the query is completed in about 2 seconds), the second it seems that the adapter.Fill(dataSet) is not waiting on the query to complete before it begins to fill the drop down with a null dataset. What should I alter so that the code execution halts until the query executes completely?
Task.Factory.ContinueWhenAll(new[]
{
One("Data Source=server;Initial Catalog=db;Integrated Security=True;MultipleActiveResultSets=True"),
Two("Data Source=server;Initial Catalog=db;Integrated Security=True;MultipleActiveResultSets=True"),
}, tasks =>
{
try
{
this.ddl1.DataSource = tasks[0].Result.Tables[0];
this.ddl1.DataTextField = "One";
this.ddl1.DataValueField = "ID";
this.ddl1.DataBind();
int indexOfLastItem = this.ddl1.Items.Count - 1;
this.ddl1.SelectedIndex = indexOfLastItem;
ddl2.DataSource = tasks[1].Result.Tables[0];
this.ddl2.DataTextField = "Two";
this.ddl2.DataValueField = "ID";
this.ddl2.DataBind();
this.ddl2.Items.Insert(0, new ListItem(Constants.All, Constants.All));
}
catch (Exception exception) { throw exception; }
}, CancellationToken.None, TaskContinuationOptions.None, TaskScheduler.FromCurrentSynchronizationContext());
public System.Threading.Tasks.Task<DataSet> One(string databaseConnection)
{
return FillDS("Select * from activeemployees", databaseConnection);
}
public System.Threading.Tasks.Task<DataSet> Two(string databaseConnection)
{
return FillDS("Select * from mastersalesdatabase", databaseConnection);
}
public System.Threading.Tasks.Task<DataSet> FillDS(string sqlQuery, string connectionString)
{
try
{
var dataSet = new DataSet();
using (var adapter = new SqlDataAdapter(sqlQuery, connectionString))
{
adapter.Fill(dataSet);
return dataSet;
}
}
catch (Exception exception) { throw exception; }
}
My query Select * from activeemployees completes in about 2 seconds and populates fine, my query Select * from mastersalesdatabase takes roughly 45 seconds and it seems the code just moves on w/o a delay to let the query execute to completion.

If you're going to do async to retrieve data into a datatable, it should look more like this:
public static async Task<DataTable> GetDataTableAsync(string connectionString, SqlCommand command)
{
using (var connection = new SqlConnection(connectionString))
{
command.Connection = connection;
await connection.OpenAsync();
using (var dataReader = await command.ExecuteReaderAsync())
{
var dataTable = new DataTable();
dataTable.Load(dataReader);
return dataTable;
}
}
}
Notice there's no need for a dataset.
Then in WebForms, we have to handle async code differently.
protected void Page_Load(object sender, EventArgs e)
{
RegisterAsyncTask(new PageAsyncTask(DoWorkAsync));
}
private async Task DoWorkAsync()
{
ActiveEmployeesDropDownList.DataSource = GetDataTableAsync(databaseConnection, new SqlCommand("select * from activeemployees"));
ActiveEmployeesDropDownList.DataBind();
}
Notice I renamed the control from ddl1 to ActiveEmployeesDropDownList because ddl1 is a horrible name. Your names should have semantic meaning.
You'll need to add the async=true attribute to your page according to MSDN.
And you should also fix your query to not take 45 seconds, but that's a separate question entirely.

Mysql Exception on ExecuteReader

Well i had a weird exception on my program so i tried to replicate it to show you guys, so what i did was to create a table with id(int-11 primary), title(varchar-255) and generated 100k random titles with 40 chars lenght, when i run my method that reads the count for each id it throws an exception check below for more.
What i found is that this was because of timeouts so i tried this for the timeouts.
set net_write_timeout=99999; set net_read_timeout=99999;
Tried pooling=true on connection
Tried cmd.timeout = 120;
I also tried adding MaxDegreeOfParallelism i played with multiple values but still the same error appears after a while.
My exception:
Could not kill query, aborting connection. Exception was Unable to
read data from the transport connection: A connection attempt failed
because the connected party did not properly respond after a period of
time, or established connection failed because connected host has
failed to respond.
public static string db_main = "Server=" + server + ";Port=" + port + ";Database=" + database_main + ";Uid=" + user + ";Pwd=" + password + ";Pooling=true;";
private void button19_Click(object sender, EventArgs e)
{
List<string> list = db.read_string_list("SELECT id from tablename", db.db_main);
//new ParallelOptions { MaxDegreeOfParallelism = 3 },
Task.Factory.StartNew(() =>
{
Parallel.ForEach(list, id =>
{
string sql = "SELECT COUNT(*) FROM tablename where id=" + id;
var ti = db.read_int(sql, db.db_main);
Console.WriteLine(ti);
});
}).ContinueWith(_ =>
{
Console.WriteLine("Finished");
});
}
public static int? read_int(string sql, string sconn)
{
var rdr = MySqlHelper.ExecuteReader(db.db_main, sql);
if (rdr.HasRows)
{
rdr.Read();
return rdr.GetInt32(0);
}
else
return null;
}
Alternate Method to read int with timeout option.
public static int? read_int2(string sql, string sconn)
{
using (var conn = new MySqlConnection(sconn))
{
using (var cmd = new MySqlCommand(sql, conn))
{
//cmd.CommandTimeout = 120;
conn.Open();
using (var rdr = cmd.ExecuteReader())
{
if (rdr.HasRows)
{
rdr.Read();
return rdr.GetInt32(0);
}
else
return null;
}
}
}
}
What can be causing this? any clues?

So finally my solution on this was to increase net_read_timeout variable (im pointing out net_write_timeout because that can happen when executing a long query too)
Run these queries *Note: After you restart your PC default values will take place again.
set ##global.net_read_timeout = 999;
set ##global.net_write_timeout = 999;
or you can add this on the connection string
default command timeout=999;
Finally i used this method to read the values.
public static int? read_int(string sql, string sconn)
{
try
{
using (MySqlDataReader reader = MySqlHelper.ExecuteReader(sconn, sql))
{
if (reader.HasRows)
{
reader.Read();
return reader.GetInt32(0);
}
else
return null;
}
}
catch (MySqlException ex)
{
//Do your stuff here
throw ex;
}
}