I'm trying to modify an existing database call so that it makes asynchronous database calls. This is my first foray into asynchronous database calls, so I've looked at this post, this post, and the first two sections of this MSDN article. This is the code that I've come up with, which is similar to what's found in the second answer of the second post:
public async Task<IEnumerable<Item>> GetDataAsync(int id)
{
using (SqlConnection conn = new SqlConnection(oCSB.ConnectionString))
{
using (SqlCommand cmd = new SqlCommand("stored_procedure", conn))
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("param", "value");
await conn.OpenAsync();
SqlDataReader reader = await cmd.ExecuteReaderAsync();
return ReadItems(reader, id).ToList();
}
}
}
private IEnumerable<Item> ReadItems(SqlDataReader reader, long id)
{
while (reader.Read())
{
var item = new Item(id);
yield return item;
}
}
The application is a Web Forms application, and the call is initiated by a jQuery ajax request to a static WebMethod in an aspx page, which then calls the GetDataAsync method. Unfortunately, the application hangs on the cmd.ExecuteReaderAsync call with no exception thrown, and I haven't been able to figure out why. I've run it both on the VS dev server and on my local IIS 8, but I get the same result. I've also tried modifying it so that it makes a very simple select on a table. I've also tried changing the code based on other posts I've come across either on MSDN or SO. Anybody know what could possibly be causing it to hang on the ExecuteReaderAsync call?
Related
TLDR; I have an ASP.NET Core 5.0 API that's sitting at AWS. It makes a large call to MSSQL db to return ~1-4k rows of data. A single request is fine, taking ~500ms, but when multiple requests come in about the same time (4-5), the request slows to ~2000ms per call. What's going on?
There's not much more to state than what I have above. I open a connection to our DB then initialize a SqlCommand.
using (var connection = new SqlConnection(dbConnection))
connection.Open();
using (SqlCommand command = new SqlCommand(strSQLCommand))
I've tried both filling a datatable with SqlDataAdapter and using a SqlDataReader to fill up a custom object, I get similar slow downs either way. As stated above the query returns ~1-4k rows of data of varying types. And Postman says the returned Json data is about 1.95MB of size after decompression. The slowdown only occurs when multiple requests come in around the same time. I don't know if it's having trouble with multiple connections to the db, or if it's about the size of the data and available memory. Paging isn't an option, the request needs to return that much data.
This all occurs within a HttpGet function
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public IActionResult Foo([FromHeader] FooRequest request)
{
///stuff
DataTable dt = new DataTable();
using (var connection = new SqlConnection(_dataDBConnection))
{
timer.Start();
connection.Open();
using (SqlCommand command = new SqlCommand(
"SELECT foo.name, bar.first, bar.second, bar.third, bar.fourth
FROM dbo.foo with(nolock)
JOIN dbo.bar with(nolock) ON bar.name = foo.name
WHERE bar.date = #date", connection))
{
command.Parameters.AddWithValue("#date", request.Date.ToString("yyyyMMdd"));
using (SqlDataAdapter adapter = new SqlDataAdapter(command))
{
adapter.Fill(dt);
}
}
timer.Stop();
long elapsed = timer.ElapsedMilliseconds;
}
///Parse the data from datatable into a List<object> and return
///I've also used a DataReader to put the data directly into the List<object> but experienced the same slowdown.
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}
Any insights would be appreciated!
--EDIT AFTER ADDITOINAL TESTING---
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public IActionResult Foo([FromHeader] FooRequest request)
{
///stuff
using (var connection = new SqlConnection(_dataDBConnection))
{
connection.Open();
///This runs significantly faster
using (SqlCommand command = new SqlCommand(#"dbo.spGetFoo", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#date", request.date.ToString("yyyyMMdd"));
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
///Add data to list to be returned
}
}
}
}
///Parse the data from datatable into a List<object> and return
///I've also used a DataReader to put the data directly into the List<object> but experienced the same slowdown.
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}
--FINAL EDIT PLEASE READ--
People seem to be getting caught up on the DataAdapter and Fill portion instead of reading the full post. So, I'll include a final example here that provides the same issue above.
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public async Task<IActionResult> Foo([FromHeader] FooRequest request)
{
///stuff
using (var connection = new SqlConnection(_dataDBConnection))
{
await connection.OpenAsync();
///This runs significantly faster
using (SqlCommand command = new SqlCommand(#"dbo.spGetFoo", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#date", request.date.ToString("yyyyMMdd"));
using (SqlDataReader reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
///Add data to list to be returned
}
}
}
}
///Parse the data from datatable into a List<object> and return
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}
First thing to note here is that your action method is not asynchronous. Second thing to note here is that using adapters to fill datasets is something I hadn't seen for years now. Use Dapper! Finally, that call to the adapter's Fill() method is most likely synchronous. Move to Dapper and use asynchronous calls to maximize your ASP.net throughput.
I think your idea is correct, it shouldn't be a database problem.
I think that Session can be one suspect. If you use ASP.NET Core
Session in your application, requests are queued and processed one by
one. So, the last request can stay holt in the queue while the
previous requests are being processed.
Another can be bits of MVC running in your pipeline and that can bring
Session without asking you.
In addition, another possible reason is that all threads in the
ASP.NET Core Thread Pool are
busy.
In this case, a new thread will be created to process a new request
that takes additional time.
This is just my idea, any other cause is possible. Hope it can help you.
The reason this is slow is that the method is not async. This means that threads are blocked. Since Asp.Net has a limited thread pool, it will be exhausted after a while, and then additional requests will have to queue, which makes the system slow. All of this should be fixed by using async await pattern.
Since SQLDataAdapter does not provide any async methods, it could be easier to use a technology which provides such an async methods, e.g. EF Core. Otherwise you could start a new task for adapter.Fill, however, this is not a clean way of doing it.
For simple lookups, I need to perform some SQL statements on a DB2 machine. I'm not able to use an ORM at the moment. I have a working example through this code, however I'm wondering if it can be optimized more as this would essentially create a connection on each request. And that just seems like bad programming.
Is there a way I can optimize this Get() request to leave a connection open? Nesting using statements seems dirty, as well. How should I handle the fact that Get() really wants to return an object of User no matter what; even in error? Can I put this connection in the start of the program so that I can use it over and over again? What are some of the best practices for this?
public class UsersController : ApiController
{
String constr = WebConfigurationManager.ConnectionStrings["DB2Connection"].ConnectionString;
public User Get([FromUri] User cst)
{
if (cst == null)
{
throw new HttpResponseException(HttpStatusCode.NotFound);
}
else
{
using (OdbcConnection DB2Conn = new OdbcConnection(constr))
{
DB2Conn.Open();
using (OdbcCommand com = new OdbcCommand(
// Generic SQL Statement
"SELECT * FROM [TABLE] WHERE customerNumber = ?", DB2Conn))
{
com.Parameters.AddWithValue("#var", cst.customerNumber);
using (OdbcDataReader reader = com.ExecuteReader())
{
try
{
while (reader.Read())
{
cst.name = (string)reader["name"];
return cst;
}
}
catch
{
throw;
}
}
}
}
return cst;
}
}
}
I found a great question that doesn't really have detailed answers, I feel like similar solutions exist for both of these questions...
And that just seems like bad programming.
Why do you think that?
The underlying system should be maintaining connections in a connection pool for you. Creating a connection should be very optimized already.
From a logical perspective, what you're doing now is exactly what you want to be doing. Create the connection, use it, and dispose of it immediately. This allows other threads/processes/etc. to use it from the connection pool now that you're done with it.
This also avoids the myriad of problems which arise from manually maintaining your open connections outside of the code that uses them.
Is there a way I can optimize this Get() request to leave a connection open?
Have you measured an actual performance problem? If not, there's nothing to optimize.
And there's a very good chance that hanging on to open connections in a static context in your web application is going to have drastic performance implications.
In short... You're already doing this correctly. (Well, except for that unnecessary try/catch. You can remove that.)
Edit: If you're just looking to improve the readability of the code (which itself is a matter of personal preference), this seems readable to me:
public User Get([FromUri] User cst)
{
if (cst == null)
throw new HttpResponseException(HttpStatusCode.NotFound);
using (var DB2Conn = new OdbcConnection(constr))
using (var com = new OdbcCommand("SELECT * FROM [TABLE] WHERE customerNumber = ?", DB2Conn))
{
com.Parameters.AddWithValue("#var", cst.customerNumber);
DB2Conn.Open();
using (OdbcDataReader reader = com.ExecuteReader())
while (reader.Read())
{
cst.name = (string)reader["name"]
return cst;
}
}
return cst;
}
Note that you can further improve it by re-addressing the logic of that SQL query. Since you're fetching one value from one record then you don't need to loop over a data reader. Just fetch a single literal and return it. Note that this is free-hand and untested, but it might look something like this:
public User Get([FromUri] User cst)
{
if (cst == null)
throw new HttpResponseException(HttpStatusCode.NotFound);
using (var DB2Conn = new OdbcConnection(constr))
using (var com = new OdbcCommand("SELECT name FROM [TABLE] WHERE customerNumber = ? FETCH FIRST 1 ROWS ONLY", DB2Conn))
{
com.Parameters.AddWithValue("#var", cst.customerNumber);
DB2Conn.Open();
cst.name = (string)com.ExecuteScalar();
}
return cst;
}
#David's answer addresses your actual questions perfectly but here's some other observations that may make your code a little more pallatable to you:
remove the try/catch block - all you're doing is re-throwing the exception which is what will happen if you don't use a try/catch at all. Don't catch the exception unless you can do something about it. (I see now that #David's answer addresses that - either it was added after I read it or I missed it - my apologies for the overlap but it's worth reinforcing)
Change your query to just pull name and use ExecuteScalar instead of ExecuteReader. You are taking the name value from the first record and exiting the while loop. ExecuteScalar returns the value from the first column in the first record, so you can eliminate the while loop and the using there.
I have an MVC setup, the Controller's Index() method launches a query against a database which then gets serialized to JSON and returned to the View. Almost each Controller is used only to display data so there are multiple such methods. As an example:
public string getSalesReport()
{
return queryForJsonResult(ConfigurationManager.AppSettings["RetrieveSalesReport"]);
}
Is the method run. The appSetting I pass to it is the query (which is definitely correct). This method then does the following:
private string queryForJsonResult(string query)
{
SqlConnection connection = new SqlConnection(_connectionString);
connection.Open();
SqlCommand command = new SqlCommand(query, connection);
SqlDataReader result = command.ExecuteReader();
string jsonResult = bufferedResultBuilder(result);
connection.Close();
return jsonResult;
}
private string bufferedResultBuilder(SqlDataReader result)
{
// Takes a result set and uses a buffer to build a JSON string.
StringWriter stringWriter = new StringWriter(new StringBuilder());
JsonTextWriter jsonTextWriter = new JsonTextWriter(stringWriter);
jsonTextWriter.WriteStartArray();
while (result.Read())
{
jsonTextWriter.WriteStartObject();
for (int i = 0; i < result.FieldCount; i++)
{
jsonTextWriter.WritePropertyName(result.GetName(i));
jsonTextWriter.WriteValue(result[i]);
}
jsonTextWriter.WriteEndObject();
}
jsonTextWriter.WriteEndArray();
return stringWriter.ToString();
}
So it takes the result set and builds a JSON string out of it, one entry at a time. When I run the project, all of this seems to work just fine, the data is loaded properly. I also have unit tests which utilize Stopwatch to test how long these Controller methods take to run. When I run the unit test, however, this one particular entry catches an error:
Result Message: Test method
Dashboard.Tests.BenchmarkTests.salesOverviewLoadTime threw exception:
System.InvalidOperationException: ExecuteReader: CommandText property
has not been initialized
The query is definitely correct, and there are several other very similar examples which differ only in the initial query that is input. All of those methods work just fine. The stack trace for this error traces it back to queryForJsonResult(), SqlCommand.ExecuteReader(), but like I said it only does this for one of the many very similar methods used to display views.
Why is it that one query, which actually loads the data correctly, catches an error when I unit test? The unit tests are set up almost identically except for the Controller that is loaded and all of the other ones work just fine.
This is an issue that is still strange to me. As #MichaelLiu pointed out, my .config files did not have the same appSettings. I copied over all of the appSettings that contained my queries and this resolved the issue. This does not necessarily explain, however, why all the other queries worked fine. I think I may have had one line of difference in the way the code exception handled e.g.
try {} catch {return;}
return;
as opposed to
try {
return;
} catch {
return;
}
Though there shouldn't be anything going wrong with return.
In any case, the solution is to make sure that both your project and it's unit test counterparts have same .config specifications, as needed.
I've the following function that allows me to pass in a object and populate that object with the returning data, if any.
I've modified the function so that it can be called asynchronously.
public static async Task<MySqlDataReader> MySQLReturnReader(string sName, List<MySqlParameter> oParameters, Action<MySqlDataReader> fn)
{
using (MySqlConnection oConn = new MySqlConnection(MasterConn))
{
await oConn.OpenAsync();
using (MySqlCommand oMySqlCommand = new MySqlCommand(sName, oConn))
{
oMySqlCommand.CommandType = CommandType.StoredProcedure;
if (oParameters != null)
{
foreach (MySqlParameter oPar in oParameters)
{
oMySqlCommand.Parameters.Add(oPar);
}
}
oMySqlCommand.Connection.Open();
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
{
fn(oReader);
}
}
}
return;
}
My class object is something like;
public class MyClass
{
public int Id {get;set;}
public string Name {get;set;}
...
}
The function can be called like
List<MyClass> oMyClassList = new List<MyClass>();
List<MySqlParameter> oParams = new List<MySqlParameter>();
List<int> Ids = new List<int>(500);
Ids.Add(1);
Ids.Add(2);
...
Ids.Add(499);
foreach(int Id in Ids)
{
MySQLReturnReader("SPCheck", oParams, oRes =>
{
while (oRes.Read())
{
MyClass oMyClass = new MyClass();
oMyClass.Id = Convert.ToInt32(oRes["Id"]);
oMyClass.Name = oRes["Name"].ToString();
}
oMyClassList.Add(oMyClass);
}
);
}
The problem is I'm getting the compilation error of 'Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader'. Where am I going wrong ?
I'm wanting to use ExecuteReaderAsync in this way, as the Stored procedure called is very complex and would prefer to run the requests in parallel.
In your code, you have:
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
The compiler error Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader' means that you need to use the await keyword to "unwrap" the task returned by ExecuteReaderAsync:
using (MySqlDataReader oReader = await oMySqlCommand.ExecuteReaderAsync())
Note however that if you're using the official MySql.Data package, this call won't actually execute asynchronously. The Async methods in the MySql.Data connector aren't asynchronous; they block on network I/O and only return when the DB operation is complete. (For a much more detailed discussion, see this question and its top answer.) MySQL bug #70111 reports this problem in the MySQL connector.
To get truly asynchronous DB operations, you'll have to wait until that bug is fixed, or switch to a different connector. I've been developing a new, fully async connector that should be a drop-in replacement for MySql.Data; to try it out, install MySqlConnector from NuGet; its source is at GitHub.
This most likely indicates that the library you're using doesn't support ExecuteReaderAsync(), so you're just calling the default implementation inherited from DbCommand. This is why it returns the general DbDataReader (instead of the MySQL-specific one). And this also means that your code won't actually be asynchronous, the default version of ExecuteReaderAsync() is just a synchronous wrapper around ExecuteReader().
So, I think you should directly use the old ExecuteReader(), until your library adds support for ExecuteReaderAsync().
I am working on an existing application. This application reads data from a huge file and then, after doing some calculations, it stores the data in another table.
But the loop doing this (see below) is taking a really long time. Since the file sometimes contains 1,000s of records, the entire process takes days.
Can I replace this foreach loop with something else? I tried using Parallel.ForEach and it did help. I am new to this, so will appreciate your help.
foreach (record someredord Somereport.r)
{
try
{
using (var command = new SqlCommand("[procname]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(…);
IAsyncResult result = command.BeginExecuteReader();
while (!result.IsCompleted)
{
System.Threading.Thread.Sleep(10);
}
command.EndExecuteReader(result);
}
}
catch (Exception e)
{
…
}
}
After reviewing the answers , I removed the Async and used edited the code as below. But this did not improve performance.
using (command = new SqlCommand("[sp]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
foreach (record someRecord in someReport.)
{
command.Parameters.Clear();
command.Parameters.Add(....)
command.Prepare();
using (dr = command.ExecuteReader())
{
while (dr.Read())
{
if ()
{
}
else if ()
{
}
}
}
}
}
Instead of looping the sql connection so many times, ever consider extracting the whole set of data out from sql server and process the data via the dataset?
Edit: Decided to further explain what i meant..
You can do the following, pseudo code as follow
Use a select * and get all information from the database and store them into a list of the class or dictionary.
Do your foreach(record someRecord in someReport) and do the condition matching as usual.
Step 1: Ditch the try at async. It isn't implemented properly and you're blocking anyway. So just execute the procedure and see if that helps.
Step 2: Move the SqlCommand outside of the loop and reuse it for each iteration. that way you don't incurr the cost of creating and destroying it for every item in your loop.
Warning: Make sure you reset/clear/remove parameters you don't need from the previous iteration. We did something like this with optional parameters and had 'bleed-thru' from the previous iteration because we didn't clean up parameters we didn't need!
Your biggest problem is that you're looping over this:
IAsyncResult result = command.BeginExecuteReader();
while (!result.IsCompleted)
{
System.Threading.Thread.Sleep(10);
}
command.EndExecuteReader(result);
The entire idea of the asynchronous model is that the calling thread (the one doing this loop) should be spinning up ALL of the asynchronous tasks using the Begin method before starting to work with the results with the End method. If you are using Thread.Sleep() within your main calling thread to wait for an asynchronous operation to complete (as you are here), you're doing it wrong, and what ends up happening is that each command, one at a time, is being called and then waited for before the next one starts.
Instead, try something like this:
public void BeginExecutingCommands(Report someReport)
{
foreach (record someRecord in someReport.r)
{
var command = new SqlCommand("[procname]", sqlConn);
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(…);
command.BeginExecuteReader(ReaderExecuted,
new object[] { command, someReport, someRecord });
}
}
void ReaderExecuted(IAsyncResult result)
{
var state = (object[])result.AsyncState;
var command = state[0] as SqlCommand;
var someReport = state[1] as Report;
var someRecord = state[2] as Record;
try
{
using (SqlDataReader reader = command.EndExecuteReader(result))
{
// work with reader, command, someReport and someRecord to do what you need.
}
}
catch (Exception ex)
{
// handle exceptions that occurred during the async operation here
}
}
In SQL on the other end of a write is a (one) disk. You rarely can write faster in parallel. In fact in parallel often slows it down due to index fragmentation. If you can sort the data by primary (clustered) key prior to loading. In a big load even disable other keys, load data rebuild keys.
Not really sure what are doing in the asynch but for sure it was not doing what you expected as it was waiting on itself.
try
{
using (var command = new SqlCommand("[procname]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
foreach (record someredord Somereport.r)
{
command.Parameters.Clear()
command.Parameters.Add(…);
using (var rdr = command.ExecuteReader())
{
while (rdr.Read())
{
…
}
}
}
}
}
catch (…)
{
…
}
As we were talking about in the comments, storing this data in memory and working with it there may be a more efficient approach.
So one easy way to do that is to start with Entity Framework. Entity Framework will automatically generate the classes for you based on your database schema. Then you can import a stored procedure which holds your SELECT statement. The reason I suggest importing a stored proc into EF is that this approach is generally more efficient than doing your queries in LINQ against EF.
Then run the stored proc and store the data in a List like this...
var data = db.MyStoredProc().ToList();
Then you can do anything you want with that data. Or as I mentioned, if you're doing a lot of lookups on primary keys then use ToDictionary() something like this...
var data = db.MyStoredProc().ToDictionary(k => k.MyPrimaryKey);
Either way, you'll be working with your data in memory at this point.
It seems executing your SQL command puts lock on some required resources and that's the reason enforced you to use Async methods (my guess).
If the database in not in use, try an exclusive access to it. Even then in there are some internal transactions due to data-model complexity consider consulting to database designer.