I've noticed something curious about reading from an IDataReader within a using statement that I can't comprehend. Though I'm sure the answer is simple.
Why is it that whilst inside the using (SqlDataReader rd) { ... } if I directly perform a yield return the reader stays open for the duration of the read. But if I perform a direct return calling a SqlDataReader extension method (outlined below) that the reader closes before the enumerable can be actualized?
public static IEnumerable<T> Enumerate<T>(this SqlDataReader rd)
{
while (rd.Read())
yield return rd.ConvertTo<T>(); //extension method wrapping FastMember
rd.NextResult();
}
To be absolutely clear of what I'm asking, I'm unsure why the following are fundamentally different:
A fleshed out example, as per #TimSchmelter's request:
/*
* contrived methods
*/
public IEnumerable<T> ReadSomeProc<T>() {
using (var db = new SqlConnection("connection string"))
{
var cmd = new SqlCommand("dbo.someProc", db);
using(var rd = cmd.ExecuteReader())
{
while(rd.Read())
yield return rd.ConvertTo<T>(); //extension method wrapping FastMember
}
}
}
//vs
public IEnumerable<T> ReadSomeProcExt<T>() {
using (var db = new SqlConnection("connection string"))
{
var cmd = new SqlCommand("dbo.someProc", db);
using(var rd = cmd.ExecuteReader())
{
return rd.Enumerate<T>(); //outlined above
}
}
}
/*
* usage
*/
var lst = ReadSomeProc<SomeObect>();
foreach(var l in lst){
//this works
}
//vs
var lst2 = ReadSomeProcExt<SomeObect>();
foreach(var l in list){
//throws exception, invalid attempt to read when reader is closed
}
Summary: Both versions of the method defer, but because ReadSomeProcExt doesn't defer execution, the reader is disposed before execution is passed back to the caller (i.e. before Enumerate<T> can run). ReadSomeProc, on the other hand, doesn't create the reader until it's been passed back to the caller, so it doesn't dispose the container until all its values have been read.
When your method uses yield return, the compiler actually changes the compiled code to return an IEnumerable<>, and the code in your method will not run until other code starts iterating over the returned IEnumerable<>.
That means that the code below doesn't even run the first line of your Enumerate method before it disposes the reader and returns a value. By the time someone else starts iterating over your returned IEnumerable<>, the reader has already been disposed.
using(SqlDataReader rd = cmd.ExecuteReader()){
return rd.Enumerate<T>();
}
But this code would execute the entire Enumerate() method in order to produce a List<> of results prior to returning:
using(SqlDataReader rd = cmd.ExecuteReader()){
return rd.Enumerate<T>().ToList();
}
On the other hand, whoever's calling the method with this code doesn't actually execute the method until the result is evaluated:
using(SqlDataReader rd = cmd.ExecuteReader()){
while(rd.Read())
yield return rd.ConvertTo<T>(); //extension method wrapping FastMember
}
But the moment they execute the returned IEnumerable<>, the using block opens up, and it doesn't Dispose() until the IEnumerable<> finishes its iterations, at which point you will have already read everything you need from the data reader.
It's because the "yield return" will return one element and continue the iteration, while the "normal" return will finish the invocation.
Related
Please see the below two methods. The first returns an IAsyncEnumerable. The second tries to consume it.
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
public static class SqlUtility
{
public static async IAsyncEnumerable<IDataRecord> GetRecordsAsync(
string connectionString, SqlParameter[] parameters, string commandText,
[EnumeratorCancellation]CancellationToken cancellationToken)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
using (SqlCommand command = new SqlCommand(commandText, connection))
{
command.Parameters.AddRange(parameters);
using (var reader = await command.ExecuteReaderAsync()
.ConfigureAwait(false))
{
while (await reader.ReadAsync().ConfigureAwait(false))
{
yield return reader;
}
}
}
}
}
public static async Task Example()
{
const string connectionString =
"Server=localhost;Database=[Redacted];Integrated Security=true";
SqlParameter[] parameters = new SqlParameter[]
{
new SqlParameter("VideoID", SqlDbType.Int) { Value = 1000 }
};
const string commandText = "select * from Video where VideoID=#VideoID";
IAsyncEnumerable<IDataRecord> records = GetRecordsAsync(connectionString,
parameters, commandText, CancellationToken.None);
IDataRecord firstRecord = await records.FirstAsync().ConfigureAwait(false);
object videoID = firstRecord["VideoID"]; //Should be 1000.
// Instead, I get this exception:
// "Invalid attempt to call MetaData when reader is closed."
}
}
When the code tries to read the resultant IDataReader (at object videoID = firstRecord["VideoID"];), I get this exception:
Invalid attempt to call MetaData when reader is closed.
This is because SqlDataReader is disposed. Can someone supply a recommended method for enumerating SqlDataReader in an asynchronous way so that each resultant record is available to the calling method? Thank you.
In this scenario, LINQ is not your friend, as FirstAsync is going to close the iterator before it returns the result, which isn't what ADO.NET expects; basically: don't use LINQ here, or at least: not in this way. You might be able to use something like Select to perform the projection while the sequence is still open, or it may be easier to just offload all the work here to a tool like Dapper. Or, to do it manually:
await foreach (var record in records)
{
// TODO: process record
// (perhaps "break"), because you only want the first
}
You can avoid this by not returning an object that depends on the connection still being open. For example, if you only need the VideoID, then just return that (I'm assuming it's an int):
public static async IAsyncEnumerable<int> GetRecordsAsync(string connectionString, SqlParameter[] parameters, string commandText, [EnumeratorCancellation]CancellationToken cancellationToken)
{
...
yield return reader["VideoID"];
...
}
Or project into your own class:
public class MyRecord {
public int VideoId { get; set; }
}
public static async IAsyncEnumerable<MyRecord> GetRecordsAsync(string connectionString, SqlParameter[] parameters, string commandText, [EnumeratorCancellation]CancellationToken cancellationToken)
{
...
yield return new MyRecord {
VideoId = reader["VideoID"]
}
...
}
Or do what Marc suggested and use a foreach and break after the first one, which would look like this in your case:
IAsyncEnumerable<IDataRecord> records = GetRecordsAsync(connectionString, parameters, commandText, CancellationToken.None);
object videoID;
await foreach (var record in records)
{
videoID = record["VideoID"];
break;
}
When you expose an open DataReader, the reponsibility of closing it along with the underlying Connection belongs now to the caller, so you should not dispose anything. Instead you should use the DbCommand.ExecuteReaderAsync overload that accepts a CommandBehavior argument, and pass the CommandBehavior.CloseConnection value:
When the command is executed, the associated Connection object is closed when the associated DataReader object is closed.
Then you can just hope that the caller will play by the rules and call the DataReader.Close method promptly, and will not let the connection open until the object is garbage collected. For this reason exposing an open DataReader should be considered an extreme performance optimization technique, that should be used sparingly.
Btw you would have the same problem if you returned an IEnumerable<IDataRecord> instead of an IAsyncEnumerable<IDataRecord>.
To add to the other answers, you could make your utility method generic and add a projection delegate ,Func<IDataRecord, T> projection, as a parameter like this:
public static async IAsyncEnumerable<T> GetRecordsAsync<T>(
string connectionString, SqlParameter[] parameters, string commandText,
Func<IDataRecord, T> projection, // Parameter here
[EnumeratorCancellation] CancellationToken cancellationToken)
{
...
yield return projection(reader); // Projected here
...
}
And then pass in a lambda or reference a method group such as this when calling:
public static object GetVideoId(IDataRecord dataRecord)
=> dataRecord["VideoID"];
such:
GetRecordsAsync(connectionString, parameters, commandText, GetVideoId, CancellationToken.None);
Way out here at the tail end of 2021 I had this exact question. I couldn't find a complete example, so I just messed around with what I could find until I got something to work.
Here is my code-complete, albeit simple (so you can expand it later) example, along with some comments detailing some of the pits I stepped into along the way:
// This function turns each "DataRow" into an object of Type T and yields
// it. You could alternately yield the reader itself for each row.
// In this example, assume sqlCommandText and connectionString exist.
public async IAsyncEnumerable<T> ReadAsync<T>( Func<SqlDataReader, T> func )
{
// we need a connection that will last as long as the reader is open,
// alternately you could pass in an open connection.
using SqlConnection connection = new SqlConnection( connectionString );
using SqlCommand cmd = new SqlCommand( sqlCommandText, connection );
await connection.OpenAsync();
var reader = await cmd.ExecuteReaderAsync();
while( await reader.ReadAsync() )
{
yield return func( reader );
}
}
Then in any other part of your (async) code you can call your function in an await foreach loop:
private static async Task CallIAsyncEnumerable()
{
await foreach( var category in ReadAsync( ReaderToCategory ) )
{
// do something with your category; save it in a list, write it to disk,
// make an HTTP call ... the whole world is yours!
}
}
// an example delegate, which I'm passing into ReadAsync
private static Category ReaderToCategory( SqlDataReader reader )
{
return new Category()
{
Code = ( string )reader[ "Code" ],
Group = ( string )reader[ "Group" ]
};
}
A couple other things I found: you can't yield from within a try, but you can stuff everything up to (and including) cmd.ExecuteReaderAsync() into a try, or a separate method that returns the DataReader. Or you can wrap the await foreach in a try block; I think the issue is in yielding to a caller outside the try (which makes sense, after you think about it).
If you use another method to generate the reader, pass the connection into that method, so you can control its lifetime. If your method creates the connection, executes the command, and returns the SqlDataReader, the connection will close (if you used a 'using`) before you can read from the reader. Again, this makes perfect sense if you think about it, but it tripped me up for a few minutes.
Good luck, and I hope this helps someone else in the future!
I have following statement using using:
using (var reader = data.CreateCommand(sql).ExecuteDataReader())
in this case data is some object which internally holds SqlConnection. CreateCommand(sql) function returns SqlCommand and ExecuteDataReader returns SqlDataReader. Since SqlCommand and SqlDataReader are both IDisposable, will they both be disposed by this use of using statement?
For now I have done it like this:
using (var cmd = data.CreateCommand(sql))
using (var reader = cmd.ExecuteDataReader())
But I would like to know if it's possible to combine them as stated above?
"if it's possible to combine them" - two using are totally fine, because both needs to be disposed.
You can combine them by extracting into method:
void RunWithReader(string sql, Action<SQLDataReader> action)
{
using (var cmd = data.CreateCommand(sql))
using (var reader = cmd.ExecuteDataReader())
action(reader);
}
Then you can use lambda
RunWithReader("whatever command", reader =>
{
... // while(reader.Read() { ... } - this could also be extracted
});
Agree with Matthew Watson's comment. You need to have both of usings statemets.
Here is the related question with additional reasoning. SqlConnection SqlCommand SqlDataReader IDisposable
The way you have presented your code the inner IDisposable (IDbCommand) is not disposed.
You have two choices:
You can put it all in one using like this:
using (IDisposable cmd = data.CreateCommand(), reader = ((IDbCommand)cmd).ExecuteReader())
{
// code here
}
But that is rather cumbersome. The other option is nested using statements:
using (var cmd = data.CreateCommand())
{
using (var reader = cmd.ExecuteReader())
{
// code here
}
}
Other than that you can get a bit complicated and write an extension method to help (sort of) clean it up for you.
public static class Ex
{
public static void Using<T, R>(this T obj, Func<T, R> create, Action<T, R> use) where R : IDisposable
{
using (var d = create(obj))
{
use(obj, d);
}
}
}
Then you can do this:
data.Using(d => d.CreateCommand(), (d, c) => c.Using(c2 => c2.ExecuteReader(), (c3, r) =>
{
// code here
}));
But perhaps that's not much of an improvement.
In General
You should Dispose every class that implements IDisposable.
However, your question is specifically dealing with a scenario where all the resources are all related, which if my understanding is correct, the Garbage Collector will dispose these resources for you automatically.
The Finalize() Method ("Destructors" or "Finalizers")
The Finalize() method makes sure that resources are disposed when
the instance is no longer referenced by the application.
In concert together with the Dispose() method
The Dispose() method will "release all resources used by the component". Which would mean that this code block (taken from MSDN's ExecuteReader Method Example) will properly dispose of all the compenents used by it...
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
SqlCommand command = new SqlCommand(queryString, connection);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
// ...
}
}
In this case, when disposing SqlConnection, it will properly
release all resources used by the component. Which means:
SqlCommand
SqlDataReader
** Note: This has always been my understanding...
Refactoring of the above code to the following:
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (SqlCommand command = new SqlCommand(queryString, connection))
{
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
// ...
}
}
}
}
The main advantage of manually disposing of each resource, would be
for a performance gain since we are not processing up to the
Finalize method (which is relatively a more expensive than doing Dispose().
However, that doesn't mean you want to Dispose() everything, as in some cases the Finalize() method is a better choice.
Assuming that I am still in good shape so far, and my understanding of the Garbage Collector's behavior is valid.
I would consider that SqlCommand (full code reference here), would then also dispose of SqlReader, which would mean that you could just get away with:
using (SqlCommand command = new SqlCommand(queryString, connection))
{
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
// ...
}
}
which in your case would translate to:
using (var cmd = data.CreateCommand(sql))
{
var reader = cmd.ExecuteDataReader();
// ...
}
I've the following function that allows me to pass in a object and populate that object with the returning data, if any.
I've modified the function so that it can be called asynchronously.
public static async Task<MySqlDataReader> MySQLReturnReader(string sName, List<MySqlParameter> oParameters, Action<MySqlDataReader> fn)
{
using (MySqlConnection oConn = new MySqlConnection(MasterConn))
{
await oConn.OpenAsync();
using (MySqlCommand oMySqlCommand = new MySqlCommand(sName, oConn))
{
oMySqlCommand.CommandType = CommandType.StoredProcedure;
if (oParameters != null)
{
foreach (MySqlParameter oPar in oParameters)
{
oMySqlCommand.Parameters.Add(oPar);
}
}
oMySqlCommand.Connection.Open();
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
{
fn(oReader);
}
}
}
return;
}
My class object is something like;
public class MyClass
{
public int Id {get;set;}
public string Name {get;set;}
...
}
The function can be called like
List<MyClass> oMyClassList = new List<MyClass>();
List<MySqlParameter> oParams = new List<MySqlParameter>();
List<int> Ids = new List<int>(500);
Ids.Add(1);
Ids.Add(2);
...
Ids.Add(499);
foreach(int Id in Ids)
{
MySQLReturnReader("SPCheck", oParams, oRes =>
{
while (oRes.Read())
{
MyClass oMyClass = new MyClass();
oMyClass.Id = Convert.ToInt32(oRes["Id"]);
oMyClass.Name = oRes["Name"].ToString();
}
oMyClassList.Add(oMyClass);
}
);
}
The problem is I'm getting the compilation error of 'Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader'. Where am I going wrong ?
I'm wanting to use ExecuteReaderAsync in this way, as the Stored procedure called is very complex and would prefer to run the requests in parallel.
In your code, you have:
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
The compiler error Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader' means that you need to use the await keyword to "unwrap" the task returned by ExecuteReaderAsync:
using (MySqlDataReader oReader = await oMySqlCommand.ExecuteReaderAsync())
Note however that if you're using the official MySql.Data package, this call won't actually execute asynchronously. The Async methods in the MySql.Data connector aren't asynchronous; they block on network I/O and only return when the DB operation is complete. (For a much more detailed discussion, see this question and its top answer.) MySQL bug #70111 reports this problem in the MySQL connector.
To get truly asynchronous DB operations, you'll have to wait until that bug is fixed, or switch to a different connector. I've been developing a new, fully async connector that should be a drop-in replacement for MySql.Data; to try it out, install MySqlConnector from NuGet; its source is at GitHub.
This most likely indicates that the library you're using doesn't support ExecuteReaderAsync(), so you're just calling the default implementation inherited from DbCommand. This is why it returns the general DbDataReader (instead of the MySQL-specific one). And this also means that your code won't actually be asynchronous, the default version of ExecuteReaderAsync() is just a synchronous wrapper around ExecuteReader().
So, I think you should directly use the old ExecuteReader(), until your library adds support for ExecuteReaderAsync().
I'm looking to figure out the best way to execute a database query using the least amount of boilerplate code. The method suggested in the SqlCommand documentation:
private static void ReadOrderData(string connectionString)
{
string queryString = "SELECT OrderID, CustomerID FROM dbo.Orders;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand command = new SqlCommand(queryString, connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
try
{
while (reader.Read())
{
Console.WriteLine(String.Format("{0}, {1}", reader[0], reader[1]));
}
}
finally
{
reader.Close();
}
}
}
mostly consists of code that would have to be repeated in every method that interacts with the database.
I'm already in the habit of factoring out the establishment of a connection, which would yield code more like the following. (I'm also modifying it so that it returns data, in order to make the example a bit less trivial.)
private SQLConnection CreateConnection()
{
var connection = new SqlConnection(_connectionString);
connection.Open();
return connection;
}
private List<int> ReadOrderData()
{
using(var connection = CreateConnection())
using(var command = connection.CreateCommand())
{
command.CommandText = "SELECT OrderID FROM dbo.Orders;";
using(var reader = command.ExecuteReader())
{
var results = new List<int>();
while(reader.Read()) results.Add(reader.GetInt32(0));
return results;
}
}
}
That's an improvement, but there's still enough boilerplate to nag at me. Can this be reduced further? In particular, I'd like to do something about the first two lines of the procedure. I don't feel like the method should be in charge of creating the SqlCommand. It's a tiny piece of repetition as it is in the example, but it seems to grow if transactions are being managed manually or timeouts are being altered or anything like that.
edit: Assume, at least hypothetically, there's going to have to be a bunch of different types of data being returned. And consequently the solution can't be just one one-size-fits-all method, there will have to be a few different ones depending, at minimum, on whether ExecuteNonQuery, ExecuteScalar, ExecuteReader, ExecuteReaderAsync, or any of the others are being called. I'd like to cut down on the repetition among those.
Tried Dapper?
Granted this doesn't get you a DataReader but you might just prefer it this way once you've tried it.
It's about the lightest-weight an ORM can be while still being called an ORM. No more methods to map between DataReader and strong types for me.
Used right here on all the StackExchange sites.
using (var conn = new SqlConnection(cs))
{
var dogs = connection.Query("select name, age from dogs");
foreach (dynamic dog in dogs)
{
Console.WriteLine("{0} age {1}", dog.name, dog.age);
}
}
or
using (var conn = new SqlConnection(cs))
{
var dogs = connection.Query<Dog>("select Name, Age from dogs");
foreach (Dog dog in dogs)
{
Console.WriteLine("{0} age {1}", dog.Name, dog.Age);
}
}
class Dog
{
public string Name { get; set; }
public int Age { get; set; }
}
If you want to roll data access on your own, this pattern of help methods could be one way to remove duplication:
private List<int> ReadOrderData()
{
return ExecuteList<int>("SELECT OrderID FROM dbo.Orders;",
x => x.GetInt32("orderId")).ToList();
}
private IEnumerable<T> ExecuteList(string query,
Func<IDataRecord, T> entityCreator)
{
using(var connection = CreateConnection())
using(var command = connection.CreateCommand())
{
command.CommandText = query;
connection.Open();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
yield return entityCreator(reader);
}
}
}
You'll have to add support for parameters and this might not compile, but the pattern is what I'm trying to illustrate.
What I typically do is use a custom class that I wrote a while back that accepts a SQL string, and optionally a list of parameters and it returns a DataTable.
Since the thing that changes between invocations is typically just the SQL that is optimal IMHO.
If you truly do need to use a DataReader you can do something like this:
public void ExecuteWithDataReader(string sql, Action<DataReader> stuffToDo) {
using (SqlConnection connection = new SqlConnection(connectionString)) {
using (SqlCommand command = new SqlCommand(sql, connection)) {
connection.Open();
using (SqlDataReader reader = command.ExecuteReader()) {
try {
while (reader.Read()) {
stuffToDo(reader);
}
}
finally {
reader.Close();
}
}
}
}
}
private static void ReadOrderData(string connectionString) {
string sql = "SELECT OrderID, CustomerID FROM dbo.Orders;";
ExecuteWithDataReader(sql, r => Console.WriteLine(String.Format("{0}, {1}", r[0], r[1])));
}
The first two line are the most important thing you need...
but if you still wish to do it, you can turn them to a database handler class, yes it will become more of code, but in refactoring concept, every thing will move to the related topic...
try to write a singleton class, that receive a command and do action, so return result of type SqlDataReader reader...
Doing this in comments was too much.
I would suggest that the boilerplate code around
using(conn = new sqlconnection)
using(cmd = new sqlcommand) {
// blah blah blah
}
isn't something to be lightly removed and instead would encourage that you keep it exactly where it's at. Resources, especially unmanaged ones, should be opened and released at the closest point to execution as possible IMHO.
In no small part due to the ease with which other developers will fail to follow the appropriate clean up conventions.
If you do something like
private SQLConnection CreateConnection()
{
var connection = new SqlConnection(_connectionString);
connection.Open();
return connection;
}
Then you are inviting another programmer to call this method and completely fail to release the resource as soon as the query is executed. I don't know what kind of app you are building, but in a web app such a thing will lead to memory / connection / resource errors of types that are difficult to debug, unless you've been through it before.
Instead, I'd suggest you look into a lightweight ORM such as Dapper.net or similar to see how they approached it. I don't use dapper, but I hear it's pretty good. The reason I don't use it is simply that we don't allow inline sql to be executed against our databases (but that's a very different conversation).
Here's our standard:
public static DataTable StatisticsGet( Guid tenantId ) {
DataTable result = new DataTable();
result.Locale = CultureInfo.CurrentCulture;
Database db = DatabaseFactory.CreateDatabase(DatabaseType.Clients.ToString());
using (DbCommand dbCommand = db.GetStoredProcCommand("reg.StatsGet")) {
db.AddInParameter(dbCommand, "TenantId", DbType.Guid, tenantId);
result.Load(db.ExecuteReader(dbCommand));
} // using dbCommand
return result;
} // method::StatisticsGet
We make heavy use of Enterprise Library. It's short, simple and to the point and very well tested. This method just returns a datatable but you could easily have it return an object collection.. or nothing.
I am working on an existing application. This application reads data from a huge file and then, after doing some calculations, it stores the data in another table.
But the loop doing this (see below) is taking a really long time. Since the file sometimes contains 1,000s of records, the entire process takes days.
Can I replace this foreach loop with something else? I tried using Parallel.ForEach and it did help. I am new to this, so will appreciate your help.
foreach (record someredord Somereport.r)
{
try
{
using (var command = new SqlCommand("[procname]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(…);
IAsyncResult result = command.BeginExecuteReader();
while (!result.IsCompleted)
{
System.Threading.Thread.Sleep(10);
}
command.EndExecuteReader(result);
}
}
catch (Exception e)
{
…
}
}
After reviewing the answers , I removed the Async and used edited the code as below. But this did not improve performance.
using (command = new SqlCommand("[sp]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
foreach (record someRecord in someReport.)
{
command.Parameters.Clear();
command.Parameters.Add(....)
command.Prepare();
using (dr = command.ExecuteReader())
{
while (dr.Read())
{
if ()
{
}
else if ()
{
}
}
}
}
}
Instead of looping the sql connection so many times, ever consider extracting the whole set of data out from sql server and process the data via the dataset?
Edit: Decided to further explain what i meant..
You can do the following, pseudo code as follow
Use a select * and get all information from the database and store them into a list of the class or dictionary.
Do your foreach(record someRecord in someReport) and do the condition matching as usual.
Step 1: Ditch the try at async. It isn't implemented properly and you're blocking anyway. So just execute the procedure and see if that helps.
Step 2: Move the SqlCommand outside of the loop and reuse it for each iteration. that way you don't incurr the cost of creating and destroying it for every item in your loop.
Warning: Make sure you reset/clear/remove parameters you don't need from the previous iteration. We did something like this with optional parameters and had 'bleed-thru' from the previous iteration because we didn't clean up parameters we didn't need!
Your biggest problem is that you're looping over this:
IAsyncResult result = command.BeginExecuteReader();
while (!result.IsCompleted)
{
System.Threading.Thread.Sleep(10);
}
command.EndExecuteReader(result);
The entire idea of the asynchronous model is that the calling thread (the one doing this loop) should be spinning up ALL of the asynchronous tasks using the Begin method before starting to work with the results with the End method. If you are using Thread.Sleep() within your main calling thread to wait for an asynchronous operation to complete (as you are here), you're doing it wrong, and what ends up happening is that each command, one at a time, is being called and then waited for before the next one starts.
Instead, try something like this:
public void BeginExecutingCommands(Report someReport)
{
foreach (record someRecord in someReport.r)
{
var command = new SqlCommand("[procname]", sqlConn);
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(…);
command.BeginExecuteReader(ReaderExecuted,
new object[] { command, someReport, someRecord });
}
}
void ReaderExecuted(IAsyncResult result)
{
var state = (object[])result.AsyncState;
var command = state[0] as SqlCommand;
var someReport = state[1] as Report;
var someRecord = state[2] as Record;
try
{
using (SqlDataReader reader = command.EndExecuteReader(result))
{
// work with reader, command, someReport and someRecord to do what you need.
}
}
catch (Exception ex)
{
// handle exceptions that occurred during the async operation here
}
}
In SQL on the other end of a write is a (one) disk. You rarely can write faster in parallel. In fact in parallel often slows it down due to index fragmentation. If you can sort the data by primary (clustered) key prior to loading. In a big load even disable other keys, load data rebuild keys.
Not really sure what are doing in the asynch but for sure it was not doing what you expected as it was waiting on itself.
try
{
using (var command = new SqlCommand("[procname]", sqlConn))
{
command.CommandTimeout = 0;
command.CommandType = CommandType.StoredProcedure;
foreach (record someredord Somereport.r)
{
command.Parameters.Clear()
command.Parameters.Add(…);
using (var rdr = command.ExecuteReader())
{
while (rdr.Read())
{
…
}
}
}
}
}
catch (…)
{
…
}
As we were talking about in the comments, storing this data in memory and working with it there may be a more efficient approach.
So one easy way to do that is to start with Entity Framework. Entity Framework will automatically generate the classes for you based on your database schema. Then you can import a stored procedure which holds your SELECT statement. The reason I suggest importing a stored proc into EF is that this approach is generally more efficient than doing your queries in LINQ against EF.
Then run the stored proc and store the data in a List like this...
var data = db.MyStoredProc().ToList();
Then you can do anything you want with that data. Or as I mentioned, if you're doing a lot of lookups on primary keys then use ToDictionary() something like this...
var data = db.MyStoredProc().ToDictionary(k => k.MyPrimaryKey);
Either way, you'll be working with your data in memory at this point.
It seems executing your SQL command puts lock on some required resources and that's the reason enforced you to use Async methods (my guess).
If the database in not in use, try an exclusive access to it. Even then in there are some internal transactions due to data-model complexity consider consulting to database designer.