We Keep Coding

There is already an open DataReader ... even though it is not

Note: I've went through millions of questions when the issue is not disposing the reader/connection properly, or when the error is because of badly handled lazy loading. I believe that this issue is a different one, and probably related to MySQL's .NET connector.
I'm using MySQL server (5.6) database extensively through its .NET connector (6.8.3). All tables are created with MyISAM engine for performance reasons. I have only one process with one thread (update: in fact, it's not true, see below) accessing the DB sequentially, so there is no need for transactions and concurrency.
Today, after many hours of processing the following piece of code:
public IEnumerable<VectorTransition> FindWithSourceVector(double[] sourceVector)
{
var sqlConnection = this.connectionPool.Take();
this.selectWithSourceVectorCommand.Connection = sqlConnection;
this.selectWithSourceVectorCommand.Parameters["#epsilon"].Value
= this.epsilonEstimator.Epsilon.Min() / 10;
for (int d = 0; d < this.dimensionality; ++d)
{
this.selectWithSourceVectorCommand.Parameters["#source_" + d.ToString()]
.Value = sourceVector[d];
}
// *** the following line (201) throws the exception presented below
using (var reader = this.selectWithSourceVectorCommand.ExecuteReader())
{
while (reader.Read())
{
yield return ReaderToVectorTransition(reader);
}
}
this.connectionPool.Putback(sqlConnection);
}
threw the following exception:
MySqlException: There is already an open DataReader associated with this Connection which must be closed first.
Here is the relevant part of the stack trace:
at MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception)
at MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex)
at MySql.Data.MySqlClient.MySqlCommand.CheckState()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader()
at implementation.VectorTransitionsMySqlTable.d__27.MoveNext() in C:\Users\bartoszp...\implementation\VectorTransitionsMySqlTable.cs:line 201
at System.Linq.Enumerable.d__3a1.MoveNext()
at System.Linq.Buffer1..ctor(IEnumerable1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
at implementation.VectorTransitionService.Add(VectorTransition vectorTransition) in C:\Users\bartoszp...\implementation\VectorTransitionService.cs:line 38
at Program.Go[T](Environment`2 p, Space parentSpace, EpsilonEstimator epsilonEstimator, ThresholdEstimator thresholdEstimator, TransitionTransformer transitionTransformer, AmbiguityCalculator ac, VectorTransitionsTableFactory vttf, AxesTableFactory atf, NeighbourhoodsTableFactory ntf, AmbiguitySamplesTableFactory astf, AmbiguitySampleMatchesTableFactory asmtf, MySqlConnectionPool connectionPool, Boolean rejectDuplicates, Boolean addNew) in C:\Users\bartoszp...\Program.cs:line 323
The connectionPool.Take returns the first connection that satisfies the following predicate:
private bool IsAvailable(MySqlConnection connection)
{
var result = false;
try
{
if (connection != null
&& connection.State == System.Data.ConnectionState.Open)
{
result = connection.Ping();
}
}
catch (Exception e)
{
Console.WriteLine("Ping exception: " + e.Message);
}
return result && connection.State == System.Data.ConnectionState.Open;
}
(This is related to my previous question, when I resolved a different, but similar issue: MySQL fatal error during information_schema query (software caused connection abort))
The FindWithSourceVector method is called by the following piece of code:
var existing
= this.vectorTransitionsTable
.FindWithSourceVector(vectorTransition.SourceVector)
.Take(2)
.ToArray();
(I need to find at most two duplicate vectors) - this is the VectorTransitionService.cs:line 38 part of the stack trace.
Now the most interesting part: when the debugger stopped execution after the exception occured, I've investigated the sqlConnection object to find, that it doesn't have a reader associated with it (picture below)!
Why is this happening (apparently at "random" - this method was being called almost every minute for the last ~20h)? Can I avoid that (in ways other then guess-adding some sleeps when Ping throws an exception and praying it'll help)?
Additional information regarding the implementation of the connection pool:
Get is intended for methods that call only simple queries and are not using readers, so the returned connection can be used in a re-entrant way. It is not used directly in this example (because of the reader involved):
public MySqlConnection Get()
{
var result = this.connections.FirstOrDefault(IsAvailable);
if (result == null)
{
Reconnect();
result = this.connections.FirstOrDefault(IsAvailable);
}
return result;
}
The Reconnect method just iterates though the whole array and recreates and opens the connections.
Take uses Get but also removes the returned connection from the list of available connections so in case of some methods that during their usage of a reader call other methods that also need a connection, it will not be shared. This is also not the case here, as the FindSourceVector method is simple (doesn't call other methods that use the DB). However, the Take is used for the sake of convention - if there is a reader, use Take:
public MySqlConnection Take()
{
var result = this.Get();
var index = Array.IndexOf(this.connections, result);
this.connections[index] = null;
return result;
}
Putback just puts a connection to the first empty spot, or just forgets about it if the connection pool is full:
public void Putback(MySqlConnection mySqlConnection)
{
int index = Array.IndexOf(this.connections, null);
if (index >= 0)
{
this.connections[index] = mySqlConnection;
}
else if (mySqlConnection != null)
{
mySqlConnection.Close();
mySqlConnection.Dispose();
}
}

I suspect this is the problem, at the end of the method:
this.connectionPool.Putback(sqlConnection);
You're only taking two elements from the iterator - so you never complete the while loop unless there's actually only one value returned from the reader. Now you're using LINQ, which will automatically be calling Dispose() on the iterator, so your using statement will still be disposing of the reader - but you're not putting the connection back in the pool. If you do that in a finally block, I think you'll be okay:
var sqlConnection = this.connectionPool.Take();
try
{
// Other stuff here...
using (var reader = this.selectWithSourceVectorCommand.ExecuteReader())
{
while (reader.Read())
{
yield return ReaderToVectorTransition(reader);
}
}
}
finally
{
this.connectionPool.Putback(sqlConnection);
}
Or ideally, if your connection pool is your own implementation, make Take return something which implements IDisposable and returns the connection back to the pool when it's done.
Here's a short but complete program to demonstrate what's going on, without any actual databases involved:
using System;
using System.Collections.Generic;
using System.Linq;
class DummyReader : IDisposable
{
private readonly int limit;
private int count = -1;
public int Count { get { return count; } }
public DummyReader(int limit)
{
this.limit = limit;
}
public bool Read()
{
count++;
return count < limit;
}
public void Dispose()
{
Console.WriteLine("DummyReader.Dispose()");
}
}
class Test
{
static IEnumerable<int> FindValues(int valuesInReader)
{
Console.WriteLine("Take from the pool");
using (var reader = new DummyReader(valuesInReader))
{
while (reader.Read())
{
yield return reader.Count;
}
}
Console.WriteLine("Put back in the pool");
}
static void Main()
{
var data = FindValues(2).Take(2).ToArray();
Console.WriteLine(string.Join(",", data));
}
}
As written - modelling the situation with the reader only finding two values - the output is:
Take from the pool
DummyReader.Dispose()
0,1
Note that the reader is disposed, but we never get as far as returning anything from the pool. If you change Main to model the situation where the reader only has one value, like this:
var data = FindValues(1).Take(2).ToArray();
Then we get all the way through the while loop, so the output changes:
Take from the pool
DummyReader.Dispose()
Put back in the pool
0
I suggest you copy my program and experiment with it. Make sure you understand everything about what's going on... then you can apply it to your own code. You might want to read my article on iterator block implementation details too.

TyCobb and Jon Skeet have correctly guessed, that the problem was the pool implementation and multi-threading. I forgot that actually I did start some tiny Tasks in the Reconnect method. The first connection was created and opened synchronously but all other where opened asynchronously.
The idea was that because I only need one connection at time, there others can reconnect in different threads. However, because I didn't always put the connection back (as explained in Jon's answer) reconnecting was happening quite frequently, and because the system was quite loaded these reconnection threads weren't fast enough, which eventually led to race conditions. The fix is to reconnect in a more simple and straightforward manner:
private void Reconnect()
{
for (int i = 0; i < connections.Length; ++i)
{
if (!IsAvailable(this.connections[i]))
{
this.ReconnectAt(i);
}
}
}
private void ReconnectAt(int index)
{
try
{
this.connections[index] = new MySqlConnection(this.connectionString);
this.connections[index].Open();
}
catch (MySqlException mse)
{
Console.WriteLine("Reconnect error: " + mse.Message);
this.connections[index] = null;
}
}

Cannot implicitly convert type DbDataReader to MySqlDataReader when using ExecuteReaderAsync

I've the following function that allows me to pass in a object and populate that object with the returning data, if any.
I've modified the function so that it can be called asynchronously.
public static async Task<MySqlDataReader> MySQLReturnReader(string sName, List<MySqlParameter> oParameters, Action<MySqlDataReader> fn)
{
using (MySqlConnection oConn = new MySqlConnection(MasterConn))
{
await oConn.OpenAsync();
using (MySqlCommand oMySqlCommand = new MySqlCommand(sName, oConn))
{
oMySqlCommand.CommandType = CommandType.StoredProcedure;
if (oParameters != null)
{
foreach (MySqlParameter oPar in oParameters)
{
oMySqlCommand.Parameters.Add(oPar);
}
}
oMySqlCommand.Connection.Open();
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
{
fn(oReader);
}
}
}
return;
}
My class object is something like;
public class MyClass
{
public int Id {get;set;}
public string Name {get;set;}
...
}
The function can be called like
List<MyClass> oMyClassList = new List<MyClass>();
List<MySqlParameter> oParams = new List<MySqlParameter>();
List<int> Ids = new List<int>(500);
Ids.Add(1);
Ids.Add(2);
...
Ids.Add(499);
foreach(int Id in Ids)
{
MySQLReturnReader("SPCheck", oParams, oRes =>
{
while (oRes.Read())
{
MyClass oMyClass = new MyClass();
oMyClass.Id = Convert.ToInt32(oRes["Id"]);
oMyClass.Name = oRes["Name"].ToString();
}
oMyClassList.Add(oMyClass);
}
);
}
The problem is I'm getting the compilation error of 'Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader'. Where am I going wrong ?
I'm wanting to use ExecuteReaderAsync in this way, as the Stored procedure called is very complex and would prefer to run the requests in parallel.

In your code, you have:
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
The compiler error Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader' means that you need to use the await keyword to "unwrap" the task returned by ExecuteReaderAsync:
using (MySqlDataReader oReader = await oMySqlCommand.ExecuteReaderAsync())
Note however that if you're using the official MySql.Data package, this call won't actually execute asynchronously. The Async methods in the MySql.Data connector aren't asynchronous; they block on network I/O and only return when the DB operation is complete. (For a much more detailed discussion, see this question and its top answer.) MySQL bug #70111 reports this problem in the MySQL connector.
To get truly asynchronous DB operations, you'll have to wait until that bug is fixed, or switch to a different connector. I've been developing a new, fully async connector that should be a drop-in replacement for MySql.Data; to try it out, install MySqlConnector from NuGet; its source is at GitHub.

This most likely indicates that the library you're using doesn't support ExecuteReaderAsync(), so you're just calling the default implementation inherited from DbCommand. This is why it returns the general DbDataReader (instead of the MySQL-specific one). And this also means that your code won't actually be asynchronous, the default version of ExecuteReaderAsync() is just a synchronous wrapper around ExecuteReader().
So, I think you should directly use the old ExecuteReader(), until your library adds support for ExecuteReaderAsync().

using statements

Today I noticed a snippet of code that looks like the one below:
public class Test
{
SqlConnection connection1 =
new SqlConnection(ConfigurationManager.ConnectionStrings["c1"].ToString());
SqlConnection connection2 =
new SqlConnection(ConfigurationManager.ConnectionStrings["c2"].ToString());
public void Method1()
{
using (connection1)
{
connection1.Open();
using (SqlCommand newSqlCommand = new SqlCommand("text",connection2))
{
// do something
}
}
}
public void Method2()
{
using (connection1)
{
// do something
}
}
}
I am just wondering why would anyone want to open the connection when creating the class and not when calling the corresponding methods inside the class?
EDIT: I should have probably posted the whole code instead. So I do see where they are opening connection1, but then they are instantiating a sqlcommand with a different sql connection (connection2) which has not been opened anywhere. What am I missing here?
Thanks,

This line only initializes a connection object, which can later be used to open a connection to a database server.
SqlConnection connection1 =
new SqlConnection(ConfigurationManager.ConnectionStrings["c1"].ToString());
What I know is that using disposes of an object (and in the case of Connection objects they are automatically closed) after its scope so I wouldn't recommend such usage because it might be problematic with other object types (other than Connection) that can't be used after having their dispose called.

connection1 = new SqlConnection(...) does not really open the connection. It just creates the connection object.
You have to call connection1.Open(); to actually open it. You can do this inside using statement block.
Refer to this MSDN page for more details.

It either
written this way to enforce only single call to a method be performed on the class
unintentionally confusing by throwing ObjectDisposed exception if you call 2 methods
contains connection re-initialization code in blocks you've ommited.

The code is dangerous.
If you call Method1 then Method2 (or visa versa) you will get an error (connection string not initialized).
The error occurs because the using statement will close the connection AND Disposes the object. I double checked what happens when dispose is called...the connection string is cleared (and possibly some other things I didn't notice).
You don't want to re-use a disposed object.
The cost of instantiating a new connection object is insignificant so I would just create the object where needed, maybe with a little factory method to reduce code duplication, so change to something like:-
private static SqlConnection GetSqlConnection()
{
return new SqlConnection(ConfigurationManager.ConnectionStrings["c1"].ToString());
}
private void Method1()
{
using (var conn = GetSqlConnection())
{
conn.Open();
// do stuff...
}
}
private void Method2()
{
using (var conn = GetSqlConnection())
{
conn.Open();
// do other stuff...
}
}
Of course there are many different ways of approaching this problem, this is just one and indeed quite a simplistic one...but it is a good starting point and is safe :-)

Enterprise Library 5.0 and Command Timeout

I am using following code and want to know whether we require to set command timeout if using CreateSprocAccessor of enterprise library , if not then how timeout is being managed?
var accessor = _sqlDatabase.CreateSprocAccessor<xyz>("uspGetxyz",
new xyzParameters(_sqlDatabase),
MapBuilder<xyz>.MapAllProperties().Build());
//Execute the accessor to obtain the results
var Data = accessor.Execute();
xyzList = Data.ToList<xyz>();

I have started using Microsoft Enterprise Library long back where in normal case the DB operation calls using provided methods of “Database” class fulfill the need. In some case, for the long running query, developer wants to set CommandTimeout property of SqlCommand (or DbCommand) class. This will allow query to be executed long time as value set in command timeout.
By default Data Access Application block does not support/take simple CommandTimeout parameter in method calls (there are many workaround samples available on net). To achieve the same with minimal changes, I have added a simple function named “WithCommandTimeOut” taking timeOutSecond parameter in “Microsoft.Practices.EnterpriseLibrary.Data.Database” class which returns same instance of “Database” class. Refer updated code snippet below for code changes. Hope this will solve timeout Problem.
//Class Level Static Variables
//Used to reset to default after assigning in "PrepareCommand" static method
static int DEFAULT_COMMAND_TIMEOUT_RESET = 30;
//Default value when "WithCommandTimeOut" not called
static int COMMAND_TIMEOUT_FOR_THIS_CALL = DEFAULT_COMMAND_TIMEOUT_RESET;
public Database WithCommandTimeOut(int timeOutSeconds)
{
COMMAND_TIMEOUT_FOR_THIS_CALL = timeOutSeconds;
return this;
}
protected static void PrepareCommand(DbCommand command, DbConnection connection)
{
if (command == null) throw new ArgumentNullException("command");
if (connection == null) throw new ArgumentNullException("connection");
//Here is the magical code ----------------------------
command.CommandTimeout = COMMAND_TIMEOUT_FOR_THIS_CALL;
//Here is the magical code ----------------------------
command.Connection = connection;
//Resetting value to default as this is static and subsequent
//db calls should work with default timeout i.e. 30
COMMAND_TIMEOUT_FOR_THIS_CALL = DEFAULT_COMMAND_TIMEOUT_RESET;
}
Ex.
Database db = EnterpriseLibraryContainer.Current.GetInstance(Of Database)("SmartSoftware");
db.WithCommandTimeOut(0).ExecuteDataSet(CommandType.Text, query);

I can't believe it what blunder enterprise library team has made, they have not given any way to set command time out in case of Accessor, it is a know issue with them
http://entlib.codeplex.com/workitem/28586
cant believe it, i have developed whole project and just came to know this a know issue :-(wtf

We can update this in the connection string, increase Connection Timeout=1000;

You can modify DbCommand timeout in your xyzParameters class in AssignParameters method:
public void AssignParameters(
System.Data.Common.DbCommand command, object[] parameterValues)
{
command.CommandTimeout = 0;
...
}

Volla!!! i have made changes in source code of Enterprise library, added a new "execute" method which will take timeOut paramerter, in Sproc accessors class and used these binaries in my project

Enterprise library manage connections

I am building an application with c# and I decided to use the Enterprise Library for the DAL (SQL Server).
I don't remember where, but I had read an article about EntLib which said that the connections are closed automatically.
Is it true?
If not, what is the best approach of managing the connections in the middle layer?
Open and close in each method?
The above is a sample method of how I am using the EntLib
public DataSet ReturnSomething
{
var sqlStr = "select something";
DbCommand cmd = db.GetSqlStringCommand(sqlStr);
db.AddInParameter(cmd, "#param1", SqlDbType.BigInt, hotelID);
db.AddInParameter(cmd, "#param2", SqlDbType.NVarChar, date);
return db.ExecuteDataSet(cmd);
}
Thanks in advance.

the ExecuteDataSet method returns a DataSet object that contains all the data. This gives you your own local copy. The call to ExecuteDataSet opens a connection, populates a DataSet, and closes the connection before returning the result
for more info:
http://msdn.microsoft.com/en-us/library/ff648933.aspx

I think you should have something like a static class used as a Façade which would provide the correct connection for your library subsystems.
public static class SystemFacade {
// Used as a subsystem to which the connections are provided.
private static readonly SystemFactory _systemFactory = new SystemFactory();
public static IList<Customer> GetCustomers() {
using (var connection = OpenConnection(nameOfEntLibNamedConnection))
return _systemFactory.GetCustomers(connection);
}
public static DbConnection OpenConnection(string connectionName) {
var connection =
// Read EntLib config and create a new connection here, and assure
// it is opened before you return it.
if (connection.State == ConnectionState.Closed)
connection.Open();
return connection;
}
}
internal class SystemFactory {
internal IList<Customer> GetCustomers(DbConnection connection) {
// Place code to get customers here.
}
}
And using this code:
public class MyPageClass {
private void DisplayCustomers() {
GridView.DataSource = SystemFacade.GetCustomers();
}
}
In this code sample, you have a static class that provides the functionalities and features of a class library. The Façade class is used to provide the user with all possible action, but you don't want to get a headache with what connection to use, etc. All you want is the list of customers out of the underlying datastore. Then, a call to GetCustomers will do it.
The Façade is an "intelligent" class that knows where to get the information from, so creates the connection accordingly and order the customers from the subsystem factory. The factory does what it is asked for, take the available connection and retrieve the customers without asking any further questions.
Does this help?

Yes, EntLib closes connections for you (actually it releases them back into the connection pool). That is the main reason why we originally started to use EntLib.
However, for all new development we have now gone on to use Entity Framework, we find that much more productive.