I have the following code shape. It seems that I'm misunderstanding the C# method return values. How is it possible that a "full" enumerator gets returned as an empty one?
class ThingDoer
{
public NpgsqlDataReader DoQuery()
{
NpgsqlCommand c = new NpgsqlCommand(...);
NpgsqlDataReader dataread = c.ExecuteReader();
return dataread; // Debugger confirms that six data are enumerable here.
}
}
...
class OtherThing
{
public void higherLevelFunction()
{
NpgsqlDataReader result = myThingDoer.DoQuery();
result.Read(); // No data! result's enumerable returns nothing!
}
}
You don't detail where your connection is coming from. Assuming it's something like:
public NpgsqlDataReader DoQuery()
{
using(NpgsqlConnection = GetConnectionCode())
{
NpgsqlCommand c = new NpgsqlCommand(...);
NpgsqlDataReader dataread = c.ExecuteReader();
return dataread;
}//Connection closes at this using-scope being left because that triggers Dispose()
}
Then change it to:
public NpgsqlDataReader DoQuery()
{
bool ownershipPassed = false;
NpgsqlConnection conn = GetConnectionCode();
try
{
NpgsqlCommand c = new NpgsqlCommand(...);
NpgsqlDataReader dataread = c.ExecuteReader(CommandBehavior.CloseConnection);
ownershipPassed = true;
return dataread;
}
finally
{
if(!ownershipPassed)//only if we didn't create the reader than takes charge of the connection
conn.Dispose();
}
}
Then where you use the reader, you have to dispose it to in turn dispose the connection's underlying connection to the database:
public void higherLevelFunction()
{
using(NpgsqlDataReader result = myThingDoer.DoQuery())
result.Read();
}
NpgsqlCommand c = new NpgsqlCommand(...);
NpgsqlDataReader dataread = c.ExecuteReader();
The above lines are very local to the method DoQuery. So as soon as the control comes out of the method, every object created inside to this method loses its scope. Hence you're losing the data because it's a reference type you're referring to in the caller method.
Related
I am running a test function to identify a memory leak:
[TestMethod]
public void DatabaseTools_Other_MemoryTest()
{
for (int i = 0; i < 100; i++)
{
try
{
var r = DatabaseTools.GetDataSet(true);
r = null;
}
catch (Exception e)
{
int EndPoint = i;
}
}
}
The goal of this method is to call DatabaseTools.GetDataSet(true) until it hits an OutOfMemoryException, which happens during the 3rd or 4th load. However, as I understand it, this shouldn't actually ever happen- this is DatabaseTools.GetDataSet:
public static DataSet GetDataSet(bool setData, string sqlText = null, string strConnection = null)
{
sqlText = sqlText ?? FullFilemakerQuery;
if (setData)
{
Database = strConnection;
Data = new DataSet();
}
DataSet dataSet = new DataSet();
using (SqlConnection connection = GetConnectionString())
{
using (SqlDataAdapter dataAdapter = new SqlDataAdapter(sqlText, connection))
{
dataAdapter.SelectCommand.CommandType = System.Data.CommandType.Text;
if (setData)
{
dataAdapter.FillSchema(Data, SchemaType.Source);
DisableAutoIncrement(Data);
dataAdapter.Fill(Data);
NameTables(Data, sqlText);
BuildDataKeysAndRelations(Database);
dataSet = null;
}
else
{
dataAdapter.FillSchema(dataSet, SchemaType.Source);
DisableAutoIncrement(dataSet);
dataAdapter.Fill(dataSet);
NameTables(dataSet, sqlText);
}
}
connection.Close();
}
return dataSet ?? Data;
}
public static void NameTables(DataSet dataSet, string sqlText)
{
for (int i = 0; i < dataSet.Tables.Count; i++)
{
dataSet.Tables[i].TableName = sqlText.Split(';')[i].Split(Dividers, StringSplitOptions.None)[1].Trim();
}
}
public static void DisableAutoIncrement(DataSet data)
{
foreach (DataTable T in data.Tables)
{
T.PrimaryKey.Select(c => { c.AutoIncrement = false; return c; }).ToList();
}
}
When passing only 'true' to this function, it sets sqlText equal to a static FullFileMakerQuery which selects everything the program could use from the database, it then gets the default database name (Database has a custom setter such that when given a null or empty value, it sets itself to default), and sets the static Data to a new DataSet. We have tried setting it to null at this point (no change) and we have tried using Data.Dispose() which caused errors. Because this function can also just return the dataset without also setting the global Data to it, we initialize a new DataSet dataSet. Then we do the standard connect, data adapter, fillschema, load data.
The weirdness: by setting breakpoints in the memory test function and saving dumps, loading the dataset once takes some amount of memory, reloading it uses less memory (by about 36,000 bytes in System.Data.Common.StringStorage) and then reloading it again uses more memory (by about 120,000 bytes in the same place as before). If we reload it one more time, it uses even more memory and crashes due to an OutOfMemoryException, and at this point I have no idea what could be causing it.
You never call .Dispose() on your DataSet, neither on the return value, nor on your static variable before you call new again.
As you are using using blocks with other classes already, I guess you know what to do to actually call .Dispose() on it.
Turn on Visual Studio Static Code Analysis, it will give you warnings should you forget to do so.
I'm learning how Dapper is working behind the scenes.
However I saw this pattern of disposing which is not understood to me.
Roughly in general — this is how QueryAsync is implemented :
/*1*/ public async Task<IEnumerable<T>> QueryAsync<T>(string sql, Func<IDataRecord, T> projector, DbConnection _conn, dynamic param = null)
/*2*/ {
/*3*/
/*4*/ DbDataReader reader = null;
/*5*/ bool wasClosed = _conn.State == ConnectionState.Closed;
/*6*/ try
/*7*/ {
/*8*/
/*9*/ using (var cmd = _conn.CreateCommand())
/*10*/ {
/*11*/ if (param!=null)
/*12*/ foreach (var prop in param.GetType().GetProperties(BindingFlags.Instance | BindingFlags.Public))
/*13*/ {
/*14*/ var parameter = cmd.CreateParameter();
/*15*/ parameter.ParameterName = prop.Name;
/*16*/ parameter.Value = prop.GetValue(param, null);
/*17*/ cmd.Parameters.Add(parameter);
/*18*/ }
/*19*/
/*20*/ await _conn.OpenAsync().ConfigureAwait(false);
/*21*/ cmd.CommandTimeout = 100000;
/*22*/ cmd.CommandText = sql;
/*23*/ reader = await cmd.ExecuteReaderAsync().ConfigureAwait(false);
/*24*/ List<T> buffer = new List<T>();
/*25*/ while (await reader.ReadAsync().ConfigureAwait(false)) buffer.Add(projector(reader));
/*26*/ return buffer;
/*27*/ }
/*28*/
/*29*/ }
/*30*/ finally
/*31*/ {
/*32*/ using (reader) { }
/*33*/ if (wasClosed) _conn.Close();
/*34*/ }
/*35*/ }
I can understand why he didn't use using over the connection , that's because he wanted to conditionally close the connection via the wasClosed variable.
For doing it - he must use the try/finally clause. ( so the conditional closing will be in the finally clause)
But my question is about line #32.
Instead of doing using at the finally clause , he could do:
using (DbDataReader reader = await cmd.ExecuteReaderAsync().ConfigureAwait(false))
{
List<T> buffer = new List<T>();
while (await reader.ReadAsync().ConfigureAwait(false)) buffer.Add(projector(reader));
return buffer;
}
So the finally clause is left with :
finally
{
//using (reader) { } //removed
if (wasClosed) _conn.Close();
}
Question
I've seen this using clause in a finally clause many times in dapper.
I must be missing something here, But what does this pattern achieve that my suggestion does not?
I'm no #MarcGravell, but I think there is one thing you're missing. The code you pasted doesn't exactly match the link you reference. The relevant code path looks like this:
try
{
if (command.Buffered)
{
List<T> buffer = new List<T>();
while (await reader.ReadAsync(cancel).ConfigureAwait(false))
{
buffer.Add((T)func(reader));
}
while (await reader.NextResultAsync().ConfigureAwait(false)) { }
command.OnCompleted();
return buffer;
}
else
{
// can't use ReadAsync / cancellation; but this will have to do
wasClosed = false; // don't close if handing back an open reader;
// rely on the command-behavior.
var deferred = ExecuteReaderSync<T>(reader, func, command.Parameters);
reader = null; // to prevent it being disposed before the caller gets to see it
return deferred;
}
}
finally
{
using (reader) { } // dispose if non-null
if (wasClosed) cnn.Close();
}
The method can either return a buffered result (indicated by the command.Buffered flag) or a deferred iterator. If Marc was to wrap the reader with a using statement and return an iterator, it (the reader) would of been disposed by the time the call-site executed it. By setting the reader to null (in the line before he returns the deferred result) he prevents the reader from being disposed, because the using in the finally block would be translated to this:
finally
{
IDisposable disposable = reader;
try
{
}
finally
{
if (dispoable != null)
{
disposable.Dispose();
}
}
}
When he sets the reader to null, it isn't disposed, and the reference exists in the iterator is still alive, pointing to the reader. This way, he can both dispose the reader in the normal codepath, but keep it alive if a deferred iterator was requested.
I have a class like
class ReadData{
public IdataReader Execute(string sql){
// Ado.net code here
return cmd.ExecuteReader();
}
}
This is the sample implementation and it works fine.
I am calling like this in caller
class caller{
void CallMethod(){
var reader = Execute("Sql query here");
while(reader.Read()){
logic here
}
//Here i need to get the out params after reading the resultset.
//But the impplementation should in the class ReadData.
//because that class has implementation to get the out params for the
//other type means, execute without resultset get only output params
}
}
Some possible ways like calling the first method with callback and in that once the data is read completely then read the out params.
I don't know how to implement that stuff.
Any possible better ways to do ?
Please help me on this..
public void Execute(string sql, Action<IDataRecord> action)
{
using(var connection = new ...)
{
connection.Open();
using(var command = new ...)
{
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
action(reader);
}
}
}
}
}
This would allow you to do something like this:
var entries = List<object>();
Execute("Sql query here", row => entries.Add(row["Field"]));
Or you could try a more linqy appraoch:
public IEnumerable<IDataRecord> Execute(string sql)
{
using(var connection = new ...)
{
connection.Open();
using(var command = new ...)
{
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
yield return reader;
}
}
}
}
}
Which would allow something like this:
var list = Execute("Sql query here").Where(row => (int)row["Field"] == 17)).ToList();
However, this has some weird effects with defered execution if you don't materialize it properly.
I'm trying to determine if there are glaring errors in a code base, or not.
The code in question calls a third party dll which returns an IDataReader. If the code uses the reader without disposing of it, it won't be explicitly returned to the pool, correct?
Here's the calling code:
IDataReader rdr = db.ExecSPGetDataReader("dbo.someStoredProcedure", paramList);
if (rdr.Read())
{
List<nameValuePair> formValues = Utils.nameValuePairs(rdr["valuepairs"].ToString());
foreach (nameValuePair nvp in formValues)
{
if (nvp.name.ToLower() == "name")
{
outString = nvp.value;
break;
}
}
}
Here's the decompiled third party dll code:
public IDataReader ExecSPGetDataReader(string sp, List<param> paramList)
{
IDataReader dataReader;
using (DbCommand dbC = this.setDBCommand(sp, paramList, true))
{
IDataReader dr = this._db.ExecuteReader(dbC);
this.setOutputParams(dbC, paramList);
dataReader = dr;
}
return dataReader;
}
It looks like the command gets disposed of, probably for the purpose of disposing the connection, but if that's true, how can anything be read from the returned IDataReader?
If the code uses the reader without disposing of it, it won't be explicitly returned to the pool, correct?
That is correct. Change the code to this:
using (IDataReader rdr =
db.ExecSPGetDataReader("dbo.someStoredProcedure", paramList))
{
if (rdr.Read())
{
List<nameValuePair> formValues =
Utils.nameValuePairs(rdr["valuepairs"].ToString());
foreach (nameValuePair nvp in formValues)
{
if (nvp.name.ToLower() == "name")
{
outString = nvp.value;
break;
}
}
}
}
The using statement will ensure that Dispose is called.
I am attempting to read a MySQL database from my C# project using the MySQL drivers for .net off the MySQL site.
Though I did a bit of research on this (including this), I am still flummoxed why this is happening. I later ran a spike and I still get the same error. (Prior to running this I populated the database with some default values.) Here's the spike code in toto.
class Program {
static void Main (string[] args) {
Console.WriteLine (GetUserAge ("john")); // o/p's -1
}
static int GetUserAge (string username) {
string sql = "select age from users where name=#username";
int val = -1;
try {
using (MySqlConnection cnn = GetConnectionForReading ()) {
cnn.Open ();
MySqlCommand myCommand = new MySqlCommand (sql, cnn);
myCommand.Parameters.AddWithValue ("#username", username);
using (MySqlDataReader reader = myCommand.ExecuteReader ()) {
DataTable dt = new DataTable ();
dt.Load (reader);
if (reader.Read ()) {
val = reader.GetInt32 (0);
}
}
}
} catch (Exception ex) {
Console.WriteLine (ex.Message);
} finally {
}
return val;
}
private static MySqlConnection GetConnectionForReading () {
string conStr = "Data Source=localhost;Database=MyTestDB;User ID=testuser;Password=password";
return new MySqlConnection (conStr);
}
}
The code above gives me the exception: "Invalid attempt to Read when reader is closed."
Later I modified the if-condition like so:
if (reader.HasRows && reader.Read ()) {
val = reader.GetInt32 (0);
}
And now the o/p is -1. (The data's in there in the table.) If for some reason the result set had zero rows, the reader should not have got into the if-block in the first place. I mean, the whole point of the Read() method is to check if there are any rows in the result set in the first place.
At my wit's end here... just cannot figure out where I'm going wrong.
Thank you for your help! :)
I think using DataTable.Load will "consume" the reader and, at the very least, position it at the end. It may even account for the closed connection (but I'm just guessing here). What if you remove that line? I don't think it makes any sense here.