We Keep Coding

What can I do to stop getting Connection Pool error

I was getting quite often the following error:
Npgsql.NpgsqlException: 'The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 15 seconds)'
Then I looked for possible causes and solutions in here and found out that I should be applying the using statement. So I reviewed all my code and did that.
However, I keep getting that error while testing a button that gets information from my database, does some calculation and writes the results in a few textboxes. It usually crashes at the fifth-ish time I click it. A piece of the code follows:
private void CalcTemp(Cable cable)
{
string sqlString = "Server=172.19.2.40; Port=5432; User Id=postgres; Password=password; Database=PROLIG;";
using (NpgsqlConnection sqlCon = new NpgsqlConnection(sqlString))
{
string cmdString = #"SELECT tempamb, elevmaxonan, elevmaxonaf, elevmaxonaf2, topoil1_2, topoil1_4, especial1factor, especial1topoil,
especial2factor, especial2topoil, especial3factor, especial3topoil, especial4factor, especial4topoil,
especial5factor, especial5topoil, especial6factor, especial6topoil FROM correntes WHERE prolig_ofs_id = #id;";
NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon);
sqlCmd.Parameters.AddWithValue("id", StartOF.MyOF.id);
NpgsqlDataAdapter sqlDa = new NpgsqlDataAdapter(sqlCmd);
DataTable dt = new DataTable();
sqlDa.Fill(dt);
//does calculation
}
Any thoughts on why this is happening and how to fix it?
Thanks a lot.

Just add using to your command creation:
using (NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon))
{
Disposing all objects which implement IDisposable is a good practice.
Since your command is not disposed of in time, your connection is not closed and returned to the pool.
It is a lot of stuff on dispose which must be executed directly (or by using).
protected override void Dispose(bool disposing)
{
if (State == CommandState.Disposed)
return;
if (disposing)
{
// Note: we only actually perform cleanup here if called from Dispose() (disposing=true), and not
// if called from a finalizer (disposing=false). This is because we cannot perform any SQL
// operations from the finalizer (connection may be in use by someone else).
// We can implement a queue-based solution that will perform cleanup during the next possible
// window, but this isn't trivial (should not occur in transactions because of possible exceptions,
// etc.).
if (_prepared == PrepareStatus.Prepared)
_connector.ExecuteBlind("DEALLOCATE " + _planName);
}
Transaction = null;
Connection = null;
State = CommandState.Disposed;
base.Dispose(disposing);
}
So you need the following code:
private void CalcTemp(Cable cable)
{
string sqlString = "Server=172.19.2.40; Port=5432; User Id=postgres; Password=password; Database=PROLIG;";
using (NpgsqlConnection sqlCon = new NpgsqlConnection(sqlString))
{
string cmdString = #"SELECT * FROM correntes WHERE prolig_ofs_id = #id;";
using (NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon))
{
sqlCmd.Parameters.AddWithValue("id", StartOF.MyOF.id);
NpgsqlDataAdapter sqlDa = new NpgsqlDataAdapter(sqlCmd);
DataTable dt = new DataTable();
sqlDa.Fill(dt);
//does calculation
} //end using command (calls dispose on command, even if exception happens)
} //end using connection (calls dispose on connection object, even if exception happens)
}
Next advice - do not use data tables in case of large amount of data. Use DataReader instead.

SqlCommand timeout in C# MMO application

I have a C# project that is working with TCP socket in an asynchronous way.
Every request comes from client and ask question from SQL Server stored procedure, opens and closes a SQL connection after ending of question.
I've used this code:
using (var con = new SqlConnection(setting.ConnectionString))
{
try
{
//some codes (edited)
SqlCommand command = new SqlCommand(con);
command.CommandText = "procedurename1";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(new SqlParameter("#name", sb.ToString()));
SqlDataAdapter adapter = new SqlDataAdapter(command);
try
{
adapter.Fill(dataSet);
}
catch (Exception ex)
{
con.Close();
con.Dispose();
throw ex;
}
finally {
con.Close();
con.Dispose();
}
}
catch(Exception ex)
{}
finally
{
con.close();
con.dispose();
}
}
I've used
netstat -a -n | find /c "1433"
to count SQL connections open and close.
Problem is SQL connections count increases and it rarely decreases and count down.
Main problem, is when my program works under lots of requests about 30 minutes, I get
SqlCommand timeout error (default 30 seconds passed)
and after restarting my C# program, the SqlCommand timeout will be gone.
Is this a problem of my program or SQL Server side?
Remember it always calls a stored procedure in SQL Server, not executing query
directly.
main method:
public void main()
{
Task.Factory.StartNew(() =>
{
allDone.Reset();
mySocket.AcceptAsync(e);
allDone.WaitOne();
});
}
public void e_Completed(object sender, SocketAsyncEventArgs e)
{
var socket = (Socket)sender;
ThreadPool.QueueUserWorkItem(HandleTcpRequest, e.AcceptSocket);
e.AcceptSocket = null;
socket.AcceptAsync(e);
}
public void HandleTcpRequest(object state)
{
//do some code and connection to SQL server
DLL.Request httprequest = new DLL.Request(dataSet.Tables[0], fileDt);
DLL.IHttpContext _context = new DLL.HttpContext(httprequest);
_context.GetResults();
}

Main problem, is when my program works under lots of requests about 30 minutes,
To isolate the root problem of the time-out, I suggest testing the sql query of the stored procedure independent of TCP socket calls for 30 minutes
and log the time-out exception details for inspection
Run the following query within 30 minutes to simulate your working environment:
public void RunQuery()
{
using (var con = new SqlConnection(setting.ConnectionString))
{
try
{
//some codes
}
catch(SqlException ex)
{
//test for timeout
if (ex.Number == -2) {
Console.WriteLine ("Timeout occurred");
// log ex details for more inspection
}
}
}
}
Read How to handle the CommandTimeout properly?
As you use async calls, I suggest you to try to use Asynchronous Database Calls With Task-based Asynchronous Programming Model (TAP)

I'm going to take a long-shot based on the way the limited Sql-related code we can see is written since we can't see "//some codes".
I'm going to guess that some of the disposable things like SqlCommand, DataReader, SqlDataAdapter, TransactionScope, etc are not in 'using' blocks, so are holding resources open on the database.
It may also be worth raising the possibility that this kind of problem could be in the code shown in the question or any other program accessing that database, including your own applications and SSMS (e.g. if a developer has an uncommitted transaction running in a window).
P.S. I would suggest deleting everything in the using block except the "//some codes" part.
UPDATE after more code was added
Here is your code after correction. this will ensure that the resources are disposed, which will prevent the leaking resources that are probably causing your problem.
using (var con = new SqlConnection(setting.ConnectionString))
{
//some codes (edited)
using (SqlCommand command = new SqlCommand(con))
{
command.CommandText = "procedurename1";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(new SqlParameter("#name", sb.ToString()));
using (var adapter = new SqlDataAdapter(command))
{
adapter.Fill(dataSet);
}
}
}
P.S. don't ever write "throw ex;" from inside a catch ever again. It causes the stack trace to be lost - just use "throw;".

.NET Core wont close my MySqlConnection

I have a .NET Core program that uses the MySqlConnection class. My Database is a ClearDB Database that is stored in Azure.
When I launch the program it is working like it should. But when I wait for like 10 minuts doing nothing, it wont connect to the database anymore(Timeout?). Restarting the program and it works again.
When looking at the connections on the ClearDB webpage it isn't closing when I close it in my program. After 10 minuts or so it closes automaticly, as I see in ClearDB webpage. But with the program still running it wont connect to the database anymore. Restarting program is only solution.
Code for now looks something like this:
private static async Task<uint> getDeviceId(string macAddress)
{
using (var connection = new MySqlConnection(ConnectionString))
{
uint returnvalue = 0;
var cmd = connection.CreateCommand() as MySqlCommand;
cmd.CommandText = #"SELECT id FROM devices WHERE mac = '" + macAddress + "'";
connection.Open();
Console.WriteLine(connection.State);
DbDataReader reader = await cmd.ExecuteReaderAsync();
using (reader)
{
while (await reader.ReadAsync())
{
returnvalue = await reader.GetFieldValueAsync<uint>(0);
}
}
reader.Dispose();
cmd.Dispose();
return returnvalue;
}
}
I have tried the following:
Using statement
Close/dispose connection,reader and command
Pooling=false in connectionstring
But none of them works. Somebody got an idea?

Assuming MySql provider is like the MSSQL provider, it does not actually close the connection in the database, it just releases it back to the pool.
You do not want to disable pooling, you will kill efficiency.
This is by design, and what you want.

The using statement from the code snippet should close your connections. However, I'm not sure how that interacts with async, or how ClearDB differs from normal MySql. Given the issues in the question and that lack of clarity, you might try this, just to see if it helps:
private static async Task<uint> getDeviceId(string macAddress)
{
uint returnvalue = 0;
MySqlConnection connection;
try
{
connection = new MySqlConnection(ConnectionString);
var cmd = connection.CreateCommand() as MySqlCommand;
//Don't EVER(!) use string concatenation like that in a query!
cmd.CommandText = #"SELECT id FROM devices WHERE mac = #macAddress";
cmd.Parameters.Add("#macAddress", MySqlDbType.VarChar, 18).Value = macAddress;
connection.Open();
Console.WriteLine(connection.State);
DbDataReader reader = await cmd.ExecuteReaderAsync();
using (reader)
{
while (await reader.ReadAsync())
{
returnvalue = await reader.GetFieldValueAsync<uint>(0);
}
}
reader.Dispose();
cmd.Dispose();
}
finally
{
connection.Close();
connection.Dispose();
}
return returnvalue;
}
A using block basically just re-writes your code as try/finally anyway, so doing this step by-hand can sometimes make debugging easier (you can log where it hits the .Close() call).
If this does resolve the problem, I wouldn't stop there, but rather start from there and see just how close to "normal" code you can get. I'm also concerned here that you have disabled connection pooling, and that this method is static.

Should ServiceBroker conversations be inside a transaction

We have some code that looks a bit like this (error handling and other things removed)
using (var tran = conn.BeginTransaction())
{
var client = new Service(...);
var dialog = client.GetConversation(null, conn, tran);
var response = dialog.Receive();
// do stuff with response, including database work
dialog.Send(message, conn, tran);
dialog.EndConversation(conn, tran);
tran.Commit();
conn.Close();
}
We've inherited this code and aren't experts in ServiceBroker, would there be problems if we moved the conversation outside of the transaction like this:
var client = new Service(...);
var dialog = client.GetConversation(null, conn, tran);
var response = dialog.Receive();
using (var tran = conn.BeginTransaction())
{
// do stuff with response, including database work
tran.Commit();
}
dialog.Send(message, conn, tran);
dialog.EndConversation(conn, tran);
conn.Close();

In this case you receive message and its gets removed from the queue. You will not be able to receive it again..
If all code is in transaction and there is error in message processing- transaction never commits and message stays in queue (by default- after 5 rollbacks queue gets disabled). So you can detect the reason of error, correct it and process message again (expected exceptions should not cause rollback, there are quite a few ways to handle them).
I would say that everything should be in transaction.

TransactionScope helper that exhausts connection pool without fail - help?

A while back I asked a question about TransactionScope escalating to MSDTC when I wasn't expecting it to. (Previous question)
What it boiled down to was, in SQL2005, in order to use a TransactionScope, you can only instance and open a single SqlConnection within the life of the TransactionScope. With SQL2008, you can instance multiple SqlConnections, but only a single one can be open at any given time. SQL2000 will always escalate to DTC...we don't support SQL2000 in our application, a WinForms app, BTW.
Our solution to single-connection-only problem was to create a TransactionScope helper class, called LocalTransactionScope (aka 'LTS'). It wraps a TransactionScope and, most importantly, creates and maintains a single SqlConnection instance for our application. The good news is, it works - we can use LTS across disparate pieces of code and they all join the ambient transaction. Very nice. The trouble is, every root LTS instance created will create and effectively kill a connection from the connection pool. By 'Effectively Kill' I mean it will instance a SqlConnetion, which will open a new connection (for whatever reason, it never reuses a connection from the pool,) and when that root LTS is disposed, it closes and disposes the SqlConnection which is supposed to release the connection back to the pool so that it can be reused, however, it clearly never is reused. The pool bloats until it's maxed out, and then the application fails when a max-pool-size+1 connection is established.
Below I've attached a stripped down version of the LTS code and a sample console application class that will demonstrate the connection pool exhaustion. In order to watch your connection pool bloat, use SQL Server Managment Studio's 'Activity Monitor' or this query:
SELECT DB_NAME(dbid) as 'DB Name',
COUNT(dbid) as 'Connections'
FROM sys.sysprocesses WITH (nolock)
WHERE dbid > 0
GROUP BY dbid
I'm attaching LTS here, and a sample console application that you can use to demonstrate for yourself that it will consume connections from the pool and never re-use nor release them. You will need to add a reference to System.Transactions.dll for LTS to compile.
Things to note: It's the root-level LTS that opens and closes the SqlConnection, which always opens a new connection in the pool. Having nested LTS instances makes no difference because only the root LTS instance establishes a SqlConnection. As you can see, the connection string is always the same, so it should be reusing the connections.
Is there some arcane condition we're not meeting that causes the connections not to be re-used? Is there any solution to this other than turning pooling off entirely?
public sealed class LocalTransactionScope : IDisposable
{
private static SqlConnection _Connection;
private TransactionScope _TransactionScope;
private bool _IsNested;
public LocalTransactionScope(string connectionString)
{
// stripped out a few cases that need to throw an exception
_TransactionScope = new TransactionScope();
// we'll use this later in Dispose(...) to determine whether this LTS instance should close the connection.
_IsNested = (_Connection != null);
if (_Connection == null)
{
_Connection = new SqlConnection(connectionString);
// This Has Code-Stink. You want to open your connections as late as possible and hold them open for as little
// time as possible. However, in order to use TransactionScope with SQL2005 you can only have a single
// connection, and it can only be opened once within the scope of the entire TransactionScope. If you have
// more than one SqlConnection, or you open a SqlConnection, close it, and re-open it, it more than once,
// the TransactionScope will escalate to the MSDTC. SQL2008 allows you to have multiple connections within a
// single TransactionScope, however you can only have a single one open at any given time.
// Lastly, let's not forget about SQL2000. Using TransactionScope with SQL2000 will immediately and always escalate to DTC.
// We've dropped support of SQL2000, so that's not a concern we have.
_Connection.Open();
}
}
/// <summary>'Completes' the <see cref="TransactionScope"/> this <see cref="LocalTransactionScope"/> encapsulates.</summary>
public void Complete() { _TransactionScope.Complete(); }
/// <summary>Creates a new <see cref="SqlCommand"/> from the current <see cref="SqlConnection"/> this <see cref="LocalTransactionScope"/> is managing.</summary>
public SqlCommand CreateCommand() { return _Connection.CreateCommand(); }
void IDisposable.Dispose() { this.Dispose(); }
public void Dispose()
{
Dispose(true); GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
if (disposing)
{
_TransactionScope.Dispose();
_TransactionScope = null;
if (!_IsNested)
{
// last one out closes the door, this would be the root LTS, the first one to be instanced.
LocalTransactionScope._Connection.Close();
LocalTransactionScope._Connection.Dispose();
LocalTransactionScope._Connection = null;
}
}
}
}
This is a Program.cs that will exhibit the connection pool exhaustion:
class Program
{
static void Main(string[] args)
{
// fill in your connection string, but don't monkey with any pooling settings, like
// "Pooling=false;" or the "Max Pool Size" stuff. Doesn't matter if you use
// Doesn't matter if you use Windows or SQL auth, just make sure you set a Data Soure and an Initial Catalog
string connectionString = "your connection string here";
List<string> randomTables = new List<string>();
using (var nonLTSConnection = new SqlConnection(connectionString))
using (var command = nonLTSConnection.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = #"SELECT [TABLE_NAME], NEWID() AS [ID]
FROM [INFORMATION_SCHEMA].TABLES]
WHERE [TABLE_SCHEMA] = 'dbo' and [TABLE_TYPE] = 'BASE TABLE'
ORDER BY [ID]";
nonLTSConnection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
string table = (string)reader["TABLE_NAME"];
randomTables.Add(table);
if (randomTables.Count > 200) { break; } // got more than enough to test.
}
}
nonLTSConnection.Close();
}
// we're going to assume your database had some tables.
for (int j = 0; j < 200; j++)
{
// At j = 100 you'll see it pause, and you'll shortly get an InvalidOperationException with the text of:
// "Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.
// This may have occurred because all pooled connections were in use and max pool size was reached."
string tableName = randomTables[j % randomTables.Count];
Console.Write("Creating root-level LTS " + j.ToString() + " selecting from " + tableName);
using (var scope = new LocalTransactionScope(connectionString))
using (var command = scope.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = "SELECT TOP 20 * FROM [" + tableName + "]";
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.Write(".");
}
Console.Write(Environment.NewLine);
}
}
Thread.Sleep(50);
scope.Complete();
}
Console.ReadKey();
}
}

The expected TransactionScope/SqlConnection pattern is, according to MSDN:
using(TransactionScope scope = ...) {
using (SqlConnection conn = ...) {
conn.Open();
SqlCommand.Execute(...);
SqlCommand.Execute(...);
}
scope.Complete();
}
So in the MSDN example the conenction is disposed inside the scope, before the scope is complete. Your code though is different, it disposes the connection after the scope is complete. I'm not an expert in matters of TransactionScope and its interaction with the SqlConnection (I know some things, but your question goes pretty deep) and I can't find any specifications what is the correct pattern. But I'd suggest you revisit your code and dispose the singleton connection before the outermost scope is complete, similarly to the MSDN sample.
Also, I hope you do realize your code will fall apart the moment a second thread comes to play into your application.

Is this code legal?
using(TransactionScope scope = ..)
{
using (SqlConnection conn = ..)
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
using (SqlConnection conn = ..) // the same connection string
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
scope.Complete();
}