On a production system, I occasionally find the following error in the log:
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
In order to remedy this I increased the maximum pool size to an outrageously high 10,000:
connectionString="metadata=res:///MyEntities.csdl|res:///MyEntities.ssdl|res://*/MyEntities.msl;provider=System.Data.SqlClient;provider connection string="data source=localhost;initial catalog=MyDb;integrated security=True;MultipleActiveResultSets=True;Max Pool Size=10000;App=My Service""
But the problem still occurs. What other causes could there be to this error, other than the connection pool being maxed out?
EDIT: before anyone else suggests it, I do always use using(...) { } blocks whenever I open a connection to the DB, e.g.:
using (var db = new MyEntities())
{
// do stuff
}
How are you connecting to the database?
Having a larger number of connections will make your application live longer, but it's likely that the root problem is that you're not releasing all of your connections properly. Check that you are closing connections after opening them. e.g.
using (SqlConnection myConnection = new SqlConnection(ConnectionString))
{
... perform database query
}
will automatically close the connection when done.
Usually this happens because you in the code close, for example, a DataReader, but you do not close its associated connection.
In the code above, there are two solutions depending on what you would like to do.
1/ Explicitly close the connection when done.
connection.Close();
2/ Use the connection in a Using block, this guarantees that the system disposes the connection (and closes it) when the code exits the block.
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
// Do work here; connection closed on following line.
}
Always .Close() the connections. That means always doing it is very important practice. If you're not doing it, you are doing it wrong. Any application can overfill the ConnectionPool. So make sure you have invoke the .Close() each time you opened it to clear the pool. Do not depend on the GC to close the connections.
Make sure you call .Close() in try blocks, also in catch blocks.
From what I have gathered from other sources like this MSDN page the garbage collection of the using (...) { } block is insufficient to prevent your application from running out of connections. The reply by William Vaughn states that clearing the connection explicitly via the Close() method returns threads to the pool far more quickly than a reliance on garbage collection.
So, while you have done nothing wrong as the using (...) { } block is proper coding, its lack of efficiency is what is leaving threads tied up too long. You may also look into the Collect() method to "force" garbage collection, but as the documentation states, this may cause performance issues (so it might be an option, or it might be trading one problem for another).
Related
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
This is the first ASP.net site I developed a long time ago, it has this code at the top of a lot of pages (and in a lot of methods).
cn = new SqlConnection(ConfigurationManager.ConnectionStrings["LocalSqlServer"].ToString());
cn.Open();
A lot of pages don't have:
cn.Close();
Also none of the pages do using(SqlConnection...), (although all the data readers are in using blocks).
First question is, is the the primary candiate for the error?
Second question is, is the best way to solve this refactoring/manually searching for unclosed connections? I know it's an ugly hack, but the time spent refactoring will be huge, but can we set a scheduled task to recycle the connection pool once a day at 3am or something?
Yes, that is the primary cause of the error. Currently, many of those connections will wait until the next GC to re-pool the underlying connection. You will exhaust the pool (and database connections) pretty quickly.
The best way of refactoring this is to add the missing using, such that the connection is scoped. Personally I'd also refactor that to a single method, i.e.
using(var cn = SomeUtilityClass.GetOpenConnection())
{...}
Closing the reader does little unless the reader is marked to close the connection; and having the data-reader close the connection (via a behaviour) sort of works, but it assumes you get as far as consuming the reader - it won't necessarily behave well in error scenarios.
Even I have encountered this error in the application that I once worked on. The problem that I identified was the same - no using statements and no close calls.
I would advise you to search the whole project for SqlConnection and then include the SqlConnection, SqlCommand and SqlDataAdapter in using statements and also do a connection.close within the sqlconnection using statement. Together with this in the config file increase the timeout of the connection within the connection string. You can also you CommandTimeout property of SqlCommand.
I have my business-logic implemented in simple static classes with static methods. Each of these methods opens/closes SQL connection when called:
public static void DoSomething()
{
using (SqlConnection connection = new SqlConnection("..."))
{
connection.Open();
// ...
connection.Close();
}
}
But I think passing the connection object around and avoiding opening and closing a connection saves performance. I made some tests long time ago with OleDbConnection class (not sure about SqlConnection), and it definitely helped to work like this (as far as I remember):
//pass around the connection object into the method
public static void DoSomething(SqlConnection connection)
{
bool openConn = (connection.State == ConnectionState.Open);
if (!openConn)
{
connection.Open();
}
// ....
if (openConn)
{
connection.Close();
}
}
So the question is - should I choose the method (a) or method (b) ? I read in another stackoverflow question that connection pooling saved performance for me, I don't have to bother at all...
PS. It's an ASP.NET app - connections exist only during a web-request. Not a win-app or service.
Stick to option a.
The connection pooling is your friend.
Use Method (a), every time. When you start scaling your application, the logic that deals with the state will become a real pain if you do not.
Connection pooling does what it says on the tin. Just think of what happens when the application scales, and how hard would it be to manually manage the connection open/close state. The connection pool does a fine job of automatically handling this. If you're worried about performance think about some sort of memory cache mechanism so that nothing gets blocked.
Always close connections as soon as you are done with them, so they underlying database connection can go back into the pool and be available for other callers. Connection pooling is pretty well optimised, so there's no noticeable penalty for doing so. The advice is basically the same as for transactions - keep them short and close when you're done.
It gets more complicated if you're running into MSDTC issues by using a single transaction around code that uses multiple connections, in which case you actually do have to share the connection object and only close it once the transaction is done with.
However you're doing things by hand here, so you might want to investigate tools that manage connections for you, like DataSets, Linq to SQL, Entity Framework or NHibernate.
Disclaimer: I know this is old, but I found an easy way to demonstrate this fact, so I'm putting in my two cents worth.
If you're having trouble believing that the pooling is really going to be faster, then give this a try:
Add the following somewhere:
using System.Diagnostics;
public static class TestExtensions
{
public static void TimedOpen(this SqlConnection conn)
{
Stopwatch sw = Stopwatch.StartNew();
conn.Open();
Console.WriteLine(sw.Elapsed);
}
}
Now replace all calls to Open() with TimedOpen() and run your program. Now, for each distinct connection string you have, the console (output) window will have a single long running open, and a bunch of very fast opens.
If you want to label them you can add new StackTrace(true).GetFrame(1) + to the call to WriteLine.
There are distinctions between physical and logical connections. DbConnection is a kind of logical connection and it uses underlying physical connection to Oracle. Closing/opening DbConnection doesn't affect your performance, but makes your code clean and stable - connection leaks are impossible in this case.
Also you should remember about cases when there are limitations for parallel connections on db server - taking that into account it is necessary to make your connections very short.
Connection pool frees you from connection state checking - just open, use and immediately close them.
Normally you should keep one connect for each transaction(no parallel computes)
e.g when user execute charge action, your application need find user's balance first and update it, they should use same connection.
Even if ado.net has its connection pool, dispatching connection cost is very low, but reuse connection is more better choice.
Why not keep only one connection in application
Because the connection is blocking when you execute some query or command,
so that means your application is only doing one db operation at sametime,
how poor performance it is.
One more issue is that your application will always have a connection even though your user is just open it but no operations.If there are many user open your application, db server will cost all of its connection source in soon while your users have not did anything.
I came across an article saying that using my sqlConnection like this :
using (SqlConnection sqlConnection = new SqlConnection(Config.getConnectionString()))
{
using (SqlDataAdapter dataAdapter = new SqlDataAdapter(query, sqlConnection))
{
dataAdapter.Fill(dataSet);
}
}
increases performance because it disposes the objects at the end of your method. So i have been coding with 'Using' for a while now, after chatting with some other developers they said that that creating and destroying the instance multiple times wont really increase performance.
What are the performance implications on the sqlserver and system resources if I am using 'Using' on all of my dataAccess methods. Will the sqlServer be hit harder because of the connection being connected and reconnected multiple times?
SqlConnection, by default, has connection pooling enabled. The Dispose() simply releases the connection to the pool sooner. This means other code can then re-use this connection, reducing the connections to the SQL server, and reducing the time to establish a physical connection.
So yes: it can improve overall performance.
The alternatives:
if your code exits cleanly and you always remember to Close() the connection, then probably no difference
if your code throws an exception (that you haven't handled), or you forget to Close() the connection, then you could be leaving unused connections lying around until there is enough memory pressure to trigger GC and the finalizer. This could mean you need more physical connections to the SQL server (a pain), and every time a new underlying connection is needed it has to take the performance hit of establishing the actual database connection
Overall, though - think of IDisposable as a contract; it is your job as a .NET developer to notice IDisposable resources, and actively Dispose() them when you are done, ideally with using if the usage is tightly scoped (like in this case).
It has no significant influence on performance in most cases.
All the using().... construct makes sure is that the SqlConnection is freed / disposed of after it's done its job. That's all there is - no magic performance boost....
Sure - creating and disposing objects does cost a bit of performance - it's either that, or then you unnecessarily keep objects in your memory and connections to your SQL Server open for much longer than needed.
I would vote for using the using() {...} approach 100% of the time - it's cleaner, it's safer, it's just better programming practice. The performance "hit" you might take is miniscule and not worth the trouble.
Marc
It increases performance only in the sense that, after your connection instance has been disposed, the physical connection in the pool can be re-used by another thread. If you kept it open, then another thread trying to open a connection would add a new physical connection to the pool.
ADO.NET has such feature as connection pooling, therefore if you intensively open connections, most likely connection will not be disposed, only returned to the pool.
If you are doing several database operations after each other, you should use the same connection instead of creating one connection for each. Otherwise you should close the connection as soon as possible, so that it is returned to the connection pool and can be reused.
You should always use a using block for the connection, so that you are sure that they are closed properly. If you fail to close a connection object it will stay in memory until the garbage collector removes it, hogging a database connection. That means that the next opreation can't reuse the connection from the pool but it has to establish a completely new connection, which takes a lot longer.
there is a performance improvement .
if you use using
E.g :
using (SqlConnection sqlConnection = new SqlConnection("ConnectionString"))
{
}
The compiler automatically add try and finally.
alt text http://img111.imageshack.us/img111/4514/using.jpg
I read that .NET uses connection pooling.
For example, if I instantiate a bunch of SqlConnection objects with the same connection string, then internally .NET will know to use the same connection.
Is this correct?
Also, in a big web-based application, any tips on the best way to harness this "power" ?
Setting up the TCP connection between your Web application and SQL Server can be an expensive operation. Connection pooling allows connections to the database to be reused for subsequent data requests. Rather than setting up a new TCP connection on each request, a new connection is set up only when one is not available in the connection pool. When the connection is closed, it is returned to the pool where it remains connected to the database, as opposed to completely tearing down that TCP connection.
Always close your connections when you're finished with them. No matter what anyone says about garbage collection within the Microsoft .NET Framework, always call Close or Dispose explicitly on your connection when you are finished with it. Do not trust the common language runtime (CLR) to clean up and close your connection for you. The CLR will eventually destroy the class and force the connection closed, but you have no guarantee when the garbage collection on the object will actually happen.
To use connection pooling optimally, there are a couple of rules to live by. First, open the connection, do the work, and then close the connection. It's okay to open and close the connection multiple times on each request if you have to, rather than keeping the connection open and passing it around through different methods. Second, use the same connection string (and the same thread identity if you're using integrated authentication). If you don't use the same connection string, for example customizing the connection string based on the logged-in user, you won't get the same optimization value provided by connection pooling. And if you use integrated authentication while impersonating a large set of users, your pooling will also be much less effective.
The .NET CLR data performance counters can be very useful when attempting to track down any performance issues that are related to connection pooling.
http://msdn.microsoft.com/en-us/magazine/cc163854.aspx
If you use the following syntax, when ever the using block is left the dispose method will be called, even if an exception occurs.
using(SqlConnection connection = new SqlConnection())
{
// Work with connection object here.
}
//connection object gets disposed here.
not sure if this is entirely related, but I just took over a project and noticed the original programming team failed to do something very important.
when you have a SQLConnection, let's call it conn and you do this:
conn.Open();
and then you perform some SQL statement, be it a select, insert or update. it is entirely possible that it will fail. So of course, you should do this:
try { conn.Open() }
catch (SqlException ex)
{
//do your logging/exception handling
}
however, people forget to add the Finally block.
finally {
if (conn.State == System.Data.ConnectionState.Open)
conn.Close();
}
you want to make sure if you have an exception that the connection does not stay open, so make sure you close it.
I am using multiple queries to pull data from the same server in my application. The issue is that I have to open a new connection every time I have a new query.
Is it even possible to:
Open the connection
Run query
Pull results
Run another query
Pull another result
Run final query
Pull another result
Close connection.
Although you may not yet know it, you are doing it correctly.
Open the connection, do your query, close it. Preferably using a using block or try/finally.
This may sound like a lot of overhead, but the connection pool in the .NET Framework Data Provider for SQL Server will actually optimize this for you.
In fact closing the connection is recommended.
Here is a quote from the documentation:
It is recommended that you always
close the Connection when you are
finished using it in order for the
connection to be returned to the pool.
This can be done using either the
Close or Dispose methods of the
Connection object. Connections that
are not explicitly closed might not be
added or returned to the pool. For
example, a connection that has gone
out of scope but that has not been
explicitly closed will only be
returned to the connection pool if the
maximum pool size has been reached and
the connection is still valid.
Here is an example of some code that does this:
try {
conn.Open();
// Perform query here
} finally {
conn.Close();
}
For reference:
http://msdn.microsoft.com/en-us/library/8xx3tyca(VS.71).aspx
If you are using ASP.NET with the same connection string you will be using a pooled connection that may never get physically closed, so you will pretty much always use an available open connection.
It's very possible. Assuming that you are talking about Connection and a DataReader. If you have to create a different connection every time, it sound like something is going wrong.
Without seeing any code, I am guessing that you are leaving the DataReader open. This is a BIG mistake. By default DataReaders completely consume the connection and leaving it unclosed can lead leaks. Close the DataReader, then execute another. I'd recommend wrapping the DataReader in a using block.
Rob
Short answer: Yes. This should be possible with most data providers.
Long answer: It depends on what you are using for your data access. However, you probably do not need to worry about it. Many data provider frameworks have connection pooling built in, so the subsequent connection creation/opening shouldn't "really" open a connection.
Sure, if you're using a SqlConnection object you can just do something like this:
connection.Open();
cmd.ExecuteReader(); // or any other form of getting the data
cmd2.ExecuteReader();
.
.
.
.
connection.Close();
I'd also like to add, if you're using a few SqlDataAdapters for your queries, although you normally don't need to open the connection by yourself, if you DO explicitly call connection.Open() it then won't close the connection for you automatically, allowing you to execute multiple queries with only one connection.
If you are using C# to open a connection. use using statement will help you clean up the resource/connection even if there is some excepion throwing out.
using (SqlConnection connection =
new SqlConnection(connectionString)
{
connection.Open();
//issue command
}
And read this:
http://msdn.microsoft.com/en-us/library/8xx3tyca(VS.71).aspx, you can "Controlling Connection Pooling with Connection String Keywords", and the system will handle pooling for you.