[Disclaimer] : I think I have read every stackoverflow post about this already
I have been breaking my head over this for quite some time now. I am getting the following exception in my asp.net web.api.
Exception thrown: 'System.InvalidOperationException' in mscorlib.dll
Additional information: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Most people suggested that I should look for leaked connections in my application. Here is my code. Now I am sure that I am not leaking any connections
public async Task<IEnumerable<string>> Get()
{
var ds = new DataSet();
var constring = "Data Source=xxx;Initial Catalog=xxx;User Id=xxx;Password=xxx;Max Pool Size=100";
var asyncConnectionString = new SqlConnectionStringBuilder(constring)
{
AsynchronousProcessing = true
}.ToString();
using (var con = new SqlConnection(asyncConnectionString))
using (var cmd = new SqlCommand("[dbo].[xxx]", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("#x1", 1);
cmd.Parameters.AddWithValue("#x2", "something");
await con.OpenAsync();
using (var rdr =await cmd.ExecuteReaderAsync())
{
if (rdr.HasRows)
{
ds.Load(rdr, LoadOption.OverwriteChanges, "MyTable");
}
rdr.Close();
con.Close();
ds.Dispose();
}
}
//I know this looks wrong, just an empty api method to show the code
return new string[] { "value1", "value2" };
}
The exception does not occur when I am using my local Sql Server. Only happens when I connect to our 'test server'. Are there anything else I can look at when trying resolve this issue. Like Sql server settings / network settings etc.
The stored procedure I call does not lock up the db I have checked for that as well. If that was the case it should have failed on my local Sql instance as well.
I am using jmeter to generate load, 1500 - threads(users). Surely I should be able to handle way more than that.
Thanks in advance
You have not specified any Connection Time out property, so it's 15 seconds default. using Max Pool Size=100 is not a good idea until you don't have proper hardware resources.
You started 1500 threads, so it seems that the some the threads keep waiting for 15 seconds to get their chance for connection opening. And as time goes out, you get the connection time out error.
So I think increasing the 'Connection Timeout' property in connection string may resolve your issue.
Related
I'm hosting an Aurora MySql instance on AWS and trying to read a table from it on a Lambda function.
This is my connection string:
Server=xxx.xxx.xxx4.xxx; port=3306; database=thedatabase; uid=theuser; pwd=thepassword; Connect Timeout=300
This is the code (.Net Core 2.1):
private static void GetFromDb()
{
LambdaLogger.Log($"Function name GetFromDb() has been called.\n");
int counter = 0;
try
{
LambdaLogger.Log($"Using {str}\n");
using (MySqlConnection conn = new MySqlConnection(str))
{
LambdaLogger.Log($"Connection is about to be opened\n");
conn.Open();
LambdaLogger.Log($"Connection was opened\n");
var text = "SELECT * FROM MarketPlace.Customers";
using (MySqlCommand cmd = new MySqlCommand(text, conn))
{
cmd.CommandTimeout = 360;
var reader = cmd.ExecuteReader();
LambdaLogger.Log($"Command was issued\n");
if (reader.HasRows)
{
LambdaLogger.Log($"reader has rows\n");
products = new List<Product>();
while (reader.Read())
{
counter++;
LambdaLogger.Log($"Reading # {counter}\n");
Product p = new Product();
p.Id = reader.GetInt32(0);
p.Name = reader.GetString(1);
products.Add(p);
}
}
reader.Close();
LambdaLogger.Log($"{counter} items readed");
}
}
}
catch (Exception ex)
{
throw new Exception($"[GetFromDb] Error {ex.ToString()}", ex);
}
}
When try to open the connection, code stops executing, no exception is caught or raised.
Log from CloudWatch:
START RequestId: 52225968-d360-4d27-8872-305e4b92e346 Version: $LATEST
...
...
Function name GetFromDb() has been called.
Using Server=xxx.xxx.xxx4.xxx; port=3306; database=thedatabase; uid=theuser; pwd=thepassword; Connect Timeout=300
Connection is about to be opened
END RequestId: 52225968-d360-4d27-8872-305e4b92e346
REPORT RequestId: 52225968-d360-4d27-8872-305e4b92e346 Duration: 30030.17 ms Billed Duration: 30000 ms Memory Size: 256 MB Max Memory Used: 107 MB Init Duration: 207.87 ms
2019-12-12T18:23:58.089Z 52225968-d360-4d27-8872-305e4b92e346 Task timed out after 30.03 seconds
I'm really stuck here. I don't have any idea of what is happening. Roles, policies, etc. are ok.
The strange thing is despite connection timeout is set to 300 seconds, it takes less the this to stop running.
Any help would be appreciated. Thanks in advance.
A timeout is usually an indication of network connectivity issues.
Assumption:
The AWS Lambda function is configured to use the same VPC as the Aurora instance
Your Security Group configurations should be:
A Security Group on the Lambda function (Lambda-SG) — Allow all Outbound
A Security Group on the Aurora database (Aurora-SG) — Allow inbound connections on the appropriate port (3306?) from Lambda-SG
That is, Aurora-SG specifically allows inbound traffic from Lambda-SG.
I have created this stored procedure in SQL Server 2008 R2:
CREATE PROCEDURE dbo.test AS
BEGIN
WAITFOR DEALY '00:00:40'
END
I have also written a C# console app to test the connect timeout property.
static void Main(string[] args)
{
OpenSqlConnection();
Console.ReadLine();
}
private static void OpenSqlConnection()
{
string connectionString = GetConnectionString();
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (var command = new SqlCommand("dbo.test", connection) { CommandType = CommandType.StoredProcedure })
{
connection.Open();
Console.WriteLine("ConnectionTimeout: {0}", connection.ConnectionTimeout);
command.ExecuteNonQuery();
Console.WriteLine("Finished");
connection.Close();
}
}
}
static private string GetConnectionString()
{
return
"Data Source=test.com;Initial Catalog=test_db;Integrated Security=True;MultipleActiveResultSets=True;Connect Timeout=10; Application Name=Test";
}
There was a connection timeout exception returned as excepted (timeout in connection string < 40s). However, when I changed the Connect Timeout property to 120, I still got the same timeout exception. After some trial and error, it seems that the timeout value is always 30s regardless of my connection string value.
Is the connection timeout controlled by somewhere else? (if yes, where?)
Why the connect timeout value in connection string has no effect on my console app?
Update
Thanks all. I have just noticed that the command timeout is different from connection timeout(timeout in connection string).
Instead of changing my app.config, I would have to specify the CommandTimeout property as Sudipta Maiti suggested and recompile my code.
There is a similar issue in:
http://forums.asp.net/t/1197160.aspx?Can+you+change+command+timeout+via+the+connection+string+
Use CommandTimeout:
using (var command = new SqlCommand("dbo.test", connection) {
CommandType = CommandType.StoredProcedure })
{
connection.Open();
Console.WriteLine("ConnectionTimeout: {0}",
connection.ConnectionTimeout);
command.CommandTimeout = 120;
command.ExecuteNonQuery();
Console.WriteLine("Finished");
connection.Close();
}
I strongly suspect the given Timeout, it may not be enough to complete your operations. So make it Connection Timeout=30;
Connect Timeout -or- Connection Timeout by default the value will be 15 seconds. Based on your operation performance you might need to extend the seconds.
When a connection is returned to the pool, its creation time is compared with the current time, and the connection is destroyed if that time span (in seconds) exceeds the value specified by Connection Lifetime. This is useful in clustered configurations to force load balancing between a running server and a server just brought online.
A value of zero (0) causes pooled connections to have the maximum connection timeout.
There is a Sample Source almost same as yours.
I am getting this error when I try to get all the username password from production copied local database, I guess it is because of not closing the connection properly, but I am not sure how . I am using the Microsoft Enterprise Library, ant thought or comment about it?
Timeout expired. The timeout period elapsed prior to
obtaining a connection from the pool. This may have occurred
because all pooled connections were in use and max pool size was reached.
this is the mothod that is getting the username and password and producing the error.
private Model.UsernameandPass GetUsernamePass(string AccountNumber)
{
Model.UsernameandPass model = null;
string myConnection = System.Configuration.ConfigurationManager.ConnectionStrings[connectionName].ToString();
SqlDatabase db = new SqlDatabase(myConnection);
using (DbCommand command = db.GetStoredProcCommand("Get_TheUsernamePassWordFromProduction"))
{
db.AddInParameter(command, "AccountNumber", DbType.String, AccountNumber);
var result = db.ExecuteReader(command);
try
{
while (result.Read())
{
model = new Model.UsernameandPass();
model.Username = result.GetString(1);
model.Password = result.GetString(2);
}
}
catch (Exception ex)
{
}
}
db = null;
return model;
}
I am getting the error in this line after program runs for a while.
var result = db.ExecuteReader(command);
You're getting that error because a connection cannot be established, not only because they aren't being closed properly. Check permissions for the user you're trying to authenticate against the database with. Also be sure to call .open()/.close() when/if you are programatically opening/closing connections.
Check this link. You may want to increase your pool size, or check for long-running queries.
I have the following method which uses a transaction.
private string getDocumentDetailsByNumber(string DocNumber)
{
SqlTransaction transaction = DALDBConnection.SqlConnection.BeginTransaction();
try
{
DataSet DocNum = new DataSet();
string sDocNumber = "";
string[] taleNamesDoc = new string[1];
taleNamesDoc[0] = "docnumber";
SqlParameter[] paramDoc = new SqlParameter[1];
paramDoc[0] = new SqlParameter("#DocumentNumber", DocNumber.ToString().Trim());
SqlHelper.FillDataset(transaction, CommandType.StoredProcedure, "spGetDocumentDetailsByNumber", DocNum, taleNamesDoc, paramDoc);
string docTitle = DocNum.Tables["docnumber"].Rows[0][0].ToString();
transaction.Commit();
return docTitle;
}
catch (Exception ex)
{
transaction.Rollback();
throw ex;
}
}
after running the method several times, user ended up getting the error message below.
the timeout period elapsed prior to obtaining a connection from the
pool
Error occurred because I haven't closed the connection and the connection pool has over flown.
I tried to close the connection before committing the transaction.
transaction.Connection.Close();
transaction.Commit();
Then got the following error.
This SqlTransaction has completed; it is no longer usable
How can I close the connection to avoid the error?
You cannot exhaust your pool by using a single connection. You need to close all connections you are using. Preferably after the transaction has ended one way or another. using blocks are your friend for almost all database related objects.
By the way:
throw ex;
This damages your exception by replacing the original stacktrace. Use:
throw;
to rethrow the exception you caught unchanged.
As mentioned, you should dispose of the connection properly. I've modified your code to demonstrate. Please note you will need to substitute the connection string with yours.
private string getDocumentDetailsByNumber(string DocNumber)
{
using (var connection = new SqlConnection("My Connection String"))
{
SqlTransaction transaction = connection.BeginTransaction();
DataSet DocNum = new DataSet();
string sDocNumber = "";
string[] taleNamesDoc = new string[1];
taleNamesDoc[0] = "docnumber";
SqlParameter[] paramDoc = new SqlParameter[1];
paramDoc[0] = new SqlParameter("#DocumentNumber", DocNumber.ToString().Trim());
SqlHelper.FillDataset(transaction, CommandType.StoredProcedure, "spGetDocumentDetailsByNumber", DocNum, taleNamesDoc, paramDoc);
string docTitle = DocNum.Tables["docnumber"].Rows[0][0].ToString();
transaction.Commit();
return docTitle;
} // Connection is disposed and cleaned up.
}
Opening new connections are cheap and should not be frowned upon. Each one you call the database you should open a new one like this. By maintaining a connection and not disposing of it, you are taking resources away from the database as well. It does not have an infinite amount of connections in its pool that can be used at once.
edit
Removed try/catch as mentioned in comments. If an exception is thrown while in the using block, a rollback will occur and the exception passed up the stack.
Have you considered CALLNIG CLOSE ON IT? Would be obvious to close a connection or?
Anything that implements IDIsposable should be disposed, btw., not just closed. And SqlConnection implements IDisposable. THis has nothing to do with SqlTransaction - you violae a fundamental rule of the .NET world by not disposing a disposable routine.
I want to set the querytimeout from the connection string.
not the connection timeout, is it possible?
No. It's per command, not per connection.
Edit, May 2013
As requested in comment:
SQLCommand.CommandTimeout for command execution
There is no matching SQLConnection property (the questions says not the SqlConnection.ConnectionTimeout property
Some more notes about commands and execution time outs in SQL Server (DBA.SE).
And more SO stuff: What happens to an uncommitted transaction when the connection is closed?
You have always been able to specify the Connect Timeout via the SqlClient connection string, this applies to establishing a connection with the database server, not executing commands / running queries.
The default Connect Timeout is 15 seconds.
With the release of Microsoft.Data.SqlClient v2.1 it's introduced the "Command Timeout" connection string property to override, if required, the default of 30 seconds for this property.
Hence it is now possible to set the default command timeout via the connection string.
In order to use this new feature, with EF Core 3 and 5, you must add an explicit dependency on the updated SqlClient package by adding the following to your project file:
<PackageReference Include="Microsoft.Data.SqlClient" Version="2.1.0" />
In addition, you must update your connection string in order to increase the default command timeout - keep in mind that this will apply to your entire application, unless overridden in code by setting the SqlCommand.CommandTimeout property.
Connection string examples:
"YourDatabaseAlias": "Server={serverURL}; Initial Catalog={db}; Integrated Security=true; Command Timeout=60"
The connection string above sets the command timeout to 1 minute (60 seconds).
Hope this is useful.
See:- ConnectionStrings content on this subject. There is no default command timeout property.
You can only set the connection timeout on the connection string, the timeout for your query would normally be on the command timeout.
(Assuming we are talking .net here, I can't really tell from your question).
However the command timeout has no effect when the command is executed against a context connection (a SqlConnection opened with "context connection=true" in the connection string).
I have tried different values for the parameter Command Timeout in the below connection string and it worked every time as expected.
Data Source=Your_Db_Server;Initial Catalog=Your_DB;Integrated Security=true;TrustServerCertificate=true;Connect Timeout=600;Command Timeout=120
Only from code:
namespace xxx.DsXxxTableAdapters {
partial class ZzzTableAdapter
{
public void SetTimeout(int timeout)
{
if (this.Adapter.DeleteCommand != null) { this.Adapter.DeleteCommand.CommandTimeout = timeout; }
if (this.Adapter.InsertCommand != null) { this.Adapter.InsertCommand.CommandTimeout = timeout; }
if (this.Adapter.UpdateCommand != null) { this.Adapter.UpdateCommand.CommandTimeout = timeout; }
if (this._commandCollection == null) { this.InitCommandCollection(); }
if (this._commandCollection != null)
{
foreach (System.Data.SqlClient.SqlCommand item in this._commandCollection)
{
if (item != null)
{ item.CommandTimeout = timeout; }
}
}
}
}
//....
}
I find answer in FollowCode:
SqlDataAdapter da = new SqlDataAdapter(Query, ConnectionString);
da.SelectCommand.CommandTimeout = queryTimeoutInSeconds;
you can set Timeout in connection string (time for Establish connection between client and sql). commandTimeout is set per command but its default time is 30 secend