When running my microservice developed with ASP.NET Core and EF Core (latest versions) I started getting this error:
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
This is very strange, as it worked fine until tomorrow. SQL database is hosted in Docker.
This exception is thrown by code:
try
{
using var c = ctx.Database.GetDbConnection();
var cconstrcc = c.ConnectionString;
var dbstate = c.State;
c.Open(); // <- exception thrown
ctx.Database.OpenConnection();
}
catch (Exception ex)
{
}
This is only test code, because application just hangs when I try to run query dbContext.SomeTable.ToList() (does not work with async version as well), and I am unable to see why and when it happens (I suspected some deadlock maybe).
This is very very strange, as it happens on the first attempt to connect to database.
I was inspecting SQL Server Monitor, SQL Server Profiler and also run stored procedure sp_who, but I don't see ANYTHING, any connection made from my app, it's like SQL pool exhausts immediately.
Also, we used dependency injection everywhere, but I have gone through all context usages and switched it to local variable with
using var ctx = new DbContext();
to make sure I close every connection. Issue is still exactly the same.
UPDATE
But I also run some test application (console app) and made connection simply by using SqlConnection object, without any ORM - it worked.
UPDATE
Also tried to go back in history with git checkout to point where it was definietely working - to my surprise, that version also hangs and does not work.
UPDATE
Changing from IIS to Kestrel resolved the issue, connections are made successfully. So it's IIS related bug ?
Related
I have a .net core application in which I am using c# and MongoDB. Within the application, I am using MongoDB driver (Version 2.7) for any database related operation and I have a MongoDB database (Version 4.0.9). I am facing one strange issue and not able to find out the root cause for it. The very first request to the database is taking significantly more time than the subsequent requests. As an example, if the first request is taking 1 second and if we make immediate more requests it is taking ~200-250 milliseconds
Does anyone know the solution to the above situation?
this is not an error. it is the default behavior of the c# driver. the driver only establishes the connection to the database server when the very first operation is initiated and will take a few hundred milliseconds to establish connection.
subsequent operations do not need to establish new connections because of the driver's connection pooling mechanisms. more connections will only be established only if they are really needed. if the app is not multi-threaded, the driver will usually open about 2 connections for the entirety of the app from what i have seen. if you inspect your mongodb log file, it will be apparent.
my suggestion is to just ignore the time it takes to initialize the connection if you're doing any kind of testing/ benchmarks.
UPDATE:
if your database is hosted across a network, something like a firewall may be interfering with idle connections. if that's the case you could try doing the following so that idle connections get recycled/renewed every minute.
MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromMinutes(1)
if all else fails to work, the only remaining option i can think of is to fire off a keep-alive task like the following:
public void InitKeepAlive()
{
Task.Run(async () =>
{
while (true)
{
await client.GetCollection<Document>("Documents")
.AsQueryable()
.Select(d => d.Id)
.FirstOrDefaultAsync();
await Task.Delay(TimeSpan.FromMinutes(1));
}
});
}
We have written several C# web services that have a connection to our internal Firebird 2.5.5 database.
Unfortunately the exception "Error reading data from the connection" is thrown more and more often and we don't know how to fix it.
We tried to disable pooling but this did not have the desired effect.
We also wrote a try catch block that reconnects and re-executes the SQL, but this does not seem to us to be the right solution.
Is there another option?
Here are some environment informations:
C# 7.0
.NET 4.5
Firebird Version 2.5.5
Firebird Driver 5.5.0
The Firebird log does not show any error messages at that time
The error happens from time to time with any sql statement
The problem is relatively simple: the network connection between client and server is interrupted or broken for some reason, but the State of the client connection remains Open - even though you cannot use that connection anymore. Unfortunately Firebird decided to not update this status to Broken automatically, which would make a lot more sense if you ask me.
You already figured out that reopening the connection "somewhat fixes" the problem, and we have discussed that you could do this only when FbException.ErrorCode is 335544726.
Unfortunately this does mean that any open transaction is also lost, and you cannot commit any data from it anymore. The only way I could think of to reliably recover from this situation is to rethrow the exception:
try
{
// ...
}
catch (FbException ex)
{
if (ex.ErrorCode == 335544726)
{
// close the connection (reopen depending on your application)
}
throw;
}
This way, you can catch this exception at a higher level in your application, and deal with it however is appropriate at that point - ie. retrying the entire transaction, or letting the user choose what to do.
I have a long running application that uses NHibernate.ISessionFactory to connect to an Oracle database.
Occasionally the database goes offline (e.g. for weekend maintenance), but even once the database is back online, subsequent queries fail with the following exception (inner exceptions also shown):
NHibernate.Exceptions.GenericADOException: could not execute query
[ select .....]
>> Oracle.ManagedDataAccess.Client.OracleException: ORA-03135: Connection lost contact
>> OracleInternal.Network.NetworkException: ORA-03135: Connection lost contact
>> System.Net.Sockets.SocketException: An established connection
was aborted by the software in your host machine
Restarting the application restores the functionality, but I would like the application to be able to automatically cope without a restart, by "resetting" the connection.
I have tried the following with my ISessionFactory when I hit this exception:
sf.EvictQueries();
sf.Close();
sf = null;
sf = <create new session factory>
but see the same exception after recreating the ISessionFactory. I assume this is because NHibernate is caching the underlying broken connection in some kind of connection pool?
How can I persuade NHibernate to create a genuinely new connection (or even just reset all state completely), and hence allow my application to fix the connection issue itself without an application restart?
EDIT:
Following A_J's answer, note that I am already calling using (var session = _sessionFactory.OpenSession()) for each database request.
I suspect you are opening ISession (call to ISessionFactory.OpenSession()) at startup and closing it at application end. This is wrong approach for any long running application.
You should manage connection at lower level of time. In web application, this is generally handled per request. In your case, you should find what that should be. If yours is windows service that does some activity after specified time then Timer_Tick event is good place.
I cannot suggest what that location could be in your application; you need to find out on your own.
Edit 1
Looking at your edit and comment, I do not think this has anything to do with NHibernate. May be that the connection pool is returning a disconnected/stale connection to NHibernate.
Refer this and this accepted answer.
So I have the following trivial code in a WebAPI controller that is published to an Azure App Service website.
using (var tx = new TransactionScope())
{
var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["OrganizationManagement"].ConnectionString);
connection.Open();
return Enumerable.Empty<TimeSessionDTO>();
}
100% of the time this is giving me a transport error exception on the Open call:
A transport-level error has occurred when receiving results from the
server. (provider: TCP Provider, error: 0 - An existing connection was
forcibly closed by the remote host.)
I have tried using ReliablSqlConnection from the Transient Fault Handling EL block using an exponential retry policy, and I just end up with a transaction timeout with that.
If I remove the surrounding TransactionScope, it works and does not throw an exception.
If I run the same code on my local machine with the connection string still pointing to the SQL Azure database, it works fine with the TransactionScope.
What could be going on that I cannot open a database connection inside of a transaction, in an Azure website?
Update: I should also note that using an Entity Framework DbContext inside of a TransactionScope was working fine. It's just choking on plain ADO.NET for some reason.
FYI I also tried it on a new MVC application on Azure, with the same result. I just don't get it :)
Wow, so the problem seems to have been with the connection string. When I first deployed the database I let the database project build the connection string from the server/database/user info, which I later added to the Web.config file in the WebAPI project. Then when I deployed the WebAPI project I guess it saved that connection string in the publish profile.
It turns out that connection string used a little different format and different options than what is supplied when you view the connection strings in the Azure portal. I had already changed it in the Web.config file, but it seems that what is in the publish profile overwrites what is in Web.config, so the change never took effect on the server.
I guess that explains why it worked when I ran it locally, but I have no idea why it only failed when it was in a transaction.
In recent times, a particular page in my web app throws the
Exception Details: MySql.Data.MySqlClient.MySqlException: Timeout
expired. The timeout period elapsed prior to completion of the
operation or the server is not responding.
Though I use Ibtais as persistence layer, this error occurs. I have restarted the MySql service instance but stil i get the same error. It didn't happen earlier but happens frequently in recent times.
All the web applications deployed on the server uses Ibatis and the DB server remains on the same machine where IIS is installed. There are about 8000 records in which around 300 to 500 would be filtered on page load
Any insights for the cause of the problem?
I encountered the same problem with yours, and I found this MySQLConnection--Specifying default command timeout. Just add "default command timeout=xxx" into your connectString, this key's value is in seconds.
I tried and it worked for me.
You could set command timeout to 0, its not a good idea though. Some requests could go on indefinitely.
There is an underlying problem that is causing the queries to time out in the first place. Are you inserting, updating, or in any way working with large binary values that would lock the table? That is the most common reason I see for an error like this on such a small amount of data.
Make your connection string look like this:
DotNet Framework:
<add key="MYSQL_CONNECTION_STRING_RDS" value="Uid=userid;Password=pass;
Server=localhost;Port=3306;
Database=dbname;default command timeout=0;SslMode=none"
/>
DotNet Core
"MYSQL_CONNECTION_STRING_RDS": {
"ConnectionString": "Server=localhost;Database=dbname;user=userid;password=pass;default command timeout=0;",
"ServerVersion": "8.0.22"
}