We have a web page which uses ODP.Net to connect to oracle database from ASP MVC page. During the heavy user load if application pool recycles for any reason, than we get few connection timeout errors from internal oracle class.
Inner Exception: StackTrace: at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPwd, Boolean bGetForApp, String affinityInstanceName, Boolean bForceMatch)
How can we handle this scenario ? what could be the reason for this error ?
If you are using a new connection string that differs per user or with some other variable parameter you can exceed your max connections. Pooling only works if you use the exact same string. Under the covers your "close" is ignored and the connection manager keeps the connection open. In SQL Server you can see a connection reset in the trace console every time an existing connection is reused.
Oracle probably has something similar.
Tracing your connections and possibly connection resets will most likely shed some light on what is creating so many connections.
Related
Im using Oracle12c with the application written in C# and using Oracle.ManagedDataAccess.dll to handle the DB Connection.
A product we have has started to occasionally throw this exception after running fine for years:
Oracle.ManagedDataAccess.Client.OracleException (0xFFFFFC0C): Pooled connection request timed out
at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPwd, Boolean bGetForApp, String affinityInstanceName, Boolean bForceMatch)
at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassword, Boolean bGetForApp, String affinityInstanceName, Boolean bForceMatch)
at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, SecureString securedPassword, SecureString securedProxyPassword)
at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
I know the cause of this error. Looking at the code neither the OracleConnection or the OracleCommand objects are being disposed. So these connections are building up until it eventually throws this exception.
The fix is straight forward. Wrap these in Using statements. I dont need help with that.
However my interests me is why this problem has started now. This software was running for years without issue. They did some database maintenance, updated other software on the same server, then this problem started. I dont know what Db maintenance they did.
The connection string in the application does not specify any pool attributes.
Is there a oracle db setting which would cause a lower amount of simultaneous connections in the database that could have caused this to start occurring?
UPATE
I wrote a little test app to check the limit.
It just loops around and opens a connection, performs a basic query and doesnt dispose the connection.
On my test system it starts throwing this exception after 640ish loops. It varies give or take 10 loops each time i run it.
What is setting this limit?
I just had the same problem.
The reason, why you get that exception is, that the Oracle Pool Manager doesn't have a free connection anymore (per default you can have up to 100 connections). Often the reason for that problem are not closed connections (so 'using' was the right ways).
Even though you already found a solid solutions for that problem, I want to add those:
Try/catch and using is always a good idea
Adjust the Min Pool Size (or Max Pool Size, but default is 100 should be enough): So there's always a connection ready to use and the connection doesn't have to get established (see https://docs.oracle.com/en/database/oracle/oracle-database/21/jjucp/optimizing-ucp-behavior.html#GUID-FFCAB66D-45B3-4D7B-991B-40F1480630FD)
Update to Oracle.ManagedDataAcess >= 21.8.0 (
Bug 34322469 - CONNECTION POOL THROWS "CONNECTION REQUEST TIMED OUT" EXCEPTION DUE TO LOOPING WITHIN POOLMANAGER.GET() is fixed there)
I have a long running application that uses NHibernate.ISessionFactory to connect to an Oracle database.
Occasionally the database goes offline (e.g. for weekend maintenance), but even once the database is back online, subsequent queries fail with the following exception (inner exceptions also shown):
NHibernate.Exceptions.GenericADOException: could not execute query
[ select .....]
>> Oracle.ManagedDataAccess.Client.OracleException: ORA-03135: Connection lost contact
>> OracleInternal.Network.NetworkException: ORA-03135: Connection lost contact
>> System.Net.Sockets.SocketException: An established connection
was aborted by the software in your host machine
Restarting the application restores the functionality, but I would like the application to be able to automatically cope without a restart, by "resetting" the connection.
I have tried the following with my ISessionFactory when I hit this exception:
sf.EvictQueries();
sf.Close();
sf = null;
sf = <create new session factory>
but see the same exception after recreating the ISessionFactory. I assume this is because NHibernate is caching the underlying broken connection in some kind of connection pool?
How can I persuade NHibernate to create a genuinely new connection (or even just reset all state completely), and hence allow my application to fix the connection issue itself without an application restart?
EDIT:
Following A_J's answer, note that I am already calling using (var session = _sessionFactory.OpenSession()) for each database request.
I suspect you are opening ISession (call to ISessionFactory.OpenSession()) at startup and closing it at application end. This is wrong approach for any long running application.
You should manage connection at lower level of time. In web application, this is generally handled per request. In your case, you should find what that should be. If yours is windows service that does some activity after specified time then Timer_Tick event is good place.
I cannot suggest what that location could be in your application; you need to find out on your own.
Edit 1
Looking at your edit and comment, I do not think this has anything to do with NHibernate. May be that the connection pool is returning a disconnected/stale connection to NHibernate.
Refer this and this accepted answer.
So, I am using MongoDB on a Azure VM and I have a web site hosted on Azure web Sites as a service.
My problem is: Sometimes I get an error like this:
"Exception: MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
After I got this error, just I access the endpoint again and I have success immediately.
Details:
ConnectionString: mongodb://xxx.aaa.net:1000, xxx.aaa.net:1001, xxx.aaa.net:1002/?readPreference=nearest
Before open the conection with Mongo I set the MaxConnectioIdleTimeout property like this "MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromSeconds(30);" , To keep alive connection, because the idle timeout for azure LoadBalancer become inactivy after 4 minutes by default.
So, I don't know what's going on.
Can anybody help me?
Yes, 4 minutes. Azure closes a connection session if it keeping on idling up to 4 minutes.But the mongodb driver don't know it，it will still get this connection form connection pool.
You can either set the MaxConnectionIdleTime(in mongodb driver settings) less than 4 minutes, or set the max session idle time of azure VM more larger by powershell (30 minutes max).
BTW, if your web site which also hosted on azure access the mongodb server with the private IP(PIP), there won't be this problem.
As I asked #apodemakels for some official documentation, I have found MongoDB Production Notes where it states:
The TCP idle timeout on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.
In addition, there's a ticket on MongoDB Jira with regard to this issue.
I hope both these documents may help with this or in case someone faces a similar situation.
The question has been asked before here in StackOverflow, but in my experience, the answers were actually wrong. At least for .NET Framework 4.0 and SQL Server 2005 they are wrong.
I would need help to sort this out once and for all.
The question is - can a stateless WCF service use database connection pooling in some way?
See Can a Stateless WCF service ...
The earlier answers essentially stated that there is no problem and no difference to any other ADO.NET scenarios. However, I have not been able to get a stateless WCF service to use the connection pooling EVER, while I can see it always work outside WCF services. No matter what connection strings or parameters I am trying to use, it does not do it.
Database connection pooling is meant to be enabled by default, so a simple connection string should get me there, for instance on SQL Server Express:
SqlConnection sqlCn = new SqlConnection("Data Source=SERVER\SQLEXPRESS; Initial Catalog = xDB; Integrated Security = SSPI;")
Using this connection, in a Windows Form application, if I do 3 consecutive rounds of sqlCn.Open() -- query the database -- sqlCn.Close(), I am getting a long delay (for instance 2 seconds) on the first sqlCn.Open(), and no delays at all on queries and open / close afterwards. Exactly what I expect with database connection pooling.
But if I make 3 calls to a WCF service containing the same sqlCn.Open() -- query the database -- sqlCn.Close() code, I am getting the 2 second initial slow startup for EVERY single call.
My guess is that the connection pooling is entirely controlled by the ADO.NET objects created by my code, and since I am instantiating any ADO.NET classes I use (such as SqlConnection etc) inside my WCF service, they get destroyed when my service call is over and the connection pool along with it.
This may not be true, but if not, is there anything wrong with what I have done?
Anyone have any experience with that?
(Please do test any assumption or theory before posting)
1) Here's the documentation:
http://msdn.microsoft.com/en-us/library/8xx3tyca.aspx
When a connection is first opened, a connection pool is created based
on an exact matching algorithm that associates the pool with the
connection string in the connection. Each connection pool is
associated with a distinct connection string. When a new connection is
opened, if the connection string is not an exact match to an existing
pool, a new pool is created. Connections are pooled per process, per
application domain, per connection string and when integrated security
is used, per Windows identity. Connection strings must also be an
exact match; keywords supplied in a different order for the same
connection will be pooled separately.
2) Per the same link, "By default, connection pooling is enabled in ADO.NET."
3) This is completely independent of whether the WCF call in question is stateless or not.
4) Finally:
We strongly recommend that you always close the connection when you
are finished using it so that the connection will be returned to the
pool. You can do this using either the Close or Dispose methods of the
Connection object, or by opening all connections inside a using
statement in C#, or a Using statement in Visual Basic. Connections
that are not explicitly closed might not be added or returned to the
pool.
I managed to resolve it myself.
I had to explicitly state "Pooling = true" (and add a non-zero "Min Pool Size") to my connection string. Then it was working consistently. If this was not set, it would sometimes factually work as expected, but mostly not.
I tested it also with different user accounts (SQL Server authentication with user name / password versus "Integrated Security = SSPI"). Both approaches work for a WCF service as long as you set "Pooling = true".
No data if this is a problem only for my installation / SQL Server version / ADO.NET version but it sure did take quite a while to resolve.
Using C# and Oracle Data Provider for .NET (ODP) I made a long query to the database, then I end the connection on the server side using TOAD. After that, the subsequent calls to the database, even creating a new OracleConnection object, throw the following error:
ORA-01012: not logged on
Process ID: xxx
Session ID: yyy Serial number: zzz
Where Process ID and Session ID are the identifiers I used to end the connection.
It seems like when I end the connection to the Oracle database on the server side, the broken connection is returned to the connection pool. And when the C# client code (using ODP) opens a new connection, the broken connection that was returned to the connection pool may be retrieved.
Any ideas on how to fix this behaviour?
BTW I'm using Oracle client 10
I solved my problem by setting to true the "Validate Connection" property in the connection string.
you can read more here
As a warning I quote the Oracle docs.
The Validate Connection attribute validates connections coming out of
the pool. This attribute should only be used when absolutely necessary
because it causes a server round-trip to the database to validate each
connection right before it is provided to the application. If invalid
connections are uncommon, developers can create their own event
handler to retrieve a new connection, rather than using Validate
Connection. This generally provides better performance.