We currently have a little situation on our hands - it seems that someone, somewhere forgot to close the connection in code. Result is that the pool of connections is relatively quickly exhausted. As a temporary patch we added Max Pool Size = 500; to our connection string on web service, and recycle pool when all connections are spent, until we figure this out.
So far we have done this:
SELECT SPId
FROM MASTER..SysProcesses
WHERE DBId = DB_ID('MyDb') and last_batch < DATEADD(MINUTE, -15, GETDATE())
to get SPID's that aren't used for 15 minutes. We're now trying to get the query that was executed last using that SPID with:
DBCC INPUTBUFFER(61)
but the queries displayed are various, meaning either something on base level regarding connection manipulation was broken, or our deduction is erroneous...
Is there an error in our thinking here? Does the DBCC / sysprocesses give results we're expecting or is there some side-effect catch? (for example, connections in pool influence?)
(please, stick to what we could find out using SQL since the guys that did the code are many and not all present right now)
I would expect that there is a myriad of different queries 'remembered' by inputbuffer - depending on the timing of your failure and the variety of queries you run, it seems unlikely that you'd see consistent queries in this way. Recall that the connections will eventually be closed, but only when they're GC'd and finalized.
As Mitch suggests, you need to scour your source for connection-opens and ensure they're localized and wrapped in a using(). Also look for possibly-long-lived objects that might be holding on to connections. In an early version of our catalog ASP page objects held connections that weren't managed properly.
To narrow it down, can you monitor connection-counts (perfmon) as you focus on specific portions of your app? Does it happen more in CRUD areas vs. reporting or other queries? That might help narrow down the source-scour you need to do.
Are you able to change the connection strings to contain information about where and why the connection was created in the Application field?
Related
I recently monitored my sql database activity I found about 400 processes in activity monitoring.Later I figured that the problem is with my connection string object which would not be cleared physically even though I completely closed and disposed it, so once I suspend my IIS all the processes from activity monitoring would disappear.
after a little searching I found that I can clean all of my connections from application pool so that all the useless processes from SMSS would be killed but
I'm really concerned about it's impact on webserver. It's true that this approach would clear useless tasks from SMSS but for every request a new connection should really be created is it worth it???
considering my application is kind of enterprise app which is supposed to handle to many requests, I'm so afraid of making IIS server down by using this approach.
Do notice that my connection string value is not completely fixed for all the requeests, I made it variable by changing only "Application Name" section of it in every request according to the request parameters for the purpose of getting requestors information in sql activity monitoring and sql profiler.
is it worth to do so considering my business scope or it's better I fix the connection string value in other word is performance lag on this approach is so severe that I have to change my logging strategy or it's just a little slower???
Do notice that my connection string value is not completely fixed for all the requeests, I made it variable by changing only "Application Name" section of it in every request according to the request parameters for the purpose of getting requestors information in sql activity monitoring and sql profiler.
This is really bad because it kills pooling. You might as well disable pooling but that comes with a heavy performance penalty (which you are paying right now already).
Don't do that. Obtain monitoring information in a different way.
Besides that, neither SQL Server nor .NET have a problem with 400 connections. That's unusually high but will not cause problems.
If you run multiple instances of the app (e.g. for HA) this will multiply. The limit is 30k. I'm not aware of any reasons why this would cause a slowdown for the app, but it might cause problems for your monitoring tools.
For the past two days I've been trying to resolve the following error:
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
As far as I have checked all of our DbContext requests are wrapped with using, and still it looks like it doesn't get disposed or something else.
Is there a way to determine if the web app is leaking connection? Ir something else I can try?!
I have tried to increase timeouts in SQL Server + increment the pool size from default value of 100 to 200 as a temporary result but it didn't helped, and that's not the solutions I want.
Any suggestions would help.
I think I have a case similar to yours. Our old project version (still using EF4) leaks connections when my code does NOT touch the context. The context is created, a new connection allocated, but on Dispose() the context does not seem to return its connection.
If I just perform one little query (.First() on any random table) on that underutilized context, the situation improves.
A more recent branch of my project uses EF6. That code works fine.
What helped me track this down was SQL Profiler. I also knew roughly what my problem user was doing, so I repeated his steps and kept an eye on Profiler's SPID column. Then I stepped through parts of the code until I got one of the ghost connections I was chasing.
(I stumbled across this just now, so I have not had time to investigate further)
Currently there is discussion as to what are the pros and cons of having a single sql connection architecture.
To elaborate what we are discussing is, at application creation open a sql connection and at application close or error closing the sql connection. And not creating another connection at all, but using just that one to talk with the DB.
We are wondering what the community thinks.
Close the connection as soon as you do not longer need it for an undefined amount of time. By doing so, the connection returns to the connection-pool (if connection pooling is enabled), and can be (re)used by someone else.
(Connections are expensive resources, and are sometimes limited).
If you keep hold on a connection for the entire lifetime of an application, and you have multiple users for that application (thus multiple instances of the app, and multiple connections), and if your DB server is limited to have only x number of concurrent connections, then you could have a problem ....
See also best practices for ado.net
Follow this simple rule... Open connection as late as possible and close it as soon as possible.
I think it's a bad idea, for several reasons.
If you have 10,000 users using your application, that's 10,000 connections open constantly
If you have to restart your Sql Server, all those 10,000 connections are invalidated and your application will suddenly - assuming you've included reconnect logic - be making 10000 near-simultaneous re-connect requests.
To expand on point 1, you should close connections as soon as you can because otherwise you're using up a finite resource for, potentially, an inifinite period of time. If you had Sql Server configured to allow a maximum of 10,001 simultaneous connections, then you can only have 10,001 users running your application at any one time. If you open/close connections on demand then your application will scale much further as the likelihood of all the active users making use of the database simultaneously is, realistically, low.
Under the covers, ADO.NET uses connection pooling to manage the connections to the database. I would suggest leaving it up to the connection pool to take care of your connection needs. Keeping a connection open for the duration of your application is a bad idea.
I use a helpdesk system called Richmond Systems that uses one connection for the life of the application, and as a laptop user, it is a royal pain in the behind. Even when I carry my laptop around open, the jumps between the wireless access points are enough to drop the DB conenction. The software then complains about the DB conenction, gets into an error state and won't close. It has to be killed manually from Task Manager.
In short, DON'T HOLD OPEN A DATABASE CONNECTION FOR LONGER THAN NECESSARY.
But on the flip side, I'd be cautious about opening and closing connections too often. This is a lot cheaper with connection pooling than without, but even with pooling, the pool manager may decide to grow or shrink the pool, turning it back into an expensive operation.
My general rule is to open a connection when the user initiates some action, do the work, then close the connection before waiting for the next user input. For any given "Update" button click or whatever, I'll generally have only one connection. But you definately do not want to keep connections open while waiting for user input if you can at all help it for all the reasons others have mentioned. You could literally wait for days before the user presses another key or touches another button -- what if he leaves his computer on and goes on vacation? Tying up a resource for unpredictable amounts of time like that is bad news. In most cases, the elapsed time waiting for user input will far exceed the time doing actual work.
I'm running a data import (using C#/Linq), and naturally I'm trying to optimize my queries as much as possible. To this end I'm running a trace on the DB using SQL Server Profiler, with my trace filtered by my SQL login name (it's a name that can uniquely be attributed to my data import process).
Strangely enough, most of my SQL statements are really quick :) - very few queries even break over the 1ms mark. But spaced in between all my queries are several rows where the EventClass is "Audit Login" or "Audit Logout" - and the duration of an "Audit Logout" can be up to a minute!
Has this got something to do with the fact that I'm using transactions in my import? If so, is there any way to find which are the big-hitting queries so I can clean those up?
If I remember correct, the duration of an Audit Logout is the amount of time the connection was open. E.g. nothing to do with speed of the command - just the amount of time the login was 'logged in'.
Login/Logout events are related to the setting up / tearing down. IIRC the time is the 'was logged in for time' as opposed to a processing duration as with other log events.
In general, one hides these events unless you suspect there's an issue with connection pool management etc.
The raw times for the batches should be sufficient to diagnose the time the actual activity is taking including the impact of any transactions etc.
The Audit Logout event class indicates that a user has logged out of (logged off) Microsoft SQL Server. Events in this class are fired by new connections or by connections that are reused from a connection pool.
it's the total time the connection was logged in for, including idle time, so it doesn't indicate a performance problem. Also profiling logins/logouts is very unlikely to cause a performance problem. You'd be better off looking for poorly performing queries, possibly long-running queries.
For more info i suggest https://msdn.microsoft.com/en-us/library/ms175827.aspx :)
Also worth noting as in this answer that Audit Login/Logout may just mean the connection is being reused from / returned to the connection pool.
You can use another field from the event to determine if it's a connection pool event or a 'real' login/logout.
We have a web service coded in C# that makes many calls to MS SQL Server 2005 database. The code uses Using blocks combined with C#'s connection pooling.
During a SQL trace, we saw many, many calls to "sp_resetconnection". Most of these are short < 0.5 sec, however sometimes we get calls lasting as much as 9 seconds.
From what I've read sp_resetconnection is related to connection pooling and basically resets the state of an open connection. My questions:
Why does an open connection need its state reset?
Why so many of these calls!
What could cause a call to sp_reset connection to take a non-trivial amount of time.
This is quite the mystery to me, and I appreciate any and all help!
The reset simply resets things so that you don't have to reconnect to reset them. It wipes the connection clean of things like SET or USE operations so each query has a clean slate.
The connection is still being reused. Here's an extensive list:
sp_reset_connection resets the following aspects of a connection:
It resets all error states and numbers (like ##error)
It stops all EC's (execution contexts) that are child threads of a parent EC executing a parallel query
It will wait for any outstanding I/O operations that is outstanding
It will free any held buffers on the server by the connection
It will unlock any buffer resources that are used by the connection
It will release all memory allocated owned by the connection
It will clear any work or temporary tables that are created by the connection
It will kill all global cursors owned by the connection
It will close any open SQL-XML handles that are open
It will delete any open SQL-XML related work tables
It will close all system tables
It will close all user tables
It will drop all temporary objects
It will abort open transactions
It will defect from a distributed transaction when enlisted
It will decrement the reference count for users in current database; which release shared database lock
It will free acquired locks
It will releases any handles that may have been acquired
It will reset all SET options to the default values
It will reset the ##rowcount value
It will reset the ##identity value
It will reset any session level trace options using dbcc traceon()
sp_reset_connection will NOT reset:
Security context, which is why connection pooling matches connections based on the exact connection string
If you entered an application role using sp_setapprole, since application roles can not be reverted
The transaction isolation level(!)
Here's an explanation of What does sp_reset_connection do? which says, in part "Data access API's layers like ODBC, OLE-DB and SqlClient call the (internal) stored procedure sp_reset_connection when re-using a connection from a connection pool. It does this to reset the state of the connection before it gets re-used." Then it gives some specifics of what that system sproc does. It's a good thing.
sp_resetconnection will get called everytime you request a new connection from a pool.
It has to do this since the pool cannot guarantee the user (you, the programmer probably :)
have left the connection in a proper state. e.g. Returning an old connection with uncommited transactions would be ..bad.
The nr of calls should be related to the nr of times you fetch a new connection.
As for some calls taking non-trivial amount of time, I'm not sure. Could be the server is just very busy processing other stuff at that time. Could be network delays.
Basically the calls are the clear out state information. If you have ANY open DataReaders it will take a LOT longer to occur. This is because your DataReaders are only holding a single row, but could pull more rows. They each have to be cleared as well before the reset can proceed. So make sure you have everything in using() statements and are not leaving things open in some of your statements.
How many total connections do you have running when this happens?
If you have a max of 5 and you hit all 5 then calling a reset will block - and it will appear to take a long time. It really is not, it is just blocked waiting on a pooled connection to become available.
Also if you are running on SQL Express you can get blocked due to threading requirements very easily (could also happen in full SQL Server, but much less likely).
What happens if you turn off connection pooling?