Basic Intro:
Connection string: Host=IP;Port=somePort;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=100
My web application has several dozen endpoints available via WS and HTTP. Every one of these endpoints opens a new NPGSQL connection (all using the same connection string as above), processes data, then closes via the using statement.
Issue:
When the application restarts for an update, there is typically 2-3,000 users all reconnecting. This typically leads to errors regarding the connection pool being full and new connections being rejected due to too many clients already. However, once it can finally come online it typically only uses between 5-10 connections at any given time.
Question:
Is the logic below the proper way to use connection pooling? With every endpoint creating a new NPGSQL connection using the same connection string specifying a connection pool of 100?
It seems that the connection pool often shoots right up to 100, but ~80/100 of those connections are shown as idle in a DB viewer with new connection requests being denied due to pool overflow.
Better option?
I could also try and force a more "graceful" startup by slowly allowing new users to re-connect, but I'm not sure if the logic for creating a new connection with every endpoint is correct.
// DB Connection String - Used for all NPGSQL connections
const string connectionStr "Host=IP;Port=somePort;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=100";
// Endpoint 1 available via Websocket
public async Task someRequest(someClass someArg)
{
/* Create a new SQL connection for this user's request using the same global connections string */
using var conn = new NpgsqlConnection(connectionStr);
conn.Open();
/* Call functions and pass this SQL connection for any queries to process this user request */
somefunction(conn, someArg);
anotherFunction(conn, someArg);
/* Request processing is done */
/* conn is closed automatically by the "using" statement above */
}
// Endpoint 2 available via Websocket
public async Task someOtherRequest(someClass someArg)
{
/* Create a new SQL connection for this user's request using the same global connections string */
using var conn = new NpgsqlConnection(connectionStr);
conn.Open();
/* Call functions and pass this SQL connection for any queries to process this user request */
somefunction(conn, someArg);
anotherFunction(conn, someArg);
/* Request processing is done */
/* conn is closed automatically by the "using" statement above */
}
// endpoint3();
// endpoint4();
// endpoint5();
// endpoint6();
// etc.
EDIT:
I've made the change suggested, by closing connections and sending them back to the pool during processing. However, the issue still persists on startup.
Application startup - 100 connections claimed for pooling. Almost all of them are idle. Application receives connection pool exhaustion errors, little to no transactions are even processed.
Transactions suddenly start churning, not sure why? Is this after some sort of timeout perhaps? I know there was some sort of 300 second timeout default in documentation somewhere... this might match up here.
Transactions lock up again, pool exhaustion errors resume.
Transactions spike and resume, user requests start coming through again.
Application levels out as normal.
EDIT 2:
This startup issue seems to consistently be taking 5 minutes from startup to clear a deadlock of idle transactions and start running all of the queries.
I know 5 minutes is the default value for idle_in_transaction_session_timeout. However, I tried running SET SESSION idle_in_transaction_session_timeout = '30s'; and 10s during the startup this time and it didn't seem to impact it at all.
I'm not sure why those 100 pooled connections would be stuck in idle like that on startup, taking 5 minutes to clear and allow queries to run if that's the case...
I had forgotten to update this post with some of the latest information. There was a few other internal optimizations I had made in the code.
One of the major ones, was simply changing conn.Open(); to await conn.OpenAsync(); and conn.Close(); to conn.CloseAsync();.
Everything else I had was properly async, but there was still IO blocking for all of the new connections in NPGSQL, causing worse performance with large bursts.
A very obvious change, but I didn't even think to look for an async method for the connections opening and closing.
A connection is released to the pool once you close it in your code. From what you wrote, you are keeping it open for the entire time of a request, so basically 1 user = 1 connection and pooling is just used as a waiting room (timeout setting, 15 seconds by default). Open/Close the connection each time you need to access the DB, so the connection is returned to the pool and can be used by another user when time is spent in .net code.
Example, in pseudo code:
Enter function
Do some computations in .net, like input validation
Open connection (grab it from the pool)
Fetch info#1
Close connection (return it to the pool)
Do some computations in .net, like ordering the result, computing an age from a date etc
Open connection (grab it from the pool)
Fetch info #2
Close connection (return it to the pool)
Do some computations in .net
Return
Related
How can I create a server side script to ping clients on mysql and windows form to see if they are still logged in?
For example ping a client and they have to do something in return to verify that they are still online.
You can't directly ping client from server, but there are some options.
About timeouts
I think in any case you have to use server side timeout (wait_timeout), this timeout can be set server wide or upon connecting from the client, ex:
SET SESSION wait_timeout = 60
This is required because client can suddenly disappear without even closing tcp connection and explicit timeout will help mysql to close connection and free resources after wait_timeout seconds of client inactivity. According to mysql manual default timeout is rather large: 28800 seconds.
There is a drawback. If during normal operation your client may be inactive for more than wait_timeout seconds then either the client should know how to deal with closed connection (reconnect when database tells it has gone away) or it should send "ping" queries (like select 1) at least every wait_timeout - 1 seconds.
Using get_lock() function
Since mysql 5.7(and also in mariadb since 10.0.2) you can use multiple user-level locks.
A lock obtained with GET_LOCK() is released explicitly by executing RELEASE_LOCK() or implicitly when your session terminates (either normally or abnormally). Locks obtained with GET_LOCK() are not released when transactions commit or roll back.
So the idea is to issue get_lock query upon client connection, ex:
SELECT GET_LOCK('logged_in_{CLIENT_ID}', timeout)
You can set timeout to 0 and immediately tell that client cannot login, or you can wait (blocking) at most wait_timeout seconds to be sure that there is really another client holding the lock.
This lock will be released automatically by server when client disconnects or after wait_timeout of inactivity.
If lock is free get_lock() will return 1 otherwise (after waiting timeout seconds) will return 0
Using process id
If you don't want to use locks, PID of process can be used.
When client connects instead of writing islogged = 'YES' you can use current CONNECTION_ID() as a value.
Before logging in you can check that there is no active process for current client like this
SELECT islogged FROM logged
INNER JOIN information_schema.processlist ON
processlist.id = logged.islogged
WHERE
client_id = ...
And if above query returns nothing you then may upsert new pid into logged table
REPLACE INTO logged SET islogged = CONNECTION_ID(), client_id = ...
I would prefer to use get_lock() because it seems easier, does not suffer from concurrency issues and allows to implement waiting.
Don't forget that timeouts are essential and you have to deal with reconnection or send regular pings to avoid unexpected "server has gone away" issues in client.
This question already has answers here:
is it safe to keep database connections open for long time
(9 answers)
Closed 4 years ago.
Every connection needs to be opened and closed. But what time is the best or it depends on the situation?
Open connection when initiate a windows(WPF), form(Windows form), webpage(ASP.NET) and close when the user close window/form/webpage.
Open connection only when user calls the database. For example the user click "Add" button and we run the SQL "INSERT..." then close the connection ASAP. Then if the user click Add/Edit again we have to reconnect to the database, run the SQL and close it ASAP again?
Both solutions have their advantages. By keeping the connection, we will make the program more performance. By keep connecting and disconnecting, the performance will drop.
The correct answer is wrap them in a using for the query or queries you want to use them for. Don't hold on to connections for longer then you immediately need. Doing so creates more problems then its worth.
In essence if you need a connection for a bunch of queries just wrap them in a using statement, if they are separated by long running tasks, close them and open them on a piecemeal basis. The worst thing you can so is try to keep them open and check for if they are still alive.
SQL Server Connection Pooling (ADO.NET)
In practice, most applications use only one or a few different
configurations for connections. This means that during application
execution, many identical connections will be repeatedly opened and
closed. To minimize the cost of opening connections, ADO.NET uses an
optimization technique called connection pooling.
Furthermore
Connection pooling reduces the number of times that new connections
must be opened. The pooler maintains ownership of the physical
connection. It manages connections by keeping alive a set of active
connections for each given connection configuration. Whenever a user
calls Open on a connection, the pooler looks for an available
connection in the pool. If a pooled connection is available, it
returns it to the caller instead of opening a new connection. When the
application calls Close on the connection, the pooler returns it to
the pooled set of active connections instead of closing it. Once the
connection is returned to the pool, it is ready to be reused on the
next Open call.
In the following example, three new SqlConnection objects are created, but only two connection pools are required to manage them.
Example from MSDN
using (SqlConnection connection = new SqlConnection("Integrated Security=SSPI;Initial Catalog=Northwind"))
{
connection.Open();
// Pool A is created.
}
using (SqlConnection connection = new SqlConnection("Integrated Security=SSPI;Initial Catalog=pubs"))
{
connection.Open();
// Pool B is created because the connection strings differ.
}
using (SqlConnection connection = new SqlConnection("Integrated Security=SSPI;Initial Catalog=Northwind"))
{
connection.Open();
// The connection string matches pool A.
}
For Controllers WCF services, and CQRS, they are usually short lived, so injecting them in a scoped life cycle is very common. However for things like button clicks in user applications, Service patterns calls, just use them when you need them. Never try to cache or pool.
ASP.NET / Web API / (...)
Usually tied to the unit of work pattern, a connection is created for an incoming request then closed when the request has been processed
deviations may occur when you have long running background tasks
WPF / Window Forms
In this case I would suggest that it's more on a per action basis, even when polling for changes - if that scenario exists
Open, Query, Close, repeat
It is better to have a separate project which we call the data layer. The project can have different files for different modules (recommended) or all in one file as you wish. The database layer will expose the various methods like insert, update, get, delete, etc. It is always better to open the connection only for the particular method call and close it as soon as you get the result. You can use the using block like:
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
//your code here insert/update/get/delete
}
The connection will get disposed automatically when the control is out of the using block.
Open the connection and do your operation such as Inserting, Update or Delete and as you get the response close the connection.
to every task open and close connection do not leave connection open in any case
Currently I'm experiencing alot of issues with Server Processes (Seen from sp_who2) that is "sleeping" instead of just, finishing (being removed), when I connection to my MSSQL Database, calls a Stored Procedure, get some data and then closing the connection.
What's the best way in C#/.NET to connect to a MSSQL database, call a Stored Procedure, retrieve data from the Stored Procedure (Data Reader) and then close the connection?
Is there a way to close/dispose so that the "sleeping" processes gets killed?
Does it have something to do with me creating new SqlConnections, opening them and closing them all the time?
My flow is as follows:
A request occur:
I create a new SqlConnection Instance that connects to my MSSQL Database.
I call a Stored Procedure, retrieve the data and presents it to the user.
I close the Connection with .Close();
I repeat all these steps for each request. Requests happen once every 5-10 seconds. (Sometimes slower, sometimes faster)
I'm sorry If this question is missing some details, but I hope this is enough to get a some what helpful answer.
Thanks!
you need to use SET XACT_ABORT ON or add some client rollback code
When a client timeout event occurs (.net CommandTimeout for example), the client sends an "ABORT" to SQL Server. SQL Server then simply abandons the query processing. No transaction is rolled back, no locks are released.
Now, the connection is returned to the connection pool, so it isn't closed on SQL Server. If this ever happens (via KILL or client reboot etc) then the transactions+locks will be cleared. Note that sp_reset_connection won't or doesn't clear them, even though it is advertised to do so
This detritus from the abort will block other processes.
The way to make SQL Server clear transactions+locks on client timeout (strictly, ABORT events) is to use SET XACT_ABORT ON.
You can verify this be opening 2 query windows in SSMS:
Window 1:
In menu Query..Query Options set a timeout of 5 seconds then run this
BEGIN TRAN
UPDATE sometable WITH (TABLOCKX) SET foo = foo WHERE 1 = 0;
WAITFOR DELAY '00:00:10' -- just has to be longer then timeout
Window 2, this will wait forever (or hit your timeout)
SELECT * FROM sometable
My program execution hangs on .Open() method longer than specified 2 seconds timeout when connecting to SQL-Server.
I managed to reproduce the problem when machine that database is on is offline or restartig. When machine that database is on get up the execution of program un-hangs.
How can I force to throw exception or whatever when 2 second timeout exceed?
I tried OleDB and SqlConnection provider, no difference.
I can ping the machine before connecting to it but there is still case that ping will be successful (let's say 1 second before machine shutdown) and then Open connection hangs...
Example code provided below.
public static IDbConnection GetConnection(IDbConnection connection)
{
connection.ConnectionString = "connectionString; Connection Timeout=2;";
connection.Open();
return connection;
}
Connection Timeout property is just the time for the connection to be created, everything after that (and there is a lot to do after connection is established) does not count and it may take indefinitely (unless there is another timeout I'm not aware of).
What you can do is to execute your own code in a separate thread with a watchdog to limit total time execution to two seconds. Using tasks it is pretty easy:
const int HardConnectionTimeoutInMilliseconds = 2000;
if (!Task.Run(() => connection.Open()).Wait(HardConnectionTimeoutInMilliseconds))
return null; // Timeout!
Just for completeness this is old-style code for this:
Thread worker = new Thread(delegate()
{
connection.Open();
});
worker.Start();
if (!worker.Join(TimeSpan.FromSeconds(2)))
return null;
Be careful with so short timeout: for a TCP connection two seconds are always a too short time and if you're using Windows authentication with AD then it may takes longer than you expect.
In my opinion you have to live with this lag (15 to 30 seconds are a safe and reasonable time for TCP connection with integrated security). You may also want to wait more and retry (because errors may be temporary, see Know when to retry or fail when calling SQL Server from C#?), note that situation you're describing (server is going down) is pretty unusual then IMO it shouldn't affect normal operations. If it's an issue for your UI then you should make your program parallel (to keep UI responsive).
My WCF service is keeping DB connections for futher sending SQL through they. Sometimes connection become broken for various reasons. Early there was special timer that checks connections every 1 minute. But it's not so good solution of problem. Could you please advice me some way to keep connections working properly or even though reconnect as soon as possible to deliver user stable service.
Thanks!
EDIT:
Database server is Oracle. I'm connecting to databse server using devart dotConnect for Oracle.
You don't have to "keep" database connections. Leave the reuse and caching of database connections to the .net framework.
Just use this kind of code and dispose the connection as soon as you are finished using it:
using(var connection = new SqlConnection(...))
{
//Your Code here
}
There is no problem in executing the code above for each call to the database. The connection information is cached and the second "new" connection to the database is very fast.
To read more about "ConnectionPooling" you might read this MSDN Articel.
Edit:
If you use pooling the connection is not really close but put back to the pool. The initial "handshake" between the client and the database is only done once per connection on the pool.
The component you are using supports the connection pooling as well:
Read1
Read 2