We Keep Coding

How can i ping mysql clients?

How can I create a server side script to ping clients on mysql and windows form to see if they are still logged in?
For example ping a client and they have to do something in return to verify that they are still online.

You can't directly ping client from server, but there are some options.
About timeouts
I think in any case you have to use server side timeout (wait_timeout), this timeout can be set server wide or upon connecting from the client, ex:
SET SESSION wait_timeout = 60
This is required because client can suddenly disappear without even closing tcp connection and explicit timeout will help mysql to close connection and free resources after wait_timeout seconds of client inactivity. According to mysql manual default timeout is rather large: 28800 seconds.
There is a drawback. If during normal operation your client may be inactive for more than wait_timeout seconds then either the client should know how to deal with closed connection (reconnect when database tells it has gone away) or it should send "ping" queries (like select 1) at least every wait_timeout - 1 seconds.
Using get_lock() function
Since mysql 5.7(and also in mariadb since 10.0.2) you can use multiple user-level locks.
A lock obtained with GET_LOCK() is released explicitly by executing RELEASE_LOCK() or implicitly when your session terminates (either normally or abnormally). Locks obtained with GET_LOCK() are not released when transactions commit or roll back.
So the idea is to issue get_lock query upon client connection, ex:
SELECT GET_LOCK('logged_in_{CLIENT_ID}', timeout)
You can set timeout to 0 and immediately tell that client cannot login, or you can wait (blocking) at most wait_timeout seconds to be sure that there is really another client holding the lock.
This lock will be released automatically by server when client disconnects or after wait_timeout of inactivity.
If lock is free get_lock() will return 1 otherwise (after waiting timeout seconds) will return 0
Using process id
If you don't want to use locks, PID of process can be used.
When client connects instead of writing islogged = 'YES' you can use current CONNECTION_ID() as a value.
Before logging in you can check that there is no active process for current client like this
SELECT islogged FROM logged
INNER JOIN information_schema.processlist ON
processlist.id = logged.islogged
WHERE
client_id = ...
And if above query returns nothing you then may upsert new pid into logged table
REPLACE INTO logged SET islogged = CONNECTION_ID(), client_id = ...
I would prefer to use get_lock() because it seems easier, does not suffer from concurrency issues and allows to implement waiting.
Don't forget that timeouts are essential and you have to deal with reconnection or send regular pings to avoid unexpected "server has gone away" issues in client.

Postgresql and .Net - Connection Pooling with Multiple Endpoints

Basic Intro:
Connection string: Host=IP;Port=somePort;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=100
My web application has several dozen endpoints available via WS and HTTP. Every one of these endpoints opens a new NPGSQL connection (all using the same connection string as above), processes data, then closes via the using statement.
Issue:
When the application restarts for an update, there is typically 2-3,000 users all reconnecting. This typically leads to errors regarding the connection pool being full and new connections being rejected due to too many clients already. However, once it can finally come online it typically only uses between 5-10 connections at any given time.
Question:
Is the logic below the proper way to use connection pooling? With every endpoint creating a new NPGSQL connection using the same connection string specifying a connection pool of 100?
It seems that the connection pool often shoots right up to 100, but ~80/100 of those connections are shown as idle in a DB viewer with new connection requests being denied due to pool overflow.
Better option?
I could also try and force a more "graceful" startup by slowly allowing new users to re-connect, but I'm not sure if the logic for creating a new connection with every endpoint is correct.
// DB Connection String - Used for all NPGSQL connections
const string connectionStr "Host=IP;Port=somePort;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=100";
// Endpoint 1 available via Websocket
public async Task someRequest(someClass someArg)
{
/* Create a new SQL connection for this user's request using the same global connections string */
using var conn = new NpgsqlConnection(connectionStr);
conn.Open();
/* Call functions and pass this SQL connection for any queries to process this user request */
somefunction(conn, someArg);
anotherFunction(conn, someArg);
/* Request processing is done */
/* conn is closed automatically by the "using" statement above */
}
// Endpoint 2 available via Websocket
public async Task someOtherRequest(someClass someArg)
{
/* Create a new SQL connection for this user's request using the same global connections string */
using var conn = new NpgsqlConnection(connectionStr);
conn.Open();
/* Call functions and pass this SQL connection for any queries to process this user request */
somefunction(conn, someArg);
anotherFunction(conn, someArg);
/* Request processing is done */
/* conn is closed automatically by the "using" statement above */
}
// endpoint3();
// endpoint4();
// endpoint5();
// endpoint6();
// etc.
EDIT:
I've made the change suggested, by closing connections and sending them back to the pool during processing. However, the issue still persists on startup.
Application startup - 100 connections claimed for pooling. Almost all of them are idle. Application receives connection pool exhaustion errors, little to no transactions are even processed.
Transactions suddenly start churning, not sure why? Is this after some sort of timeout perhaps? I know there was some sort of 300 second timeout default in documentation somewhere... this might match up here.
Transactions lock up again, pool exhaustion errors resume.
Transactions spike and resume, user requests start coming through again.
Application levels out as normal.
EDIT 2:
This startup issue seems to consistently be taking 5 minutes from startup to clear a deadlock of idle transactions and start running all of the queries.
I know 5 minutes is the default value for idle_in_transaction_session_timeout. However, I tried running SET SESSION idle_in_transaction_session_timeout = '30s'; and 10s during the startup this time and it didn't seem to impact it at all.
I'm not sure why those 100 pooled connections would be stuck in idle like that on startup, taking 5 minutes to clear and allow queries to run if that's the case...

I had forgotten to update this post with some of the latest information. There was a few other internal optimizations I had made in the code.
One of the major ones, was simply changing conn.Open(); to await conn.OpenAsync(); and conn.Close(); to conn.CloseAsync();.
Everything else I had was properly async, but there was still IO blocking for all of the new connections in NPGSQL, causing worse performance with large bursts.
A very obvious change, but I didn't even think to look for an async method for the connections opening and closing.

A connection is released to the pool once you close it in your code. From what you wrote, you are keeping it open for the entire time of a request, so basically 1 user = 1 connection and pooling is just used as a waiting room (timeout setting, 15 seconds by default). Open/Close the connection each time you need to access the DB, so the connection is returned to the pool and can be used by another user when time is spent in .net code.
Example, in pseudo code:
Enter function
Do some computations in .net, like input validation
Open connection (grab it from the pool)
Fetch info#1
Close connection (return it to the pool)
Do some computations in .net, like ordering the result, computing an age from a date etc
Open connection (grab it from the pool)
Fetch info #2
Close connection (return it to the pool)
Do some computations in .net
Return

LINQ to SQL timeout on one server and not other

I'm using LINQ to SQL to access a stored procedure. This works fine on my local sql server. But once I point to a different server within the internal network, I get a timeout. In SQL Profiler, I can see the query that is sent to the problem server. This should mean the query is hitting the server and executing right?
I can run the LINQ output directly on the problem server without issue. Is this probably not a code issue and instead some problem with the sql server server configuration? What should I check to troubleshoot it?

The connect timeout is different from the command timeout.
see: What is the difference between SqlCommand.CommandTimeout and SqlConnection.ConnectionTimeout?
The summary is connect timeout is the amount of time to establish a connection.
Command timeout is the amount of time to let the command run. Not just how long it takes to get the first record, but how long it takes to grab the entire dataset.
If you see the query showing up in SQL Profiler, then yes it is being sent. So you're essentially hitting the command timeout default limit of 30 seconds.
I would suggest starting with changing the command timeout to 500 and see where that takes you. Then I would start doing some performance analysis to determine why it takes that long. Maybe you can trim out some of the data requested or even add additional indexes to better support the query.

Handling DB connection breaks

My WCF service is keeping DB connections for futher sending SQL through they. Sometimes connection become broken for various reasons. Early there was special timer that checks connections every 1 minute. But it's not so good solution of problem. Could you please advice me some way to keep connections working properly or even though reconnect as soon as possible to deliver user stable service.
Thanks!
EDIT:
Database server is Oracle. I'm connecting to databse server using devart dotConnect for Oracle.

You don't have to "keep" database connections. Leave the reuse and caching of database connections to the .net framework.
Just use this kind of code and dispose the connection as soon as you are finished using it:
using(var connection = new SqlConnection(...))
{
//Your Code here
}
There is no problem in executing the code above for each call to the database. The connection information is cached and the second "new" connection to the database is very fast.
To read more about "ConnectionPooling" you might read this MSDN Articel.
Edit:
If you use pooling the connection is not really close but put back to the pool. The initial "handshake" between the client and the database is only done once per connection on the pool.
The component you are using supports the connection pooling as well:
Read1
Read 2

Check if a server is available

I'm looking for a way to check if a server is still available.
We have a offline application that saves data on the server, but if the serverconnection drops (it happens occasionally), we have to save the data to a local database instead of the online database.
So we need a continues check to see if the server is still available.
We are using C# for this application
The check on the sqlconnection.open is not really an option because this takes about 20 sec before an error is thrown, we can't wait this long + I'm using some http services as well.

Just use the System.Net.NetworkInformation.Ping class. If your server does not respond to ping (for some reason you decided to block ICMP Echo request) you'll have to invent your own service for this. Personally, I'm all for not blocking ICMP Echo requests, and I think this is the way to go. The ping command has been used for ages to check reachability of hosts.
using System.Net.NetworkInformation;
var ping = new Ping();
var reply = ping.Send("google.com", 60 * 1000); // 1 minute time out (in ms)
// or...
reply = ping.Send(new IPAddress(new byte[]{127,0,0,1}), 3000);

If the connection is as unreliable as you say, I would not use a seperate check, but make saving the data local part of the exception handling.
I mean if the connection fails and throws an exception, you switch strategies and save the data locally.
If you check first and the connection drops afterwards (when you actually save data), then you still would still run into an exception you need to handle. So the initial check was unnecessary. The check would only be useful if you can assume that after a succesfull check the connection is up and stays up.

From your question it appears the purpose of connecting to the server is to use its database. Your priority must be to check whether you can successfully connect to the database. It doesn't matter if you can PING the server or get an HTTP response (as suggested in other answers), your process will fail unless you successfully establish a connection to the database. You mention that checking a database connection takes too long, why don't you just change the Connection Timeout setting in your application's connection string to a more impatient value such as 5 seconds (Connection Timeout=5)?

If this is an sql server then you can just try to open a new connection to it. If the SqlConnection.Open method fails then you can check the error message to determine if the server is unavailable.

What you are doing now is:
use distant server
if distant server fails, resort to local cache
How to determine if the server is available? Use a catch block. That's the simplest to code.
If you actually have a local database (and not, for example, a list of transactions or data waiting to be inserted), I would turn the design around:
use the local database
regularly synchronize the local database and the distant database
I'll let you be the judge on concurrency constraints and other stuff related to your application to pick a solution.

Since you want to see if the database server is there either catch any errors when you attempt to connect to the database or use a socket and attempt a raw connection to the server on some service, I'd suggest the database as that is the resource you need.