I am a relative novice to MySql and I've run out of possible ideas so looking for a bit of help.
We recently started using MySql and we are seeing in multiple environments a consistent latency issue with calls to MySql. The latency is not present when running the queries using Workbench.
Although it is happening with many calls, I cut the code down to very simple calls like this for testing.
cmd.CommandText =
"SELECT 'test'";
cmd.CommandType = CommandType.Text;
Console.WriteLine("before execute:{0}",timer.ElapsedMilliseconds);
cmd.ExecuteNonQuery();
I even changed the call to NonQuery to make sure that as little as possible was being returned. Consistently every call is taking 200-300ms when I use our dev/qa servers. If I run it locally, runs in just a millisecond or two. I also tried multiple executes in a row without closing connection to eliminate connections and pools. I believe the bad performance is specifically when the command is being executed.
After a week of back and forth, this issue was isolated to our VIP. I am not a network person so apologies if I don't get this description quite correct.
The VIP apparently had a small packet size set for the network. This caused the select statement packets to become fragmented and adding the latency. We just made a change to our VIP and the problem has gone away.
Marking question answered.
Related
I have a console batch application which includes a process that uses SqlDataAdapter.Fill(DataTable) to perform a simple SELECT on a table.
private DataTable getMyTable(string conStr)
{
DataTable tb = new DataTable();
StringBuilder bSql = new StringBuilder();
bSql.AppendLine("SELECT * FROM MyDB.dbo.MyTable");
bSql.AppendLine("WHERE LEN(IdString) > 0");
try
{
string connStr = ConfigurationManager.ConnectionStrings[conStr].ConnectionString;
using (SqlConnection conn = new SqlConnection(connStr))
{
conn.Open();
using (SqlDataAdapter adpt = new SqlDataAdapter(bSql.ToString(), conn))
{
adpt.Fill(tb);
}
}
return tb;
}
catch (SqlException sx)
{
throw sx;
}
catch (Exception ex)
{
throw ex;
}
}
This method is executed synchronously, and was run successfully in several test environments over many months of testing -- both when started from the command-line or started under control of an AutoSys job.
When moved into production, however, the process hung up -- at the Fill method as nearly as we can tell. Worse, instead of timing out, it apparently started spawning new request threads, and after a couple hours, had consumed more than 5 GB of memory on the application server. This affected other active applications, making me very unpopular. There was no exception thrown.
The Connection String is about as plain-vanilla as they come.
"data source=SERVER\INSTANCE;initial catalog=MyDB;integrated security=True;"
Apologies if I use the wrong terms regarding what the SQL DBA reported below, but when we had a trace put on the SQL Server, it showed the Application ID (under which the AutoSys job was running) being accepted as a valid login. The server then appeared to process the SELECT query. However, it never returned a response. Instead, it went into an "awaiting command" status. The request thread appeared to remain open for a few minutes, then disappeared.
The DBA said there was no sign of a deadlock, but that he would need to monitor in real time to determine whether there was blocking.
This only occurs in the production environment; in test environments, the SQL Servers always responded in under a second.
The AutoSys Application ID is not a new one -- it's been used for several years with other SQL Servers and had no issues. The DBA even ran the SELECT query manually on the production SQL server logged in as that ID, and it responded normally.
We've been unable to reproduce the problem in any non-production environment, and hesitate to run it in production without a server admin standing by to kill the process. Our security requirements limit my access to view server logs and processes, and I usually have to engage another specialist to look at them for me.
We need to solve this problem sooner or later. The amount of data we're looking at is currently only a few rows, but will increase over the next few months. From what's happening, my best guess is that it involves communication and/or security between the application server and the SQL server.
Any additional ideas or items to investigate are welcome. Thanks everyone.
This may be tied to permissions. SQL Server does some odd things instead of giving a proper error message sometimes.
My suggestion, and this might improve performance anyway, is to write a stored procedure on the server side that executes the select, and call the stored procedure. This way, the DBA can ensure you have proper access to the stored procedure without allowing direct access to the table if for some reason that's being blocked, plus you should see a slight performance boost.
Though it may be caused by some strange permissions/ADO.NET issues as mentioned by #user1895086, I'd nonetheless would recommend to recheck a few things one more time:
Ensure that queries run manually by DBA and executed in your App are the same - either hardcode it or at least log just before running. It is better to be safe than sorry.
Try to select only few rows - it is always a good idea to not select the entire table if you can avoid it, and in our case SELECT TOP 1(or 100) query may not exhibit such problems. Perhaps there is just much more data than you think and ADO.Net just dutifully tries to load all those rows. Or perhaps not.
Try SqlDataReader to be sure that SqlDataAdapter does not cause any issues - yes, it uses the same DataAdapter internally, but we would at least exclude those additional operations from a list of suspects.
Try to get a hand on the dump with those 5 GB of memory - analyzing memory dumps is not a trivial task, but it won't be too difficult to understand what is eating those hefty chunks of memory. Because I somehow doubt that ADO.NET will just spawn a lot of additional objects for no reason.
We are experiencing an issue where several hundred threads are trying to update a table ID, similar to this post, and sometimes encountering errors such as:
Cannot insert duplicate key in object dbo.theTable. The duplicate
key value is (100186).
The method that is being executed hundreds of times in parallel executes several stored procedures:
using (var createTempTableCommand = new SqlCommand())
{
createTempTableCommand.CommandText = createTempTableScript;
createTempTableCommand.Connection = omniaConnection;
createTempTableCommand.ExecuteNonQuery();
}
foreach (var command in listOfSqlCommands)
{
using (var da = new SqlDataAdapter(command))
{
da.Fill(dtResults);
}
}
In order to recreate such an environment/scenario, is it advisable to simply record a trace and then simply replay it?
How do we recreate an environment with high concurrency?
You can avoid all deadlocks/dirty read only when you will rewrite your solution into sequencial instead of paralel.
You can accept some error and create appropriate error handling. Blocked or wrong run with duplicate key can be started again.
You can try rewrite your solution without touching the same rows with more thread at the same time. You have to change your transaction isolation level (https://msdn.microsoft.com/en-us/library/ms709374(v=vs.85).aspx), change your locking to rowlocking (probably combination of ROWLOCK, UPDLOCK hints). This solution will minimalize your errors, but cannot handle all errors.
So I recommend 2. In some solutions is better way to run command without transation - you can handle it without blocking other threads and enforce relations in next step.
And for "similar post" - the same way. Error handling will be better in your app. Prevent to use cursor solutions like in similar post, because in goes against database fundamendals. Collect data into sets and use sets.
I don't think tracing is a good approach to reproducing a high concurrency environment, because the cost of tracing will itself skew the results and it's not really designed for that purpose. Playback won't necessarily be faithful to the timing of the incoming events.
I think you're better off creating specific load tests to hopefully exercise the problems you're encountering, rent some virtual machines and beat the heck out of a load-test db.
Having said that, tracing is a good way to discover what the actual workload is. Sometimes, you're not seeing all the activity that's coming against your database. Maybe there are some "oh yeah" jobs running when the particular problems present themselves. Hundreds of possibilities I'm afraid - and not something that can be readily diagnosed without a lot more clues.
I recently monitored my sql database activity I found about 400 processes in activity monitoring.Later I figured that the problem is with my connection string object which would not be cleared physically even though I completely closed and disposed it, so once I suspend my IIS all the processes from activity monitoring would disappear.
after a little searching I found that I can clean all of my connections from application pool so that all the useless processes from SMSS would be killed but
I'm really concerned about it's impact on webserver. It's true that this approach would clear useless tasks from SMSS but for every request a new connection should really be created is it worth it???
considering my application is kind of enterprise app which is supposed to handle to many requests, I'm so afraid of making IIS server down by using this approach.
Do notice that my connection string value is not completely fixed for all the requeests, I made it variable by changing only "Application Name" section of it in every request according to the request parameters for the purpose of getting requestors information in sql activity monitoring and sql profiler.
is it worth to do so considering my business scope or it's better I fix the connection string value in other word is performance lag on this approach is so severe that I have to change my logging strategy or it's just a little slower???
Do notice that my connection string value is not completely fixed for all the requeests, I made it variable by changing only "Application Name" section of it in every request according to the request parameters for the purpose of getting requestors information in sql activity monitoring and sql profiler.
This is really bad because it kills pooling. You might as well disable pooling but that comes with a heavy performance penalty (which you are paying right now already).
Don't do that. Obtain monitoring information in a different way.
Besides that, neither SQL Server nor .NET have a problem with 400 connections. That's unusually high but will not cause problems.
If you run multiple instances of the app (e.g. for HA) this will multiply. The limit is 30k. I'm not aware of any reasons why this would cause a slowdown for the app, but it might cause problems for your monitoring tools.
I'm not optimistic about getting an answer to this one, since it's such a weird and seemingly machine-specific issue, but I'm posting in the hopes that someone else has figured this one out.
I'm developing an application in C# using Visual Studio and the MySQL.Data ADO.Net driver (version 6.9.8) for database access. Twice, I've run into a situation where very simple SQL queries that worked only moments before start timing out. From that point forward, they never succeed again -- but only on my machine. Every other query in the application (and there are hundreds) works fine, but the one query, even if I change it slightly, never works again. If I push the code and another developer runs it, the code works perfectly on his machine.
As an example of how simple these queries are, here's the one I'm most recently having issues with:
SELECT * FROM user_preferences WHERE (user_id=#user_id) LIMIT 1
The table I'm selecting from has only 1 row and 4 columns. It couldn't be simpler. While the query is waiting to time out, it does not appear in the process list for MySQL, so I have to assume that either it never made it to the server or the server returned instantly and the MySQL connector is just failing to return the response. I'd like to figure out which, but I doubt it would make a difference either way.
So to summarize, that query worked fine this morning. I changed nothing except some unrelated code, and it briefly started returning no data. I started debugging it, and now it just times out. Another developer runs the code, and it works perfectly.
I'm at a complete loss, here. Short of figuring this out, I fear I'm going to have to re-image my laptop, and I'd really like to avoid that if at all possible. Any help would be appreciated, though like I said, it's probably quite a shot in the dark.
Edit: The C# code in question:
using (var connection = new MySqlConnection(connectionSettings.GetConnectionString()))
{
await connection.OpenAsync(cancellationToken);
using (var reader = await MySqlHelper.ExecuteReaderAsync(connection, sql.Query.ToString(), cancellationToken, sql.Parameters.ToArray()))
{
while (await reader.ReadAsync(cancellationToken))
{
readerAction(reader);
}
}
}
I'm having some trouble when I select large tables full of strings data using Npgsql over a high latency connection (200-300 ms).
The total fields value size is something like 256kb.
I'm totally sure that the problem is related to the transfer over the network beacause if I execute the same query locally it's executed in very short time (10 ms - 20 ms), instead if the query is executed via the slow connection it takes 20-30 seconds.
Also if i'm measuring the table using length(fields) the query is executed in a decent time (1-2 seconds).
I'm experiencing this problem on different pc, different postgresql version and npgsql.
I think that the problem is related to the size of packets.. maybe increasing the buffer could solve my problem but, how to do it? in postgres? or in npgsql?
Per comments, you're using an SSH tunnel.
TCP-in-TCP tunneling like SSH does is absolutely awful for carrying lots of data. Expect it to perform terribly. Congestion control and window scaling don't work properly, retransmits are a problem, reordering and reassembly is inefficient, slow-start doesn't work well, etc.
Just don't do that. Use direct TCP/IP with SSL, or use a UDP-based tunnel/VPN.
PostgreSQL's protocol is quite friendly to pipelining and it requires zero round trips per field or per row fetched when getting results. You just keep reading until there's nothing more to read. So round-trip latencies shouldn't be the issue here.
It's very likely to be problems caused by the tunnel.