I'm not optimistic about getting an answer to this one, since it's such a weird and seemingly machine-specific issue, but I'm posting in the hopes that someone else has figured this one out.
I'm developing an application in C# using Visual Studio and the MySQL.Data ADO.Net driver (version 6.9.8) for database access. Twice, I've run into a situation where very simple SQL queries that worked only moments before start timing out. From that point forward, they never succeed again -- but only on my machine. Every other query in the application (and there are hundreds) works fine, but the one query, even if I change it slightly, never works again. If I push the code and another developer runs it, the code works perfectly on his machine.
As an example of how simple these queries are, here's the one I'm most recently having issues with:
SELECT * FROM user_preferences WHERE (user_id=#user_id) LIMIT 1
The table I'm selecting from has only 1 row and 4 columns. It couldn't be simpler. While the query is waiting to time out, it does not appear in the process list for MySQL, so I have to assume that either it never made it to the server or the server returned instantly and the MySQL connector is just failing to return the response. I'd like to figure out which, but I doubt it would make a difference either way.
So to summarize, that query worked fine this morning. I changed nothing except some unrelated code, and it briefly started returning no data. I started debugging it, and now it just times out. Another developer runs the code, and it works perfectly.
I'm at a complete loss, here. Short of figuring this out, I fear I'm going to have to re-image my laptop, and I'd really like to avoid that if at all possible. Any help would be appreciated, though like I said, it's probably quite a shot in the dark.
Edit: The C# code in question:
using (var connection = new MySqlConnection(connectionSettings.GetConnectionString()))
{
await connection.OpenAsync(cancellationToken);
using (var reader = await MySqlHelper.ExecuteReaderAsync(connection, sql.Query.ToString(), cancellationToken, sql.Parameters.ToArray()))
{
while (await reader.ReadAsync(cancellationToken))
{
readerAction(reader);
}
}
}
Related
Recently I came across kinda funny issue. Generally I am working on .NET web app hosted on IIS and using MS SQL server.
Having said that I wrote a simple piece of code to update specific record in db. The record should already exist and no further operations should occur on it (or actually even the entire table) for considerable time (no concurrent operation should appear to overwrite update).
var record = db.InputDataRecords.Where(i => i.Id == inputDataId).Single();
record.InputData = updatedInputData;
db.SaveChangesAsync();
So I wrote the above code, tested it on my local machine, everything worked perfectly fine so I just left it as it was.
Recently the code finally got to the test server (pretty slow sucker) and to my surprise I got info from the test crew that the records do not update accordingly. What is even worse no errors were logged although they should if something went wrong plus I am pretty sure the above code did not throw any errors as, for the sake of peace, I forced it to log message with some details after it is executed (I cannot debug on that machine unfortunately so I couldn't get any better feedback).
I struggled with this issue for couple of hours doing some small adjustments here and there, but nothing was really helping. Finally switching to synchronous SaveChanges() did the trick.
Did anyone of you come across similar issue? I am pretty curious why it could occur.
We started developing an application in C# recently and decided to try Entity Framework 6.1.3 with a code-first approach to handle persistence. The data is being stored in a SQL Server Express 2012 instance which is running on a local server. The application is still very small. Currently we are only persisting two POCOs and have two tables in the database: one with 5 fields (about 10 rows of data), and one with 4 fields (2 rows).
We are experiencing a 3 second delay when Entity Framework processes the first query with subsequent queries being extremely quick. After some searching we found that this is normal for Entity Framework so, although it seemed excessive for such a small database, we've been coping with it.
After doing some work today, none of which was specifically related to persistence, suddenly I found that the first query was only taking a quarter of a second to run. I couldn't see anything obvious in the changes I'd made so I uploaded them to source control and asked my colleague to download everything. When he builds and runs it on his computer, it still takes 3 seconds for the first query. I've compiled the application, tried it on two test computers and they experience the initial 3 second delay.
There seems to be no correlation between the problem and the computers/operating systems. My computer is running Windows 7 SP1 x64. My colleague's development computer is running Windows 10 x64. The other two test computers are running Windows 7 SP1 x86. They are all a similar specification (Core i5, 4GB/8GB RAM).
In my research of the delay I found that there are several things you can do to improve performance (pre-generating views etc) and I have done none of this. I haven't installed anything or made any changes to my system, although I suppose it's possible an update was installed in the background. Nothing has changed on the database server or in the database itself or in the POCOs. We are all connecting to the same database on the same server.
This raises a few obviously related questions. If it's possible for it to start up in a quarter of a second, why has it been taking 3 seconds up until now? What happened on my computer to suddenly improve performance and how can I replicate this on the other computers that are still slow?
Can anyone offer any advice please?
EDIT
I turned on logging in Entity Framework to see what queries were being run and how long they were taking. Before the first and, for testing purposes, only query is run, EF runs 3 queries to do with migration. The query generated to retrieve my data follows that and is as follows:
SELECT
[Extent1].[AccountStatusId] AS [AccountStatusId],
[Extent1].[Name] AS [Name],
[Extent1].[Description] AS [Description],
[Extent1].[SortOrder] AS [SortOrder]
FROM [dbo].[AccountStatus] AS [Extent1]
-- Executing at 28/01/2016 20:55:16 +00:00
-- Completed in 0 ms with result: SqlDataReader
As you can see it runs really quickly which is hardly surprising considering there are only 2 records in that table. The 3 migration queries and my query take no longer than 5ms to run in total on both my computer and my colleague's computer.
Copying that query in to SSMS and running it from various other machines produces the same result. It's that fast it doesn't register a time taken. It certainly doesn't look like the query causes the delay.
EDIT 2: Screenshots of diagnostic tools
In order to give a good comparison I've altered the code so that the query runs at application start. I've added a red arrow to indicate the point at which the form appears. I hadn't noticed before but when my colleague runs the application the first time after starting Visual Studio, it's about a second quicker. All subsequent times are slower.
1) Colleague's computer - first run after loading Visual Studio
2) Colleague's computer - all subsequent runs
3) My computer - all runs
So every time my colleague runs the application (apart from the first time) there is a second's pause in addition to the usual delay. The first run immediately after starting Visual Studio seems to eliminate this second's pause but it's still no where near the speed on my computer.
Incidentally, there is a normal delay of around a quarter of a second caused by the application starting. If I change the application to require a button click for the first query, the second's pause and usual 2 second delay happen only after the button is clicked.
Another thing of note is the amount of memory the application uses. Most of the time on my computer it will use around 40MB but on the other computer it never seems to use more than 35MB. Could there be some kind of memory optimisation going on that is slowing things down for the other computer? Maybe my computer is loading some additional/cached information in to memory that the others are having to generate. If this is possible, any thoughts on where I might look for this?
EDIT 3
I've been holding off making changes to the model and database because I was worried the delay would come back and I'd not have anything to test against. Just wanted to add that after exhausting all other possibilities, I've tried modifying a POCO and the database and it's still quick on my computer but slow on others.
I've altered the title of this question to more accurately reflect the problem I'm trying to solve.
Query plans in SQL Server can change over time. It may be that your machine has cached a good query plan, while your co workers machine has not. In other words, it may have nothing to do with EF. You could potentially confirm or deny this theory by running the same query by hand in management studio.
In order to tackle performance problems related to EF generated queries, I advice you to use Profiler or an alternative (as Express Editions do not have it) to see how much of your time is actually consumed running the query.
If most of your time is used for running the query, as already suggested by jackmott, you can run it in SSMS by toggling Actual Execution Plan to see the generated plan in each situation.
If time is spent on something else (some C# warming up or something similar), Visual Studio 2015 has builtin performance analysis that can be used to see where it is spending that time.
Today I got an emergency call from the users on our ASP.NET production system. Some users (not all) were unable to enter certain data. The user posted the data, and the system then froze; the call never returned.
We tried to repro the problem on the QA system (which has a fresh restore of production data), and could not. I then ran from my dev environment and connected directly to the production DB, masquerading as one of the affected users. Again, no problem. Conclusion: there must be some kind of issue in the production environment, probably somewhere in the IIS process that's hosting the website.
So I fired up Visual Studio on the production server, and attached to the IIS process (Kids, don't do this at home!), set a breakpoint in the offending code, logged in as the user, and attempted to save the data. Hit the breakpoint and stepped line by line, until I hit a line of code like this:
try
{
...
using (var db = new MyDataContext())
{
...
var fooToUpdate = db.Foos.Single(f => f.ID == fooId); // <-- THIS LINE
...
}
}
catch (Exception ex)
{
// some error logging
}
After hitting "step" on that line, the thread simply vanished. Disappeared without a trace. I put a sniffer on the database, and no query was fired; needless to say there was no DB locking involved. No exception was thrown. The code entered Entity Framework and never left.
The way the data is is that every user has a different and unique fooId for every day, so no other user will have the same fooId. Most users were able to load their Foo, but a select handful of users fail consistently to load their personal Foo. I tried running the query to load the Foo in a SSMS window; no trouble at all. The only time it fails is in this particular IIS process, on the production server.
Now, I could just recycle the app pool or restart IIS, and that would probably paper over the problem. But something similar happened a week ago, and we couldn't trace it then, either. So we reset IIS then, hoping the problem would go away. And it did, for a week. And now it's back.
Does anyone have any ideas how it is possible for a thread to simply vaporize like this? Is Norman Bates hiding behind the EF door?
Given the fact that the thread did not magically vaporize, we could speculate some of the more likely options:
The debugger had a hard time following the production code compiled in Release mode. Just because debugging Release code works 90% of the time, don't fall under the illusion that it is dependable. Optimized code can very quickly throw the debugger off the track of actual execution. When this happens, it will look like the thread just vanished.
Assuming the thread does legitimately enter the call and not return (which seems to be supported by the original complaint of the application "freezing"), then the most likely scenario is a deadlock of some type. EntityFramework deadlocks are not common, but not unheard of either. The most common issues I'm aware of usually involve TransactionScope or CommitableTransaction. Are you using any transactions in the omitted code sections?
Turns out that the EF part was a red herring after all. I went and downloaded Telerik's JustDecompile and JustCode, in the hope of stepping into the EF code, but when I stepped in to that line, I found myself not in the Single() extension method, but inside one of my own method calls - that I thought I had executed on the previous line. Evidently the code was not perfectly in sync with the version in production.
LESSON 1: If you attach to a process, your execution point may not be where you think it is, if your code is not identical to the code that was
compiled into that process.
So anyway, now that I could step into the code without decompiling anything, the first thing I noticed was:
lock (_lockObj)
{
...
}
And when I tried to step into it, it froze there. Smoking gun.
So somewhere, some other thread is locking this object. Looked at other places where the lock is invoked, leading to a spaghetti of dependencies, along with another code-locked segment, with several DB calls and even a transaction boundary. It could be a code lock / db transaction deadlock, though a brief scan of the code in the DB transaction failed to pick up any contenders within the life of the transaction for blocking anything else. Plus, there's the evidence of the DB not showing any blocking or open transactions. Rather, it may just be the the fact of a few hundred queued up long-running processes, all inside code locks inside code locks, and in the end it all looks something like the West Side Highway at 17:05 on a Friday, with a jackknifed trailer truck lying across 3 lanes approaching the GW bridge.
LESSON 2: Code locks are dangerous, not only - but especially - when used in conjunction with DB transactions. Try to find ways to make your code thread safe without using code locks. And if you really must use code locks, make sure you get in and out as quickly as possible. Don't give your thread a magazine to read while it's occupying the only stall, so to speak.
I am a relative novice to MySql and I've run out of possible ideas so looking for a bit of help.
We recently started using MySql and we are seeing in multiple environments a consistent latency issue with calls to MySql. The latency is not present when running the queries using Workbench.
Although it is happening with many calls, I cut the code down to very simple calls like this for testing.
cmd.CommandText =
"SELECT 'test'";
cmd.CommandType = CommandType.Text;
Console.WriteLine("before execute:{0}",timer.ElapsedMilliseconds);
cmd.ExecuteNonQuery();
I even changed the call to NonQuery to make sure that as little as possible was being returned. Consistently every call is taking 200-300ms when I use our dev/qa servers. If I run it locally, runs in just a millisecond or two. I also tried multiple executes in a row without closing connection to eliminate connections and pools. I believe the bad performance is specifically when the command is being executed.
After a week of back and forth, this issue was isolated to our VIP. I am not a network person so apologies if I don't get this description quite correct.
The VIP apparently had a small packet size set for the network. This caused the select statement packets to become fragmented and adding the latency. We just made a change to our VIP and the problem has gone away.
Marking question answered.
I am trying to connect to a Domino Session which works fine when testing in isolation, however as soon as I put it in a thread I run into problems.
This works fine when I put in a Unit test, so I know I can connect to the server fine.
NotesSession session = new NotesSession();
string DominoPassword = Helpers.GetConfigSetting("DominoPassword");
session.Initialize(DominoPassword);
However when I put this in a thread (want to be able to connect to Domino several times) I
get the following error
"{Unable to evaluate expression because the code is optimized or a native frame is on top of the call stack.}"
A couple of years ago I definitely got farther than this with testing of multi-threaded Domino Interop in C#. I don't recall seeing that error. It might be due to a difference between our respective dev/test environments. It's long enough ago, though, that I don't recall what my environments were when I ran my tests.
But whether or not the Domino objects are thread-safe remains an open question. See here. As far as I know, IBM has never said that they are. (Also, last time I checked, IBM said that the classes are not supported on Win64 platforms even in Win32 applications, so that might be entering into your problem.