Over the last few weeks I've been subject to a sudden and significant performance deterioration when browsing locally hosted ASP.NET 3.5 MVC web applications (C#). Load times for a given page are on average 20 seconds (regardless of content); start up is usually over a minute. These applications run fast on production and even test systems (Test system is comparable to my development environment).
I am running IIS 6.0, VS2008, Vista Ultimate, SQL2005, .NET 3.5, MVC 1.0, and we use VisualSVN 1.7.
My SQL DB is local and IPv6 does not seem to be the cause. I browse in Firefox and IE8 outside of Debug mode using loopback, machine name, and 'localhost' and get the exact same results every time (hence DNS doesn't seem to be the issue either).
Below are screen shots of my dotTrace output.
http://www.glowfoto.com/static_image/28-100108L/3123/jpg/06/2010/img4/glowfoto
This issue has made it near impossible to debug/test any web app. Any suggestions very much appreciated!
SOLUTION: Complete re-installation of Windows, IIS, Visual Studio, etc. It wasn't the preferred solution, but it worked.
Surely the big red flag on that profiler output is the fact that AddDirectory is called 408 times and AddExistingFile is called 66,914 times?
Can you just confirm that there's not just a shed load of directories and files underneath your MVC app's root folder? Because it looks like the framework is busying itself trying to work out what files it needs to build (or add watches to) on startup.
[I am not au fait with MVC and so maybe this is not what is happening but 67k calls to a function with a name like "AddExistingFile" does smell wrong].
I've learnt that it's usually a "smell" when things fail near a power of two ...
Given
Over the last few weeks I've been subject to a sudden and significant performance deterioration
and
AddExistingFile is called 66,914 times
I'm wondering if the poor performance hit at about the time as the number of files exceeded 65,535 ...
Other possibilities to consider ...
Are all 66,914 files in the same directory? If so, that's a lot of directory blocks to access ... try a hard drive defrag. In fact, it's even more directory blocks if they're distributed across a bunch of directories.
Are you storing all the files in the same list? Are you preseting the capacity of that list, or allowing it to "grow" naturally and slowly?
Are you scanning for files depth first or breadth first? Caching by the OS will favor the performance of depth first.
Update 14/7
Clarification of Are you storing all the files in the same list?
Naive code like this first example doesn't perform ideally well because it needs to reallocate storage space as the list grows.
var myList = new List<int>();
for (int i=0; i<10000; i++)
{
myList.Add(i);
}
It's more efficient, if you know it, to initialize the list with a specific capacity to avoid the reallocation overhead:
var myList = new List<int>(10000); // Capacity is 10000
for (int i=0; i<10000; i++)
{
myList.Add(i);
}
Update 15/7
Comment by OP:
These web apps are not programmatically probing files on my hard disk, at least not by my hand. If there is any recursive file scanning, its by VS 2008.
It's not Visual Studio that's doing the file scanning - it is your web application. This can clearly be seen in the first profiler trace you posted - the call to System.Web.Hosting.HostingEnvironment.Initialize() is taking 49 seconds, largely because of 66,914 calls to AddExistingFile(). In particular, the read of the property CreationTimeUTC is taking almost all the time.
This scanning won't be random - it's either the result of your configuration of the application, or the files are in your web applications file tree. Find those files and you'll know the reason for your performance problems.
Try creating a new, default MVC2 application in a new web folder. Build and browse it. If your load times are okay with the new app, then there's something up with your application. If not, it's outside of the context of the app and you should start looking at IIS config, extensions, hardware, network, etc.
In your app, back up your web config and start with a new, default web.config. That should disable any extensions or handlers you've installed. If that fixes your load times, start adding stuff from the old web.config into the new one in small blocks until the problem reappears, and in that way isolate the offending item.
I call this "binary search" debugging. It's tedious, but actually works pretty quickly and will most likely identify the problem when we get stuck in one of those "BUT IT SHOULD WORK!!!" modes.
Update Just a thought: to rule out IIS config, try running the site under Cassini/built-in dev server.
The solution was to format and do a clean install of Vista, SQL Server 2005, Visual Studio 2008, IIS6 and the whole lot. I am now able to debug, without consequence, the very same webapp(s) I was experiencing the problems with initially. This leads me to believe the problem lay within one of the installations above and must have been aggravated by a software update or by the addition of software.
You could download Fidler to measure how long each call takes and get some measurements.
Link
This video might help...
Related
I'm working with an open source ecommerce system written using C# and ASP.NET MVC 5. This package makes heavy use of autofac and IoC. It presents a plugin architecture that developers who work with this ecommerce package can use to create plugins, and it uses that plugin architecture itself.
Under some very mysterious circumstances (not just when the application domain recycles) it's taking a full minute to perform a page load. I've been struggling for a while to figure what's taking so long.
I haven't been able to get the application to run from within Visual Studio using the built in web server, and I haven't even been able to get the Visual Studio debugger to attach to an IIS process that is running it. So I haven't been able to use profilers that require Visual Studio.
I have managed to get Glimpse working, but Glimpse is not capturing the event that is taking a long time. In the time line, it shows "Start Request", which takes 0 ms, and then 7 to 50 seconds later, the next event starts, which is, for example, "Authorization: Home.Index", which it says took .01 ms.
What in the world is going on during that 7 to 50 second interval in between? How can I find that out?
Here's a somewhat strange clue: Today, after no change that we're aware of at all, suddenly the problem has more-or-less vanished (most of the time, at least temporarily) when the browser is on our internal network, but when the browser is on a computer outside our network, the problem persists. It's as if some plugin is doing some kind of DNS look up that is taking much less time (to succeed or fail) internally than it takes externally.
So conceivably, a controller is getting instantiated and some dependency or another from some plugin or another is doing some kind of network look up at construction time or something like that. So I'm trying to figure out how to measure how long it's taking MVC to construct each of the dependencies of the Home controller. So far, the solution has elluded me.
I guess I can maybe track down all the implementations of each of the dependencies and figure out which implementation is ending up getting used, and put log messages into their constructor code. Except some of the plugins don't provide source code. At least I can do it for the ones that do, anyway, but is there some profiler or tool I can use that doesn't require Visual Studio and works with ASP.NET MVC processes running under IIS?
Environment:
Visual Studio 2015 Update 1
IIS Express 10
Moderately sized MVC web application
.NET Framework 4.6.1 x64 debug builds
Newer Core I7 laptop, plenty of ram, SSD drive
Making a small change in one projects .cs file, hit the green arrow to test out the change. It takes about 8 seconds for the build to finish and Chrome to pop open a new tab. Not bad. But then it takes about ~30 seconds for the first page to show up.
What can be done to reduce that delay? Would pre-compiled views be the first order improvement here? What are some of the best current techniques to achieve that?
Try installing Glimpse. Correctly set up it will show you where the delay is including the database calls and their duration etc.
Install Redgate Ants and step through the code locally (potentially pointing at the production database if this is a live problem), this tool should be able to tell you where any slow down is.
One of the features is:
Jump straight to the slowest activity
The call tree in the .NET performance profiler shows you data for
every method and identifies the most expensive methods, database
queries, and web requests
There is a 14 day free trial, which should be enough time to diagnose your problem.
I am having a rather time-consuming issue with the Sitecore CMS.
Is it possible to shorten the time it takes for the application pool to recycle every time a change is made in the bin folder or web.config?
It takes the server 2 to 5 minutes to respond right after a change.
Any ideas?
There is one article from alex shyba from sitecore here on Reducing Sitecore Startup time
Summary of article is
In machine.config: disable process to check assembly signed
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
Disable performance counters in Sitecore web.config
<setting name="Counters.Enabled" value="false" />
There are plenty of articles on the improving sitecore performance I have listed few links below:
Sitecore SDN - Optimizing Performance in Sitecore
Sitecore Slides on performance improvements - download pdf
Analyzing and identifying the slow causing culprit - Sitecore Startup basics
He needs to restart the application pool in order to load the new dll's in.
You could try and minimize the things you do at startup(but i'm guessing it's mostly sitecore stuff which you can't influence) so the best suggestion i can give is to have 2 webservers with a content switch.
You would run your application on 2 servers and the content switch decides which server handles which request(be carefull with session and statics because each server will know nothing of the other).
If at some point you need to release a new version you just instruct your content switch to direct all trafic to webserver A. You then deploy to webserver B, open the website via a direct url that doesn't pass the content switch and make sure it's working properly + warmed up.
Then you tell the content switch to point all trafic to B and you have all the time in the world to update webserver A and switch the content switch back to normal mode.
I had this issue as well.
For whatever reason my local copy would take around 5 minutes to fully spin up after any change or starting the website fresh. It was absolutely dreadful for testing anything. It was only an issue locally. My development server environment and production didn't seem to be effected.
I found out a profiling tool that allowed me to find an outlier control that was taking an obscene amount of time to render. Try this...
Fire up your website after it compiles.
Log into this admin page: http://local.example.com/sitecore/admin/stats.aspx
You'll see a screen like this: Look for anything with an obscenely high Avg or Max time. Your problem will likely be in the logic there.
As a bonus, if you can keep hitting pages after your long recompile without much trouble, you should notice the Avg time roughly halving itself every page hit for your problem control if your problem is anything like mine was.
In my case, I found the largest issue came from one control that was doing an expensive search for one item.
It looked like this -> Sitecore.Context.Database.SelectItems("" + Sitecore.Context.Item.Paths.Path + "/ancestor-or-self::*[##templateid='" + templateId + "']");
This was mostly a local issue because the SQL server is remote to my machine, while the other servers were in the same building. Hence dev and production were relatively unaffected.
Good luck!
You can follow Alex Shyba's post here
But 2-5 minutes sounds extreme. This isn't normal. Are your hardware up to date?
Also I would look in to your prefetch cache. In your development environment you probably don't want i to fetch to much on start up, as this isn't needed in development.
I would also look into the initialize pipeline and global.asax to see if you are doing any custom start up jobs.
You can reduce the pre-fetch cache size on core, master, and web databases, but this is not recommended, do this only on development machines, not on production servers.
I have an ASP.NET application that does a large database read. It loads up a gridview inside an update panel. In VS2008, just running on my local machine, it runs fantastically. In production (identical code, just published and put on one of our network servers), it runs slow as dirt.
Debug is set to false, so this is not the cause of the slow down. I'm not an experienced web developer, so besides that, feel free to suggest the obvious.
I have been using Firebug to determine what's going on, and here is what that has turned up:
On production, there are around 500 requests. The timeline bar is very short. The size column varies from run to run, but is always the same for the duration of the run.
Locally, there are about 30 requests. The timeline bar takes up the entire space.
Can anyone shed some light on why this is happening and what I can do to fix it? Also, I can't find much of anything on the web about this, so any references are helpful too.
EDIT: I forgot to mention that I am using the exact same database for both local and production environments.
EDIT: __EVENTTARGET points to the timer that updates the progress.
EDIT: Timer appears to be working - I set it to a larger interval, which shows me that the local app is processing ~50000 records/second and that the production app (same database, same code!) is processing at best ~5000 records/sec.
Back to square one, mysteriously slow app. :(
My guess would be the size of the database you're using on your dev box vs. production. Is it possible there is something happening for each row of your gridview that only happens 30 times on your box becuase it has less data than production?
Systematic debugging - strip it down and add things back one at a time until you see the change.
Brandi,
Can you check the Event Viewer logs for the web server? Do you have anything failing in there?
Are you running the app in VS2008 off of IIS or the dev server?
Anytime I've experienced strange discrepancies between seemingly identical dev and prod code, I've learned to have a quick look at the machine.config file in production.
If you have access to it, you might compare it to your local copy, and specifically review these settings (although the linked post doesn't refer to your exact problem per se.)
I've come to accept that this may be a processor issue based on this other post:
ASP.NET Website Slow Performance on production server
Thanks to everyone who helped answer.
I am developping a (relatively small) website in ASP.Net 2.0. I am also using nAnt to perform some easy tweaking on my project before delivering executables. In its current state, the website is "precompiled" using
aspnet_compiler.exe -nologo -v ${Appname} -u ${target}
I have noticed that after the IIS pool is restarted (after a idle shutdown or a recycle), the application takes up to 20 seconds before it is back online (and Application_start is reached).
I don't have the same issue when I am debugging directly within Visual Studio (it takes 2 seconds to start) so I am wondering if the aspnet_compiler is really such a good idea.
I couldn't find much on MSDN. How do you compile your websites for production?
Make sure that:
You are using a Web Application project rather than a Web Site project, this will result in a precompiled binary for your code behind
You have turned off debug code generation in the web.config file - I guess if this is different to when you used aspnet_compiler the code may be recompiled
If you've tried those, you could maybe try running ngen over your assembly thus saving the JIT time?
For ultimate reponsiveness, don't allow your app to be shutdown.
The first method is to make sure that it's incredibly popular so that there's always someone using it.
Alternatively, fetching a tiny keep-alive page from somewhere else as a scheduled activity can be used to keep your site 'hot'.
If your website is compiled as updatable, you'll see a bunch of .ASPX files in your virtual directory. These must be compiled on startup. That's so you can come in and alter the web UI itself. This is default for both websites and web applications.
Make sure this is set in web.config <compilation debug=false>. In my case, I also have a batch file which issue Get requests for all the main pages before giving to users (page loading simulation).
The key is to make sure the IIS Application Pool never shuts down. This is where the code is actually hosted. Set the "Idle Timeout" (Under Advanced Settings) to something really high, like 1440 minutes (24 hours) to make sure it's not shut down as long as somebody hits your site once a day.
You are still going to have the JIT time whenever you deploy new code, or if this idle timeout period is exceeded witout any traffic.
Configuring IIS 7.x Idle Timeout
#Simon:
The project is a Web Application. Websites are then slower to startup (I had no idea it had an incidence, beside the different code organization)?
I checked, and while I edit the web.config after aspnet_compiler is called, I don't touch the debug value (I will however check the website is not faster to startup if I don't touch the web.config, just to make sure)
(And I will definitely have a look at ngen, I was not aware of that tool.)