I'm working on a legacy application built on ASP.NET Webforms and .NET Framework 4.6.2.
The application's initial page is very slow to start on my local machine and I'm trying to troubleshoot it. I put a breakpoint on Global.asax Application_Start method, and it's taking quite some time even before the breakpoint is hit. I'd like to know how can I track exactly what happens before the Application_Start event (what calls are made, external libraries used, etc.), from the moment I make the first request to the server, so I can get a starting point to improve the performance. Thanks in advance.
Well, while in VS create (or open) some aspx page with JUST hello world on it. Does that page take a long time to load?
And to be fair, create a brand new blank asp.net webforms project, and find out if that project also is slow to load.
In other words, this just might mean your computer is slow.
When you hit f5, then:
The project code is compiled. Then a bunch of information is "copied", setup, then configured for IIS to launch. Then a WHOLE web server is launched (IIS express), then the web page is loaded, and then IIS processes that page, and then it is rendered.
In other words, a BOATLOAD of startup stuff occurs. Once that WHOLE big web server is spooled up, then and only then does your application start and all that jazz start.
If you can navigate to another page (say from the menu bar (assuming your application has one), and then BACK to that supposed page that loads slow? Does that page load slow the 2nd time you hit that page?
If that is the case, then your computer might be slow, starved for memory, or even starved for processing to start up the WHOLE system of stuff required to run that whole web server.
if once the page is displayed, and everything runs normal, then this is not a slow loading web page, but in fact a slow load WHOLE web site, and in actual use and production of the system, then it not really slow anymore is it?
So, say if your computer only has 8 gig, and say a low or slow laptop processor, then startup of all the "enormous" moving parts required to run a whole web server system might be the issue, and NOT that the page load is really slow.
And since the production system will not be re-loading from 100% scratch each time a whole web server system, then you may well not have a slow system, but only a slow "first time" startup.
So, do other web site projects based on .net framework, and asp.net pages start out slow?
And is this a asp.net web site, or is this a asp.net web site (application) project?
So, quite a bit of "hand stands" will occur on first run and load of the web site. So, this could be your computer, and not really that the web site is slow.
Having stated the above?
I have some found web site projects RATHER slow to load, and this is especially in the case of a asp.net web site application. (as opposed to web site's, which tend to load a WHOLE lot faster - about 1 second, and the other was taking about 18-23 seconds on my old laptop (it was starved for processing).
On a older laptop, I was seeing about 18-23 seconds delay after hitting f5. On my newer laptop, I have about 2, sometimes 3 seconds delay from the time I hit f5 in code or markup to see the results and web site running.
So, often IIS has to be loaded and started, maybe SQL server started, and even more bits and parts. All of these things can take a LONG time on your dev box, but on a production server, all of those things are running all the time.
So, it not clear if you have a very slow web site, slow loading "specific" web page, or you just talking about the overall slow loading time for "any" first page to display, and then after that, everything is fine? In other words, it is f5 to start things going is the slow point in time.
And you didn't even note how long of a delay and time here?
Are you talking about 5 seconds, or 50 seconds?
Right click on your task bar, task manager.
Here is me hitting f5 on my laptop. Note the 6 cores go 100% max speed and power during that startup.
I do know that on my laptop, when I hit f5, then all 6 cores (12 threads) jumps about about 2ghz to 4 ghz, and all 6 cores are pinned for those 2-3 seconds while that whole web server is created, configured, and THEN started, and then the page in question has to load.
So, here you can see that hitting f5, the web page "starts" almost instant, and then about 4 seconds more.
(the processor usage is somewhat high - part of that is due to running the screen capture for this screen shot).
So, as "soon" as I hit f5, you can see the processor jump above 20%. I count about < 1 second for the browser to appear. Then about 4 more seconds. And the above is a slower to load page. And as soon as the page displays, then you see the cpu drop down.
As noted, try creating a new asp.net web site "application". A web site, they tend to load < 1 second. In fact above is the "by far" slowest startup I have. Most are about 1-2 seconds from hitting f5.
A asp.net web site application? They take longer - the whole site has to be compiled each time, and the .dll's have to be created.
Sometimes, it simple things like some virtual folder mapped to a location that does not exist (so on startup, a non existing folder or path name can take some time to "time out".
so, how fast does a new test project (create a new asp.net web site application (create a application, not a web site). I find that web sites load fast - even on computers with limited hardware.
However, asp.net web "applications" tend to load a WHOLE lot slower. (and thus more ram, and more cpu, and more cpu cores makes a HUGE difference in startup times).
Performance is a real art form in our industry. But, a few more details such as do all "application" sites start slow on your dev box, or JUST this one, and by how much compared to other sites.
Often, a delay can be network connection to a database, and say little or nothing to do with startup, but some network delay in say connecting to sql server which may well be some kind of network connection.
So, all first pages slow, or just one page?
does a new asp.net web site application (application is important here - not just web site) also startup slow? (if yes, then this is not specific to your site, is it???).
And using task manager - do you see high cpu usage during that startup (f5), or is it low? (this then suggests network and connection delays - not cpu and memory and performance issues.
So, a lot more trouble shooting here is required, but then again, we have little idea as to what testing you done so far, such as testing another web site application on your computer - and does it have things like authentication and sql server logon's involved.
I mean, we can have some issues with a car that will not start. So, we don't just yet swap out the engine, re-do the fuel injection system, and then ONLY to find out we forgot to put in a new battery, and that was the issue all along.
So, do a few tests - get some base-lines of say a test site - use one of the templates that has the "default" bootstrap menu, master pages etc. How fast does that start? Since if that site ALSO starts slow, then attempts at troubleshooting the existing site will be in vain, since all sites load slow - not just the "one" site your working on.
My C# ASP.net MVC site allows users to upload many photos at a time (using dropzone.js). After uploading about 80 files the user started receiving a 503 Server Unavailable error.
They waited a few minutes, retried the files that failed and the rest uploaded (about 20).
How can I prevent the server from doing this?
When it happened, the rest of the site remained operational.
TIA!
A 503 Server Unavailable error under load typically means you've crossed some resource limit on your server. There's two ways to fix this:
Change the client, so it doesn't reach this limit (or at least not quickly). For dropzone specifically I'd add throttling logic to it so that when a user drops a 100 photos all at once-- your logic will process these in batches (e.g. five at a time).
Change the server, so this limit is higher. Without more information about your infrastructure it's difficult to tell what's causing the current bottleneck. It could be as simple as scaling up the CPUs/RAM of your host.
503 Server unavailable means, your application is not scalable and breaking to handle further requests. Couple of suggestions to fix this issue.
Vertical scaling
Horizontal Scaling
Revisit the Architecture
Even before going through these points, I suggest to "load test" your current app and create bench-marking. Means start testing application for 100 files per minute, 200, 300 ... and identify at which number it is breaking/throwing 503 error. You can use simple JMeter kind of tools for this.
And also define your target requirement. like 10K files per hour or 100 files per minute...
Vertical Scaling:
Based on your bench marking results from load test, you can start fine tuning RAM/CPU/IO capacity of the existing machine. Example if you are using containers , you can increase the allocated hardware resources. And repeat the test until it satisfies your target requirement.
Horizontal Scaling
Based on your bench marking results from load test, start adding more nodes to your cluster. And repeat the test.
Revisit the Architecture
The above two options talks about infra fine tuning without modifying application architecture. But your requirement is "file uploading" which is IO task not CPU bound. So to achieve greater scalability with optimal resources you can consider re architecture the application either using "NodeJS" or "C# Reactive X" (https://github.com/dotnet/reactive). These are Reactive style programming which will provide you non-blocking async processing capabilities.
Background
I recently came across an out of memory exception when users would visit few pages of my Kentico website. Fast forward - I found that the allocated memory (System > General) was over 2 GB! I then went to Debug > Clear cache and then noticed the allocated memory sitting roughly around 400 MB (phew..). Now, when the users would visit the page, it would work without any out of memory exception.
Question
Is there a way I could get these memory statistics via code (ideally C#)? I'm thinking of being able to regularly monitor these memory statistics and trigger an alert (send an email/post to webhook from my C# code) when the allocated memory gets too high.
Additonal information
Kentico version 9.0.42, hosted in Azure, scaled to 2 instances.
The App Service Plan's (in Azure) memory usage was roughly at 50% through out - this rules out setting an alert at that level.
Thanks!
If you look at the code on the System page, you'll find your answer. Go to /CMSModules/System/Controls/System.ascx.cs file and search for Memory.Text. You'll find several SystemHelper methods to get the values for you.
SystemHelper.GetVirtualMemorySize()
SystemHelper.GetWorkingSetSize()
SystemHelper.GetPeakWorkingSetSize()
You can use SystemHelper class to get statistics and memory data. It still mostly relies on .NET Process and its properties. If you are on azure, you can use above along with PerformanceCounters to log your own sets of information into application insights and create your own alerts based on these counters.
It recommended by Kentico to restart you app once per 24 hours. I suggest you schedule restart of your instances one after another some time during the night.
There doesn't seem to be anything in the Kentico API that would access the memory statistics, but maybe this answer will help you.
I have a very big problem with an Azure Webapp and would like to hear suggestions.
What we experience:
When I goto our website it's quite fast. The average load is around 1 second and respond as expected.
However, once pr. 10-20 minutes, we get a very cold load - where it's 30-60 seconds+.
This would make sense if the website didn't have visitors and it was cold, but we have 10+ visitors online at the website all business hours with 3-5 requests pr minute at a minimum.
This extra load is completely unacceptable of course.
Any ideas?
Our setup:
We have two azure web-apps. One for production and one for develop.
The production is a "STANDARD SMALL", with auto scale when CPU hits 65-85%.
We database is S2 with 10 gb.
It's a quite simple standard ASP.NET MVC site with some texts, forms and a few remote connections.
The only "non-standard" is 3 million indexed pages that looks into a database (page load is around 1s). This receives a lot of visitors from Google. We also receive some crawls from Google as we have a sitemap with 3mio+ pages.
Data from monitor:
EDIT: data from new monitor.
Web-app:
Database:
The configurations:
Production web-app:
Database:
Our attempts:
1: Always on.
We have tried always on multiple times, but then sometime within the first 30 min to 6 hours, the site just goes down and doesn't come back**. This is of course a huge problem and is not a solution.
2: Running on a VM.
We have a pretty stable and fine setup on a VM on Azure (4gb RAM) which worked ok. We had quite slow responses, but it worked decent. However, we would like to use the web app to "outsource" the scaling and platform to Azure - we just cannot accept this speed :)
**
It goes unresponsive forever until a timeout. I have tried two scenarioes: one where stopping and starting web-app worked, a second where I had to do a redeploy
To help further isolate what the bottleneck is, could you please use the new Ibiza portal at http:portal.azure.com.
The older portal, screenshots above, is showing 54-minute averages. Obviously with 5-minute averages and the DTUs at ~80% there are likely to be periods where all of the DTUs are consumed and that could be the bottleneck.
Using the new portal, these DTU graphs are 15-second averages and this finer granularity could point to the bottleneck. Can you change to the new portal and paste some more graphs?
Thanks Guy
I found a solution.
The solution wasn't just in one place, but in multiple places.
Let me try to dive in.
The main challenge was the 3 million pages we have indexed. Google crawls between 50-150k pages pr day, which we could see in Google Webmaster tools:
99.9% Of these pages were a unique type of address pages. I dived into these, and found out these took 1.5-2s on default (!). It was even slow when running towards test.
Step #1 was to make a new index and optimize the code. 5x Performance improvement there.
Now, that doesn't solve the issue by itself. I also upgraded both the database to the new S3... Didn't solve issue totally (but still better).
I also upgraded our Azure Web App to the 7gb version - and THEN things started to perform.
However, we still had a small issue every 30th minute. I went onto our VM and found an old console job that kept some content in order.... I paused that job.
Neither of these findings could stand alone - but after all these were fixed - we're good again and website responds acceptable!
Hurray!
I have an ASP.NET app running in IIS. The first time a call to the application is made, it can sometimes take extremely long (e.g. 80 seconds), whereas the second time it's very quick
I know this has to do with the app first starting and possibly needing to gather resources etc. However, the problem is that I can run the same identical app on another machine and the load time for the first call is significantly less.
So I'm wondering what factors on the machine would affect this load time?
Thanks for any assistance
I agree with Steve's comment. FYI this slow response on initial request will happen every time the app pool has been idle for a while too. You can combat this by disabling the app pool from shutting down when idle. I think the default is 20 mins, this is a setting in IIS.
Then you will only suffer the problem every time the app pool recycles. You can stop this happening but I don't advise it. Interesting article on this here http://weblogs.asp.net/owscott/archive/2013/04/06/why-is-the-iis-default-app-pool-recycle-set-to-1740-minutes.aspx Recycling the app pool every now and again protects you from memory leaks. However you can pro actively spin up the app pool by setting up a scheduled task that runs a batch file to make a request to the website on detection of an app pool recycle.
This ensures that your site is always spinning and good to go for every request.
ASP.NET only compiles when the page is requested for the first time. This means that on first load the page is being compiled and then displayed. This can be solved by following the precompile instructions from Microsoft.
http://msdn.microsoft.com/en-us/library/ms227972%28v=vs.90%29.aspx
EDIT: I realized that I didn't answer the question that you were really asking.
There are a few things that could affect the first load:
1) The browser you are using may not be as efficient at displaying the type of content on the page (assuming different browsers).
2)If the machines aren't running on the same internet connection(and even if they are speeds vary between wifi/ethernet) this could be affecting the overall speed.
3)The specs on the machines themselves can be making a difference, browsers still take up resources to run, and as such a faster computer could display quicker (although it wouldn't make a giant difference).
4)You said the app was running on IIS, but you didn't specify whether it was a local (test) server or a deployed server. If it's local the specs of the machine again come into play, and in a giant way. Booting an IIS server, deploying the app and then displaying pages (what happens when you click run in VS or similar) can take very different amounts of time depending on the machine.