Azure web app have slow and unwarm loads (30s+ load) - c#

I have a very big problem with an Azure Webapp and would like to hear suggestions.
What we experience:
When I goto our website it's quite fast. The average load is around 1 second and respond as expected.
However, once pr. 10-20 minutes, we get a very cold load - where it's 30-60 seconds+.
This would make sense if the website didn't have visitors and it was cold, but we have 10+ visitors online at the website all business hours with 3-5 requests pr minute at a minimum.
This extra load is completely unacceptable of course.
Any ideas?
Our setup:
We have two azure web-apps. One for production and one for develop.
The production is a "STANDARD SMALL", with auto scale when CPU hits 65-85%.
We database is S2 with 10 gb.
It's a quite simple standard ASP.NET MVC site with some texts, forms and a few remote connections.
The only "non-standard" is 3 million indexed pages that looks into a database (page load is around 1s). This receives a lot of visitors from Google. We also receive some crawls from Google as we have a sitemap with 3mio+ pages.
Data from monitor:
EDIT: data from new monitor.
Web-app:
Database:
The configurations:
Production web-app:
Database:
Our attempts:
1: Always on.
We have tried always on multiple times, but then sometime within the first 30 min to 6 hours, the site just goes down and doesn't come back**. This is of course a huge problem and is not a solution.
2: Running on a VM.
We have a pretty stable and fine setup on a VM on Azure (4gb RAM) which worked ok. We had quite slow responses, but it worked decent. However, we would like to use the web app to "outsource" the scaling and platform to Azure - we just cannot accept this speed :)
**
It goes unresponsive forever until a timeout. I have tried two scenarioes: one where stopping and starting web-app worked, a second where I had to do a redeploy

To help further isolate what the bottleneck is, could you please use the new Ibiza portal at http:portal.azure.com.
The older portal, screenshots above, is showing 54-minute averages. Obviously with 5-minute averages and the DTUs at ~80% there are likely to be periods where all of the DTUs are consumed and that could be the bottleneck.
Using the new portal, these DTU graphs are 15-second averages and this finer granularity could point to the bottleneck. Can you change to the new portal and paste some more graphs?
Thanks Guy

I found a solution.
The solution wasn't just in one place, but in multiple places.
Let me try to dive in.
The main challenge was the 3 million pages we have indexed. Google crawls between 50-150k pages pr day, which we could see in Google Webmaster tools:
99.9% Of these pages were a unique type of address pages. I dived into these, and found out these took 1.5-2s on default (!). It was even slow when running towards test.
Step #1 was to make a new index and optimize the code. 5x Performance improvement there.
Now, that doesn't solve the issue by itself. I also upgraded both the database to the new S3... Didn't solve issue totally (but still better).
I also upgraded our Azure Web App to the 7gb version - and THEN things started to perform.
However, we still had a small issue every 30th minute. I went onto our VM and found an old console job that kept some content in order.... I paused that job.
Neither of these findings could stand alone - but after all these were fixed - we're good again and website responds acceptable!
Hurray!

Related

How to track ASP.NET Webforms before Application_Start?

I'm working on a legacy application built on ASP.NET Webforms and .NET Framework 4.6.2.
The application's initial page is very slow to start on my local machine and I'm trying to troubleshoot it. I put a breakpoint on Global.asax Application_Start method, and it's taking quite some time even before the breakpoint is hit. I'd like to know how can I track exactly what happens before the Application_Start event (what calls are made, external libraries used, etc.), from the moment I make the first request to the server, so I can get a starting point to improve the performance. Thanks in advance.
Well, while in VS create (or open) some aspx page with JUST hello world on it. Does that page take a long time to load?
And to be fair, create a brand new blank asp.net webforms project, and find out if that project also is slow to load.
In other words, this just might mean your computer is slow.
When you hit f5, then:
The project code is compiled. Then a bunch of information is "copied", setup, then configured for IIS to launch. Then a WHOLE web server is launched (IIS express), then the web page is loaded, and then IIS processes that page, and then it is rendered.
In other words, a BOATLOAD of startup stuff occurs. Once that WHOLE big web server is spooled up, then and only then does your application start and all that jazz start.
If you can navigate to another page (say from the menu bar (assuming your application has one), and then BACK to that supposed page that loads slow? Does that page load slow the 2nd time you hit that page?
If that is the case, then your computer might be slow, starved for memory, or even starved for processing to start up the WHOLE system of stuff required to run that whole web server.
if once the page is displayed, and everything runs normal, then this is not a slow loading web page, but in fact a slow load WHOLE web site, and in actual use and production of the system, then it not really slow anymore is it?
So, say if your computer only has 8 gig, and say a low or slow laptop processor, then startup of all the "enormous" moving parts required to run a whole web server system might be the issue, and NOT that the page load is really slow.
And since the production system will not be re-loading from 100% scratch each time a whole web server system, then you may well not have a slow system, but only a slow "first time" startup.
So, do other web site projects based on .net framework, and asp.net pages start out slow?
And is this a asp.net web site, or is this a asp.net web site (application) project?
So, quite a bit of "hand stands" will occur on first run and load of the web site. So, this could be your computer, and not really that the web site is slow.
Having stated the above?
I have some found web site projects RATHER slow to load, and this is especially in the case of a asp.net web site application. (as opposed to web site's, which tend to load a WHOLE lot faster - about 1 second, and the other was taking about 18-23 seconds on my old laptop (it was starved for processing).
On a older laptop, I was seeing about 18-23 seconds delay after hitting f5. On my newer laptop, I have about 2, sometimes 3 seconds delay from the time I hit f5 in code or markup to see the results and web site running.
So, often IIS has to be loaded and started, maybe SQL server started, and even more bits and parts. All of these things can take a LONG time on your dev box, but on a production server, all of those things are running all the time.
So, it not clear if you have a very slow web site, slow loading "specific" web page, or you just talking about the overall slow loading time for "any" first page to display, and then after that, everything is fine? In other words, it is f5 to start things going is the slow point in time.
And you didn't even note how long of a delay and time here?
Are you talking about 5 seconds, or 50 seconds?
Right click on your task bar, task manager.
Here is me hitting f5 on my laptop. Note the 6 cores go 100% max speed and power during that startup.
I do know that on my laptop, when I hit f5, then all 6 cores (12 threads) jumps about about 2ghz to 4 ghz, and all 6 cores are pinned for those 2-3 seconds while that whole web server is created, configured, and THEN started, and then the page in question has to load.
So, here you can see that hitting f5, the web page "starts" almost instant, and then about 4 seconds more.
(the processor usage is somewhat high - part of that is due to running the screen capture for this screen shot).
So, as "soon" as I hit f5, you can see the processor jump above 20%. I count about < 1 second for the browser to appear. Then about 4 more seconds. And the above is a slower to load page. And as soon as the page displays, then you see the cpu drop down.
As noted, try creating a new asp.net web site "application". A web site, they tend to load < 1 second. In fact above is the "by far" slowest startup I have. Most are about 1-2 seconds from hitting f5.
A asp.net web site application? They take longer - the whole site has to be compiled each time, and the .dll's have to be created.
Sometimes, it simple things like some virtual folder mapped to a location that does not exist (so on startup, a non existing folder or path name can take some time to "time out".
so, how fast does a new test project (create a new asp.net web site application (create a application, not a web site). I find that web sites load fast - even on computers with limited hardware.
However, asp.net web "applications" tend to load a WHOLE lot slower. (and thus more ram, and more cpu, and more cpu cores makes a HUGE difference in startup times).
Performance is a real art form in our industry. But, a few more details such as do all "application" sites start slow on your dev box, or JUST this one, and by how much compared to other sites.
Often, a delay can be network connection to a database, and say little or nothing to do with startup, but some network delay in say connecting to sql server which may well be some kind of network connection.
So, all first pages slow, or just one page?
does a new asp.net web site application (application is important here - not just web site) also startup slow? (if yes, then this is not specific to your site, is it???).
And using task manager - do you see high cpu usage during that startup (f5), or is it low? (this then suggests network and connection delays - not cpu and memory and performance issues.
So, a lot more trouble shooting here is required, but then again, we have little idea as to what testing you done so far, such as testing another web site application on your computer - and does it have things like authentication and sql server logon's involved.
I mean, we can have some issues with a car that will not start. So, we don't just yet swap out the engine, re-do the fuel injection system, and then ONLY to find out we forgot to put in a new battery, and that was the issue all along.
So, do a few tests - get some base-lines of say a test site - use one of the templates that has the "default" bootstrap menu, master pages etc. How fast does that start? Since if that site ALSO starts slow, then attempts at troubleshooting the existing site will be in vain, since all sites load slow - not just the "one" site your working on.

Asp.Net Core Razor Pages Site Very Slow

I created my WebSite on Asp.Net Core and I hosted it on SmarterAsp.Net
http://greensaucegames.com/
I don't know what the problem is, if it was something I did, or if the hosting server is slow.
But the site is very slow, sometimes it doesn't even open the page I just clicked, the one from nowhere it reloads.
Does anyone know a way for me to try to diagnose what may be making it so slow and intermittent? How do you do it? Or some advice on how can I improve the speed?
Are Razor Pages really slow? Should I use something else?
I do use Blazor for the Admin Area, can this be interfering with the rest of the site?
Because your hosting subscription plan is very cheap. Use a high resources server will see acceptable performance.
I went to SmarterAsp.Net's homepage, and even IT loaded very slowly. I clicked on their "Why Us" link and watched in horror as a medium-sized image scanned in over about 15 seconds. The words "total garbage" immediately came to mind.
Then I tried to open one of your asset images in a new browser tab. Just the image itself, which is not a large one, took like 20 seconds to load.
Conclusion-- 100% chance that the shared hosting you have is the problem.
When I first put up a real business website, it took me quite a while to realize that shared hosting is a no-go. Even if the site works great at first, new users will come to that host, and soon enough you'll be stuck in glue again. Upgrading to "premium" packages and so on didn't help either.
Then I opened up my wallet, shelled out for an Azure Windows Server VM, and added a couple years to my life thanks to my immediate and total lack of stress. It's nice when everything just works.
There are many factors that make your site working slowly. For example
How effective is your code
How effective is your SQL query
Have you optimized your image
If you believe that you have optimized your site, then you need to check your external factors like your hosting provider. You may need to upgrade your plan to higher plan to get more stable server. Their hosting service is cheap but it might impact to your webiste performance. To test it, you can upgrade to higher plan. If your site still working slowly, you may need to find other hosting provider. I personally use Asphostportal.com and I can recommend them if you need .NET Core hosting.

ASP.NET MVC Slow start-up

Sometimes my MVC 4 application starts very slowly, but all the following requests come up quickly. It's running on IIS 8 and it uses Forms authentication.
The first start-up might take 20 seconds or so. I'm not 100% sure how long does it take to get a slow start-up again, but I guess it's more than an hour.
It's the same issue as described here:
MVC slow if site has been idle
So checking out the Application Pool recycling thing, I stopped the application pool, started it again, then browsed to the address, but it still came up quickly. I then ran the Powershell command (Get-Process -Id ).StartTime on the IIS, and it told me that last recycling for this application pool was when I started it.
I suppose that exludes the pool recycling?
The project is using Devexpress MVC layout, and I have removed all the assemblies/references that I don't need, but I didn't notice much difference afterwards.
The other applications on this IIS are made with Web forms, and they always come up quickly. The other applications also don't have the Forms authentication.
As a workaround I'm about to make a service that opens the address every 30 minutes or so, but still would be interested to figure out the real cause.
Any ideas?
Its may happen sometimes if configuration is not made properly.
You can use following techniques for improving performance.
1. Enable compression;
2. Optimize caching;
3. Optimize CSS;
4. Optimize HTML;
5. Optimize images;
6. Optimize callback management;
7. Optimize data management.
Please find a link here for more details how can we improve performance of website while using devexpress.
Devexpress support describes a lot about each small points regarding performance.
https://www.devexpress.com/Support/Center/Question/Details/K18541

asp.net crash: IIS8 Requests Execting increases and CPU stays low

I'm stuck figuring out what will be the next trouble shooting step to take or research.
I've got one web application running on one IIS8 webserver. Lately we are struggling with the performance of the webapplication. This morning the webapplication 'crasht' again. With this I mean that it didn't respond to any new requests.
Because Perfmon is your friend, I fired off the following counters:
Processor Time (%)
Requests Executing
Requests/Sec
See the image I've included.
What I find interessting is that "Requests Executing" only increased at this point. And... the CPU was not running 100% to get rid of all these requests.
As I'm the main developer of this web application, I already optimized many high-CPU webpages.
With this SO thread I'm hoping to find the bottle neck. Maybe with tips for extra logging. Application code analysis etc. I can provide more information about the setup or application when needed.
Hope you can help. Many thanks in advance.
Some specifications:
Windows Server 2012R2
IIS8
Intel Xeon Quad core CPU
Total of 4 gig memory
ASP.net 4.0 web application
User specs:
Google Analytics Real Time visitors between 100-500
Normal requests/sec average of 30
'Peak' requests/sec of 200 / 300
EDIT:
I run a Perfmon Data Collection Set for about a hour. Exactly at the moment when the website crashed. I used the tool PAL to analyze it. It only had many warnings about the memory being below 5%.
This memory issue is clearly an issue we will be resolving soon.
Another thing I noticed is that the list of "current requests" in IIS8 was enormous. It contained hundreds of "current requests". I would expect these requests to reach a time out value and send a request Timed Out to the users.
Here is a print-screen of the current healthy situation:
This list was infinite long.
And, again another thing I just noticed, was one request taking more then 10 minutes(!) to deliver a byte[] to a user. The 'state' was SendResponse. I persume this user was on a low-bandwitdh device. As long as this user is downloading - one workerprocess is taken. How should we prepare this long pending requests?
Have a look at PAL at https://pal.codeplex.com/ - this will guide you in collecting a number of stats and then analyse them for you.
Along with CPU, other bottlenecks are memory, network, and disc IO. PAL should help you in collecting the appropriate stats - it has several pre-defined sets for e.g. Web Server, DB Server, etc.

Sitecore - No App Pool Recycle

I am having a rather time-consuming issue with the Sitecore CMS.
Is it possible to shorten the time it takes for the application pool to recycle every time a change is made in the bin folder or web.config?
It takes the server 2 to 5 minutes to respond right after a change.
Any ideas?
There is one article from alex shyba from sitecore here on Reducing Sitecore Startup time
Summary of article is
In machine.config: disable process to check assembly signed
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
Disable performance counters in Sitecore web.config
<setting name="Counters.Enabled" value="false" />
There are plenty of articles on the improving sitecore performance I have listed few links below:
Sitecore SDN - Optimizing Performance in Sitecore
Sitecore Slides on performance improvements - download pdf
Analyzing and identifying the slow causing culprit - Sitecore Startup basics
He needs to restart the application pool in order to load the new dll's in.
You could try and minimize the things you do at startup(but i'm guessing it's mostly sitecore stuff which you can't influence) so the best suggestion i can give is to have 2 webservers with a content switch.
You would run your application on 2 servers and the content switch decides which server handles which request(be carefull with session and statics because each server will know nothing of the other).
If at some point you need to release a new version you just instruct your content switch to direct all trafic to webserver A. You then deploy to webserver B, open the website via a direct url that doesn't pass the content switch and make sure it's working properly + warmed up.
Then you tell the content switch to point all trafic to B and you have all the time in the world to update webserver A and switch the content switch back to normal mode.
I had this issue as well.
For whatever reason my local copy would take around 5 minutes to fully spin up after any change or starting the website fresh. It was absolutely dreadful for testing anything. It was only an issue locally. My development server environment and production didn't seem to be effected.
I found out a profiling tool that allowed me to find an outlier control that was taking an obscene amount of time to render. Try this...
Fire up your website after it compiles.
Log into this admin page: http://local.example.com/sitecore/admin/stats.aspx
You'll see a screen like this: Look for anything with an obscenely high Avg or Max time. Your problem will likely be in the logic there.
As a bonus, if you can keep hitting pages after your long recompile without much trouble, you should notice the Avg time roughly halving itself every page hit for your problem control if your problem is anything like mine was.
In my case, I found the largest issue came from one control that was doing an expensive search for one item.
It looked like this -> Sitecore.Context.Database.SelectItems("" + Sitecore.Context.Item.Paths.Path + "/ancestor-or-self::*[##templateid='" + templateId + "']");
This was mostly a local issue because the SQL server is remote to my machine, while the other servers were in the same building. Hence dev and production were relatively unaffected.
Good luck!
You can follow Alex Shyba's post here
But 2-5 minutes sounds extreme. This isn't normal. Are your hardware up to date?
Also I would look in to your prefetch cache. In your development environment you probably don't want i to fetch to much on start up, as this isn't needed in development.
I would also look into the initialize pipeline and global.asax to see if you are doing any custom start up jobs.
You can reduce the pre-fetch cache size on core, master, and web databases, but this is not recommended, do this only on development machines, not on production servers.

Categories