I have a website hosted under IIS 7 on Window 2008 x64. IIS is running in 64 bit mode and the site has its own Application Pool 64 bit etc. The website appears to run fine most of the time and then all of a sudden each hour it freezes the users request. They don't get a timeout message, it just hangs and appears to wait for about 2-3 minutes before returning the page.
I have monitored the Worker Process on that application pool during and see the processor is at a very steady 25%. Memory is fine and not increasing in any scary way.
I have setup Failed Request Tracing to show me every issue where a request takes more than 30 seconds and yes it records it but with no errors.
Other websites in different application pools on the same server are working fine during the outage.
Any suggestion of how I might debug this issue?
Do you have IIS set to recycle worker processes on that application pool on a given schedule? You indicate you monitored it, but you didn't indicate whether or not you found it to be recycling excessively, just that the memory allocated wasn't increasing in an untoward way.
Do the IIS logs show anything unusual during the time period? Try an app like Fiddler to help debug requests to the web server.
Turn out we were using a control called i-Load to resize images. It has a function to delete temp files after 3 hours. This was locking the IO and causing out entire web-app to halt. Switch it off and all work fine now. Hope this helps someone.
Does the Application depend on a Database that has some Job running every hour?
In case the DB is under heavy load, it would take longer for the queries to execute on your DB and therefor take longer for your web-app to process the pages.
Yes IO process can block the application pool pending thread once not completed other one. So create another thread of IO process and proper handle the cancel token source.
Related
I have an IIS website which connects to MySQL. I have a link in the website which do the intensive calculation. When this link is triggered, it uses around 20% of CPU resource. Then, if I click other links, the other links are responding very slow. Until the link that does calculation finished, then other links responding time will resume. I am curious why only 20% CPU is used, but the other links still respond slowly. It seems the calculation link occupies the CPU resource. But isn't it different link are run in parallel, how it should be affected by intensive calculation of one link? How can I improve that, or should I tune something in IIS of my code?
Increase the maximum number of worker processes in your IIS application pool for your website. This will enable the server handle multiple requests at the same time. Below is how.
Open IIS Manager and Navigate to Application pools.
Right click the application pool for your website and select Advanced Settings.
In the Advanced Settings Dialog Box under Process Model, change the value of maximum worker processes to 10. [You may enter another number (greater than or less than 10)] 1 is default and means only one worker process will be started to handle all your requests. Meaning that only one request will be processed at a time.
Click Ok to save your changes
Restart your application pool [Not mandatory but advised]
Thats all. Your Website should now be able to process multiple requests at the same time
I've been trying to diagnose an issue pertaining to thousands of hung/stuck EndRequest requests in IIS. This is becoming a large problem for us as we're hitting the concurrent connection cap after about a week or two and have to recycle the whole application pool to clear the request list.
Because this is a live application, I have limited troubleshooting options, so anything that would halt or bring down the application pool I am not allowed to do.
IIS Information
Concurrent connection cap is set to its maximum of 65535.
Configuration debug in the web config is set to false and we have a
timeout set at 110 seconds.
Windows Server 2012 R2 Version 6.2 (Build 9200)
IIS Version 8.5.9600.16384
The long running requests have 0 data transfer, checked with
WireShark.
I'm pretty much at a loss on why these aren't timing out. I've set all the appropriate settings - the ones I could find from MSDN and other sources. We have a very, very hard time replicating this on our development environment so it's been blind testing for the most part. I've found articles and such on other state hangs, but I cannnot find anything on why a request in the EndRequest state will not time out.
Advanced Settings Page:
https://postimg.org/image/gxec32kmt/
Application Pool Requests Page:
https://postimg.org/image/qupcw57o5/
Web Config:
https://postimg.org/image/5xt4rh1xh/
Update 1
I did a bit of digging into our fallback that is supposed to close connections after an hour of no usage. We seem to currently have 10,153 sessions still active with a last active time of 3 days ago. I've stepped through this function quite a bit and it seems to be working as intended. It goes through the list of sessions and anyone over an hour of inactivity has their WebSocketHandler.Close() method called. However it seems some sessions are refusing to close after the method is being called. We have logging in place to tell us if any exceptions are being thrown during the run but it seems as though it's running as expected.
This was my mistake. I was running against an old sessions data pull. A current pull of the session data shows no sessions running greater then their specified time. This means that the WebSocketHandler.Close() was called on them and they were removed from our in-memory list.
Update 2
NETSTAT using netstat -s on pastebin: https://pastebin.com/embed_js/qBbZ4gJ1
Update 3
Correction to update 1. Can a connection close be called and fail? If so, then we're accidentally orphaning the reference to the connection in our server. I would still expect the IIS timeout to kick in however, there must be some catches to it collecting requests.
The Problem
I have an ASP.NET web application connecting to an SQL server database.
Occasionally, the application hangs when some specific users try to log in. I can replicate this when I try to log in as that user on the live site from my machine, but the problem doesn't appear in an identical site (the same code, the same machine, the same database and user account - only the worker process and application pool are different).
The problem also disappears (at least temporarily) on an application pool recycle. When it happens again it tends to be the same users.
Normally I'd assume this was a problem in the application-specific code or database, and I'm still looking for a way that that's possible, but the fact it doesn't happen in an identical site running against the same database makes me doubt it.
More information
From some logging, the hang seems to happen after the Page_Load of the requested page, but before the Page_Load of the user controls on that page.
The hanging requests can be seen in the current requests of the site's worker process in IIS, the state shows as ExecuteRequestHandler and the module is the IsapiModule.
The SQL server's activity monitor shows no long running queries during the time that the application is hanging, the profiler shows no frequent identical queries coming from the application during the hang.
Most users can continue to use the application without problems.
The application stores user details with the in process ASP.NET session state.
On some occasions the problem has disappeared on its own after 10-20 minutes, other times it persists until an application pool recycle.
Occasionally the logging shows that after ~5 minutes of hanging after the requested page's Page_Load, the Page_Load of the first child user control is called, after which there is another hang.
Summary
I'm basically wondering if there is any chance that this might be due to a problem with ASP.NET session locking (are there any known bugs, or dangers of deadlock on session?)
Could it be a problem with the request's thread being assigned a low priority for processor time?
If anybody has any ideas about what this could be I'd be interested, if not, then might debugging a memory dump of the worker process help me? I don't have any experience of or know anything about how safe it is to take a memory dump of a live, running worker process.
I have an MVC3 razor page that essentially does this
[HttpPost]
public ActionResult Index(Mymodel model)
{
//run background process
return(model);
}
public backgroundprocesscompleted()
{
//write to database
}
The run background process part runs an analysis via a library then once it is done runs backgroundprocesscompleted. This whole thing works for small files (46 rows) but when I do it on bigger files (11k) it runs on IIS and appears to time out. I have changed numerous settings but still it times out. Essentially what I would like to do is force the web application to never timeout. Any good ideas on how to fix this? Thanks
There are several reasons why a longer-running background process in ASP.Net might fail, including
An unhandled exception in a thread not associated with a request will take down the process.
The AppDomain your site runs in can go down for a number of reasons and take down your background task with it.
When you modify web.config, ASP.NET will recycle the AppDomain
IIS will itself recycle the entire w3wp.exe process every 29 hours (by default)
In a shared hosting environment, many web servers are configured to tear down the application pools after some period of inactivity
http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx
The article linked above has some suggestions for mitigating those risks. However, I would recommend doing long-running processing entirely outside of ASP.Net. One straightforward approach is to write the work into a transactional MSMQ Queue, and have a Windows Service read work items off of that queue and process them.
Intro
I am developing a web application using ASP.NET MVC 3, C#, targeting IIS 7.0+. We have a number of long-running async requests (utilizing AsyncController/Async action features). I also use HttpRuntime.Cache quite often (and in some non-standard way which doesn't really matter here).
Question
Can IIS application pool be suddenly recycled if there are active long-running requests present? I want to prevent that behavior if possible as I don't want to lose data from cache.
Do I really need to use some persistent storage (i.e. Database) to overcome possible issues?
Normally IIS won't recycle an application which has pending requests for it due to period of inactivity. But IIS could recycle your application if you hit memory or CPU tresholds. But this is something that you could configure in your IIS management console.
When a recycle is triggered there is a "shutdown time limit" that defaults to 90 seconds. That's how long processes have to finish before they will be shutdown forcibly. This is configurable through IIS, you likely want to increase this to a large value in addition to removing non-desired recycle triggers (memory, CPU, time, schedule, requests, etc.)