I've been trying to diagnose an issue pertaining to thousands of hung/stuck EndRequest requests in IIS. This is becoming a large problem for us as we're hitting the concurrent connection cap after about a week or two and have to recycle the whole application pool to clear the request list.
Because this is a live application, I have limited troubleshooting options, so anything that would halt or bring down the application pool I am not allowed to do.
IIS Information
Concurrent connection cap is set to its maximum of 65535.
Configuration debug in the web config is set to false and we have a
timeout set at 110 seconds.
Windows Server 2012 R2 Version 6.2 (Build 9200)
IIS Version 8.5.9600.16384
The long running requests have 0 data transfer, checked with
WireShark.
I'm pretty much at a loss on why these aren't timing out. I've set all the appropriate settings - the ones I could find from MSDN and other sources. We have a very, very hard time replicating this on our development environment so it's been blind testing for the most part. I've found articles and such on other state hangs, but I cannnot find anything on why a request in the EndRequest state will not time out.
Advanced Settings Page:
https://postimg.org/image/gxec32kmt/
Application Pool Requests Page:
https://postimg.org/image/qupcw57o5/
Web Config:
https://postimg.org/image/5xt4rh1xh/
Update 1
I did a bit of digging into our fallback that is supposed to close connections after an hour of no usage. We seem to currently have 10,153 sessions still active with a last active time of 3 days ago. I've stepped through this function quite a bit and it seems to be working as intended. It goes through the list of sessions and anyone over an hour of inactivity has their WebSocketHandler.Close() method called. However it seems some sessions are refusing to close after the method is being called. We have logging in place to tell us if any exceptions are being thrown during the run but it seems as though it's running as expected.
This was my mistake. I was running against an old sessions data pull. A current pull of the session data shows no sessions running greater then their specified time. This means that the WebSocketHandler.Close() was called on them and they were removed from our in-memory list.
Update 2
NETSTAT using netstat -s on pastebin: https://pastebin.com/embed_js/qBbZ4gJ1
Update 3
Correction to update 1. Can a connection close be called and fail? If so, then we're accidentally orphaning the reference to the connection in our server. I would still expect the IIS timeout to kick in however, there must be some catches to it collecting requests.
Related
I have an application where it seems that for different users there is a different timeout and for some users the timeout is too soon.
It is a C#/.NET application that runs an ASPX based Website. Basically, you fill out a form and click on submit and then must wait a few minutes and you get the results. When I am connected to the system and perform this action it takes around nine minutes and I see the results.
But if another user performs this action after five minutes he gets displayed a “This page can’t be displayed” message in is browser (internet explorer – the rest of the message is “Make sure the web address http:// … is correct, Look for the page with bing, Refresh the page in a few minutes”).
A little more background:
First the nine minutes seem long, but there is a huge amount of data (collection of datapoints over a span of a year) that is processed and displayed, and basically it works, so I don’t think it is necessary to discuss this point. For a lesser amount of data there are no problems.
The data from the database is obtained via C# and it performs the queries to a MS SQL database.
First I thought it would help to set in the web.config in the httpRuntime tag the executionTimeout attribute. But the compilation tag has the debug flag set to true (and the application is deployed as release via visual studio on the server) – so the debug flag (as far I understand) overrides the executionTimeout anyways.
The Server on which the application is running is Windows Server 2012 R2 and for the site I also set under IIS in the advanced settings in the behavior / limits area the connection timeout on a higher value. But this hasn’t any effect.
I think I am missing some point, because as far I understand the deployed application doesn’t have any timeouts set (because of the debug flag) and this behavior seems user specific.
Do you have any hints or ideas where I can look for?
Edit
In the comments was suggested to check the SQL Server logs for errors, the log was error free.
As it turned out, it was a problem due to an older version of the browser that had some strict timeout settings. By switching to another version, this behavior doesn't show anymore.
I am running a ASP.NET 4.5 Web API application on IIS 8.5. Under worker process, I am seeing certain request are queued up by IIS and are never being served. This could be a memory leak issue. I am still investigating. During investigation I read about thread pool settings.
The article talks about Recommended Threading Settings for Reducing Contention. The article says make these changes in machine.config. My question is can I make these changes in web.config ?
Do we have any other recommendation for fixing he queuing issue.
Machine.config specify the configuration for all .Net applications in this windows instance. Web.config specific the configuration for specific application.
If this issue only happen in single IIS web application, you could specify the configuration in Web.config. Otherwise, it is suggested to set them in Machine.config.
In my experience, Your problem should be an application pool hang or low performance issue if request get stuck in queue.
First of all, please check whether any error message was logged in IIS application or system event log.
To troubleshooting performance issue, please try to capture dump file with Procdump or debug diagnostic tool.
We need to check these managed stack traces via these dump file. So that we got to know which method or request are getting slow or hung.
With WINDBG mex extension, we will know：
how many request are being processed
How's the status of each thread
Is there any thread get frozen or dead locked
If there is a deadlock, Which thread is get locked and what's the address of your dead lock.
If we need to know which configuration or solution should be applied in your server, find the root cause or characteristic is also necessary.
If you don't know how to Analyze dump file, Debug diagnostic analysis tool or WINDBG analyze -v command would help.
It's something in your application code that is hanging or just running too long.
The request monitor in IIS Manager shows all currently running requests - not just requests that have not been served yet. I confirmed this by running a project in Visual Studio that I have setup to debug in IIS. I set a breakpoint in Visual Studio and initiated a request in my browser. When it hit that breakpoint, the request monitor showed that request and the time kept climbing as long as I did not continue execution of the code.
While sitting at the breakpoint, the "State" I saw in IIS Manager was "ExecuteRequestHandler", so if that's the state you see, then the request is surely being served by your application, and it's your application being slow.
If it only happens with one specific API call, you can look through your code for possible deadlocks or long-running queries. Jokies' answer might help you pinpoint where and why easier (maybe). If it makes SQL queries, you could also look at pending queries on your SQL server to see what it's doing.
(Side note: I didn't even know that request monitor existed before this, so this was interesting to look into!)
Update: You can enabled Failed-Request Tracing in IIS to hunt down exactly where it's stalling. Create a timed rule (log a trace when requests take longer than x seconds). The only downside is that it has to change your web.config to enable this, so it'll recycle your app pool, which may or may not restrict when you can do this.
There is more information in this article about tracing long-running requests, including that you can modify your code to use Trace.Write to write information from your code into the traces that IIS picks up.
This question is really around analysis - how do I analyze what's causing the problem?
Situation - we have a C# webservice configured through IIS 7.5, and a website in the same intranet domain hitting the webservice with POST and GET methods. Server is windows 2008 r2 64-bit, C# is 4.0. The first call of the day is slow (30-60 seconds), though I have not checked if I try again later in the day is it slow as well. Subsequent calls are 2-3 seconds. When checked using FireFox web console or firebug, the time is spent on "Waiting" for the webservice.
Things I've tried :-
Setting no recycle time for the webservice AppPool
Setting no idle time-out for the webservice AppPool
Setting proxy bypassonlocal = true and usesystemdefault = false in case it's a proxy look up issue
Nothing's worked so far. My thought is that even if it's C# compile to machine code on first run, it shouldn't 'expire' if the AppPool does not time-out or recycle, yet it is slow everyday.
So flat out of options, how do I go about trying to find the source of the problem? Any diagnostics I can run on the server to check what the webservice is doing?
From your description it sounds as if your webservice might be going to sleep given the first call after a period of inactivity is slow and subsequent calls are faster.
I've seen this behaviour occur on many of my .NET IIS applications, can be frustrating to say the least!
This is however, default .NET behaviour, but there are ways of keeping your application awake, especially as of .NET4.
I refer you to the following article as a first step. Give it a go and see if this makes any difference for you:
https://www.simple-talk.com/blogs/2013/03/05/speeding-up-your-application-with-the-iis-auto-start-feature/
Good luck!
(Sorry if this is a really long question, it said to be specific)
The company I work for has a number of sites, which have been running for some time with no problems. The applications are a mix of ASP.NET 2.0, 3.5, and 4.0, all using an ADO.NET to connect to a SQL Server Standard instance (on the same webserver) all being hosted with IIS7.
The problem began when we moved to an upgraded webserver. We made every effort to set up the server, db instance and IIS with the exact same settings (except for the different machine name, and the fact that we had upgraded from SQLExpress to Standard), and as far as we could tell, we did. Both servers are running Windows Server 2008 R2 (all current updates applied), and received a default install.
The problem is very apparent when starting up one of these applications. When you reach the login page of our application, the page itself loads extremely fast. This is true even when you load the page from a new machine that could not possibly have the page cached, with IIS caching disabled. The problem is actually visible when you enter your login information and click the login button. Because of the (not great)design of our databases, the login process must access a number of databases, theoretically up to 150 separate DBs, but in practice usually 2. The problem occurs even when only 2 databases (the minimum) are opened. Not a great design, but we have to live with it for now.
When trying to initially open a connection to the database, the entire process stops for about 20 seconds every time, regardless of whether you are connecting to 2 dbs or 40. I have run a .NET profiler (jetbrains dottrace) against the process, and the only information I could take from it was that one or all of the calls to sqlconnection.open() was accounting for 90% of the time. This only happens on first-use of the application, but the problem is compounded by the fact that IIS seems to disregard the recycling settings we have set for it, and recycles the application after a few minutes of idle, causing the problem to occur again.
I also tried to use the SQL Server profiler to see which database operations were the cause of the slowdown, but because of all the other DB activity, (and the fact that I had to do this on our production server, because the problem doesnt occur in our test environments) I couldn't pin down the exact operation that was causing the stoppage. I will try coming in late at night and shutting down the production sites to run the SQL profiler, but I might not be able to do this right away.
In the course of researching the problem, I have tried a couple solutions
Thinking it might be a name resolution problem, I tried modifiying both the hosts file on the webserver as well as giving the connectionstrings an IP address instead of the servername to resolve, with no difference. I have heard of the LLMNR protocol causing problems like this, but I think trying to connect by IP or resolving with the hosts file should have eliminated that possibility, tho i admit I never tried actually turning off LLMNR.
I have increased the idle timeouts, recycling intervals etc in IIS, but this doesn't even seem to be respected, much less solving the problem. This leads me to believe there is a setting overriding the IIS application settings on the machine.
multiple other code fixes, none of which made any difference. Is a SqlServer setting causing the problem?
other stuff that i forgot by now.
Any ideas, experience or whatevers would be greatly appreciated in helping me solve this problem!
I would advise using a non-tcp connection if you are still running the SQL instance on the local machine. SQL Server supports several protocols, tcp, named pipes, and shared memory are the more common.
Named Pipes
Data Source=np:computer\instance
Shared Memory
Data Source=lpc:computer\instance
Personally I prefer the Shared Memory. Remember you need to enable these protocols, and to avoid configuration mistakes I suggest you disable all you are not using.
see http://msdn.microsoft.com/en-us/library/ms187892.aspx
IIS Reset
In IIS7 there are two ways to configure the idle-timeout. Both begin by clicking on the "Application Pools" section and right-clicking the appropriate app domain. If you click the "Recycling..." option there is one setting. The other is in "Advanced Settings..." under the section for "Process Model" you will find "Idle Time-out (minutes)" which set to zero disables the process timeout. This later option is the one that works for us.
If I were you I'd solve this problem first as restarting the appdomain and/or worker process is always painful even if you don't have a 20 second lag.
Some ideas:
from the web server, can you ping the db server and get a "normal"
response, or are you seeing a similar delay?
if you're seeing a delay, run a tracert to see if you can nail down where the slowness is occurring
try using a tool like QueryExpress (http://www.albahari.com/queryexpress.aspx) which doesn't require an install to run. You can download this EXE and run it from your web server. See if you can connect to your db using this and run queries in a normal fashion.
Try something like SysInternals' TcpView (http://technet.microsoft.com/en-us/sysinternals/bb897437) to take a look at your open connections and see what activity is happening on your server and how much data is being sent to and received from your db server.
Just some initial thoughts on where I'd start to look based upon your problem description. I hope this helps. Good luck with things!
With IIS not respecting recycling settings: did restarting IIS/rebooting change the behavior?
I have a website hosted under IIS 7 on Window 2008 x64. IIS is running in 64 bit mode and the site has its own Application Pool 64 bit etc. The website appears to run fine most of the time and then all of a sudden each hour it freezes the users request. They don't get a timeout message, it just hangs and appears to wait for about 2-3 minutes before returning the page.
I have monitored the Worker Process on that application pool during and see the processor is at a very steady 25%. Memory is fine and not increasing in any scary way.
I have setup Failed Request Tracing to show me every issue where a request takes more than 30 seconds and yes it records it but with no errors.
Other websites in different application pools on the same server are working fine during the outage.
Any suggestion of how I might debug this issue?
Do you have IIS set to recycle worker processes on that application pool on a given schedule? You indicate you monitored it, but you didn't indicate whether or not you found it to be recycling excessively, just that the memory allocated wasn't increasing in an untoward way.
Do the IIS logs show anything unusual during the time period? Try an app like Fiddler to help debug requests to the web server.
Turn out we were using a control called i-Load to resize images. It has a function to delete temp files after 3 hours. This was locking the IO and causing out entire web-app to halt. Switch it off and all work fine now. Hope this helps someone.
Does the Application depend on a Database that has some Job running every hour?
In case the DB is under heavy load, it would take longer for the queries to execute on your DB and therefor take longer for your web-app to process the pages.
Yes IO process can block the application pool pending thread once not completed other one. So create another thread of IO process and proper handle the cancel token source.