Website crashed every two months. App pool recycle fixes the issue - c#

We have a ASP.NET 2.0 (i know it's old!) website hosted on a Windows 2003 server and IIS 6.0. The application periodically crashes after every two months.
We have tried to analyse the memory consumption and other things by using Telerik and very recently Riverbed.
Any idea what should we be looking for?
UPDATE: After searching the WWW I stumbled upon this a fix where they tell that we can increase the request queue limit to stop this issue. Although the root cause analysis is still desirable.

Related

Huge amount of close_wait after .NET CORE webapp recycled on IIS behing Windows NLB

Trying to find out the root cause of this issue for weeks without success, hope someone will get new ideas where to look.
We have an Inprocess .NET Core application running on Windows 2012RC2 with IIS Ver 8.5 (.net core hosting 2.2.6).
This application is duplicated on 2 servers sharing the same IP through Windows NLB.
Both machines are physical with recent HW and 1GB link
The traffic managed by the application is IMHO not that high at all (300 requests per minute !!)
However, randomly when IIS the application pool is recycling, the application is not able to send the ACK from the TCP sequence to acknowledge the CLOSE_WAIT. Ending up with exponential number of close_wait connection close wait comparison (see traffic picture of the 2 servers at issue time)
The server at one point is not able to serve anything anymore.
This will end-up either with the default keepalivetimeout of windows occurs or if we manually recycle again the apppool.
Looking into tcpview tool at that time, the close wait are linked to the system process , so not able to know if it is the new worker process or the closing one which triggered this issue.
What I don't understand is why suddenly the app can't ACK the new close_wait calls ?
Because of the "VIP" (NLB) shared by the servers ? For us configuration and HB traffic seems ok.
Something wrong with IIS and .net core Inprocess model ? Didn't see anyone complaining about that
Any httpclient definition in the code wrongly done ? devs had a look ad didn't find anything wrong there too
Is the application is not able to send the ACK or not able to detect the FIN_ACK2 ?quite hard to see when the close wait increase so quickly, even with wireshark dump on big server
Mitigation was to reduce Windows keepalived time and controlling the recycling at very low traffic time, but I would like to know if anyone has any other idea where to dig into, or had same issue...

ASP.NET .NET 4.5 Application crashes in IIS periodically and I can't figure out the cause

I have a .net 4.5 ASP.NET WebAPI application. Deployed in IIS using 1 worker on an 8gig VM with 4 CPUs.
I made changes to it recently (upgraded ServiceStack.Interfaces, ServiceStack.Common, ServiceStack.Redis and a bunch of dependencies) and started noticing that the IIS app pool this app is deployed on recycles about once an hour (give or take a few minutes).
There is nothing in my application logs that show any kind of issues. I collect metrics using telegraf and I do NOT see memory metrics increase at all, as far as all the metrics I look at everything looks absolutely normal and then the app pool recycles.
I looked at the event viewer and filtered the logs by WAS source and see event with ID 5011. Which basically means the IIS worker crash as I understand.
So then I used the DebugDiag and ran it on my local box with the app deployed on my box (I can reproduce the issue locally). It ran for a while and finally got the same event in the event viewer. Looked at the crash analysis logs from DebugDiag and all I see if a bunch of exceptions logged but nothing concrete right before the crash.
At this point I'm not entirely sure what else I can to figure out what's causing the crash so hoping there are more suggestions on what I can do to get more transparency.
What I think is happening is, there is some incompatibility with one of my dependencies and some of the upgraded packages which cause an exception to be thrown which is not handled by anything and crashes the IIS worker.
My application is working perfectly fine, as far as all API endpoints functions wit no issues, memory is NOT increasing, CPU is fine. So as far as I can tell there are no issues upto the crash.
Wondering if anyone knows any tricks to find whats causing the crash and/or handle it, prevent this exception from escaping and crashing the worker.
I was able to narrow down with some confidence that the issue lies somewhere within the ServiceStack.Redis RedisPubSubServer. What is the actual issue, I don't know as that would take a lot more time to dig and I've wasted too much time already.
However, piggybacking on some existing code I had (from before ServiceStack supported sentinel) I created a new implementation of the redis client wrapper for the which I call LazySentinelServiceStackClientWrapper; instead of using the built-in sentinel manager, it relies on a custom sentinel provider which I created LazySentinelApiSentinelProvider this implementation attempts to interrogate the available sentinel hosts in random order for master and slave nodes and then I construct a pool using the retrieved read/write and readonly hosts and this pool is used to run the redis operations. The pool is refreshed whenever an error occurs (after a failover). Opposed to the builtin sentinel manager that comes with ServiceStack.Redis which instantiates Redis pubsub server and listens for messages from sentinel whenever configuration changes such as fail-overs occur and updates the managed redis connection pool.
I installed my version of this redis client wrapper into my application has seen no app pool recycle events since (other than the scheduled ones).
Above is the log of app pool recycle events before I disabled the ServiceStack.Redis sentinel manager.
And here's the log of app pool recycle events after installing my new lazy sentinel manager
The first spike is me recycling the app manually and second one is the scheduled 1am recycle. So clearly the issue is solved.
What is the actual reason why the sentinel manager via redis pub sub server is causing IIS rapid fail protection to fire and recycle the app pool I do not know. Maybe someone with much more redis experience and/or IIS experience can attest to that. Also I did not test this in .net core and only tested for a .net 4.5.1 application deployed in IIS but on many different machines including local development machine and beefy production machines.
Finally one last note, that first image which shows all the recycle events, that's on my CI machine which is barely taking any traffic, maybe 1 request every few minutes. So this means the issue is not some memory leak or some resource exhaustion. Whatever the issue is, it happens regardless of traffic, CPU load, memory load, it just happens periodically.
Needless to say I will not be using the builtin sentinel manager at least for now.

A worker process serving application pool 'xxx v4.0 (Classic)' has requested a recycle because it reached its private bytes memory limit

I have a shared hosting account with 128MB of RAM and my site is in its own app pool.
The site is small and gets low traffic, but I keep getting the following error:
A worker process serving application pool 'xxx v4.0 (Classic)' has requested a recycle because it reached its private bytes memory limit.
This is happening frequently, which restarts the app pool. If the app pool restarts too often, eventually it will stop. Then I'll get a 503 error when I go to the site.
The site is written using c#, with data access from ef and ado.net. All my database connections are in using statements and I am confident they are being opened and closed correctly.
I have spoken to the host and I can upgrade the RAM to 256MB which does appear to make the site run nicely. But I am a bit concerned that just upgrading the RAM is only masking the problem temporarily.
Debug is set to false in the web config and I before I copy the files to the server I am building for release.
When I run the solution in visual studio my IIS Worker Process hovers around 100 MB.
I think my questions are:
Is there any way I can replicate my hosting environment on my local machine?
Is it normal for a fairly small website to exceed 128MB of RAM?
I am at a bit of a loss of what to try. Any help or guidance would be greatly appreciated.
Other potentially important info:
.NET Framework is 4.5
Web Forms
AjaxControlToolkit is used (only the scripts I need are loaded)
I've looked at many blog posts and similar questions but I can't seem to make any progress.
Thanks
Jim
That message is about hitting the configured limit within IIS itself, it does not necessarily have anything to do with the amount of RAM on the host itself (although the settings you do set within IIS should take your aggregate RAM into account, so there is an indirect link).
Open IIS
Left Click on "Application Pools"
Find your dedicated pool and right click on it, selecting "Recycling..."
Check the "Private memory usage (in KB):" value
That is what you are exceeding
[These instructions are based on IIS 7.5 but are similar for other versions]

ASP.net Web Application Pages Stop Responding (Web Forms) on IIS 8

Well, we have an web application deployed on:
Windows Server 2012
IIS 8.​
Target .net Framework 4.0
DevExpress Ver: 13.1.8 (third-party controls)
Now the issue is this, some of the pages stop responding when we try to load them. (after few time- not specific)
To make them in working again we have to restart 'World Wide Web Publishing Service' or sometime 'IIS' as well.
But after few hours, again the pages stop responding. Browser does't even give any error message, just keep trying
to load page and go in 'Waiting for Server Response' state.
And this behavior is not for all page. Even under this scenario some other pages still work. It means application is running
but some pages are note working.
I have also check that there is no dead lock at database level.
Also, IIS 8 is enable with both 3.5 and 4.5 options.
Note: We have some other applications on the same server running fine.
Can you please suggest me that how can I resolve this issue,
Thanks
Qazi
​
I suggest to create a Hello World! page and load it.
If the Hello World page even doesn't load, analyse the IIS log .
especially look at the value of "Win32Status" . If its not 0, use - net helpmsg [Win32StatusCode] in command line . This should give you enough clue for troubleshooting.
I suspect it is the HTTP Compression that is causing the issue; dump the HTTP Compression in Web Extensions, and then it will behave better.
Check also your CPU and RAM workload.

web service Threads not starting in IIS

I have looked at similar articles on this topic without actually getting a solution to the problem i'm facing.
I have a c# web service that contains two threads and they all started and worked perfectly when tested on the ASP.NET development server on my development machine, which might suggest that the code is in order.
Since I intend to deploy it on IIS 7.5, I decided to test it on a local copy of IIS. On IIS, I noticed that the threads refuse to start. Before I resorted to using threads, I have used System.Timers Timer, and System.Threading Timer with the same behaviour, that is, the Timers worked on my ASP.NET development server but refused to work when tested on IIS. Please, What could be the reason for this and what do I do to solve this bottleneck.
Thanks so much for your contributions.
I just learnt the hard way that I wasn't supposed to use threads in the web service. After removing the threads, it still worked fine.

Categories