Sudden IIS application's shutdown - c#

I have a multiple versions of an ASP.net website, all of them are working fine except the latest version which recently started to stop suddenly. Each version runs in a different application pool. Our investigations resulted in the following:
Application_End event is getting fired after couple minutes from starting the web application. However, the Application_Start is not being triggered after that.
After the application is stopped, most pages works with no styles, as our styles are bundled using "BundleTable". All bundles return 404 error, but the login process which relies on a web service works fine, and moves me to the home page (with no styles due to 404 response)
Requesting any physical resource (image/js/css/svc) works fine
After the Application_End event get fired, the w3wp process related to this application's pool remains running. Killing it with task manager makes no change. it will start again without causing the application_start to get fired
Stop/restart/recycle IIS/pool doesn't cause the Application_Start to get fired (I'm checking using custom logging). Deleting asp temp files sometimes works but not everytime. Now I've tried everything but couldn't make it start again!
List item
I'm using windows server 2012 64x, with IIS 8.5 and .net V4.5. Any hints are appreciated
Update-1:
Here is a snapshot for the page with errors.
Update-2:
After making few tests and adding more logs here and there, I can summarize the issue in a simpler way:
First the issue shows only and only in the case I enable the "precompile during publishing" option while publishing my website, and merging all output DLLs in single assembly.
After publishing with the above configuration, the Application_Start event will triggered on first page request, which is normal, and the main code I have in this event is to register all CSS & JS bundles, and configure URL routing. The application will work fine for approx. 30 minutes, then the Application_End will be triggered (with "HostingEnvironment initiated shutdown" shutdown reason) but the Application_Start won't be triggered again!. The Strange thing that the application will continue to serve subsequent requests (Ex: connect to database, do logging, authenticate users, serving WCF services requests...) but all bundles and URL routings that are registered in the Application_Start are lost. Another hint that general errors (Ex: 404) which are processed in the global.asax file are not getting processed after the Application_End is triggered! It seems like the application just ignore or forget to process this file again!

The issue is finally solved after fixing all compilation warnings. I can't tell which one has fixed the issue, but thought to share my experience anyways. The warnings found was as for the below:
Removing unused variables (about 10) (shouldn't affect)
Duplicated namespace importing (about 10) (shouldn't affect)
We had an outdated WCF service proxy. So we generated the new one (couple new methods were not added, and some changes on data types. However, this service is usually being called by some external extension that was out of our testing scope
A warning related to a conflict in culture settings (AssemplyInfo.cs-> AssemblyCulture) between referenced projects (also not sure if this could affect, as when doing normal publish without merging pages & controls assemblies we don't have any issue)
Regards,

Related

Dev Tools shows endless network loop for Blazor deployed application

I am working on a blazor application. When I debug locally, I have no issues.
However, when I deploy it to IIS (locally or on a server), the network tab of dev tools in Chrome shows a endless loop.
local screenshot:
deployed in IIS screenshot (thousands of requests and growing):
My question(s): how can I figure out where this call is coming from? Can I turn of the encoding or something?
I added logging to my razor page events, and found the loop.
The OnAfterRenderAsync called another method which included a StateHasChanged. This resulted in an endless loop.
Reading about the component lifecycles here:
https://blazor-university.com/components/component-lifecycles/
We can see OnAfterRenderAsync fires any time the state changes. So I had to basically remove any logic from OnAfterRenderAsync that would cause the state of the component to change.

ASP.NET .NET 4.5 Application crashes in IIS periodically and I can't figure out the cause

I have a .net 4.5 ASP.NET WebAPI application. Deployed in IIS using 1 worker on an 8gig VM with 4 CPUs.
I made changes to it recently (upgraded ServiceStack.Interfaces, ServiceStack.Common, ServiceStack.Redis and a bunch of dependencies) and started noticing that the IIS app pool this app is deployed on recycles about once an hour (give or take a few minutes).
There is nothing in my application logs that show any kind of issues. I collect metrics using telegraf and I do NOT see memory metrics increase at all, as far as all the metrics I look at everything looks absolutely normal and then the app pool recycles.
I looked at the event viewer and filtered the logs by WAS source and see event with ID 5011. Which basically means the IIS worker crash as I understand.
So then I used the DebugDiag and ran it on my local box with the app deployed on my box (I can reproduce the issue locally). It ran for a while and finally got the same event in the event viewer. Looked at the crash analysis logs from DebugDiag and all I see if a bunch of exceptions logged but nothing concrete right before the crash.
At this point I'm not entirely sure what else I can to figure out what's causing the crash so hoping there are more suggestions on what I can do to get more transparency.
What I think is happening is, there is some incompatibility with one of my dependencies and some of the upgraded packages which cause an exception to be thrown which is not handled by anything and crashes the IIS worker.
My application is working perfectly fine, as far as all API endpoints functions wit no issues, memory is NOT increasing, CPU is fine. So as far as I can tell there are no issues upto the crash.
Wondering if anyone knows any tricks to find whats causing the crash and/or handle it, prevent this exception from escaping and crashing the worker.
I was able to narrow down with some confidence that the issue lies somewhere within the ServiceStack.Redis RedisPubSubServer. What is the actual issue, I don't know as that would take a lot more time to dig and I've wasted too much time already.
However, piggybacking on some existing code I had (from before ServiceStack supported sentinel) I created a new implementation of the redis client wrapper for the which I call LazySentinelServiceStackClientWrapper; instead of using the built-in sentinel manager, it relies on a custom sentinel provider which I created LazySentinelApiSentinelProvider this implementation attempts to interrogate the available sentinel hosts in random order for master and slave nodes and then I construct a pool using the retrieved read/write and readonly hosts and this pool is used to run the redis operations. The pool is refreshed whenever an error occurs (after a failover). Opposed to the builtin sentinel manager that comes with ServiceStack.Redis which instantiates Redis pubsub server and listens for messages from sentinel whenever configuration changes such as fail-overs occur and updates the managed redis connection pool.
I installed my version of this redis client wrapper into my application has seen no app pool recycle events since (other than the scheduled ones).
Above is the log of app pool recycle events before I disabled the ServiceStack.Redis sentinel manager.
And here's the log of app pool recycle events after installing my new lazy sentinel manager
The first spike is me recycling the app manually and second one is the scheduled 1am recycle. So clearly the issue is solved.
What is the actual reason why the sentinel manager via redis pub sub server is causing IIS rapid fail protection to fire and recycle the app pool I do not know. Maybe someone with much more redis experience and/or IIS experience can attest to that. Also I did not test this in .net core and only tested for a .net 4.5.1 application deployed in IIS but on many different machines including local development machine and beefy production machines.
Finally one last note, that first image which shows all the recycle events, that's on my CI machine which is barely taking any traffic, maybe 1 request every few minutes. So this means the issue is not some memory leak or some resource exhaustion. Whatever the issue is, it happens regardless of traffic, CPU load, memory load, it just happens periodically.
Needless to say I will not be using the builtin sentinel manager at least for now.

IIS Application Initialization with IIS 10.0 and WebAPI

Hello and good morning everyone,
I ran into an issue where the services once started, are no longer subscribing to the service discovery. In testing they were running under IIS Express. However once I flipped it over to IIS and verified everything was still working; the service no longer hits the Application_Start() within the Global.asax.cs.
Doing a bit of research, it appears that IIS requires a little configuration to get this to do some initialization work once the Application Pool is started. I am running this in a Windows 10 Professional environment with IIS 10.0.17763.1
The goal: Get the service starting up to subscribe to the service discovery was it was running under IIS Express.
I first found on the MSDN the documentation for setting all of this up including prerequisites here IIS 8.0 Application Initialization
After going through all the steps and updating the required config entries, it appears I am still not Initializing my web service when the Application Pool starts.
I have created a new endpoint on the controller for testing purposes to get this to work.
AccountController:
[System.Web.Http.HttpGet]
public IHttpActionResult Init()
{
// here is where I am registering the service thats coming up
}
Route Definition file:
{"Name": "Account - Init",
"RouteTemplate": "api/{controller}/init",
"Defaults": "controller = AccountController, action = Init"}
Web.config
<applicationInitialization skipManagedModules="false" doAppInitAfterRestart="true" remapManagedRequestsTo="/api/account/init">
<add initializationPage="/api/account/init" />
</applicationInitialization>
I am under the assumption that the Initialization will only happen ONCE, and that happens when the Application Pool first starts, or is recycled. IIS internally will make a 'managed' call to the Web Application against the mapped resource it finds in the application's web.config file. Is my understanding of how it should be working incorrect? If so, could someone please clarify why?
I am not simply trying to 'get this to work'. I would like to understand why its 'not' currently working for me.
Thanks to all in advanced who take the time to respond here.
that is correct, your web.config will only be read once when the application starts.
you can add some logging messages to see it happen.
You can actually hit debugger points if you manage to connect your Visual studio to the application pool of the IIS process in time. There is an option to attach the debugger to a process, so you can use that.
This is painfully obvious in systems where the AppSettings are in a separate file, not web.config. if you make a change, nothing will happen, until you touch web.config which triggers an application restart.
Following up,
I fixed my own problem. It appears the code IS working correctly, that its hitting the Global.asax.cs Application_Start as it should. I had an exception being thrown not related to IIS after a change I made which was resulting in what appeared to "not work".
Following the instructions on the MSDN does work.

Auto restart IIS Website from console application

We have an ASP.Net application used by many customers installed on their own servers. Because it is installed on their end with each having different databases and URL bindings etc I created a console application a while ago that gets a zip file and extracts it to c:\inetpub location to append the latest application changes. This console app is added to scheduled tasks to create an automated update.
Obviously when anyone accesses the site for the first time after it does this they have to wait a little longer whilst the site rebuilds. I changed the console application to include a Process.Start(urlofapp) so that it should hopefully do this as part of the update so the next morning that first user doesn't have to wait for the rebuild.
I have tested it ok running myself but not yet released as my concern is that this url process is kept open. Can anyone enlighten me as to whether this would be the case as I don't want this to happen or can give me any ideas as to how to rebuild the site manually as part of the console app.
IIS has had an auto-start apps feature for quite some time. You just have to enable it. You can find more info from the Gu, and the IIS site.
Safer instead to use a start page which loads everything on the server in the background. If you do this logic in an async method which in turn is called from an MVC controller for example then this can potentially run whilst the main page has finished. Alternatively use a threaded Task and show a start page that is only returned whilst setup is taking place.

Displaying a stub page while ASP.NET app starts

Can I display a specific page while ASP.NET application is still in the process of initialization?
We've got a huge ASP.NET application that takes about 30 seconds to complete its Application_Start() handler after each redeploy or AppDomain restart. Showing a nice self-reloading "temporarily unavailable" page during that time will greatly improve the experience for first users.
I tried to extract the initializer code into a different thread, but it has a lot of dependencies: the HttpContext, Server and several other classes are unavailable from derived threads. The code becomes intervowen and incomprehendable, so I'm looking for a better solution. Maybe some IIS configuration or extension?
Well, since IIS 7.5 you can use Application Initialization for that (http://www.iis.net/downloads/microsoft/application-initialization) that will make it possible to either show a "splash-Screen" or to just pre-start Application-Pools.
Here is the explanation for IIS8: http://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-application-initialization It will work similar with IIS7.5, but there you have to install the module beforehand.
And here a link taken from the comments: http://blogs.msdn.com/b/abhisk/archive/2013/08/16/configuring-application-initialization.aspx
You could load a static html page or page that is not effected by the system startup, then via ajax poll/check something (value, etc) that will only be available once the application starts, this way you could have an animation or some loading information on the page.
I think you're looking for something that is impossible. So you want to create an Application_Start that takes a long time to finish, and you want your application to be responsive during that time. Application_Start is a synchronous event, meaning that not a single request will begin processing before it completes. It is its job after all.
You have to loosen up your requirements somehow. Here are some ideas.
A new HTTP layer
As I said, you cannot run .NET code in your application before Application_Start finishes. But you can run another, different application.
So let's create another application that faces the clients. A reverse proxy or similar. I'm sure there are reverse proxy implementations out there which allow you to do some scripting and solve the issue. On the other hand, I have myself written a simple reverse proxy in C#. When it received an HTTP request, it created an HttpWebRequest and relayed it to another URL. It supported GET and POST, it is working in production for years now. If you are interested, I may be able to share some details about it.
The only thing you have to solve is how your backend application communicates with the frontend. That's easy, you can use WCF, IPC, create a simple 0-byte marker file somewhere, anything at all. When your Application_Start starts, you can create that 0-byte file, and when it completes, you delete it.
Note that many things happen after you shut down your application, restart it, and before the Application_Start runs. IIS initialization can take a few seconds. So instead of the 0-byte marker file, which only has 2 states (exists/not exists), you can expose a simple WCF service, with named pipes for example. It can have 3 states: doesn't respond at all (the application is stopped), it responds that the application is starting up, or it responds that the application is running. This WCF service has to be self-hosted and run in the background, because IIS won't respond obviously.
HTML/JS magic
It is basically the same idea as above. It is easier because you don't have to set up another application, but less flexible. You have to create an HTML landing page for your application and you have to make sure that users don't bookmark any other page. If you can do that, you are in luck. You can use the ideas from the above solution about how your application can communicate with the outside world while it's starting up. You can not use simple web services hosted in IIS, but you can host your own WCF services without any problems (create your own ServiceHost, etc.). Of course, this self-hosted WCF service won't run at the same host:port as your application. But it's pretty easy to expose a JSON service in WCF.
So, in your HTML landing page, you write some JS AJAX code that queries your self-hosted JSON service. If there is no response, you can tell the user that the application is not running at all. If the service says that it is starting up, you can tell that to the user. If the service says that the application is running, then you redirect your user.
This will not work if your users are browsing your site already when you shut it down. See, that's why you need an entirely new layer.
IIS magic
This is a somewhat uncharted territory for me. In integrated mode, .NET is an integral part of IIS, so I think it's hard to work around it. In classic mode, however, .NET is run as an ISAPI extension. So you can, in theory, write a new ISAPI extension that runs before .NET. The bad news is that it has to be written in C/C++. But it is obvious, because like I said, you cannot run .NET code before Application_Start finishes. But IIS is not dead during that time, so solve it at the IIS level. This can get ugly and I'm only 99% sure that it is actually possible.
Refactoring
The above are all workarounds. You really should refactor your code so that Application_Start finishes quickly. It is not meant to be a heavyweight function. There is already a framework for preloading ASP.NET apps. If you tell us why you need HttpContext and Server in your initialization code, we are here to help you with those problems.

Categories