I'm using Windows 7, IIS 7.5.7600.16385 and currently .NET 4.6.1 is installed and we have a MVC application.
Some days ago we had some strange behavior at our application. Unfortunately a service which is called inside Application_Start was not available and an unhandled exception was thrown inside. My expected behavior was that the Application_Start() is called again with the next request or the the next request is starting directly with Application_BeginRequest() like mentioned in What happens if an unhandled exception is thrown in Application_Start?.
Unfortunately I get the following result:
In case of exception inside Application_Start() I get an error 500 at the first request. That's ok.
After this all other requests are returning the exception which is thrown at the first request. I verified it by throwing an exception with timestamp inside at my local environment. Each response contains the exception with the timestamp from first request and the HTTP answer is still 500. It has no dependency which url is called. At our code no breakpoint is hit but the IIS log show the requests. It seems that the answer is cached somewhere.
Personally I like the behavior because the application doesn't respond to the requests with undefined initialization status.
And yes I know that calling other service resources inside Application_Start() is not the best idea and we will probably remove it next time :)
My Questions:
Is it possible to configure the behavior in case of an exception is thrown at Application_Start()?
Maybe somebody know when this behavior was changed or does it exists already a long time?
Well I analyzed this scenario and search through many sites, but coould not find any info about it. However, I managed to observe so behavior:
When unhandled error is thrown inside Application_Start then IIS returns error page and web app starts to shut down.
During the shutdown (in my case that was 10 sec.) any new request are handled by IIS and the response is the same as in the first request. If you think about it its logical, because IIS knows that website is shutting down so its obvious that last error cause it.
After some time application raises Application_End event to let know that shutting down is complete. After that event the next request to the website will raise Application_Start again and new response will be generated.
I don't think you can alter this behavior, because application just need some time to restart.
Today I had some time to check the behavior again. We introduced Serilog some releases ago and it seems that the configuration has an effect to the restarting behavior.
protected void Application_Start()
{
SerilogManager.Configure(); //own class
using (LogContext.PushProperty(SerilogManager.PROPERTY_NAME_ComponentName, "xxx")){}
throw new Exception(DateTime.Now.ToString("hh:mm:ss"));
}
If I remove the PushProperty line from Application_Start then the restart will work without any problem. With this line no Application_End will be called.
Now I can reproduce it on private und business computer. Not sure why my demo application didn't call Application_end on my business machine last time.
Related
I am assuming it has something to do with my app pool/IIS but why is this error only being thrown while in Production?
To Start- I have read every related question about this topic on SO.
I cannot reproduce this error in my Dev and/or Test environments. I have sent hundreds of POST requests using postman(DEV), I have also used my iOS application pointing to my Test Environment and sent thousands of requests and I am still in the same boat.
About my Project: I am using WebAPI and my application is running on iOS10 written in swift. I only receive this error occasionally and cannot reproduce it on command. I am wondering if anyone else has run into this issue and if so, what were the steps you took to take care of this problem?
Error #1
The source was not found, but some or all event logs could not be searched.
To create the source, you need permission to read all event logs to make
sure that the new source name is unique. Inaccessible logs: Security.
Note: This error is not causing me to lose any data when I receive it, all records are still being committed to my database. So it is not a huge issue since I only get the error 1 out of about 100 times a new record is submitted. But curiosity is killing me and wanted to see if anyone had any suggestions.
I started seeing errors in a .Net MVC web app hosted on Appharbor whilst a background thread was running - after careful analysis - I can't work out the cause.
Firstly, the exception I noticed is a ThreadAbortException.
However, this is really just signifying that the thread is being killed. Before the thread is killed, you can see a new worker is created by IIS and Application_Start is called on the same machine. Once the new application is up and running, IIS kills the old app and new requests are handled as expected.
At the same time, IIS logs a message of:
ShutDown Message: IIS configuration change
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
ShutDown Stack: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()
at System.Web.Hosting.PipelineRuntime.StopProcessing()
In .Net Health Monitor Logging you get a:
Message: Application is shutting down. Reason: Configuration changed.
Event Detail Code: 50004
A quick google reveals the source code I suspect is the reason for the error:
if (!HostingEnvironment.StopListeningWasCalled && !HostingEnvironment.ShutdownInitiated) {
// If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a configuration change.
HttpRuntime.SetShutdownReason(ApplicationShutdownReason.ConfigurationChange, "IIS configuration change");
}
source: https://github.com/Microsoft/referencesource/blob/master/System.Web/Hosting/IPipelineRuntime.cs
My first thought was to check timestamps for file changes, both in the bin folder and the main application directory - however, this error is thrown without any file changes. Given it only happens on Appharbor, I can't attach to the process and debug that way. I've also monitored memory usage, and don't see any issues there.
The source code states:
If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a
configuration change.
Hence, what else could be causing the error and application recycle, if the web.config / other config files aren't changing?
There are many reasons, which are listed by this helpful blog entry.
Application pool settings
The processModel element of machine.config
Memory limit
Request limit
Timeout
IIS configuration change should happen when something (anything) changes in your IIS application configuration or code. Examples:
Change in web.confg
Change in any dll, aspx, etc...
In any of those cases, IIS application will recycle. In addition, IIS will recycle your process every 29 hours by default, but that will probably not be called "IIS configuration change"
It looks like this was a Microsoft bug.
Unexpected ASP.Net application shutdown after many App_Data file
changes occur on a server that is running Windows Server 2012 R2
Hotfix: https://support.microsoft.com/en-us/kb/3052480
Last Review: 09/08/2015 16:29:00
Once this hotfix was applied, the errors went away!
I may have a problem in understating the behaviour of windows services or the life itself.
THE PROBLEM:
The service stopped unexpedetly and no recovery actions fired despite being set.
The service stopped after ServiceHelper.ChangeStartMode method call
try
{
normalnekurwalogowanie(Constants.Values.service_name);
ServiceController svc = new ServiceController(Constants.Values.service_name);
if (svc != null)
{
ServiceHelper.ChangeStartMode(svc, (automatic ? ServiceStartMode.Automatic : ServiceStartMode.Manual));
svc.Close();
}
else
normalnekurwalogowanie("null");
}
catch (Exception ex)
{
//Logger.Instance.Error("Error message: {0}\nError Stack Trace: {1}", new object[] { ex.Message, ex.StackTrace });
normalnekurwalogowanie(ex.ToString());
}
In my log file there was an error Open Service Manager Error:Unable to open Service Manager
Now, few interesting facts:
- As you can see the exception was caught and printed to the file yet the service stopped
- The error occured only after reboot; it doesn't occur after service installation before system reboot
THE SOLUTION:
After I removed reference to external Logger class (not written by me, I don't have the code) the problem disappeared. I don't know why.
THE QUESTIONS:
How can caught exception still crash the service (and in a way recovery actions are not performed)?
How can code perform differently after reboot? It goes thru exactly same sequence.
Even if the external class may have impact on my code it wasn't called anywhere before the line that threw the exception.BTW the externall class used in winforms app works fine, in service before reboot works fine.
I will try to acquire external class code and update the question.
How can caught exception still crash the service (and in a way recovery actions are not performed)?
The original exception may not crash the service, but if you have a second exception in your first exception handler, the service will crash. I'd check if Logger.Instance.Error() is throwing an exception by putting a try/catch around it.
How can code perform differently after reboot? It goes thru exactly same sequence.
While it might be the same sequence in your code, we don't know what residual state was left over on disk before the last crash. This might be the reason for the difference.
I am fairly new to Windows services. I created an installer for my c# Windows service and the installation on the server (Windows Server 2003) appears to have worked. When it's started, it writes Service started successfully to the log. When it's stopped, it writes Service stopped successfully. However, sometimes the service stops running without writing anything to the log, so I start it back up manually. When I look at the log afterward, it says Service started successfully as expected. It's weird seeing that in the log twice in a row being that it's obviously missing an entry where the service had somehow stopped running.
What could be the potential causes for this? I have the service set up as automatic and installed it to run for all users. I was under the impression that this means the service starts automatically whenever the machine boots up. How can I find out why it stopped? Do services that crash automatically write to the event log or do I have to handle exceptions in such a way that they log their own reason for the crash?
Edit: Some additional info:
I have it set up to log on as Local System Account
Under Recovery options, I have it set up to restart on first failure. I don't have anything for second or subsequent failures.
Update: An answerer recommended a global exception handler. While I won't implement this as a permanent fix, it will at least help me figure out where the problem is occurring. I actually tested this with my installed service and it works. I found out that unhandled exceptions actually do crash the service without writing anything to the log at all. I thought it'd at least report some application error, but it doesn't.
static void Main()
{
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
//other code here
}
static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
Utilities.WriteIt(e.ExceptionObject as Exception);
}
It's always best to handle the exceptions. At least use a global exception handler and write it to a logfile
It sounds like your service is failing unexpectedly without doing any form of exception-handling and/or logging. Windows services do not automatically write exceptions to the Event Log - it's up to you to handle exceptions and (if they're fatal) write them out somewhere so that you can diagnose the problem.
At the very least, I'd recommend a logfile somewhere (perhaps in the service executable folder, or preferably somewhere else that's easy to get to and won't run afoul of permissioning issues) and a standard logging method that all your exception-handlers call to write their messages to.
If a service quits unexpectedly because of some exception, I am not sure it would end up in the Event Log automatically.
I would highly recommend a logging suite like log4net for more thorough logging. You'll be able to provide a multitude of logging 'levels' (debug traces to see if you reached some code, info traces for important events, error traces to log exceptions).
You can look here for an example of a EventLogAppender. However, I would suggest starting with getting a FileAppender, one of the easiest logs to create, working first and then add a second appender for the Event Log.
I am using the Mathematica .Net/Link platform to create a web service to format and calculate math problems. However I am unable to get it working.
I create it using this code:
_Log.IpDebug("Starting the Kernel Link");
if (string.IsNullOrEmpty(_MathLinkArguments))
_InternelKernel = MathLinkFactory.CreateKernelLink();
else
_InternelKernel = MathLinkFactory.CreateKernelLink(_MathLinkArguments);
_Log.IpDebug("Kernel Link Started");
_InternelKernel.WaitAndDiscardAnswer();
The value of _MathLinkArguments is -linkmode launch -linkname \"C:\\Program Files\\Wolfram Research\\Mathematica\\7.0\\Math.exe\".
This piece of code is called from the Application_Start method of the global.asax.cs file.
When it gets to the WaitAndDiscardAnswer() call it gives the server error:
Error code: 11. Connected MathLink program has closed the link, but there might still be data underway.
Note: The SampleCode given with the .NET/Link package (both a console app and a WinForms app) works.
Edit:
I copied the console app sample code given with Mathematica into an asp.net page and it gave me the same error the first load and then on subsequent loads it gave me:
Error code: 1. MathLink connection was lost.
Edit2:
I forgot to mention that when I have procmon and task manager open while running my app, I can tell that Math.exe starts but it immediately exits, which makes those error code make complete sense...but doesn't explain why that happened.
To allow the .Net/Link to work in Asp.net (at least in IIS 7.5) you need to enable the property loadUserProfile on the app pool for the web site.
I am not entirely sure why this is the case, but from what I found while trying to debug this, there are some things that are gotten from the user's profile. I know for a fact that the default location of the kernel is, which explains why I couldn't use it with no arguments, and so I can only assume that other things are needed as well and without the profile it couldn't determine that.
But whatever the reason is this is required, it is, or at least it is a fix if you are getting similar problems like this in your own application.
I got the same error in a .Net WinForm application.
mathKernel = new MathKernel();
mathKernel.Compute("<< XYZ`XYZGraphs`");
The error occurred on loading the package straight after instantiating the MathKernel.
To resolve it you can wait a couple of seconds and then instantiating the MathKernel works fine. During this state where there might still be data underway the following conditions are both false:
if (!MathKernel.IsConnected)
{
MathKernel.Connect();
}
if (MathKernel.IsComputing)
{
MathKernel.Abort();
}
Edit:
I've recieved the error again and this time was able to determine the problem.
Using a command line open the MathKernel.exe and view the error message: