I have a webservice on one of my websites which restarts the websites app pool.
The problem Im getting is the IIS7 app pool keeps stopping with the following warning and then error:
Warning:
A process serving application pool 'appPoolNameHere' exceeded time limits during shut down.
Error:
Application pool 'appPoolNameHere' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
The code I am using to do this is:
try
{
var serverManager = new ServerManager();
var currentPoolName = SettingsManager.AppPoolName;
if (!string.IsNullOrEmpty(currentPoolName))
serverManager.ApplicationPools[currentPoolName].Recycle();
HttpRuntime.Close();
}
catch (Exception ex)
{
var exceptionManager = ExceptionManagerFactory.GetExceptionManager();
exceptionManager.LogException(ExceptionManager.Severities.Critical, ex,
"App Pool restart failed");
}
I am not getting any exceptions being thrown here, but the app pool is stopping, this doesnt happen all the time and is very hard to replicate, I have tried several different things even just hitting the hell out of that service and sometimes its fine other times it dies.
Is there anything I can do to fix this?
The reason I have to restart it is due to an external config file to the webconfig which deals with the url redirects which is updated from the admin system.
Thanks for any help with this.
Well I seem to have sorted (not fixed) the issue, with some tape and sticky-back-plastic.
The solution
Changed app pool setting: Rapid-Fail Protection to false
This stops IIS from killing the app pool after several failed requests to it.
I also added in retry system, so if the recycle request fails it waits for 10secs and tries again, it will do this 5 times before giving up and sending out an exception notice.
This has fixed the problem, and everything seems to be working well again, although something about this screams this is wrong!! at me.
Only if IIS would monitor other files in a website and restart the app pool when it saw a change.
I didn't work with Microsoft.Web.Administration namespace, but what about unloading application domain and sending request to your website so that to start application again?
Good example (2nd point) by Peter Bromberg how to do this.
Update.
Reading error stack
System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
at Microsoft.Web.Administration.Interop.IAppHostMethodInstance.Execute()
I thought of most common reason: insufficient rights. All administrative actions require administrative rights. When code above is executed, which user account is impersonated? IUSR? Switch to any admin for that block and it should work without exception.
Related
I have a windows service run with local system account
What my program does is :
if (File.Exists(outputPath))
{
File.Delete(outputPath);
}
File.Move(archivePath, outputPath);
Relevant folder is application folder of iis where its application pool's identity is ApplicationPoolIdentity located under c:\MyAppFolder.
My windows service does its few times a day, and my clients checks if any new version exists every 5 minutes(0,5,10,15...) and download that file.
Time to time, file is somehow get "locked" on filesystem then
iis gives 401 error
File cannot be deleted
My first question how can I repro this situation?
One patch is done by colleagues is:
var fs = File.GetAccessControl(outputPath);
fs.SetAccessRuleProtection(false, false);
File.SetAccessControl(outputPath, fs);
Although this patch, it seems error occured again,
I may apply, this solution as well.
Are those solutions are enough or necessary?
Again my first question is important "repro issue" and understand why this happens.
The C# console application running .NET 4.5.2 (app1) opens a COM application (app2) and does some work with that app2's API. So far all of the work is successful, but sometimes when app1 attempts to close app2, app2 hangs permanently.
If the process for app2 is ended with task manager then app1 reports access denied. Does that occur because the terminated process is no longer available or does it occur because it was blocking a thread in app1 and it was unable to report the error until the thread was allowed to continue?
The code used to terminate app2 is
private static void CloseSW(SldWorks swApp, Process sw_proces)
{
// Close with API call
if (Task.Run(() => { swApp.CloseAllDocuments(true); swApp.ExitApp(); }).Wait(TimeSpan.FromSeconds(20)))
return;
// Kill process if API call failed
if (Task.Run(() => { SWHelper.CloseSW(sw_proces); }).Wait(TimeSpan.FromSeconds(20)))
return;
// Unable to close SolidWorks, ignore error and continue
// This will eventually cause SolidWorks to crash and the crash handler will take over
}
This code should never take much more than 40 seconds to complete, but maybe the COM interop is causing some unexpected behaviour?
I am unable to reproduce this error on a development machine. What it the best way to trace the exact point of failure? It is possible that the failure is not in CloseSW but some point before this. Is there a better way to trace the error than to write each line to a log file?
It is also worth noting that this code works for 60 - 150 runs before it has any errors and both applications are closed between each run.
I have control of the remote environment so remote debugging is an option, but I've never set that up before.
Typically with COM interops causing issues is that IIS is having issues with the object using the current ISAPI.dll. Please verify that your permissions are configured within your assembly to work with your current version of IIS>
A few questions to help assist would be, which framework version are you using, which version of IIS and what is your Application Pool using for a framework.
HTH
I started seeing errors in a .Net MVC web app hosted on Appharbor whilst a background thread was running - after careful analysis - I can't work out the cause.
Firstly, the exception I noticed is a ThreadAbortException.
However, this is really just signifying that the thread is being killed. Before the thread is killed, you can see a new worker is created by IIS and Application_Start is called on the same machine. Once the new application is up and running, IIS kills the old app and new requests are handled as expected.
At the same time, IIS logs a message of:
ShutDown Message: IIS configuration change
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
ShutDown Stack: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()
at System.Web.Hosting.PipelineRuntime.StopProcessing()
In .Net Health Monitor Logging you get a:
Message: Application is shutting down. Reason: Configuration changed.
Event Detail Code: 50004
A quick google reveals the source code I suspect is the reason for the error:
if (!HostingEnvironment.StopListeningWasCalled && !HostingEnvironment.ShutdownInitiated) {
// If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a configuration change.
HttpRuntime.SetShutdownReason(ApplicationShutdownReason.ConfigurationChange, "IIS configuration change");
}
source: https://github.com/Microsoft/referencesource/blob/master/System.Web/Hosting/IPipelineRuntime.cs
My first thought was to check timestamps for file changes, both in the bin folder and the main application directory - however, this error is thrown without any file changes. Given it only happens on Appharbor, I can't attach to the process and debug that way. I've also monitored memory usage, and don't see any issues there.
The source code states:
If GL_STOP_LISTENING wasn't triggered, the reset is likely due to a
configuration change.
Hence, what else could be causing the error and application recycle, if the web.config / other config files aren't changing?
There are many reasons, which are listed by this helpful blog entry.
Application pool settings
The processModel element of machine.config
Memory limit
Request limit
Timeout
IIS configuration change should happen when something (anything) changes in your IIS application configuration or code. Examples:
Change in web.confg
Change in any dll, aspx, etc...
In any of those cases, IIS application will recycle. In addition, IIS will recycle your process every 29 hours by default, but that will probably not be called "IIS configuration change"
It looks like this was a Microsoft bug.
Unexpected ASP.Net application shutdown after many App_Data file
changes occur on a server that is running Windows Server 2012 R2
Hotfix: https://support.microsoft.com/en-us/kb/3052480
Last Review: 09/08/2015 16:29:00
Once this hotfix was applied, the errors went away!
I am trying to launch a process from a web page's back-end code/app pool. This process will launch an App that i built myself.
For some reason, the process only works / runs when i start it from VS2013... it never works when i launch it from IIS(7.5) itself.
I am on a Windows 7 machine (both IIS host, and App location), and I've setup my web site to only be accessible via internal network.
Here's the code, followed by the config / attempts to fix the issue:
protected void btn_DoIt_Click(object sender, EventArgs e)
{
string file_text = this.txt_Urls.Text;
if (!String.IsNullOrWhiteSpace(file_text))
File.WriteAllText(ConfigurationManager.AppSettings["filePath"], file_text);
ProcessStartInfo inf = new ProcessStartInfo();
SecureString ss = GetSecureString("SomePassword");
inf.FileName = #"........\bin\Release\SomeExecutable.exe";
inf.Arguments = ConfigurationManager.AppSettings["filePath"];
inf.UserName = "SomeUserName";
inf.Password = ss;
inf.UseShellExecute = false;
//launch desktop app, but don't close it in case we want to see the results!
try
{
Process.Start(inf);
}
catch(Exception ex)
{
this.txt_Urls.Text = ex.Message;
}
this.txt_Urls.Enabled = false;
this.btn_DoIt.Enabled = false;
this.txt_Urls.Text = "Entries received and process started. Check local machine for status update, or use refresh below.";
}
Here are the things I've tried to resolve the issue:
Made sure the executing assembly was built with AnyCPU instead of
x86
Ensured that the AppPool that runs the app, also runs under the same account (SomeUsername) as the ProcessStartInfo specified.
Ensured that the specific user account has full access to the executable's folder.
Ensured that IIS_USR has full access to the executable's folder.
Restarted both the app pool and IIS itself many times over implementing these fixes
I am now at a loss as to why this simply will not launch the app... when i first looked into the event log, i saw that the app would die immediately with code 1000:KERNELBASE.dll, which got me on the AnyCPU config instead of X86 fix... that fixed the event log entries but the app still doesn't start (nothing comes up in task manager), and i get no errors in the event log...
if someone could help me fix this problem i would really appreciate it. This would allow me to perform specific tasks on my main computer from any device on my network (phone, tablet, laptop, etc etc) without having to be in front of my main PC...
UPDATE
The comment to my OP, and ultimate answer from #Bradley Uffner actually nailed the problem on the head: My "app" is actually a desktop application with a UI, and in order to run that application, IIS would need to be able to get access to the desktop and the UI, just like if it were a person sitting down in front of the PC. This of course is not the case since IIS is running only as a service account and it makes sense that it shouldn't be launching UI programs in the background. Also see his answer for one way of getting around this.
Your best bet might be to try writing this as 2 parts. A web site that posts commands to a text file (or database, or some other persistent storage), and a desktop application that periodically polls that file (database, etc) for changes and executes those commands. You could write out the entire command line, including exe path command arguments, and switches.
This is the only way I can really think of to allow a service application like IIS to execute applications that require a desktop context with a logged in user.
You should assign a technical user with enough high priviliges to the running application pool. By default the application pool is running with ApplicationPoolIdentity identy which has a very low priviliges.
I'm working in .NET 3.5 and I have the problem with stopping some service using ServiceController. I searched whole internet and I found no solution to my problem ;)
Here's how I do that:
using (ServiceController service = new ServiceController("Service"))
{
try
{
TimeSpan timeout = TimeSpan.FromSeconds(timeoutSeconds);
service.Stop();
service.WaitForStatus(ServiceControllerStatus.Stopped, timeout);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
I'm waiting until my service stops. Then I want to replace it's libraries (update it). And the exception is thrown:
UnauthorizedAccessException
Exception Details: System.UnauthorizedAccessException: Access to the path 'blablabla' is denied.
I'm sure that I have access to this path, because I run my application as an Administrator.
What is interesting when this piece of code executes, this Service disappear from the List of current services (in Task Manager). So it actually stops it, but some thread has to still use it, so I can not move any Service's files. Even when I try to move files by myself (I go to the Service directory and try to move its files using mouse) I can not do it. But when I stop the Service manually (Task Manager and 'End Task') I can do whatever I want with its files. So what's the difference between stopping it from C# code (using ServiceController) and stopping it using Task Manager?
I don't know if it's important but I run this application (and this Service) on Windows Server 2008 in Virtual Box. (I needed to run it on some other machine then my computer).
Any ideas how can I solve this problem? ;)
Thanks for any help.
Best wishes,
Pete.
Ok, I solved my problem.
First I used an Administrator Command-Prompt Command of Net Stop to stop the Service and it worked, but I started to wonder why. The answer is it took a lot of time!
ServiceController actually stopped it, but some processes was still using it.
service.WaitForStatus(ServiceControllerStatus.Stopped, timeout);
Also workerd fine, because it imidaitely changed the status of the service (it's just one change in Windows Services list), but closing politely all threads using certain Service takes some time :)
So all You need to do is just wait, and then check again if it's still running. If it is, then kill it with Process class.