We Keep Coding

ASP.NET .NET 4.5 Application crashes in IIS periodically and I can't figure out the cause

I have a .net 4.5 ASP.NET WebAPI application. Deployed in IIS using 1 worker on an 8gig VM with 4 CPUs.
I made changes to it recently (upgraded ServiceStack.Interfaces, ServiceStack.Common, ServiceStack.Redis and a bunch of dependencies) and started noticing that the IIS app pool this app is deployed on recycles about once an hour (give or take a few minutes).
There is nothing in my application logs that show any kind of issues. I collect metrics using telegraf and I do NOT see memory metrics increase at all, as far as all the metrics I look at everything looks absolutely normal and then the app pool recycles.
I looked at the event viewer and filtered the logs by WAS source and see event with ID 5011. Which basically means the IIS worker crash as I understand.
So then I used the DebugDiag and ran it on my local box with the app deployed on my box (I can reproduce the issue locally). It ran for a while and finally got the same event in the event viewer. Looked at the crash analysis logs from DebugDiag and all I see if a bunch of exceptions logged but nothing concrete right before the crash.
At this point I'm not entirely sure what else I can to figure out what's causing the crash so hoping there are more suggestions on what I can do to get more transparency.
What I think is happening is, there is some incompatibility with one of my dependencies and some of the upgraded packages which cause an exception to be thrown which is not handled by anything and crashes the IIS worker.
My application is working perfectly fine, as far as all API endpoints functions wit no issues, memory is NOT increasing, CPU is fine. So as far as I can tell there are no issues upto the crash.
Wondering if anyone knows any tricks to find whats causing the crash and/or handle it, prevent this exception from escaping and crashing the worker.

I was able to narrow down with some confidence that the issue lies somewhere within the ServiceStack.Redis RedisPubSubServer. What is the actual issue, I don't know as that would take a lot more time to dig and I've wasted too much time already.
However, piggybacking on some existing code I had (from before ServiceStack supported sentinel) I created a new implementation of the redis client wrapper for the which I call LazySentinelServiceStackClientWrapper; instead of using the built-in sentinel manager, it relies on a custom sentinel provider which I created LazySentinelApiSentinelProvider this implementation attempts to interrogate the available sentinel hosts in random order for master and slave nodes and then I construct a pool using the retrieved read/write and readonly hosts and this pool is used to run the redis operations. The pool is refreshed whenever an error occurs (after a failover). Opposed to the builtin sentinel manager that comes with ServiceStack.Redis which instantiates Redis pubsub server and listens for messages from sentinel whenever configuration changes such as fail-overs occur and updates the managed redis connection pool.
I installed my version of this redis client wrapper into my application has seen no app pool recycle events since (other than the scheduled ones).
Above is the log of app pool recycle events before I disabled the ServiceStack.Redis sentinel manager.
And here's the log of app pool recycle events after installing my new lazy sentinel manager
The first spike is me recycling the app manually and second one is the scheduled 1am recycle. So clearly the issue is solved.
What is the actual reason why the sentinel manager via redis pub sub server is causing IIS rapid fail protection to fire and recycle the app pool I do not know. Maybe someone with much more redis experience and/or IIS experience can attest to that. Also I did not test this in .net core and only tested for a .net 4.5.1 application deployed in IIS but on many different machines including local development machine and beefy production machines.
Finally one last note, that first image which shows all the recycle events, that's on my CI machine which is barely taking any traffic, maybe 1 request every few minutes. So this means the issue is not some memory leak or some resource exhaustion. Whatever the issue is, it happens regardless of traffic, CPU load, memory load, it just happens periodically.
Needless to say I will not be using the builtin sentinel manager at least for now.

WinService sometimes fails to start after boot

I have a few Windows services (all written in C#) that all show the same strange behaviour.
I have them set to delayed auto start so that they get started after the boot (delayed because well they are not critical).
They all host WCF services as parts of Client-Server applications and were installed using WiX if that matters.
I noticed that sometimes they just don't start.
If you look into the Services window fast enough after the OS is ready they have status "Starting". If you then refresh the view they are no longer starting but not "Started" either.
You can then start them manually without any problem whatsoever.
This produces no error messages and no log entrys. And to make it even better this only occurs if the machine has been shut down and turned on again. Reboot works perfectly fine every time (tried it about 20 times on two different machines)
If you set the failure actions to restart the service after failure it seems it will eventually start the service successfully but surely this can not be the ideal solution.
OSs are Windows 7 and WinServer 2008 R2
What am I missing here? Why do they fail to be started automatically(the first time at least)? And why does it make a difference if the computer boots following a reboot or a shutdown?
EDIT:
I was wrong about the failure actions. The did not fix the problem.
EDIT 2:
I have added exception handling around everything to log possible exceptions. But so far no exceptions have been logged.

Might it be the WCF Services take a long time to start? afaik, the windows service has to come up in a certain time (best practices is 30 seconds, technical limit I don't know) to not time out. That could explain why your service is in status "starting" but does not start.

Please see my answer from the duplicate. A windows service typically shouldn't have access to the desktop for security reasons. But it certainly should have a good amount of logging in it. You probably have a race condition. The only thing you could do about this in WiX would be to express a dependency on another service to get the service control manager to wait awhile before starting the service. But it really would be better if your code was more robust. An example would be the OnStart event fire up a background worker process and then return success. The background thread could then keep attempting to host the WCF endpoint and everything do a fair amount of logging in the process.

Machine retains file exists/locks on client-side power outage

Our app running on client server A and creates a file on the server 2008 R2 file-server using:
CreateFile(LockFileName,
GENERIC_READ or GENERIC_WRITE,
FILE_SHARE_READ, nil,
CREATE_ALWAYS,
FILE_FLAG_WRITE_THROUGH or FILE_FLAG_DELETE_ON_CLOSE,
0);
The client is testing a disaster situation and powering off 'server A' and leaving it off.
They're reporting that our app running on 'server B' using the same filename and the same code fragment above fails (ie the file continues to exist) for at least 15 minutes until, we believe, they browse to folder containing the file in Windows Explorer at which point the file is deleted automatically.
Is anyone aware of how this is supposed to behave in this situation, where the creating server has gone away, should the handles be released and the file removed automatically? And why does looking at the file cause it to delete?
Interestingly, on another supposedly similar setup the issue does not occur.

[...] where the creating server has gone away, should the handles be released and the file removed automatically?
Eventually yes, but not immediately. As you are running Windows Server 2008 R2 (and thus SMBv2, note that I assume that both server and client are running on Windows Server 2008 R2) the client will request a durable file handle. According to the SMBv2 specification, section 3.3.6.2 and 3.3.7.1 the server must start the durable open scavenger timer (set to 16 minutes on Windows Server by default). Once the timer expires the server must examine all open handles and close those that have not been reclaimed by a client.
In your scenario of course, an open question is whether the server detects the connection loss to the client at all, as the client (i.e. the whole server, not just the process) according to your description is killed immediately.
Now assume that another client is trying to open the file while the durable timeout is still running/the server still considers the file to be open by the first client. Then it is supposed to send an oplock break notification (section 2.2.23.1) to the client that initially opened the file. As the client is unable to respond (it has been killed) the server will wait for the oplock break acknowledgment timeout to expire (section 3.3.2.1, 35 seconds by default) before it will grant the new client access to the file.
There is one other thing to note: The behavior will be different if the second client accesses the file via a local path rather than via an UNC path. In this case the client won't have to wait for the oplock break ack timeout to occur. Windows will grant him access to the file immediately while it will try to send a close request to the first client.
This is how the system is supposed to behave. As to why you are experiencing the issues described I cannot tell. I wouldn't be surprised if you'd stumbled upon a bug in the Fileserver implementation of Win Server 2008. I would try to troubleshoot the issue using the tools mentioned in the other answers (procmon is really nice) and Wireshark helps a lot too.

There is nothing to say there should no longer be any handles when the creating servers goes down. In order for a handle to be removed, something has to initiate that removal. If a server abruptly goes down, it cannot remove its handles, so those handles remain open. As far as the server still up is concerned, all is good and well, and no file handles should be forcibly closed.
Until you actually try to act upon the file handle. Suddenly, the server notices that the host of the file handle is gone, because it tries to initiate communications with said host. Once it realizes this, the file handle gets forcibly closed.
Thus, to answer your question, this seems like perfectly predictable and expected behavior to me.
The reason file handles get closed immediately in another environment probably has to do with something keeping those servers in constant communication: something is constantly accessing a remote file. That's just a guess, though.
Update
Sysinternals, bought out by Microsoft a few years ago, has a great tool called Process Explorer that allows you to search processes' open file handles. This might be of use to you in determining which program(s) are refreshing the file handle(s).
Sysinternals also has Process Monitor, which allows you to see in real-time as programs act upon file handles. This could be another useful program in troubleshooting the issue.
Edit: Oh, and if you really want to have fun, there's Handle, too.

This looks so far like a non issue to me. Or one that can not be handled outside of Microsoft's programming AND one that has side effects when hnandled. Basically you ahve to account for small disruptuons of communication between client and server and optimize network traffic, so the server can not permanently exchange packets with the client just to see whether the client is still around.
Computer programming must take that into account as far as possible, but timeouts like that are normal unless the client application handles that properly. THe main question (totally not answered) is whether this is an issue at all - so far it looks like "standard behavior".
Is anyone aware of how this is supposed to behave in this situation, where the creating server has
gone away, should the handles be released and the file removed automatically?
How would the server know?
And why does looking at the file cause it to delete?
P9ossibly it is the reading that triggered a refresh that timed out, so - at the end - this tirggered the defined behavior (DELETE_ON_CLOSE).
I would hint that any access to certain elements of the file would trigger this, but the tester did not do that excpt just refreshing the explorer.

windows service working on windows 7 but not on windows server 2003

Solved the problem see the bottom of my post.
So I have a simple windows service that watches a specific folder and upploads files that come into it to a server using a web service.
It's working fine on my machine using windows 7 but when I try to start it on a windows server 2003, I receive an error: Error 1053: the service did not respond to start or control request in a timely fashion. But I get this message after only a few seconds.
I have created the ServicesPipeTimeout and set it to 60000 milliseconds.
I have tried running it from command line using sc query command and found out that the WIN32_EXIT_CODE is 0, which I think means that the service doesn´t even try to start because it find an error before it starts.
In the event viewer I get errors 7000 and 7009.
I am the Administrator on the windows server.
The only thing I haven´t tried is a hotfix I found from microsoft but I don´t want to use it because as I understand it, it is for when the service actually times out.http://support.microsoft.com/kb/886695
I have tried everything I can think of, is there anything that I am missing?
Gísli
EDIT: Re-installed the .NET framework and now I get a new error, Saying that the service controller can not be found.
EDIT: I setup the service with a setup project, not using the installutil command. This is because I need to get user input during the installation and save that in the registry.
EDIT: I have installed the .NET 4.0 framework, it wasn´t possible to install the service with out doing that.
In addition to what I wrote above I have also tried:
Rebooting.
Re-installing.
I have tried to change the permissions on the files that the service needs to access.
Changing permissions in the registry editor.
Edited the code so that the onStart function only starts one thread.
I think it is some kind of permission problem but I don´t have much experience dealing with Windows server.
Solution:
It turned out to be two seperate problems. The .NET framwork had to be repaired and I had to remove the try/catch clause that I had when starting the service. For some reason (unknown to me) the try catch block did something that made it impossible to start the service in a windows server 2003 but it ran fine on windows 7.
It would be very interesting to know why this is.
Thanks for all the help.
Gísli

Have you installed the right version of .NET Framework on the Server 2003 PC? What comes as standard on Windows 7 will need to be installed manually on an older OS.
You say "I have tried everything I can think of". Please edit the question to show what you have already tried so we don't suggest something you have already done.
EDIT:
Try also the Fusion Log viewer. Set it to log failures then start your service. Hit refresh then see if any errors are logged. Double-click a line for more details.

I'd guess that you installed the .NET 3.5 service using .NET 2.0? InstallUtil will work since the CLR is the same, but the service won't start because of .NET 3.5 dependencies.
See Also:
How to install a Windows service developed in .NET 3.5?

In your OnStart code you could wrap everything in a try-catch block and write the exception message and stack trace to a file using something like:
File.WriteAllText(#"C:\Temp\MyServiceLog.txt", exp.Message + exp.StackTrace);
This will help you analyze the problem. Also, I'm using a very simple way of debugging my services: I'm wrapping them in a WinForms wrapper if the session is interactive. For that, I need the following:
A form that creates an instance of the service class
New methods DoStart and DoStop in my service class, which are public and call the protected OnStart and OnStop methods
New code in above form's OnLoad and OnClose handlers, so the DoStart and DoStop methods of the service instance are called accordingly
Then I add the following code to the Main method in Program.cs:
if (Environment.UserInteractive)
{
Application.Run(new FormServiceHoster());
}
else
{
// ... old code to create service instance.
}
This way you're making the "service" run as an application when you just hit F5 in Visual Studio or double click the EXE in the Explorer. When the service is actually run as a service, the Forms-code is ignored.
Usually you can debug services by attaching to the process, however, that doesn't work for debugging the start and stop code. This method helps a great deal debugging the service start and stop code.
You will need to add references to System.Windows.Forms and System.Drawing manually.

This is the point where I bet you wish you had used some logging package like log4net, which might help tell how far your program gets.
Is it possible that the code in the OnStart event is actually taking longer than you expect. If so, you could move that code to a thread, so that the OnStart event finishes (and so Windows reckons the service has started) while your thread continues working.
Are you running the service as admin? If so, that usually rules out permissions issues. (If it still goes wrong, it will rule out the KB 886695 issue as that seems to only apply to local system).
Since you are using .NET 4.0 are you sure that either, the full version is installed, rather than the Client Profile, or that your application works with the client profile?

"A timeout was reached while waiting for the service to connect" error after rebooting

I have a custom-written Windows service that I run on a number of Hyper-V VMs. The VMs get rebooted a couple times an hour as part of some automated tests being run. The service is set to automatic start and almost all of the time, it starts up fine.
However, maybe 5% of the time, with no pattern that I can discern, the service fails to start. When it fails, I get an error in Event Viewer saying
A timeout was reached (30000 milliseconds) while waiting for the My Service Name service to connect.
When this occurs, I can start the service manually, or restart again, and the service will start fine.
The thing I can't figure out is that the 30 second timeout doesn't appear to be occurring in my code. The very first line of my service class's OnStart() method logs "Starting..." to its log4net log. When the service fails to start, I don't even get anything logged at all, which indicates to me that either log4net can't log for whatever reason, or the timeout is occurring before my OnStart() gets called.
The service runs on a variety of OSes, from XP all the way up to Win7 and 2008R2, and I know that setting the service to delayed start may solve this for Vista and later, but that seems like a hack.
I haven't been able to remote debug this because of the fact that it happens so intermittently and during system startup, and I'm at a loss as to further ways to try to figure out what's going on. Any ideas?

My guess - and that's all it is - is that the disk is thrashing hard during startup, to the point where the .NET Framework itself isn't starting in the 30 seconds that Windows allocates for services to start.
A kludgy workaround may be to set the service to start manually, then write a very small stub service in unmanaged code (e.g. C++, Delphi) to start the service.
Another approach may be to start the service remotely from another machine. The sc command should do the job nicely.

I was seeing this error in the Event Viewer when trying to install a service with powershell.
The problem I had was that I had different values for "Service Name" and "Service Display Name" in my powershell script to those that I had specified in the program.cs file of my Console Application.

For what it's worth, I discovered that I received this message (almost immediately upon service startup) because I did not have version 4.5 of the .NET framework installed on the target machine. I rolled back the version I was using to version 4.0 (which was already installed on the target machine) and the service worked as expected.

I think I may have also found another contributing factor to this kind of does not start on reboot error.
It appears that if the Windows Event Log is set to Overwrite Events > 7days.. size 512kb.. But a lot of activity has occurred within this window, then Event Log is effectively full because it can't overwrite the number of events generated inside that timeframe. If you set the eventlog to a much larger size OR to Overwrite as needed then you won't experience this issue

My issue with the same error was that the .Net installation on the server was not working correctly.
To figure this out:
I made a small console app with identical logic as the executing service, and I made a try-catch around the whole code piece, dumping it all out to console.
Not sure why the information didn't bubble up, but we saw the valuable messages about the Framework errors that we would never have seen otherwise.

We are having the same problem on Windows 2016 Server.
A fix that seems to be working is changing the user under which the service running from Local Service Account to local Administrator (not sure what's the cause).