After moving a WCF service from one production server to another, where the configuration is very similar, custom event logging via Event Trace for Windows has stopped working, but just for one app.
The error is being thrown in the ctor of the EventProvider class and it is a Win32 "Not enough storage" error.
The WCF service is a 'concurrency mode multiple, instance context mode per call' thread per call configuration. At time of monitoring 60 threads belonged to the process. The EventProvider ctor is invoked per call. It is IIS/WAS hosted with AppFabric.
Another app on the same server is working OK.
I have no idea how to diagnose this. If anyone can even suggest a starting point I'd be grateful.
OK this turned out to be related to VMWare configuration. The machine is a 12Gb server, but was configured to have 6Gb permanently reserved, with 6Gb taken from a pool. With a lot of memory pressure and swapping on the physical level random Win32 exceptions started to get thrown in the VM. The solution is to make more memory available.
UPDATE: The above was coincidence, it is not related to VMWare most likely.
The issue returned after a month. It seems that what something on the server has changed which slows down garbage collection, and my per-call wcf service is not disposing EtwRegistration handles explicitly (ie. I am not explicitly Disposing the EventProvider). Experiments show that there is a limit of 1000 EventProviders per process. The change in performance on the server resulted in a handle leak that hit that limit.
Further update: If anyone would like to increase the number of providers, instead of forcing cleanup for whatever reason, I think this might help http://support.microsoft.com/kb/2583244
Related
I used to have some NET Framework WCF service. It worked like a clockwork. Once NET5 and CoreWCF were released, I migrated the service to, well, NET5 and CoreWCF.
Now it hangs up after some time. If the load is light, then it might work for a day or so (and then randomly die), but if load becomes heavier, then it may die just in an hour or so. When it dies, then I can see that it starts consuming a lot of processing power.
The clients work fine and even if I restart the service, then they will pick up the connection (after complaining for a while that the service is unavailable).
The service runs as a singleton.
Logging and then monitoring when did the logging die seems to be the only way to figure out what's going on. Unfortunately, such logging produces an outrageous amount of data and it seems producing some data even after the "core" of the service no longer operates properly.
Switching to gRPC is possible. However, this will require rewriting all clients.
Debugging is a no go because the service dies, let's say, somewhere after between one hour and one day and when it handles multiple connections and timer events.
I wonder if anyone has any ideas. Thanks a lot!
Suggestion: Run the service in a debugger and then see where it dies and/or starts consuming CPU cycles.
If you can't use a debugger, then I think logging/monitoring is your best bet. Perhaps you can reduce the amount of data logged to focus only the "core" of the service.
I have a .Net 4.0 WCF program which host in Windows service.
The program uses about 40GB of memory.
I found that there was no WCF call during the GC Time.
The GC lasts about 60 seconds.
In the Event log, I found a lot of Event-5152(The Windows Filtering Platform blocked a packet) during that time.
But, interestingly, another program which runs in the same way (and uses about 20GB of memory) doesn't have this problem.
Both programs have no config for GC. (It should be workstation GC).
I'm wondering what causes Event-5152. Is it GC or other things? Or should I restore Windows?
Additon1: Yes, I cache billions of data in memory, don't doubt it. It was designed mony years ago.
Event 5152 are general Windows Firewall security audit.The reason this is happening is that the firewall was most likely disabled in the services control panel. Just ENABLING WINDOWS FIREWALL will immediately stop the blocking experienced as long as the firewall rules are set accordingly to allow/disallow traffic.
We have a server farm of about 40 servers that we roll code to every couple weeks. One thing we noticed when we roll the code live is after deploying the assemblies and performing an IIS reset and put it back in the BigIp (F5) and it receives traffic the server will lockup for about 10 minutes and clients will just spin until an eventual timeout.
Looking at the perfmon we can see a dramatic spike in number of finally's and number of pinned objects btw which lead me to investigate memory issues.
So one thing I started looking into it our Unity IoC configuration. In the global.asax.cs we are registering about 15 interfaces where most are using the ContainerControlledLifetimeManager to manage the lifetime. Normally there is never a problem with the code except in this ten minute window so my first thought was a memory or resource management issue.
Does anyone know if you have to explicitly Dispose() of your Unity Container or is this handled by Unity automagically somehow? I noticed today that there was no Dispose wiring in place for Application_End so my thought was maybe when the servers are brought back on after the IIS reset there is a Unity or object resource issue until the GC comes around and frees the memory (the ten minutes it takes to come up).
Any help is appreciated!
Performing an iisreset will kill the currently running w3wp.exe process, so it's unlikely that not properly disposing of unity objects in Application_End would cause performance issues on startup. It is possible that the old web process doesn't properly release file system or other resources the new web process depends upon, but I think you'd see file access or some other errors if that were the case.
Since you're performing an iisreset, I would look closely at the code that runs when the application starts for the first time. Maybe there are some components that take alot of time to start up (i.e., say there is a singleton type class that downloads and caches a bunch of stuff from the database) that are causing the slow down, possibly only when combined with the stress of handling all of the waiting HTTP requests. Also, keep in mind that ASP.NET will incur a bunch of overhead as it compiles the application to be used the first time. Since it seems that your web application is behind a load balancer, you may want to come up with a way to "prime" the application on each individual web server before you add that web server back to the load balancer, which could be accomplished by just loading a page locally on that web server. Priming the application would allow the web app initialize itself without having to handle any outside requests, which should improve the startup time.
Long story short, I would investigate startup issues and see what I could tune there before I focused on shutdown issues.
I have a web application using ASP.NET, that is connecting to Oracle CRM as a back end. The ASP.Net uses some business objects to call into the Oracle CRM webservices, and this works fine.
Except, however, Oracle CRM has a limitation where they only allow you to make 20 web service calls per second (or one call per 50mS), and if you exceed this rate a SOAPException is returned "The maximum rate of requests was exceeded. Please try again in X ms."
The traffic to the site has increased recently, so we are now getting a lot of these SOAPExceptions, but as the code that calls the webservice is wrapped up in a business object, I thought I would modify it to ensure that the 50ms limit is never breached.
I use the following code
private static object lock_obj = new object();
lock (lock_obj)
{
call webservice;
System.Threading.Thread.Sleep(50);
}
However, I am still getting some SOAP Exceptions. I did try writing the code using mutexes instead of lock(), but the performance impact proved to be a problem.
Can anyone explain to me why my solution isn't workinf, and perhaps suggest an alternative?
Edit: Moved to answer. Possible due to > 1 IIS worker process. I don't think object locking spans worker processes so subsequent simultaneous threads could be started but I could be wrong
http://hectorcorrea.com/Blog/Log4net-Thread-Safe-but-not-Process-Safe
My suggestion would be an application variable which stores the tick of the last request, then from that you can work out when it's safe to fire the next.
As long as your application is running with only one ASP.NET worker process you should be ok with what you have, but there are a few things to potentially consider.
Are you using a Web Garden? If so this creates multiple worker processes and therefore a lock is only obtained per/process
Are you in a load balanced environment? If so you will need to go to a different method.
OK, it turns out that a compounding issue was that we have a windows service running on the same server that was also calling into some of the same objects every 4 minutes (running on a different process of course). When I turn it off (and having bumped the sleep up to 100 as per Mitchel's suggestion) the problem seems to have gone away almost entirely.
I say almost, because every so often I still get the odd mysterious soapexception, but I think by and large the problem is sorted. I'm still a bit mystified as to how we can get any of these Exceptions, but we will live with it for now.
I think Oracle should publicise this feature of Oracle CRM on Demand a little more widely.
We have an IIS hosted web method which is randomly dying on us about 10% of the time. In trying to debug this we've added Log.Debug() messages in front of every real code line and it appears to be dying on random lines.
Has anyone seen this or have an idea on how to debug this?
[Additional Details]
We've spent a lot of time looking at it and have discovered the following...
We have a seperate self-hosted WCF Service that access the same database and lives on the same machine. When it is under heavy load the web method croaks every time. If it's not under load then things usually work fine (but not 100%).
High CPU doesn't seem to be part of the problem. We ran a small app that created a high cpu load and the web service did not die.
The web service dies when we either new up an XmlSerializer (without doing the sgen precomp) OR have NHibernate create a SessionFactory. The only two things these things have in common is that they 1) seem like things people commonly do.. 2) seem like they would be fairly intensive.
We've added a Global.asax to try to capture Application_End and Application_Error but neither event gets fired. This to me implies that we're not dealing with a normal application pool resetting?
Sounds like it might be a threading issue. You are using informative debug messages -- you should try to reproduce the issue while running the debugger and breaking on all exceptions. Make sure you check all the windows logs for information on why the app pool crashed.
Per comment: It's hard to say, but many things can cause a thread to appear to "just die." Memory issues: are you doing any interop? Improper marshaling: are you touching data on another thread? But, I will play the probabilities and ask if you're sure your handling any exception that might be happening and logging it. Are you sure you are not gobbling up an exception and not reporting it? Somewhere down low? Is this a permissions issue? Are you running partial trust or on a low privilege user account?
Figured it out.. two problems really..
We added Global.asax but it didn't get copied over which explains why we weren't seeing any messages. We fixed this and found out that...
Our WCF log was being written out to the bin directory of the IIS Web Service. In retrospect this is kind of silly since the WS is an old school web service. The WCF stuff is in the same directory only for some reason that is unknown to us since the initial person who set things up is gone..
Lesson learned.. Somewhere there is a message that explains everything.. you just have to find it.