We Keep Coding

How to troubleshoot problems under ASP.NET out of process Session State

We are migrating a monolithic ASP.NET MVC web application from single server to farm under load balancer. It relies heavily on Session data, so we are moving from In Process to Out, either the ASP.NET State Service or SQL Server. The behaviour I describe happens with both.
The first problem we encountered was serialization. A bunch of data is stuffed into the Session with a bunch of techniques (old program, many hands over the years). Sometimes objects with a deep graph. The answer appeared to be to attribute each class in the graph with [Serializable]. Before doing so, requests would return 500 and garbage data. After attributing all of the offending classes, 200 came back and real data. Hooray!
Except: after partially loading the home page, everything clears and the home page begins loading again. Then clears again and reloads. Infinite loop.
By merely changing back to In Process, the application behaves perfectly. Changing either to State Server or SQL Server causes the same loop to occur.
My question is not how to fix our web application - I have not provided enough details for that. My question is how do we diagnose this? Does the behaviour I describe provide a clue where we should look? Is there a way to debug/trace while under out of proc session management? Are there tools that can provide insight?
So far we have resorted to "let's try this and see if it works" Versions of "this" include adding [Serializable] to everything, dumping "I am here" to a log, rubbing lucky totems. No clues (let alone solutions) so far.
To be clear: we are not under LB yet. This is a single server and the only change we are making is switching between ( ) In process, ( ) State Server, and ( ) SQL Server So I don't think I'm worrying about Machine Key or AppDomainID yet. I'm fairly confident both State Service and SQL server are set up correctly. We've gotten past connection errors etc., and when using SQL Server there are rows in the ASPStateTempSessions table (and one in the ASPStateTempApplications table)
How can I track down this strange looping behaviour and make the app behave under Out Of Process state management the way it does when it is In Process?

Ways of isolating cause of unresponsive Winforms GUI

I have a large-ish Winforms application written in C# which is periodically unresponsive. The issue seems to occur once the application has been use for an hour or so. Exact timings are difficult to gather as users often go off to work on something selse, get back to it and find it has become unresponsive.
I think a memory leak can be ruled out as I'm not seeing excessive memory usage (I've asked users to send a screenshot of the task manager and memory usagage is the same as I would see when the application is runnning normally)
Similarly, CPU usage is normal (single digit %)
As I've so far been unable to recreate the issue on mydevelopment PC I am planning on sitting next to one of the affected users and mirror every action the user performs in order to recreate this. (I'll be setting up a laptop to RDP in to my main PC)
Recreating the issue is one thing, but I'll need to find out what is actually going on in the application.
Could anyone tell me if running in debug mode (through visual studio) will be sufficient or will I need to do something different?
I've searched through a few posts and I've seen mention of profiling software, however I'm not sure if this would only help with general performance issues or memory management issues.
Alternatively, if anyone has come across similar freezing issues then do you have any suggestions of the kind of causes for this?
Some technical details: Aplication is C#, compiled against .NET 3.5, winforms GUI. There are a few external libraries (most significant is ComponentFactory Krypton Suite). Data access is to a Microsoft SQL Server 2005 database. The solution contains 39 projects, I'm not sure if that might have something to do with it?
Any suggestions/pointers would be greatly appreciated.

The application is working much more reliably now, freezing issues still occur on occasion but nowhere near as often as before.
The issue appears to be related to the endpoint security (in this case, Cisco Security Agent) present in the environment I'm working in, application has been whitelisted and has has significantly rediced the instances of application hangs. The development system I work on does not have this endpoint security present, so it didn't show up in early stages of testing.
Thanks for all your feedback, I think there are still threading and garbage collection issues that need cleaning up, hopefully this should sort out the last few issues.

Unity Not Being Disposed Causing Server Lockup?

We have a server farm of about 40 servers that we roll code to every couple weeks. One thing we noticed when we roll the code live is after deploying the assemblies and performing an IIS reset and put it back in the BigIp (F5) and it receives traffic the server will lockup for about 10 minutes and clients will just spin until an eventual timeout.
Looking at the perfmon we can see a dramatic spike in number of finally's and number of pinned objects btw which lead me to investigate memory issues.
So one thing I started looking into it our Unity IoC configuration. In the global.asax.cs we are registering about 15 interfaces where most are using the ContainerControlledLifetimeManager to manage the lifetime. Normally there is never a problem with the code except in this ten minute window so my first thought was a memory or resource management issue.
Does anyone know if you have to explicitly Dispose() of your Unity Container or is this handled by Unity automagically somehow? I noticed today that there was no Dispose wiring in place for Application_End so my thought was maybe when the servers are brought back on after the IIS reset there is a Unity or object resource issue until the GC comes around and frees the memory (the ten minutes it takes to come up).
Any help is appreciated!

Performing an iisreset will kill the currently running w3wp.exe process, so it's unlikely that not properly disposing of unity objects in Application_End would cause performance issues on startup. It is possible that the old web process doesn't properly release file system or other resources the new web process depends upon, but I think you'd see file access or some other errors if that were the case.
Since you're performing an iisreset, I would look closely at the code that runs when the application starts for the first time. Maybe there are some components that take alot of time to start up (i.e., say there is a singleton type class that downloads and caches a bunch of stuff from the database) that are causing the slow down, possibly only when combined with the stress of handling all of the waiting HTTP requests. Also, keep in mind that ASP.NET will incur a bunch of overhead as it compiles the application to be used the first time. Since it seems that your web application is behind a load balancer, you may want to come up with a way to "prime" the application on each individual web server before you add that web server back to the load balancer, which could be accomplished by just loading a page locally on that web server. Priming the application would allow the web app initialize itself without having to handle any outside requests, which should improve the startup time.
Long story short, I would investigate startup issues and see what I could tune there before I focused on shutdown issues.

IIS Session hangs - How to resolve "website leaking resources to the finalizers"?

I am experiencing the exact same issue as a user reports on eggheadcafe, but don't know what steps to take after reading the following answer.:
Two problems you should chase down:
1. Why is the website leaking resources to the finalizers. That is
bad
2. What is Oracle code waiting on -- work with Oracle's support on it
This is the issue:
I have an intermittent problem with a
web site hosted on IIS6 (w2k3 sp2).
I appears to occur randomly to users
when they click on a hyperlink within
a page. The request is sent to the
web server but a response is never
returned. If the user tries to
navigate to another hyperlink they are
not able to (i.e. the web site appears
to hang for that user). Other users
of the website at the time are not
affected by this hang and if the user
with the problem opens a new http
session (closing IE and opening the
web site again) they no longer
experience the hang.
I've placed a debugger (IISState) on
the w3wp process with the following
output. Entries with "Thread is
waiting for a lock to be released.
Looking for lock owner." look like
they might be causing the issue. Can
anyone tell what lock the process is
waiting on?
Thanks
http://www.eggheadcafe.com/software/aspnet/33799697/session-hangs.aspx
In my case my .Net C# MVC application runs against a MySQL database for data and a MS SQL database for .Net membership.
I hope someone with more knowledge of IIS can help resolve this problem.

It sounds like you have a race condition in your database calls resulting in a deadlock at the database level. You may want to look at the settings you have in your application pool for database connections. Likely you will need to put some checks in somewhere or redefine procedures in order to reduce the likelihood of the race:
http://msdn.microsoft.com/en-us/library/ms178104.aspx

I would explain the experienced hang due to session serialization. Not the part about saving/loading it from some source, but that ASP.NET does not allow the same session to execute two parallel pages simultaneously, unless they execute with a readonly-session. The later is done either in the page directive, or in web.config, by setting EnableSessionState="ReadOnly".
Your problem still exists, this wont change that the first thread hangs. I would verify that your database connections are disposed correctly. However, you never mention any Oracle database in your question (only Mysql and SQL Server). Why are you using the Oracle drivers at all? (This seems like a valid place to start debugging.)
However, as stated by David Wang in his answer in your linked question, part two of your problem is a lock that's never released. You'll need support from Oracle (or their source code) to debug this further.

IIS hang is not something surprising. IISState is out of date, and you may use Debug Diag,
http://support.microsoft.com/kb/919791 (if CPU usage is high)
http://support.microsoft.com/kb/919792 (otherwise)
The hang dumps should tell you what is the root cause.
Microsoft support can help analyze the dumps, if you are not familiar with the tricks. http://support.microsoft.com

Some questions coming from application programming (C#/Visual C++) to ASP.NET (C#)

At the new place I am working, I've been tasking with developing a web-application framework. I am new (6 months ish) to the ASP.NET framework and things seem pretty straight forward, but I have a few questions that I'd like to ask you ASP professionals. I'll note that I am no stranger to C#.
Long life objects/Caching
What is the preferred method to deal with objects that you don't want to re-initialize every time a page is it? I noticed that there was a cache manager that can be used, but are there any caveats to using this? For example, I might want to cache various things and I was thinking about writing a wrapper around the cache that prefixed cache names so that I could implement different caches using the same underlying .NET cache manager.
1) Are there any design considerations I need to think about the objects that I am want to cache?
2) If I want to implement a manager of some time that is around during the lifetime of the web application (thread-safe, obviously), is it enough to initialize it during app_start and kill it in app_end? Or is this practiced frowned upon and any managers are created uniquely in the constructor/init method of the page being served.
3) If I have a long-term object initialized at app start, is this likely to get affected when the app pool is recycled? If it is destroy at app end is it a case of it simply getting destroyed and then recreated again? I am fine with this restriction, I just want to get a little clearer :)
Long Life Threads
I've done a bit of research on this and this question is probably redundant. It seems it is not safe to start a worker thread in the ASP.NET environment and instead, use a windows service to do long-running tasks. The latter isn't exactly a problem, the target environments will have the facility to install services, but I just wanted to double check that this was absolutely necessary. I understand threads can throw exceptions and die, but I do not understand the reasoning behind prohibiting them. If .NET provided a a thread framework that encompassed System.Thread, but also provided notifications for when the Application Server was going to recycle the App-Pool, we could actually do something about it rather than just keel over and die at the point we were stopped.
Are there any solutions to threading in ASP.NET or is it basically "service"?
I am sure I'll have more queries, but this is it for now.
EDIT: Thankyou for all the responses!

So here's the main thing that you're going to want to keep in mind. The IIS may get reset or may reset itself (based on criteria) while you're working. You can never know when that will happen unless it stops rendering your page while you're waiting on the response (in which case you'll get a browser notice that the page stopped responding, eventually.
Threads
This is why you shouldn't use threads in ASP.NET apps. However, that's not to say you can't. Once again, you'll need to configure the IIS engine properly (I've had it hang when spawning a lot of threads, but that may have been machine dependent). If you can trust that nobody will cause ASP.NET to recompile your code/restart your application (by saving the web.config, for instance) then you will have less issues than you might otherwise.
Instead of running a Windows service, you could use an ASMX or WCF service which also run on IIS/.NET. That's up to you. But with multiple service pools it allows you to keep everything "in the same environment" as far as installations and builds are concerned. They obviously don't share the same processpool/memoryspace.
"You're Wrong!"
I'm sure someone will read this far and go "but you can't thread in ASP.NET!!!" so here's the link that shows you how to do it from that venerable MSDN http://msdn.microsoft.com/en-us/magazine/cc164128.aspx
Now onto Long life objects/Caching
Caching
So it depends on what you mean by caching. Is this per user, per system, per application, per database, or per page? Each is possible, but takes some contrivance and complexity, depending on needs.
The simplest way to do it per page is with static variables. This is also highly dangerous if you're using it for user-code-stuff because there's no indication to the end user that the variable is going to change, if more than one users uses the page. Instead, if you need something to live with the user while they work with the page in particular, you could either stuff it into session (serverside caching, stays with the user, they can use it across multiple pages) or you could stick it into ViewState.
The cachemanager you reference above would be good for application style caching, where everyone using the webapp can use the same datastore. That might be good for intensive queries where you want to get the values back as quickly as possible so long as they're not stale. That's up to you to decide. Also, things like application settings could be stored there, if you use a database layer for storage.
Long term cache objects
You could initialize it in the app_start with no problem, and the same goes for destroying it at the end if you felt the need, but yes, you do need to watch out for what I described at first about the system throwing all your code out and restarting.
Keel over and die
But you don't get notified when you're (the app pool here) going to be restarted (as far as I know) so you can pretty much keel over and die on anything. Always assume the app is going to go down on you before your request, and that every request is the first one.
Really tho, that just leads back into web-design in the first place. You don't know that this is the first visitor or the fifty millionth (unless you're storing that information in memory of course) so just like the app is stateless, you also need to plan your architecture to be stateless as much as possible. That's where web-apps are great.
If you need state on a regular basis, consider sticking with desktop apps. If you can live with stateless-ness, welcome to ASP.NET and web development.

1) The main thing about caching is understanding the lifetime of the cache, and the effects of caching (particularly large) objects in cache. Consider caching a 1MB object in memory that is generated each time your default.aspx page is hit; and after a year of production you're getting 10,000 hits an hour, and object lifetime is 2 hours. You can easily chew up TONS of memory, which can affect performance, and also may cause things to be prematurely expired from the cache, which in turn can cause other issues. As long as you understand the effects of all of this, you're fine.
2) Starting it up in Application_Start and shutting it down in Application_End is fine. You can also implement a custom HttpApplication with an http module.
3) Yes, when your app pool is recycled it calls Application_End and everything is shutdown and destroyed.
4) (Threads) The issue with threads comes up in relation to scaling. If you hit that default.aspx page, and it fires up a thread, and that page gets hit 10,000 in 2 minutes, you could potentially have a ton of threads running in your application pool. Again, as long as you understand the ramifications of firing up a thread, you can do it. ThreadPool is another story, the asp.net runtime uses the ThreadPool to process requests, so if you tie up all the threadpool threads, your application can potentially hang because there isn't a thread available to process the request.

1) Are there any design considerations I need to think about the objects that I am want to cache?
2) If I want to implement a manager of some time that is around during the lifetime of the web application (thread-safe, obviously), is it enough to initialize it during app_start and kill it in app_end? Or is this practiced frowned upon and any managers are created uniquely in the constructor/init method of the page being served.
There's a difference between data caching and output caching. I think you're looking for data caching which means caching some object for use in the application. This can be done via HttpContext.Current.Cache. You can also cache page output and differentiate that on conditions so the page logic doesn't have to run at all. This functionality is also built into ASP.NET. Something to keep in mind when doing data caching is that you need to be careful about the scope of the things you cache. For example, when using Entity Framework, you might be tempted to cache some object that's been retrieved from the DB. However, if your DB Context is scoped per request (a new one for every user visiting your site, probably the correct way) then your cached object will rely on this DB Context for lazy loading but the DB Context will be disposed of after the first request ends.
3) If I have a long-term object initialized at app start, is this likely to get affected when the app pool is recycled? If it is destroy at app end is it a case of it simply getting destroyed and then recreated again? I am fine with this restriction, I just want to get a little clearer :)
Perhaps the biggest issue with threading in ASP.NET is that it runs in the same process as all your requests. Even if this weren't an issue in and of itself, IIS can be configured (and if you don't own the servers almost certainly will be configured) to shut down the app if it's inactive (which you mentioned) which can cause issues for these threads. I have seen solutions to that including making sure IIS never recycles the app pool to spawning a thread that hits the site to keep it alive even on hosted servers