I am experiencing the exact same issue as a user reports on eggheadcafe, but don't know what steps to take after reading the following answer.:
Two problems you should chase down:
1. Why is the website leaking resources to the finalizers. That is
bad
2. What is Oracle code waiting on -- work with Oracle's support on it
This is the issue:
I have an intermittent problem with a
web site hosted on IIS6 (w2k3 sp2).
I appears to occur randomly to users
when they click on a hyperlink within
a page. The request is sent to the
web server but a response is never
returned. If the user tries to
navigate to another hyperlink they are
not able to (i.e. the web site appears
to hang for that user). Other users
of the website at the time are not
affected by this hang and if the user
with the problem opens a new http
session (closing IE and opening the
web site again) they no longer
experience the hang.
I've placed a debugger (IISState) on
the w3wp process with the following
output. Entries with "Thread is
waiting for a lock to be released.
Looking for lock owner." look like
they might be causing the issue. Can
anyone tell what lock the process is
waiting on?
Thanks
http://www.eggheadcafe.com/software/aspnet/33799697/session-hangs.aspx
In my case my .Net C# MVC application runs against a MySQL database for data and a MS SQL database for .Net membership.
I hope someone with more knowledge of IIS can help resolve this problem.
It sounds like you have a race condition in your database calls resulting in a deadlock at the database level. You may want to look at the settings you have in your application pool for database connections. Likely you will need to put some checks in somewhere or redefine procedures in order to reduce the likelihood of the race:
http://msdn.microsoft.com/en-us/library/ms178104.aspx
I would explain the experienced hang due to session serialization. Not the part about saving/loading it from some source, but that ASP.NET does not allow the same session to execute two parallel pages simultaneously, unless they execute with a readonly-session. The later is done either in the page directive, or in web.config, by setting EnableSessionState="ReadOnly".
Your problem still exists, this wont change that the first thread hangs. I would verify that your database connections are disposed correctly. However, you never mention any Oracle database in your question (only Mysql and SQL Server). Why are you using the Oracle drivers at all? (This seems like a valid place to start debugging.)
However, as stated by David Wang in his answer in your linked question, part two of your problem is a lock that's never released. You'll need support from Oracle (or their source code) to debug this further.
IIS hang is not something surprising. IISState is out of date, and you may use Debug Diag,
http://support.microsoft.com/kb/919791 (if CPU usage is high)
http://support.microsoft.com/kb/919792 (otherwise)
The hang dumps should tell you what is the root cause.
Microsoft support can help analyze the dumps, if you are not familiar with the tricks. http://support.microsoft.com
Related
I have just recently updated a web application on an IIS server, but after the update my users were receiving an incorrect input format error. This error is because the code is trying to convert a user's string input into a double which obviously doesn't work if the user passes in something like 55.5D.
Besides the bad coding practice I'm going to fix anyway, that's not the real issue. The issue is that the user was sending correct values (I watched and have input the exact same values myself) yet the program was still throwing this error. I reverted back to a previous version and the error disappeared. I haven't changed this section of code since the previous version. Anybody know what is going on here? I can't get the problem to repeat on development servers either without intentionally feeding the program bad input.
EDIT: I have tried clearing the user's and server's cache after the update, but that still received the same error, even after I put checks on the areas that were breaking (I missed some elsewhere in the code too). However, it worked great when only a few users were using it at a time, but it was breaking when many users were using it. Do ASP.NET controls have issues when many users are hitting the site?
Fully managed components in IIS do not require an application pool restart for changes to be applied. On the contrary, it will reload and JIT things again even if only web.config changes. If you are in doubt, you can always test this by including a test page with your application with a visible differentiator.
The most likely reason for your case is caching. Client-side is more likely culprit. Have you monitored http codes to verify that requests are really hitting your server and getting "live"-results (HTTP 200)?
Server-side in-process cache should be reset automatically but if you have persisted it somewhere out-of-process, they could be picked up after app upgrade..
And of course reverify that you really are testing exactly those dlls you think you are testing. If things don't make sense then verifying a wider set of assumptions can help.
We are migrating a monolithic ASP.NET MVC web application from single server to farm under load balancer. It relies heavily on Session data, so we are moving from In Process to Out, either the ASP.NET State Service or SQL Server. The behaviour I describe happens with both.
The first problem we encountered was serialization. A bunch of data is stuffed into the Session with a bunch of techniques (old program, many hands over the years). Sometimes objects with a deep graph. The answer appeared to be to attribute each class in the graph with [Serializable]. Before doing so, requests would return 500 and garbage data. After attributing all of the offending classes, 200 came back and real data. Hooray!
Except: after partially loading the home page, everything clears and the home page begins loading again. Then clears again and reloads. Infinite loop.
By merely changing back to In Process, the application behaves perfectly. Changing either to State Server or SQL Server causes the same loop to occur.
My question is not how to fix our web application - I have not provided enough details for that. My question is how do we diagnose this? Does the behaviour I describe provide a clue where we should look? Is there a way to debug/trace while under out of proc session management? Are there tools that can provide insight?
So far we have resorted to "let's try this and see if it works" Versions of "this" include adding [Serializable] to everything, dumping "I am here" to a log, rubbing lucky totems. No clues (let alone solutions) so far.
To be clear: we are not under LB yet. This is a single server and the only change we are making is switching between ( ) In process, ( ) State Server, and ( ) SQL Server So I don't think I'm worrying about Machine Key or AppDomainID yet. I'm fairly confident both State Service and SQL server are set up correctly. We've gotten past connection errors etc., and when using SQL Server there are rows in the ASPStateTempSessions table (and one in the ASPStateTempApplications table)
How can I track down this strange looping behaviour and make the app behave under Out Of Process state management the way it does when it is In Process?
Technology: Asp.Net 4.5
Type: MasterPage Application (Transaction/Items/Configurator)
We are in Load Test Phase.
The application locks up when there is more than one user using the application on the hosting environment (M6.Net). The application uses session variables to store datatables. Also, almost all the pages have insert, update, and delete statements to sql tables in the code behind. I have Option Strict turned on. I'm trying to figure out if its an SQL Server issue, memory issue, or something else. Once one user gets locked up, the rest get locked up and eventually goes to custom_error page but there is no error exception message. Has anyone experienced this issue and what is a good way to trouble shoot?
Would Glimpse be able to detect the issues? I have no experience with it.
Note: I did not post any code because I am at a loss as to what code I should post.
The application was running out of memory. I didn't realize the provider account I was writing the app for only had 32MB of memory. So when several users were on and the memory peeked, the app pool would recycle wiping out the session variables and that was causing the errors.
We tested the application on another web server with a large amount of memory and the application worked as intended without any issues. We will be moving to a cloud environment with a significant amount of memory which will allow 500 - 1000 simultaneous users. During this troubleshooting process I did find some memory leaks and fixed those. I will be testing the session variables for values on postbacks to make sure the app pool didn't recycle as well.
#MatthewMartin - Thanks for all your input.
Here is my problem:
I have just been brought onto a massive asp.net C# project and I've been charged with fixing some performance issues (not my area of expertise). More specifically after 5 - 7 redirects/ajax calls the web server stops responding and the whole page (and eventually the browser) freezes.
I don't think this is a coding issue as I've set up break points in a few pages (Page_Load method) and after the 5 requests it does not even reach the break points.
I don't believe this is related to this issue as I've increased the browser's maximum connections per server parameter and I got the same behavior. Furthermore after these 5 request in one browser IE, the application stops working in FF as well.
This is not a resource issue as the w3wp.exe process never exceeds 500MB memory.
One thing I've noticed when using Fiddler and other tools to monitor the requests is that the server takes a very long time when loading image files (png, jpg). I don't know if this is relevant.
I've enabled failed request tracing on the server and the only thing I've noticed is that some request fail with a 401 error even dough I've set Anonymous Authentication to enabled.
Here is the exact message
MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName ManagedPipelineHandler
Notification 128
HttpStatus 401
HttpReason Unauthorized
HttpSubStatus 0
ErrorCode 0
ConfigExceptionInfo
Notification EXECUTE_REQUEST_HANDLER
ErrorCode The operation completed successfully. (0x0)
This message is sometimes thrown with ModuleName: ScriptModule
I have already wasted 2 days on this thing and I'm running out of ideas so any suggestions would be appreciated.
Like any large generic problem, your best bet in diagnosing the issue is to figure out how to break down the issue into smaller parts, how to hypothesize the issues, and how to validate or invalidate your hypotheses. My first inclination would be to hypothesize that the server-side processes in this particular are taking a long time, causing your client requests to block, making the whole thing seem frozen.
From there, I would attempt to replicate the long running server side processes by creating isolated client side tests - perhaps if the URLs are HTTP gets, I would test the same URLs individually. If they were HTTP posts, I'd create an isolated test form if feasible to see what happens with each request. If a long running server side process is found then you have a starting point.
If there are no long running server side processes then it may be JavaScript / client side coding issues that need to be looked into. But definitely when you're working a large, unfamiliar project, your best bet is to figure out how to break down the issue into smaller components that can then be tested
I solved the issue finally. Here is what I did:
Experimented with IIS settings and App_Pool recycling and noticed that there is nothing wrong with the way it handles requests that actually reach it.
I focused on the Http.sys module and noticed that in the log files there were a lot of Timer_ConnectionIdle and Client_Reset errors.
After some more experimentation and a lot of Google searches, I accidentally found this answer and it solved my issue. As the answer suggests the problem was caused by the AVG antivirus installed and incorrectly configured on the server.
Thanks for all the help and suggestions.
If it's ajax calls that are causing your browser to freeze, make sure they are not blocking ajax calls.
Just appending to Shan's answer, which is a good one.
First off, there is obviously a code issue as this is by no means 'normal' behavior for IIS.
That said, you must isolate it as Shan indicated. For example, given the server itself no longer accepts connections then we can pretty well eliminate javascript as the source of the problem and relegate it to being just a symptom.
Typically when a worker process spins into space like this it is due to either an infinite loop or an issue where multiple threads are trying to lock the same resource. I bet if you let it run long enough IIS itself will timeout, kill and restart the process.
With that in mind you want to look for any type of multithreaded garbage (which I highly recommend you don't do in a web server) or for anything that indicates a tight infinite loop. A loop is going to become apparent if you execute the requests individually. A multi-threaded issue will only show up if you happen to get a collision.
Run various performance counters on the web server. Also, once it locks up, let it sit that way for awhile. Once IIS performs it's own reset on the worker process go look for indicators in the event log.
We ran into strange sql / linq behaviour today:
We used to use a web application to perform some intensive database actions on our system. Recently we moved to a winforms interface for various reasons.
We found out that performance has seriously decreased: an action that used to take about 15 minutes now takes as long as one whole hour. The strange thing is that It's the exact same method being called. The method performs quite a bit of read / write using linq2sql, and profiling on the client machine showed that the problematic section is on the SQL action itself, in the linq's "Save" method.
The only difference between the cases is that on one case the method is called from a web application's code behind (MVC in this case), and on the other from a windows form.
The one idea I could come up with is that SQL performance has something to do with the identity of the user accessing the db, but I could not find any support for that assumption.
Any ideas?
Did you run both tests from the same machine? If not hardware differences could be the issue... or network... one could be in a higher speed section of your network... like in the same vlan as the sql server. Try running the client code on the same server the web app was running on.
Also if your app is updating progress in a sycronous manner the app could be waiting a long time for display to update... as apposed to working with a stream ala response.write.
If you are actually outputting progress as you go you should make sure that the progress updates are events and that the display of those happens on another thread so that the processing isn't waiting on display. Actually you probably should put the processing on its own thread... and just have an event handler take care of the updates... that is a whole different discussion. The point is that your app could be waiting to update the display of progress.
It's a very old issue but I happened to run into the question just now. So for whom is may concern nowadays, the solution (and there-before the problem) was frustratingly silly. Linq2SQL was configured on the dev machines to constantly write a log to console.
This was causing a huge delay due to the simple act of outputing large amount of text to the console. On the web server the log was not being written, and therefore - no performance drawback. There was a colossal face-palming once we figured this one out. Thanks for the helpers, I hope this answer will help someone solve it faster next time.
Unattended logging. That was the problem.