We are migrating a monolithic ASP.NET MVC web application from single server to farm under load balancer. It relies heavily on Session data, so we are moving from In Process to Out, either the ASP.NET State Service or SQL Server. The behaviour I describe happens with both.
The first problem we encountered was serialization. A bunch of data is stuffed into the Session with a bunch of techniques (old program, many hands over the years). Sometimes objects with a deep graph. The answer appeared to be to attribute each class in the graph with [Serializable]. Before doing so, requests would return 500 and garbage data. After attributing all of the offending classes, 200 came back and real data. Hooray!
Except: after partially loading the home page, everything clears and the home page begins loading again. Then clears again and reloads. Infinite loop.
By merely changing back to In Process, the application behaves perfectly. Changing either to State Server or SQL Server causes the same loop to occur.
My question is not how to fix our web application - I have not provided enough details for that. My question is how do we diagnose this? Does the behaviour I describe provide a clue where we should look? Is there a way to debug/trace while under out of proc session management? Are there tools that can provide insight?
So far we have resorted to "let's try this and see if it works" Versions of "this" include adding [Serializable] to everything, dumping "I am here" to a log, rubbing lucky totems. No clues (let alone solutions) so far.
To be clear: we are not under LB yet. This is a single server and the only change we are making is switching between ( ) In process, ( ) State Server, and ( ) SQL Server So I don't think I'm worrying about Machine Key or AppDomainID yet. I'm fairly confident both State Service and SQL server are set up correctly. We've gotten past connection errors etc., and when using SQL Server there are rows in the ASPStateTempSessions table (and one in the ASPStateTempApplications table)
How can I track down this strange looping behaviour and make the app behave under Out Of Process state management the way it does when it is In Process?
Related
I have just recently updated a web application on an IIS server, but after the update my users were receiving an incorrect input format error. This error is because the code is trying to convert a user's string input into a double which obviously doesn't work if the user passes in something like 55.5D.
Besides the bad coding practice I'm going to fix anyway, that's not the real issue. The issue is that the user was sending correct values (I watched and have input the exact same values myself) yet the program was still throwing this error. I reverted back to a previous version and the error disappeared. I haven't changed this section of code since the previous version. Anybody know what is going on here? I can't get the problem to repeat on development servers either without intentionally feeding the program bad input.
EDIT: I have tried clearing the user's and server's cache after the update, but that still received the same error, even after I put checks on the areas that were breaking (I missed some elsewhere in the code too). However, it worked great when only a few users were using it at a time, but it was breaking when many users were using it. Do ASP.NET controls have issues when many users are hitting the site?
Fully managed components in IIS do not require an application pool restart for changes to be applied. On the contrary, it will reload and JIT things again even if only web.config changes. If you are in doubt, you can always test this by including a test page with your application with a visible differentiator.
The most likely reason for your case is caching. Client-side is more likely culprit. Have you monitored http codes to verify that requests are really hitting your server and getting "live"-results (HTTP 200)?
Server-side in-process cache should be reset automatically but if you have persisted it somewhere out-of-process, they could be picked up after app upgrade..
And of course reverify that you really are testing exactly those dlls you think you are testing. If things don't make sense then verifying a wider set of assumptions can help.
We are troubled by a server that bogs down after a while, and I have no clue in what direction I should be troubleshooting anymore. Hence I am looking for a technical answer to nudge me into the right direction. There are these clues that I find quiete specific, but I don't know what to do with them. Maybe you do. Let me clarify the situation below.
There is this piece of software that communicates with SugarCRM (or factually SuiteCrm 7.3.2) through the SOAP API. Written in using C#.Net, which takes the WSDL and creates a strong typed client. Now all is functioning, data is fetched and can be stored. But after a while, the server seems to bog down. I haven't figured out what is causing the problem. Though I have one specific question, but first, let me state some specifics I gathered while throubleshooting the problem. Let me remind you that this only happens after a while, after a fresh server (re)start, all is functioning well and fast.
When the problem occurs:
The server seems to time out when a 'set_entries' function is called. The data given to this API function is persisted, but the server doens't answer the call, hence the client thinks it times out.
It seems API session related. Because when it happens with API session id 'X', no data can be stored or fetched anymore using this session, though the session is qualified as valid by the server when asked. But as soon as a new session 'Y' is created, data can be fetched again. But as soon as data is stored with session 'Y', the problem occurs again, and session 'Y' seems to have become unusable as well.
A server restart clears all issues.
No PHP errors in the log seem to be related.
Sugarcrm log isn't usefulll either, not with 'info' level set. It persists the given data, but simply doesn't return an answer.
Now the specific question: How can this be API session id related? How can it be that it seems to work again with a new session id (a bit slower than after a fresh server restart), even though the session is still valid (says the API, and the session is less than 15 secs old).
I am looking for a technical answer that can point in the right direction.
Technology: Asp.Net 4.5
Type: MasterPage Application (Transaction/Items/Configurator)
We are in Load Test Phase.
The application locks up when there is more than one user using the application on the hosting environment (M6.Net). The application uses session variables to store datatables. Also, almost all the pages have insert, update, and delete statements to sql tables in the code behind. I have Option Strict turned on. I'm trying to figure out if its an SQL Server issue, memory issue, or something else. Once one user gets locked up, the rest get locked up and eventually goes to custom_error page but there is no error exception message. Has anyone experienced this issue and what is a good way to trouble shoot?
Would Glimpse be able to detect the issues? I have no experience with it.
Note: I did not post any code because I am at a loss as to what code I should post.
The application was running out of memory. I didn't realize the provider account I was writing the app for only had 32MB of memory. So when several users were on and the memory peeked, the app pool would recycle wiping out the session variables and that was causing the errors.
We tested the application on another web server with a large amount of memory and the application worked as intended without any issues. We will be moving to a cloud environment with a significant amount of memory which will allow 500 - 1000 simultaneous users. During this troubleshooting process I did find some memory leaks and fixed those. I will be testing the session variables for values on postbacks to make sure the app pool didn't recycle as well.
#MatthewMartin - Thanks for all your input.
I am experiencing the exact same issue as a user reports on eggheadcafe, but don't know what steps to take after reading the following answer.:
Two problems you should chase down:
1. Why is the website leaking resources to the finalizers. That is
bad
2. What is Oracle code waiting on -- work with Oracle's support on it
This is the issue:
I have an intermittent problem with a
web site hosted on IIS6 (w2k3 sp2).
I appears to occur randomly to users
when they click on a hyperlink within
a page. The request is sent to the
web server but a response is never
returned. If the user tries to
navigate to another hyperlink they are
not able to (i.e. the web site appears
to hang for that user). Other users
of the website at the time are not
affected by this hang and if the user
with the problem opens a new http
session (closing IE and opening the
web site again) they no longer
experience the hang.
I've placed a debugger (IISState) on
the w3wp process with the following
output. Entries with "Thread is
waiting for a lock to be released.
Looking for lock owner." look like
they might be causing the issue. Can
anyone tell what lock the process is
waiting on?
Thanks
http://www.eggheadcafe.com/software/aspnet/33799697/session-hangs.aspx
In my case my .Net C# MVC application runs against a MySQL database for data and a MS SQL database for .Net membership.
I hope someone with more knowledge of IIS can help resolve this problem.
It sounds like you have a race condition in your database calls resulting in a deadlock at the database level. You may want to look at the settings you have in your application pool for database connections. Likely you will need to put some checks in somewhere or redefine procedures in order to reduce the likelihood of the race:
http://msdn.microsoft.com/en-us/library/ms178104.aspx
I would explain the experienced hang due to session serialization. Not the part about saving/loading it from some source, but that ASP.NET does not allow the same session to execute two parallel pages simultaneously, unless they execute with a readonly-session. The later is done either in the page directive, or in web.config, by setting EnableSessionState="ReadOnly".
Your problem still exists, this wont change that the first thread hangs. I would verify that your database connections are disposed correctly. However, you never mention any Oracle database in your question (only Mysql and SQL Server). Why are you using the Oracle drivers at all? (This seems like a valid place to start debugging.)
However, as stated by David Wang in his answer in your linked question, part two of your problem is a lock that's never released. You'll need support from Oracle (or their source code) to debug this further.
IIS hang is not something surprising. IISState is out of date, and you may use Debug Diag,
http://support.microsoft.com/kb/919791 (if CPU usage is high)
http://support.microsoft.com/kb/919792 (otherwise)
The hang dumps should tell you what is the root cause.
Microsoft support can help analyze the dumps, if you are not familiar with the tricks. http://support.microsoft.com
Situation: multiple front-ends (e.g. Silverlight, ASP) sharing a single back-end server (WCF RIA or other web service).
I am looking for a standard to prevent multiple people from editing the same form. I understand that this is not an easy topic, but requirements are requirements.
Previously I used the DB last modified date against the submitted data and give a warning or error if the data was modified since it was loaded. The initial system simply overrode the data without any warning. The problem is that I have a new requirement to prevent both these situations. There will be many UIs, so a locking system might be a challenge, and there is obviously no guarantee that the client will not close the window/browser in the middle of an edit.
I would appreciate any help.
If I'm correct, it seems what you are talking about is a form of check-out/edit/check-in style workflow. You want when one user is editing a record, no other users can even begin to edit the same record.
This is a form of pessimistic concurrency. Many web and data access frameworks have support for (the related) optimistic concurrency - that is, they will tell you that someone else already changed the record when you tried to save. Optimistic has no notion of locking, really - it makes sure that no other user saved between the time you fetched and the time you save.
What you want is not an easy requirement over the web, since the server really has no way to enforce the check-in when a user aborts an edit (say, by closing the browser). I'm not aware of any frameworks that handle this in general.
Basically what you need is to hold checkout information on the server. A user process when editing would need to request a checkout, and the server would grant/deny this based on what they are checking out. The server would also have to hold the information that the resource is checked out. When a user saves the server releases the lock and allows a new checkout when requested. The problem comes when a user aborts the edit - if it's through the UI, no problem... just tell the server to release the lock.
But if it is through closing the browser, powering off the machine, etc then you have an orphaned lock. Most people solve this one of two ways:
1. A timeout. The lock will eventually be released. The upside here is that it is fairly easy and reliable. The downsides are that the record is locked for a while where it's not really in edit. And, you must make your timeout long enough that if the user takes a really, really long time to save they don't get an error because the lock timed out (and they have to start over).
2. A heartbeat. The user has a periodic ping back to the server to say "yep, still editing". This is basically the timeout option from #1, but with a really short timeout that can be refreshed on demand. The upside is that you can make it arbitrarily short. The downside is increased complexity and network usage.
Checkin/checkout tokens are really not that hard to implement if you already have a transacted persistant store (like a DB): the hard part is integrating it into your user experience.