I have a problem with one async handler in distributed ASP.NET web app. First let me explain a use case:
application uses IIS 8 on win 2012 machine with .NET Framework 4.5.2
application has disabled Session and authentication modules via web.config like this
<system.webServer>
....
<modules>
<remove name="WindowsAuthentication" />
<remove name="Session" />
<remove name="FormsAuthentication" />
</modules>
</system.webServer>
application uses custom async web handler to serve the specific request
application has very heavy traffic (about 50k requests per minute per server, async handler has about 10k requests per minute per server all tracked from NewRelic)
application is distributed via multiple w3wp processes (2 w3wp processes) and multiple virtual servers (about 10 servers)
application has high amount of connections
All normal (sync requests) are working fine but async request that does a little more work (that's why we use async request) is often slow but NewRelic reports that it is slow because of "AcquireRequestState". Now I've looked on google and stack overflow and this event is connected to creating a Session but we have sessions disabled in web.config. Does anyone know what else could "AcquireRequestState" could be doing? Are we missing some place to remove session state? Adding that from web.config to machine.config did nothing...
Here is a snippet from a request in NewRelic:
**Slowest components Count Duration % **
AcquireRequestState 1 12,600 ms 100% --> WTF?
ExecuteRequestHandler 1 5.01 ms 0%
Integrated Pipeline 1 0.334 ms 0%
UpdateRequestCache 1 0.3 ms 0%
EndRequest 1 0.168 ms 0%
AuthenticateRequest 1 0.161 ms 0%
Total time 12,600 ms 100%
EDIT:
I have <sessionState mode="Off" /> in web.config (<system.web> section) so that is not it.
I've looked into this because we had similar issues, I found this forum post, the reply that is interesting is this:
Unfortunately, the issue is not as simple as just turning sessionState off. In fact, one of the key phrases when describing the challenges with AcquireRequestState is the phrase for example, session state when it comes to when this event is raised. In digging deeper into this (actually looking at the .NET source) we can see that this is called when an EventHandler is executed or a RequestNotification object is created. I daresay there are other methods and/or events that, when called, will raise an AcquireRequestState event. Tracking down all of them represents something of a struggle. It seems this is something not talked about a lot outside of the more normalized session state discussions.
The most common place we see this event raised is certainly related to session state management. But there are very obviously outliers where these event can still be raised. It can even be called directly from application code. The thing the agent grapples with is that it can identify the event, but rarely the source. When it is raised as part of the ASP pipeline, the only notification the agent gets is that this is one segment of the transaction. The source, or the methods executed inside of the event, is something the agent is rarely instrumenting by default.
I wish we could offer more insight for you on this. There are a lot of moving parts inside of a .NET application, some of which involve the operating system itself, IIS, the version of .NET, whether or not the methods are asynchronous, application pool settings, execution modes, permissions, etc.
While I don't want to open a second can of worms here, this harkens to the issue with the lack of stack traces for 500 errors. The agent provides a stack trace when it is offered and/or available. Where the stack trace, if one even exists, occurs within the ASP pipeline is extremely important. Sometimes it occurs before any actual application code is executed. In such cases the error is reported to the application, which in turn lets the .NET agent see and report that an error occurred, but no other details are provided. The agent simply sees that it happened and reports as much information as is possible. Beyond that the agent simply has no further details it can offer.
We gave up, so I'd be interested to know if you figure it out!
AcquireRequestState is also used by the Profile Module. Did you try disabling that?
<modules>
<remove others unwanted modules... />
<remove name="Profile" />
</modules>
So I had this same issue. I had a lot of concurrent ajax requests that all used variables stored in ASP.NET ClaimsIdentity, which is effectlively using the SessionState. This meant, that a lot of the server response time was just waiting for the SessionState to be unlocked.
I solved it by decreasing the Lock Polling interval:
var sessionStateModuleType = typeof(SessionStateModule);
var pollingIntervalFieldInfo = sessionStateModuleType.GetField("LOCKED_ITEM_POLLING_INTERVAL", BindingFlags.NonPublic | BindingFlags.Static);
pollingIntervalFieldInfo.SetValue(null, 30); // default 500ms
var pollingDeltaFieldInfo = sessionStateModuleType.GetField("LOCKED_ITEM_POLLING_DELTA", BindingFlags.NonPublic | BindingFlags.Static);
pollingDeltaFieldInfo.SetValue(null, TimeSpan.FromMilliseconds(15.0)); // default 250ms
The solution above was disclosed in this blog article
Related
I have a .net MVC action that takes 2000ms to complete.
The 2000ms is recorded in the IE Dev tools Network tab, Glimpse and when I use a HttpModule to time the difference between the context_BeginRequest and context_EndRequest.
However, when I record the timings of the action method itself, so from the first line of code in the action to the last line, the time is only 300ms.
My last line of code generates the View, then I log the time and then I return the View - so it seems that even the View generation is within the 300ms:
var view = View("~/Views/Home/Index.cshtml");
LogMethodTime("HomeController.Index", stopwatch);
return view;
I have logged the time when the relevant Controller methods have completed - and they show that the time from the Constructor to the EndExecute only takes 534ms:
14:40:26,554 [13] INFO - Constructor
14:40:26,561 [13] INFO - OnAuthentication
14:40:26,608 [13] INFO - OnAuthentication
14:40:26,988 [13] INFO - OnActionExecuted
14:40:27,024 [13] INFO - OnResultExecuted
14:40:27,044 [13] INFO - EndExecuteCore
14:40:27,088 [13] INFO - EndExecute
What else in the asp.net pipeline could be swallowing 1700ms please?
I'm running this locally through IISExpress.
Please note that the Network tab of most browser's development tools don't provide granular reporting of all that's going on--particularly when your web server allows the browser to keep connections alive to transfer all the data needed. Hint, anything designed for production use does. To that end, what you are seeing are the following:
The page itself (as you measured should take ~500ms or less)
CSS stylesheets from the same server
JavaScript files from the same server
Images from the same server
Modern browsers would rather keep the one connection open and request all the resources from the same host (server) at once. This adds a lot of hidden cost to the connection. The network tab doesn't always break down the individual network time for the bundle of resources.
To see a better indication of how much time is spent for just the MVC page to return (fully transformed) then you will want to use a tool like JMeter or curl.
I managed to save 500ms by setting the following in the web.config:
<compilation debug="false"
Also, by adding more Trace statements, it has highlighted that the bulk of the time occurs between the Application_BeginRequest and the Controller constructor - so I have created a new question to ask that specific detail.
Time taken between Application_BeginRequest and MVC Controller constructor is too long
In ASP.NET on IIS 7.5, how can I close the session after an amount of time (lets say 5 hours), even if the user is still working and there is no idle time?
Is there a property in IIS or should I do it in code?
Have you tried the web.config?
<configuration>
<system.web>
<sessionState
mode="InProc"
cookieless="true"
timeout="30" />
</system.web>
</configuration>
I'm also assuming you have a basic site with no load balancing or session state servers. Those require different approaches.
I haven't tried such a solution before but here's what come into mind :
In your global.asax handle SessionStart and when a new session is started create a new (Asynchronous )Task that starts a timer (a timer that execute once after 5 hours and calls Session.Abandon() :
Something like this :
new Task(()=>{
var timer=new Timer{Interval=DateTime.Now.AddHours(5), Elpased+=(obj,args)=>{
session.Abandon();
((Timer)obj).Stop();
((Timer)obj).Dispose();
}
}).Start()
As I told you I haven't test it and since everything is working asynchronously here it might led to unknown situations.
(BTW as far as I know timer will already run in another thread so no Task is needed :) )
Update
OK, It seems that we can't hold a reference to a HtmlStateSession. Thus here's another approach that I think it can help: In this approach we hold a registery of session id and start time of each session in a static dictionary and when a request begins we check if its session (according to its session id) has been out there more than 5 hours and if that is the case we Abonden the session.(and remove the id from dictionary of course)
There's no need to have thousands of timers this way and we don't expire the sessions right away but we wait until the next time that user sends us a request.
What about setting up windows task scheduler to run an aspx page that will check how long the user is logged in and according to it, to decide if to close the session or not.
(You can save in your database, a log-in time for each user).
We have a web service using ServiceStack (v3.9.60) that is currently gets an average (per New Relic monitoring) of 600 requests per minute (load balanced with two Windows 2008 web servers.)
The actual time spend in the coded request Service (including Request Filter) takes about an average of 5ms (From what we see from recorded log4net logs.) It is offloading the request to an ActiveMQ endpoint and automatic have ServiceStack generate a 204 (Return204NoContentForEmptyResponse enabled with "public void Post(request)")
On top of that we have:
PreRequestFilters.Insert(0, (httpReq, httpRes) =>
{
httpReq.UseBufferedStream = true;
});
since we use the raw body to validate a salted hash value (passed as a custom header) during a Request Filter for approval reasons that it comes from a correct source.
Overall we see in New Relic that the whole web service call takes an average around 700ms, which is a lot compared to the 5ms it actually takes to perform the coded process. So when we looked deeper in the data New Relic reports we saw some requests periodically take quite some time (10-150 seconds per request.) Drilling down in the reporting of New Relic we see that Applying the Pre-Request Filter takes time (see image below.) We were wondering why this could be the case and if it was related to the buffered stream on the Http Request object and what possibly could be done to correct this?
EDIT
Have been playing around with this some and still haven't found an answer.
Things done:
Moved the Virtual Folder out from a sub-folder location of the actual site folder (there are about 11 other Web Services located under this site)
Assigned this Web Service to use its own Application Pool so it is not shared with the main site and other Web Services under the site
Added the requirement to Web.Config for usage of Server GC as Phil suggested
Disabled the pre-request filter that turned on the usage of buffered stream (and bypass the code that used the RawBody)
Added more instrumentation to New Relic for a better drill-down (see image below)
I'm starting to wonder if this is a Windows Server/IIS limitation due to load. But would like to hear from someone that is more familiar with such.
I am getting the following exception when calling a web service operation in my .NET application:
System.Net.WebException: The operation has timed out at
System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String
methodName, Object[] parameters) at
org.wvi.hrmgmt.HRManagementServiceserviceagent.LoadReducedTemplate(LoadReducedTemplateRequest
LoadReducedTemplateRequest) at
org.wvi.mgmt.hr._ServiceClient.UploadReducedTemplate()
Basically, a user uploads a file into the web server and the binary contents of this file is being passed as one of the input parameters when calling a web service operation. The size of the file is 1,397 KB and if I choose to process this file, I am getting the exception. Whereas, processing a file with a relatively smaller size, supposed 127 KB, gives a successful response.
I have done the below steps in order to solve the dillema:
Setting the web service timeout to infinite. -> Not getting an exception and any response
Adding Response.Close() -> Not getting any response
Editing the web.config in the IIS server and adding the following lines: -> Still getting the exception
<system.web>
<customErrors mode="Off"/>
<httpRuntime executionTimeout="600" maxRequestLength="16384"/>
<sessionState timeout="540"/>
</system.web>
However, to no avail, I'm still getting the above exception. I am using a Javascript plugin called AjaxUpload from GitHub and I can't figure out whether they have file upload size limit. Besides, I have checked the file if it is being uploaded on the server and it does, therefore, the issue may not be because of that plugin.
Another issue is that I am getting the same exception when uploading the file after a period of time the web page has been opened. I surmise that this has something to do with session timeout so I added an additional line in my web.config file specifying the maximum timeout. Please advise me if I'm on the right track.
Are there additional settings I need to apply in my IIS server to make the operation successful? Otherwise, are there anything I am missing? If you have experienced this before, kindly inform me what to do.
Thank you so much,
Ralph
This might be due to lot of proxy opened before. So I suggest you to delete Service.Reference.Config along with web service reference and add the service reference again and see whether it solves your problem.
Adding on to the previous answer, you may try setting up the execution time out on the proxy class , Refer this discussion
And you service call is performing a long running task you may prefer to have a lengthy time out for any webservice call by setting the timeout in proxy class as said before.
i have a web application which which proccesses some request on some data that client selects, when the client selects more than 20 objects and clicks on proceed the client recieves this error, because the server takes a long time to process, however if the records are less and hence a timely response is recieved, no such error comes can someone help me on this?
i have increased the sessiontimeout as well as set the
Try adjusting the executionTimeout in your web.config...this only applies if debug is set to false however.
<httpRuntime
executionTimeout="some number"
/>
If this alone does not solve your issue, check out this blog post which goes into a bit more depth on how to structure your timeouts. Note the IIS reference towards the bottom...