Unfortunately i cannot share the code but the main issue here is that a device calls the API which takes longer than 20 seconds to process which results in the call being made again from the back-end. I have confirmed that it is not the device making a second call.
According to the following article:
http://www.codeproject.com/Tips/338142/RESTful-API-in-WCF-get-called-more-than-one-time-b
something which i found is that IIS has some setting for HTTP GET as
40sec replay, so if within 40sec if server does not get response it
again query itself.
In my case it is 20 seconds.
From that article i suspect that IIS is the culprit. Does anyone have a solution for this?
Related
I have a strange issue I'm trying to triage having to do with the new HttpClient on .NET Core 2.1. From this article here (https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/) I know that the HttpClient has been completely re-written to use a different low level library for handling HTTP requests. I'm wondering if anyone has seen any issues with the new implementation.
What I'm seeing is a strange case where my application (.NET Core 2.1) which sends a POST request to some API periodically (every 10 seconds) a few of times every 15 min it will throw an exception with the error: An error occurred while sending the request.; The server returned an invalid or unrecognized response.
No other details are available, it's just an exception when I make a call like this:
using (var res = await _httpClient.PostAsync(uriBuilder.Uri, new StringContent(serializedRequestBody, Encoding.UTF8, "application/json")))
{
//Do something here
}
The exception caught is a System.Net.Http.HttpRequestException and it has some inner exception with the above error message.
So as I mentioned this does NOT happen all time, it happens seemingly at random, or at least I can not discern any particular pattern. All I can say is these POST requests are made once every 10 seconds 24/7 and anywhere between 5% and 10% of the POST requests fail with the above exception.
So used tcdump and piped it into wireshark to examine the requests to see what's actually happening when the requests fail and what i see is the following:
On a good POST I see: my app sends the request to server, server sends response back, my app sends ACK to server and server responds with FIN,ACK. Done. Good Stuff.
On POST which gets the above exception I see the following: my app sends the request to server, and almost immediately after (like a few milliseconds after) my application sends FIN, ACK to server.
This seems consistent with what I see in my application logs, which show that the request duration is 0 before the exception is thrown.
So what it looks like to me is, my application sends the request and then immediately after closes the connection for some reason. However, I don't understand why this happens. I tried comparing the raw HTTP requests (good POST vs bad POST) to see any differences and I can not see any difference.
One last thing to mention, is that I ONLY see this in applications running on .NET Core 2.1. When I run my application on .NET 2.0 I do not see this problem. Also when I use the same library (where the HTTP call is being made) in the .NET 4.5.1 application (I use multi-targeting to compile the library targeting .net standard and net451) I also do NOT see this problem. So it seems to affect only .NET Core 2.1
Any ideas of where I can go from here? Is there something else I should look for ? How would someone go about trying to triage this type of issue ?
[EDIT] I added a screenshot of the wireshark output which shows the last POST request the server never does not respond before the client sends FIN,ACK
[EDIT]
#Svek Pointed out something in the comments about the sequence of ACKs. I think there maybe something here, because (in the screenshot) after the very last POST there is a FIN, ACK and it shows Ack=7187, so I look back I see the previous FIN,ACK had sequence=7186. Now, I'm by far not an expect in TCP or networking so I maybe saying something completely dumb, but does that mean that the last FIN,ACK (which comes from my host to server) is essentially my host FIN,ACK'ing the previous FIN,ACK (from server to my host) and essentially closing the connection.
So since the next POST is made to the same host:port, using the same connection and yet the connection is closed (via that last FIN,ACK) that's why I never get a response back?
I have a .net MVC action that takes 2000ms to complete.
The 2000ms is recorded in the IE Dev tools Network tab, Glimpse and when I use a HttpModule to time the difference between the context_BeginRequest and context_EndRequest.
However, when I record the timings of the action method itself, so from the first line of code in the action to the last line, the time is only 300ms.
My last line of code generates the View, then I log the time and then I return the View - so it seems that even the View generation is within the 300ms:
var view = View("~/Views/Home/Index.cshtml");
LogMethodTime("HomeController.Index", stopwatch);
return view;
I have logged the time when the relevant Controller methods have completed - and they show that the time from the Constructor to the EndExecute only takes 534ms:
14:40:26,554 [13] INFO - Constructor
14:40:26,561 [13] INFO - OnAuthentication
14:40:26,608 [13] INFO - OnAuthentication
14:40:26,988 [13] INFO - OnActionExecuted
14:40:27,024 [13] INFO - OnResultExecuted
14:40:27,044 [13] INFO - EndExecuteCore
14:40:27,088 [13] INFO - EndExecute
What else in the asp.net pipeline could be swallowing 1700ms please?
I'm running this locally through IISExpress.
Please note that the Network tab of most browser's development tools don't provide granular reporting of all that's going on--particularly when your web server allows the browser to keep connections alive to transfer all the data needed. Hint, anything designed for production use does. To that end, what you are seeing are the following:
The page itself (as you measured should take ~500ms or less)
CSS stylesheets from the same server
JavaScript files from the same server
Images from the same server
Modern browsers would rather keep the one connection open and request all the resources from the same host (server) at once. This adds a lot of hidden cost to the connection. The network tab doesn't always break down the individual network time for the bundle of resources.
To see a better indication of how much time is spent for just the MVC page to return (fully transformed) then you will want to use a tool like JMeter or curl.
I managed to save 500ms by setting the following in the web.config:
<compilation debug="false"
Also, by adding more Trace statements, it has highlighted that the bulk of the time occurs between the Application_BeginRequest and the Controller constructor - so I have created a new question to ask that specific detail.
Time taken between Application_BeginRequest and MVC Controller constructor is too long
We have a web service using ServiceStack (v3.9.60) that is currently gets an average (per New Relic monitoring) of 600 requests per minute (load balanced with two Windows 2008 web servers.)
The actual time spend in the coded request Service (including Request Filter) takes about an average of 5ms (From what we see from recorded log4net logs.) It is offloading the request to an ActiveMQ endpoint and automatic have ServiceStack generate a 204 (Return204NoContentForEmptyResponse enabled with "public void Post(request)")
On top of that we have:
PreRequestFilters.Insert(0, (httpReq, httpRes) =>
{
httpReq.UseBufferedStream = true;
});
since we use the raw body to validate a salted hash value (passed as a custom header) during a Request Filter for approval reasons that it comes from a correct source.
Overall we see in New Relic that the whole web service call takes an average around 700ms, which is a lot compared to the 5ms it actually takes to perform the coded process. So when we looked deeper in the data New Relic reports we saw some requests periodically take quite some time (10-150 seconds per request.) Drilling down in the reporting of New Relic we see that Applying the Pre-Request Filter takes time (see image below.) We were wondering why this could be the case and if it was related to the buffered stream on the Http Request object and what possibly could be done to correct this?
EDIT
Have been playing around with this some and still haven't found an answer.
Things done:
Moved the Virtual Folder out from a sub-folder location of the actual site folder (there are about 11 other Web Services located under this site)
Assigned this Web Service to use its own Application Pool so it is not shared with the main site and other Web Services under the site
Added the requirement to Web.Config for usage of Server GC as Phil suggested
Disabled the pre-request filter that turned on the usage of buffered stream (and bypass the code that used the RawBody)
Added more instrumentation to New Relic for a better drill-down (see image below)
I'm starting to wonder if this is a Windows Server/IIS limitation due to load. But would like to hear from someone that is more familiar with such.
i have a web application which which proccesses some request on some data that client selects, when the client selects more than 20 objects and clicks on proceed the client recieves this error, because the server takes a long time to process, however if the records are less and hence a timely response is recieved, no such error comes can someone help me on this?
i have increased the sessiontimeout as well as set the
Try adjusting the executionTimeout in your web.config...this only applies if debug is set to false however.
<httpRuntime
executionTimeout="some number"
/>
If this alone does not solve your issue, check out this blog post which goes into a bit more depth on how to structure your timeouts. Note the IIS reference towards the bottom...
I've got several web-services: asmx,wcf. At couple of them there are some methods, which take a lot of time for processing, but size of input data for these methods are small and it takes not much time to transfer on the wire. I want move to not sync model. Client passes data to service, service answers that data transfer was correct and process it at background thread witout connection with client. So agter transfering connection should be closed. IS it possible? Can u help me with articles or may be just google request.
John is right - Once you close an http connection, it is done. You can't get back to the same process.
So if you can use another technology that allows duplex on one connection (e.g. WCF), do it!
However,
if you have no choice but to use webservices,
here are three ways to make it work. You may get timeouts on any of them.
Option 1:
Forget the part about 'client answers data was correct.' Just have each thread make its request and wait for the data.
Option 2:
Now, assuming that won't work and you must do the validation, this way requires the client to make 2 requests.
First request: returns valid/invalid.
Second request: returns the long-running results.
Variation of option 2:
If you have timeout problems, you could have the first request generate a GUID or unique database key and start another process, passing it this key, and return the key to the client. (if you can get the server to allow you to start a process - depends on security settings/needs - if not you may be able to start an async thread and have it keep running after the websvc one ends?) The process will do the long task, update the row in the database w/ the unique id when finished, revealing the results plus a 'done' flag. The second request by the client could always return immediately and if the processing is not done, return that, if it is, return the results. The client will repeat this every 5 sec or so until done.
Hacks, I know, but we don't always have a choice for the technology we use.
Don't do this with ASMX web services. They weren't designed for that. If you must do it with ASMX, then have the ASMX pass the data off to a Windows Service that will do the actual work, in the background.
This is more practical with WCF.
We have been writing stuff to interact with the UK gov website and the way they handle something similar is that you send your request and data to the server and it responds saying, roughly, "thanks very much - we're processing it now, please call back later using this id" - all in an XML message. You then, at some point later, send a new http request to the service saying, essentially, "I'm enquiring about the status of this particular request id" and the server returns a result that says either it has processed OK, or processed with errors, or is still processing, please try again in xx seconds.
Similar to option 2 described previously.
It's a polling solution rather than a callback or 2 way conversation but it seems to work.
The server will need to keep, or have access to, some form of persistent table or log for each request state - it can contain eg, the id, the original request, current stage through the workflow, any error messages so far, the result (if any) etc. And the web service should probably have passed the bulk of the request off to a separate Windows service as already mentioned.