We Keep Coding

Azure Long-running external web service calls

I've written a windows service in C# that interfaces with a third party via external web service calls (SOAP). Some of the calls respond quickly and some slowly
(for example, quick = retrieving a list of currencies; slow = retrieving a historical value of currencies per product over many years)
The external service works fine when I run it on my local machine - the quick calls runs for about 20 seconds; the slow calls runs for about 30 minutes. I can't do anything about the speed of the 3rd party service and I don't mind the time it takes to return an answer..
My problem is that when I deploy my service to my Azure Virtual Machine: The quick call works perfectly like it does locally; the slow ones just never returns anything. I have tried exception handling, logging to files, logging to eventLog.
There is no clear indication of what goes wrong - it just seems that for whatever reason - the long running web service calls, never return successfully to Azure.
I've read somewhere that there is some sort of connection-recycling happening every 4 minutes, which I suspect is somehow causing the external web service response to land up somewhere in a void with the load balancer or whatever not knowing any longer whom requested the content.
I start by creating the request with the relevant SOAP envelope, like this:
HttpWebRequest tRequest = (HttpWebRequest)WebRequest.Create(endpoint);
Then i set the stuff, like this:
tRequest.ClientCertificates.Add(clientCertificate);
tRequest.PreAuthenticate = true;
tRequest.KeepAlive = true;
tRequest.Credentials = CredentialCache.DefaultCredentials;
tRequest.ContentLength = byteArray.Length;
tRequest.ContentType = #"text/xml; charset=utf-8";
tRequest.Headers.Add("SOAPAction", #"http://schemas.xmlsoap.org/soap/envelope/");
tRequest.Method = "POST";
tRequest.Timeout = 3600000;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; }; //the SSL certificate is bad
Stream requestStream = tRequest.GetRequestStream();
requestStream.Write(byteArray, 0, byteArray.Length);
requestStream.Close();
requestStream.Dispose(); //works fine up to this point.
WebResponse webResponse = tRequest.GetResponse(); //The slow calls never make it past this. Fast one does
Anyone else experienced something similar and any suggestions how to solve it please?
Many thanks
http://www.fourtimesfour.co.za

When you deploy to Azure (Cloud Service, Virtual Machine) there is always the Azure Load Balancer, which sits between your VMs and the Internet.
In order to keep resources equally available to all the cloud users, Azure Load Balancer will kill idle connections. What idle for Azure LB is - default is 4 minutes of no communication sent over established channel. So if you call a web service and there is absolutely no response on the pip for 4 minutes - your connection will get terminated.
You can configure this timeout, but I would really argue the need for keeping connection open that long. Yes, you can do nothing about it, besides looking for a service which has better design (i.e. either returns responses faster, or implements asynchronous calls where the first call to the service will just give you a task id, using which you can poll periodically to get a result)
Here is a good article on how to configure the Azure Load Balancer timeout. Be aware, the maximum timeout for Azure LB is 30 minutes.

httpWebRequest (The underlying connection was closed: The connection was closed unexpectedly.)

I am developing an C# application which logs data from a webserver. It sends the following post request to the webserver and awaits for the response.
/// <summary>
/// Function for obtaining testCgi data
/// </summary>
/// <param name="Parameters"></param>
/// <returns></returns>
private string HttpmyPost(string Parameters)
{
string str = "No response";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriTestCGI);
request.Method = "POST";
byte[] bytes = Encoding.UTF8.GetBytes(Parameters);
request.ContentLength = bytes.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
try
{
var result = reader.ReadToEnd();
stream.Dispose();
str = result.ToString();
reader.Dispose();
}
catch (WebException ex)
{
//System.Windows.Forms.MessageBox.Show(ex.Message);
System.Diagnostics.Trace.WriteLine(ex.Message);
}
finally
{
request.Abort();
}
return str;
}
I am getting the error
> "The underlying connection was closed: The connection was closed
> unexpectedly"
I have tried to debug the error, and I have used fiddler in order to check the post request as given from Firefox. To my surprise whenever Fiddler was my program was working perfectly. When I close fiddler I am having the same error.
I suspect that since Fiddler is acting as a proxy it may change some of the settings.
I have tried using webclient and the result was the same.
When I tried coding the request in python, everything worked as it should without no problem. Of cource I have the option of installing IronPython and wrapping that particular function, however I consider this overkill and lacking elegance, so I am pursuing a leaner approach. I suspect this is nothing more that a setting adjustment.
I have tried modifying and in my case it is indifferent.
request.Accept
request.ReadWriteTimeout
request.Timeout
request.UserAgent
request.Headers
request.AutomaticDecompression
request.Referer
request.AllowAutoRedirect
//request.TransferEncoding
request.Expect
request.ServicePoint.Expect100Continue
request.PreAuthenticate
request.KeepAlive
request.ProtocolVersion
request.ContentType
With or without the above adjustments the code works when Fiddler is capturing data.
Also it might be noteworthy that the program yields the error at
WebResponse response = request.GetResponse();
UPDATE:
Following #EricLaw suggestions I looked into Latency.
I found this article
HttpWebRequest gets slower when adding an Interval
which suggested turning of the Nagle Algorithm.
Now there are no closed connections, although there is a small lag in the overall response (when I use winforms, and not async).

I write a bit about how Fiddler can "magically" fix things here: http://blogs.telerik.com/fiddler/posts/13-02-28/help!-running-fiddler-fixes-my-app-
The issue you're encountering is actually a bug in the .NET Framework itself. The rules of HTTP are such that the server may close a KeepAlive connection at any time after sending the first response (e.g. it doesn't need to accept another request on the connection, even if the client requested KeepAlive behavior).
.NET has a bug where it expects that the server will include a Connection: close response header if it will close the connection after the response is complete. If the server closes the connection without the Connection: Close header (entirely valid per RFC2616), .NET will encounter the closed connection when attempting to send the next request on the connection and it will throw this exception. What .NET should be doing is silently creating a new connection and resending the request on that new connection.
Fiddler resolves this problem because it doesn't care if the server closes the connection, and it keeps the connection to the client alive. When the client sends its second request, Fiddler attempts to reuse its connection to the server, notices that it's closed, and silently creates a new connection.
You can mitigate this problem in your code by:
Disabling keepalive on the request (this hurts performance)
Catching the exception and retrying automatically
Changing the server to keep connections alive longer
Approach #3 only works if you control the server and because the client may be behind a gateway/proxy that closes connections after use, you should probably use approach #2 as well.

a suggestion and a question:
1) if you really want to see what's going on install Wireshark. It will show you exactly what's being sent / received. and it will make it possible to compare with the fiddler.
I guess you are missing a header, like request.ContentType = "...." but only wireshark will show you which one (is sent via your working alternative, while not sent by your HttpWebRequest).
2) are you getting the error inside the http response content, or is it an exception, and if it's an exception is it caught in your catch, or does it occur during the request, before your try statement?

Fiddler works as an Internet Proxy. If your code works while Fiddler is running, (and maybe also from a browser), then you may have a problem with your proxy settings.

C# HttpWebRequest to Socket 3 second delay in writing to Request Stream

I have a set up with a custom HTTP server using sockets, and a client that uses HttpWebRequest to connect to that server. The client is multi-threaded, using this code:
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 48;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = verb;
request.Timeout = timeout;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Proxy = null;
request.SendChunked = false;
/*offending lines*/
var writer = new StreamWriter(request.GetRequestStream(),Encoding.UTF8);
writer.Write(stringData);
writer.Close();
/*end offending lines*/
var response = (HttpWebResponse)request.GetResponse();
I have been running thousands of requests through this code, and it works very well most of the time. The average response time is about 200ms, including some work done on the server.
The problem is that about 0.2% of the requests are very slow. When measuring I notice that the slow requests are pausing for 3 seconds (+- 20ms) when getting and writing to the request stream. GetResponse() is still fast, and nothing on the server takes extra time.
The server is implemented as a listener Socket listening asyncronously on a port, using Listen() and BeginAccept(). After a connection is accepted, BeginRecieve is called on the new Socket (aquired from EndAccept()) and BeginAccept is called again on the listener socket. From what I've read, this is the way to do it.
So my question is: what happens during GetRequestStream() and writing to it? Is data actually being sent here or is it flushed on GetResponse()? How come some of the requests take 3 seconds longer without any apparent reason? Is my server implementation flawed in some way? The weird thing is that it is always 3 seconds, except sometimes it can be 9 seconds. It seems to be a multiple of 3, for some reason.
A few possible scenarios:
The connection is initiated on GetRequestStream() and the server does not accept it right away. How can I test if this is the case?
Something is going on behind the scenes on GetRequestStream() or writing to it.
Waiting for network hardware?
Waiting for DNS resolution?
Something else?
Any help is very appreciated.
EDIT: I've tried connecting through a web proxy, and then the delay moves from GetRequestStream() to GetResponse(), indicating that the problem indeed lies in the web server. It seems like it is not accepting the connection. I realize that there might be some delay between EndAccept() and BeginAccept(), but it should not amount to a 3 second delay, right?
I've also tried running my client single-threaded, and the problem seems to disappear. This suggests that the connection delay only occurs when multiple requests are made at the same time.

I guess I'm a bit too late, but I've encountered similar problem and found and article, which describes what seems to be pretty low-level reason for such an issue - http://www.percona.com/blog/2011/04/19/mysql-connection-timeouts/

Maybe its because you don't dispose the request stream. Try to replace your offending lines with the following:
var stream = request.GetRequestStream();
var buffer = Encoding.UTF8.GetBytes(stringData);
stream.Write(buffer, 0, buffer.Length);
stream.Dispose();

Preliminary testing suggests that the issue was indeed connection crowding on the server. With some restructuring of Socket code, I managed to get rid of the slow connections. I will run more testing tonight to make sure, but it looks promising!

Why does HttpWebRequest GetResponse block for such a long time?

I have some fairly simple code that uploads a photo or video to an endpoint (using HTTP PUT or POST). Every so often I see connection closed exceptions thrown, and in reality the photo/video was uploaded just fine, it's calling GetResponse where the exception occurs.
One thing I've noticed is that GetResponse can take an awful long time to process. Often longer than the actual upload time of the photo to the server. My code writes to the web server using RequestStream.Write.
I did a little test and uploaded about 40 photos/videos to the server that range in size from 1MB to 85MB and the time for GetResponse to return was anywhere from 3 to 40 seconds.
My question is, is this normal? Is this just a matter of how long the server I am uploading these files to is taking to process my request and respond? In looking at Fidder HTTP traces it seems to be the case.
FYI, my uploads are HTTP 1.0, Timeout values set to Infinite (both Timeout and ReadWriteTimeout)

If the server is genuinely taking a long time to return any data (as shown in Fiddler) then that's the cause of it. Uploading an 85MB attachment would take a long time to start with, and then the server has to process it. You can't do a lot about that - other than to use an asynchronous method if you're able to get on with more work before the call returns.
It's not entirely clear what Fiddler's showing you though - is it showing a long time before the server sends the response? If so, there's not much you can do. I'm surprised that the connection is being closed on you, admittedly. If, however, you're not seeing your data being written to the server for a while, that's a different matter.
Are you disposing the response returned? If not, you may have connections which are being kept alive. This shouldn't be a problem if it's explicitly HTTP 1.0, but it's the most common cause of "hanging" web calls in my experience.
Basically, if you don't dispose of a WebResponse it will usually (at least with HTTP 1.1 and keepalive) hold on to the connection. There's a limit to the number of connections which can be open to a single host, so you could end up waiting until an earlier response is finalized before the next one can proceed.
If this is the problem, a simple using statement is the answer:
using (WebResponse response = request.GetResponse())
{
...
}

Yes, the response time may be a lot longer than just the upload time. After the request has been sent to the server it has to be processed and a response has to be returned. There may be some time before the request is processed, and then the file typically is going to be saved somewhere. After that the server will create the response page that is sent back.
IIS handles only one request at a time from each user, so if you start another upload before the first one is completed, it will wait until the first one completes before it even starts to process the next.

HttpWebRequest not returning, connection closing

I have a web application that is polling a web service on another server. The server is located on the same network, and is referenced by an internal IP, running on port 8080.
Every 15 secs, a request is sent out, which receives an xml response with job information. 95% of the time, this works well, however at random times, the request to the server is null, and reports a "response forcibly closed by remote host."
Researching this issue, others have set KeepAlive = false. This has not solved the issue. The web server is running .NET 3.5 SP1.
Uri serverPath = new Uri(_Url);
// create the request and set the login credentials
_Req = (HttpWebRequest)WebRequest.Create(serverPath);
_Req.KeepAlive = false;
_Req.Credentials = new NetworkCredential(username, password);
_Req.Method = this._Method;
Call to the response:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
_ResponseStream = response.GetResponseStream();
The method for this is GET. I tried changing the timeout, but the default is large enough to take this into account.
The other request we perform is a POST to post data to the server, and we are getting the same issue randomly as well. There are no firewalls affecting this, and we ruled out the Virus scanner. Any ideas to help solving this is greatly appreciated!

Are you closing the response stream and disposing of the response itself? That's the most frequent cause of "hangs" with WebRequest - there's a limit to how many connections you can open to the same machine at the same time. The GC will finalize the connections eventually, but if you dispose them properly it's not a problem.

I wouldn't rule out network issues as a possible reason for problems. Have you run a ping to your server to see if you get dropped packets that correspond to the same times as your failed requests?

Set the timeout property of FtpWebRequest object to maximum i tried it with 4 GB File and it's working great.