We Keep Coding

How To Prevent HttpWebRequest with long response from timing-out Azure Web App

We are using an HttpWebRequest in C# to get data from an internet resource in our Azure Web App. The problem is that Azure has a limitation on how long it keeps the connection alive (around 240 seconds). Due to the nature of our application, the response will sometimes take longer than 240 seconds. When this happens, the webpage will go white, and the "View Source" will show zero source code (which has made this issue difficult to debug).
Here's some sample code to illustrate:
webRequest = WebRequest.Create(PAGE_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
webRequest.Timeout = Timeout.Infinite;
webRequest.KeepAlive = true;
StreamWriter requestWriter2 = new
StreamWriter(webRequest.GetRequestStream());
requestWriter2.Write(postString);
requestWriter2.Close();
WebResponse response = webRequest.GetResponse();
Stream stream = response.GetResponseStream();
Adding webRequest.Timeout and webRequest.KeepAlive did not solve the issue.
jbq on this thread mentioned he had a workaround by sending a "newline character every 5 seconds", however did not explain how to accomplish this exactly. He was answering a question about an Azure VM, but I think an Azure Web App would have similar behaviors with respect to what I believe are the load balancers that are responsible for the timeout.
The Question:
How can I send one HttpWebRequest, and then send another HttpWebRequest while the previous one is running with a blank line to maintain the connection and prevent the Azure load balancer(?) from timing out the azure application? Would a new session variable need to be used? Perhaps an asynchronous method? Do I need to send the "pinging" request before the main request? If so, how would this look in implementation? Or is it something else entirely? Please provide some source code as an example :)
Note: you do not need to use an HttpWebRequest to replicate this issue. Attach a debugger from within Visual Studio to a live Azure Web App. Place a breakpoint within Visual Studio at any piece of code. When that breakpoint is hit, after roughly 4 minutes you'll see the page in your browser stop loading and go white with an empty source. So, it's not specifically related to HttpWebRequest, but that is an operation that would typically cause this sort of issue since some responses take longer than 4 minutes.
*EDIT: I think what I am looking for is an implementation of Asynchronous methods. I will update this post as I find a satisfactory implementation.

If you are making an HttpWebRequest to an Azure Website then you use ServicePointManager.SetTcpKeepAlive on your client code which uses HttpWebRequest.
https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.settcpkeepalive(v=vs.110).aspx
The 4 minute timeout that you are talking about is an IDLE timeout over the TCP layer and setting this will ensure that your client (who is using HttpWebRequest) sends ACK packet over TCP so that the connection doesn't get idle.
If your web application is making a HttpWebRequest to some other service, you can still use this function but that will just ensure that the Idle timeout is not hit when calling the remote service. Your actual HTTP request to the Azure Webapp may still hit the 4 minute time and if the client to your Azure web app is not HttpWebRequest, then again the 4 minute idle timeout will bite you...
The best thing to do here is to change the code a bit to implement a JOB Model kind of pattern where-in you make a server call which returns a JOBID. The client then queries the server using this JOBID in a polling fashion and when the job completes on the server the status of this JOBID can be set to COMPLETED in which case the client can then retrieve the data. You can use Webjobs in Azure Webapps to achieve something like this.
Hope this helps...

C# Enforcing HttpWebRequest to use Tls12 instead of SSLv3

I have App that makes use of some web service and acquire data via JSON, all was working fine for quite long time, up until latest discoveries about SSLv3 being vulnerable to man-in-the-middle attacks and server owners turning off SSLv3 for good. My application started to have problems connecting and returned error "Request was aborted: cannot establish secure SSL/TLS connection". I've tried to look for solution and found information i got to add this code before creating web request:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback = delegate{
return true;
};
Unfortunately no luck here, app acts the same as before, and I have no clue if this code does nothing or there is still some problem with server. Error information is pretty vague and i have problem figuring where things go wrong.
Here is my code
...
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = GetRequestContentType();
request.Method = method.ToString();
request.Credentials = GetCredential(url);
request.PreAuthenticate = true;
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
...
I want to ask how to set Tls12 to be used as default and ensure that at my end request I make is with desired protocol.
If I confirm that my app at my end works fine, is there way to get more detailed information from server response and pinpoint precise reason of error?
Thanks for all answers and suggestions.
EDIT
Second part of question is solved, I found this tool http://www.telerik.com/download/fiddler it pretty much allows to see what is going on with outgoing and incoming data. There is also thing that this tool allow to decode SSL connections, enabling this option makes that my application starts to work. I assume that this app does something that make communication between my app and destination host possible. But i do still have no idea what it could be. And how to make my app to handle these connections properly by itself.

Being desperate made me to inspect whole source code (part responsible for getting data of the internet was 3rd party and until it worked fine there was no reason to change it) and I discovered that line
request.Credentials = GetCredential(url);
called method that in its body had
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
So all my attempts to change that value before creating httpwebrequest was overwritten. Changing SecurityProtocolType to Tls12 makes it all work now.

Why does HttpWebRequest.GetResponse() fail in an ASP.NET page under IIS?

I built a set of API's for one of our developers to consume in our web application. I did this in a .NET 4.0 class library project and wrote integration tests to ensure the API integrated with the backend service correctly. In the integration tests (a unit test project), as well as a console application, the API's work correctly and return all the expected results. However, when we execute the same API's from a ASP.NET web page that is running under IIS, the API fails at the following line of code:
using (HttpWebResponse response = (HttpWebRequest)request.GetResponse())
The failure is a WebException with a status of SendFailure and a socket error of ConnectionReset (10054) in the inner exception. The error is The underlying connection was closed: An unexpected error occurred on a send. This is using HTTPS, as well (hence the X509).
I already know that this is actually when the request is made, but I'm trying to pin-point what is different about an IIS environment that would prevent the stream from being able to write bytes over the network. I know that this is actually the web service server closing the connection before we get a chance to send our data, but I want to urge, again, that this same API works fine under a integration or unit test, or console application all day long.
I have already exhausted as many articles and posts on the internet that are related that I could find, including extensive Msdn documentation such as checking things related to the certificate, modifying HTTP headers service point properties. I'm truly at a loss because the code is not complicated, I've written web request code too many times to count, but here it is:
private string ExecuteServiceMessages(Uri serviceUrl, X509Certificate clientCertificate, string requestBody)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serviceUrl);
request.ClientCertificates.Add(clientCertificate);
request.Date = DateTime.Now;
request.Method = WebRequestMethods.Http.Post;
request.ContentType = MediaTypeNames.Text.Xml;
request.UserAgent = "******";
request.KeepAlive = false;
using (Stream requestStream = request.GetRequestStream())
using (StreamWriter writer = new StreamWriter(requestStream))
{
writer.Write(requestBody);
writer.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream responseStream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(responseStream))
{
string data = reader.ReadToEnd();
reader.Close();
return data;
}
}
The certificate is being loaded in by X509Certificate.CreateFromCertFile, where our certificate for testing is just in a directory of the website. We're not pulling it directly from the certificate store (unless we don't fully understand how certificates work when loaded from a file).

Is there a delay of any kind before you get the error (e.g. something that might indicate a timeout is occuring)?
A lot happens when the framework tries to validate that certificate. One of the things it may do (depending on the X509 policy you're using) is check the Certificate Revocation List, which requires a connection to the internet. This has bitten us a couple times when we tried to run the code on a server that is in an environment with limited internet access-- it spins for exactly 60 seconds then returns the error you are seeing (not very helpful). If this is it, you can change your CRL policy to offline, or edit your hosts file and override DNS so that the CRL check is performed using the loopback address-- it'll fail but at least you won't get a timeout.

You may be connecting internet through a proxy check your IE lan settings.
from c# you need to add proxy settings
var request = (HttpWebRequest)WebRequest.CreateHttp(url);
WebProxy proxy = new WebProxy("http://127.0.0.1:8888", true);
proxy.Credentials = new NetworkCredential("ID", "pwd", "Domain");
request.Proxy = proxy;
request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
request.Timeout = 1000 * 60 * 5;
request.Method = method;
return request.GetResponse();

C# Sending an HttpWebRequest without the servername

Basically what I've been trying to do is download a file off a server. The server sends a redirect automatically which is fine, but through packet sniffing a program that does successfully download the file I've found that the Headers (for the second request) are:
GET /path/to/file.txt
...
Host: server.com
Rather than the current response being generated (what I thought was standard):
GET www.server.com/path/to/file.txt
Using the normal HttpWebRequest method results in a 500 server error, and I get exceptions thrown when trying to use just the relative path as one would expect.
Using AllowAutoRedirect does not work for this scenario as the cookies are not handled properly, but even if I handle it manually the same error occurs.
How does one go about doing this (preferably without sockets :D)?

To be honest, I'm really not sure what you are asking, but you mentioned cookie troubles. As a total shot in the dark guess, are you setting the CookieContainer on your WebRequest?
request.CookieContainer = new CookieContainer();
request.AllowAutoRedirect = true;

HttpWebRequest doesn't work except when fiddler is running

This is probably the weirdest problem I have run into. I have a piece of code to submit POST to a url. The code doesn't work neither throws any exceptions when fiddler isn't running, However, when fiddler is running, the code posts the data successfuly. I have access to the post page so I know if the data has been POSTED or not. This is probably very non-sense, But it's a situation I am running into and I am very confused.
byte[] postBytes = new ASCIIEncoding().GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://myURL);
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10";
req.Accept = "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
req.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3");
req.Headers.Add("Accept-Language", "en-US,en;q=0.8");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = postBytes.Length;
req.CookieContainer = cc;
Stream s = req.GetRequestStream();
s.Write(postBytes, 0, postBytes.Length);
s.Close();

If you don't call GetResponseStream() then you can't close the response. If you don't close the response, then you end up with a socket in a bad state in .NET. You MUST close the response to prevent interference with your later request.

Close the HttpWebResponse after getting it.
I had the same problem, then I started closing the responses after each request, and Boom, no need to have fiddler running.
This is a pseudo of a synchronous code:
request.create(url);
///codes
httpwebresponse response = (httpwebresponse)request.getresponse();
/// codes again like reading it to a stream
response.close();

I had a similar problem recently. Wireshark would show the HTTPWebRequest not leave the client machine unless Fiddler was running. I tried removing proxy settings, but that didn't fix the problem for me. I tried everything from setting the request to HttpVersion.Version10, enabling/disabling SendChuck, KeepAlive, and a host of other settings. None of which worked.
Ultimately, I just checked if .Net detected a proxy and had the request attempt to ignore it. That fixed my issue with request.GetResponse() throwing an immediate exception.
IWebProxy proxy = request.Proxy;
if (request.Proxy != null)
{
Console.WriteLine("Removing proxy: {0}", proxy.GetProxy(request.RequestUri));
request.Proxy = null;
}

In my case when I had the same situation (POST only works when Fiddler is running) the code was sending the POST from an application running on IISExpress in a development environment behind a proxy to an external server. Apparently even if you have proxy settings configured in Internet Options the environment IIS is running in may not have access to them. In my work environment I simply had to update web.config with the path to our proxy's configuration script. You may need to tweak other proxy settings. In that case your friend is this MSDN page that explains what they are: http://msdn.microsoft.com/en-us/library/sa91de1e.aspx.
Ultimately I included the following in the application's web.config and then the POST went through.
<configuration>
<system.net>
<defaultProxy>
<proxy scriptLocation="http://example.com:81/proxy.script" />
</defaultProxy>
</system.net>
</configuration>

Well i have faced similar problem few weeks back and the reason was that when fiddler is running it changes the proxy settings to pass the request through Fiddler but when its closed the proxy somehow still remains and thus doesn't allow your request to go ahead on internet.
I tried by setting the IE's and Firefox's network settings to not to take any proxy and it worked.
Try this, may it be the same problem...

I ran into the same problem with Python - requests to a local server were failing with a 404, but then when I ran them with Fiddler running they were working correctly.
The real clue to the problem here is that Fiddler works by acting as a proxy for HTTP traffic so that all requests from the local machine go through Fiddler rather than straight out into the network.
In the exact situation I was in, I was making requests to a local server, regular traffic passes through a proxy and in Local Area Network (LAN) Settings for the network connection the Proxy server pane the Bypass proxy server for local addresses option was checked.
My suspicion is that the "Bypass proxy server for local addresses" is not necessarily picked up by the programming language, but the proxy server details are. Fiddler is aware of that policy, so requests through Fiddler work but requests direct from the programming language don't.
By setting the proxy for the request for the local server to nothing, it worked correctly from code. Obviously, that could be a gotcha if you find yourself moving from an internal to external server during deployment.

I faced the same scenario : I was POSTing to an endpoint behind Windows Authentication.
Fiddler keeps a pool of open connections, but your C# test or powershell script does not when it runs without fiddler.
So you can make the test/script to also maintain a pool of open authenticated connections, by setting the property UnsafeAuthenticatedConnectionSharing to true on your HttpWebRequest. Read more about it here, microsoft KB. In both cases in that article, you can see that they are making two requests. The first one is a simple GET or HEAD to get the authentication header (to complete the handshake), and the second one is the POST, that will use the header obtained before.
Apparently you cannot (sadness) directly do the handshake with POST http requests.

Always use using construct. it make sure all resource release after call
using (HttpWebResponse responseClaimLines = (HttpWebResponse)requestClaimLines.GetResponse())
{
using (StreamReader reader = new StreamReader(responseClaimLines.GetResponseStream()))
{
responseEnvelop = reader.ReadToEnd();
}
}
add following entries to webconfig file
<system.net>
<connectionManagement>
<add address="*" maxconnection="30"/>

I found the solution in increasing the default number of connections
ServicePointManager.DefaultConnectionLimit = 10000;