We Keep Coding

C# HttpWebRequest - Stop waiting for response

I am making a HEAD request to an external server to identify if my credentials are valid. However, the external server (that I don't have control over) is processing my request as if it's a GET (and starts generating a CSV to stream that can take up to 30 minutes to generate.)
If my credentials are incorrect, I immediately get back a 401. If the request takes a long time (more than 10 seconds), I know my credentials are correct b/c the external server has started to generate the CSV.
I want my request to timeout after 10 seconds. But I'm having issues w/ HttpWebRequest.Timeout and HttpWebRequest.ReadWriteTimeout... the values don't seem to be acknowledged & my request will wait seemingly forever for the full response.
How do I get the timeouts to actually happen?
Here's what I've got:
public async Task<bool> GetResponse(string url, string username, string secret)
{
HttpWebRequest _request = new HttpWebRequest(url);
NetworkCredential _credential = new NetworkCredential(username, secret);
CredentialCache _credentialCache = new CredentialCache { { url, "Basic", _credential } };
_request.PreAuthenticate = true;
_request.Credentials = _credentialCache;
// I set all 3 of these timeouts while testing but none of them seem to be acknowledged.
_request.ContinueTimeout = 10000;
_request.ReadWriteTimeout = 10000;
_request.Timeout = 10000;
try
{
using (HttpWebResponse _response = await _request.GetResponseAsync())
{
if (_response.StatusCode == HttpStatusCode.OK && _response.ContentType.Contains("text/csv")
{
return true; // successfully retrieved response.
}
return false; // failed to retrieve successful response.
}
}
catch (WebException ex)
{
if (ex.Status = WebExceptionStatus.Timeout)
{
return true; // timeout == success.
}
return false;
}
}

It took me way too long to figure out... HttpWebRequest.GetResponseAsync() does not acknowledge the Timeout. But HttpWebRequest.GetResponse() does. You have to use the synchronous version.
Also, I only needed to set the Timeout, not the ReadWriteTimeout or ContinueTimeout.
EDIT
I ultimately decided to use HttpClient instead of HttpWebRequest. Here's what that looks like:
public sealed class MyClass : IMyClass
{
private static readonly HttpClient _httpClient = new HttpClient();
public async Task<bool> MyMethod(string url, string username, string secret)
{
try
{
CancellationTokenSource _cts = new CancellationTokenSource(Timespan.FromSeconds(10));
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Basic",
Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{secret}"))));
HttpResponseMessage _response = await _httpClient.SendAsync(
new HttpRequestMessage(HttpMethod.Head, model.ApiUrl),
HttpCompletionOption.ResponseHeadersRead,
_cts.Token);
switch (_response.StatusCode)
{
case HttpStatusCode.OK:
return true;
default:
return false;
}
}
catch (TaskCanceledException)
{
return true;
}
}
}

ASP.NET web API - System.Threading.Tasks.TaskCanceledException: A task was canceled dilemma

I have an asp.net web API. (net framework 4.6.1) I am calling a 3rd party rest API (Product) in one of my actions. In functional tests, everything works as expected, no problem. But when I do spike tests (let's say, 100 users, 10 seconds ramp-up time) I am getting System.Threading.Tasks.TaskCanceledException: A task was canceled error after some successful responses. I googled and came across that it might be due to timeout. I added TimeSpan.FromSeconds(600); but still getting the same error.
Here is how I call this 3rd party API. Is there any problem with an async call? Would you please have a look at my code?
public class UtilitiesTest
{
private static readonly HttpClient _httpClient = new HttpClient();
//some not relevant code here
public static async Task<HttpResponseMessage> CallRazer(GameRequest gameRequest, string url)
{
try
{
FormUrlEncodedContent content = null;
if (url == "Product/")
{
try
{
//some code here
//Timeout
_httpClient.Timeout = TimeSpan.FromSeconds(600);
//Call Game
var response = await _httpClient.PostAsync("https://test.com/" + url, content);
return response;
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
finally
{
content.Dispose();
}
}
else
{
try
{
//some code here
//Call Game
var response = await _httpClient.PostAsync("https://test.com/" + url, content);
return response;
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
finally
{
content.Dispose();
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
}
and here is how I call Razer:
private async Task<HttpResponseMessage> CallProducts()
{
//some code here
#region Call Razer for products
//**Call Razer**
var response = await Utilities.CallRazer(products, "Product/");
var htmlResponse = await response.Content.ReadAsStringAsync();
var model = JsonConvert.DeserializeObject<ProductResponseDto>(htmlResponse);
// some code here
return response;
}
Somebody in another forum, suggested me to use HTTPWeb Request instead of Httpclient. So I changed my code like below and all my troubles are gone.
//HTTPWebRequest
var request = (HttpWebRequest) WebRequest.Create("http://test.com" + url);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
var keyValueContent = productRequest.ToKeyValue();
var formUrlEncodedContent = new FormUrlEncodedContent(keyValueContent);
var urlEncodedString = await formUrlEncodedContent.ReadAsStringAsync();
using (var streamWriter = new StreamWriter(await request.GetRequestStreamAsync()))
{
streamWriter.Write(urlEncodedString);
}
HttpWebResponse httpResponse = (HttpWebResponse) (await request.GetResponseAsync());
response = new HttpResponseMessage
{
StatusCode = httpResponse.StatusCode,
Content = new StreamContent(httpResponse.GetResponseStream()),
};
return response;

If it's still timing out after 600 seconds, then your web service just can't keep up.
But yes, a timeout will generate a TaskCanceledException, which is not intuitive at all. This will actually be changing in .NET Core at least (just fixed last week), but that doesn't help you.
Here is code I've used to rethrow a TimeoutException instead. The only other reason a TaskCanceledException could be thrown is if you passed a CancellationToken that ended up being cancelled, but you're not. So it's definitely a timeout.
} catch (TaskCanceledException) {
//Could have been caused by cancellation or timeout if you used one.
//If that was the case, rethrow.
//cancellationToken.ThrowIfCancellationRequested();
//HttpClient throws TaskCanceledException when the request times out. That's dumb.
//Throw TimeoutException instead and say how long we waited.
string time;
if (_httpClient.Timeout.TotalHours > 1) {
time = $"{_httpClient.Timeout.TotalHours:N1} hours";
} else if (_httpClient.Timeout.TotalMinutes > 1) {
time = $"{_httpClient.Timeout.TotalMinutes:N1} minutes";
} else if (_httpClient.Timeout.TotalSeconds > 1) {
time = $"{_httpClient.Timeout.TotalSeconds:N1} seconds";
} else {
time = $"{_httpClient.Timeout.TotalMilliseconds:N0} milliseconds";
}
throw new TimeoutException($"No response after waiting {time}.");
}

Task fromAsync timeout

I have this piece of code to make an asynchronous HTTP request:
public static void MakeRequest(Uri uri, Action<RequestCallbackState> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null).ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(new RequestCallbackState(response.GetResponseStream()));
responseStream.Close();
response.Close();
});
}
It works, but I need to set a request timeout. I tried to use request.Timeout but don't seem to do anything. Is there a way to set up a task timeout in this code?
Edited to add a new timeout callback.
New code:
public static void MakeRequest(Uri uri, Action<RequestCallbackState> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
IAsyncResult t = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null).ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(new RequestCallbackState(response.GetResponseStream()));
responseStream.Close();
response.Close();
});
ThreadPool.RegisterWaitForSingleObject(t.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), request, 1000, true);
}
private static void TimeoutCallback(object state, bool timedOut)
{
if (timedOut)
{
Console.WriteLine("Timeout");
WebRequest request = (WebRequest)state;
if (state != null)
{
request.Abort();
}
}
}
Testing with:
HttpSocket.MakeRequest(new Uri("http://www.google.comhklhlñ"), callbackState =>
{
if (callbackState.Exception != null)
throw callbackState.Exception;
Console.WriteLine(GetResponseText(callbackState.ResponseStream));
});
Thread.Sleep(10000);

From the Timeout documentation:
The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.
This is because the framework forces you to handle the timeout yourself. You should be able to use the example code here, except pass the Task returned from the FromAsync call to the ThreadPool.RegisterWaitForSingleObject method.
Edit:
You need to put the registration on the original task, not the continuation:
public static void MakeRequest(Uri uri, Action<Stream> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
const int TimeoutPeriod = 1000;
Task<WebResponse> t = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
ThreadPool.RegisterWaitForSingleObject((t as IAsyncResult).AsyncWaitHandle, TimeoutCallback, request, TimeoutPeriod, true);
t.ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(response.GetResponseStream());
responseStream.Close();
response.Close();
});
}
This works for me (if I set the timeout duration very short, and hit this, I always timeout appropriately).

Check to see if site is online, limit the timeout

I'm trying to make a function that checks if a site is online or not, but is having some problem with the timeout. I want to limit it to a max 3 sec, if there is no respons within 3 sec I should see the page as offline.
My try:
class OnlineCheck
{
public static bool IsOnline(string url)
{
try
{
WebClient webclient = new WebClient();
webclient.Headers.Add(HttpRequestHeader.KeepAlive, "1000");
webclient.OpenRead(url);
}
catch { return false; }
return true;
}
}

The WebClient doesn't support timeout. But you can use the HttpWebRequest!
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Endpoint);
request.Timeout=3000;
request.GetResponse();

If you want to check that the site is online, you are not really interested in the content of the page, just that you get a response. To make that more efficient, you should only request the http headers. Here is a quick example on how you could do:
private static IEnumerable<HttpStatusCode> onlineStatusCodes = new[]
{
HttpStatusCode.Accepted,
HttpStatusCode.Found,
HttpStatusCode.OK,
// add more codes as needed
};
private static bool IsSiteOnline(string url, int timeout)
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
{
if (request != null)
{
request.Method = "HEAD"; // get headers only
request.Timeout = timeout;
using (var response = request.GetResponse() as HttpWebResponse)
{
return response != null && onlineStatusCodes.Contains(response.StatusCode);
}
}
}
return false;
}

Use HttpWebRequest rather than WebClient. HttpWebRequest class has a timeout property.

You can try this code:
System.Net.WebRequest r = System.Net.WebRequest.Create("http://www.google.com");
r.Timeout = 3000;
System.Net.WebProxy proxy = new System.Net.WebProxy("<proxy address>");
System.Net.NetworkCredential credentials = new System.Net.NetworkCredential();
credentials.Domain = "<domain>";
credentials.UserName = "<login>";
credentials.Password = "<pass>";
proxy.Credentials = credentials;
r.Proxy = proxy;
try
{
System.Net.WebResponse rsp = r.GetResponse();
}
catch (Exception)
{
MessageBox.Show("Is not avaliable");
return;
}
MessageBox.Show("Avaliable!");

static bool isOnline (string URL)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Timeout = 3000;
try
{
WebResponse resp = request.GetResponse();
}
catch (WebException e)
{
if (((HttpWebResponse)e.Response).StatusCode == HttpStatusCode.NotFound)
{
return false;
}
}
return true;
}

can I check if a file exists at a URL?

I know I can locally, on my filesystem, check if a file exists:
if(File.Exists(path))
Can I check at a particular remote URL?

If you're attempting to verify the existence of a web resource, I would recommend using the HttpWebRequest class. This will allow you to send a HEAD request to the URL in question. Only the response headers will be returned, even if the resource exists.
var url = "http://www.domain.com/image.png";
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "HEAD";
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
/* A WebException will be thrown if the status of the response is not `200 OK` */
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
Of course, if you want to download the resource if it exists it would most likely be more efficient to send a GET request instead (by not setting the Method property to "HEAD", or by using the WebClient class).

If you want to just copy & paste Justin's code and get a method to use, here's how I've implemented it:
using System.Net;
public class MyClass {
static public bool URLExists (string url) {
bool result = false;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try {
response = (HttpWebResponse)webRequest.GetResponse();
result = true;
} catch (WebException webException) {
Debug.Log(url +" doesn't exist: "+ webException.Message);
} finally {
if (response != null) {
response.Close();
}
}
return result;
}
}
I'll keep his observation:
If you want to download the resource, and it exists, it would be more efficient to send a GET request instead by not setting the Method property to "HEAD" or by using the WebClient class.

Below is a simplified version of the code:
public bool URLExists(string url)
{
bool result = true;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
try
{
webRequest.GetResponse();
}
catch
{
result = false;
}
return result;
}

If you are using a unc path or a mapped drive, this will work fine.
If you are using a web address (http, ftp etc) you are better off using WebClient - you will get a WebException if it doesn't exist.

public static bool UrlExists(string file)
{
bool exists = false;
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(file);
request.Method = "HEAD";
request.Timeout = 5000; // milliseconds
request.AllowAutoRedirect = false;
try
{
response = (HttpWebResponse)request.GetResponse();
exists = response.StatusCode == HttpStatusCode.OK;
}
catch
{
exists = false;
}
finally
{
// close your response.
if (response != null)
response.Close();
}
return exists;
}

I had the same problem to solve in asp.net core, I've solved with HttpClient
private async Task<bool> isFileExist(string url)
{
using (HttpClient client = new HttpClient())
{
var restponse = await client.GetAsync(url);
return restponse.StatusCode == System.Net.HttpStatusCode.OK;
}
}

My version:
public bool IsUrlExist(string url, int timeOutMs = 1000)
{
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "HEAD";
webRequest.Timeout = timeOutMs;
try
{
var response = webRequest.GetResponse();
/* response is `200 OK` */
response.Close();
}
catch
{
/* Any other response */
return false;
}
return true;
}

WebRequest will waiting long time(ignore the timeout user set) because not set proxy, so I change to use RestSharp to do this.
var client = new RestClient(url);
var request = new RestRequest(Method.HEAD);
request.Timeout = 5000;
var response = client.Execute(request);
result = response.StatusCode == HttpStatusCode.OK;

Thanks for all answers.
And I would like to add my implementation which includes default state when we get errors, for specific cases like mine.
private bool HTTP_URLExists(String vstrURL, bool vResErrorDefault = false, int vTimeOut = 1200)
{
bool vResult = false;
WebRequest webRequest = WebRequest.Create(vstrURL);
webRequest.Timeout = vTimeOut; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)webRequest.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) vResult = true;
else if (response.StatusCode == HttpStatusCode.NotFound) vResult = false;
else vResult = vResErrorDefault;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
{
var resp01 = (HttpWebResponse)ex.Response;
if (resp01.StatusCode == HttpStatusCode.NotFound)
{
vResult = false;
}
else
{
vResult = vResErrorDefault;
}
}
else
{
vResult = vResErrorDefault;
}
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
return vResult;
}

Anoter version with define timeout :
public bool URLExists(string url,int timeout = 5000)
{
...
webRequest.Timeout = timeout; // miliseconds
...
}

This works for me:
bool HaveFile(string url)
{
try
{
using (WebClient webClient = new WebClient())
{
webClient.DownloadString(url);
}
return true;
}
catch (Exception)
{
return false;
}
}