C# HttpWebRequest - Stop waiting for response - c#

I am making a HEAD request to an external server to identify if my credentials are valid. However, the external server (that I don't have control over) is processing my request as if it's a GET (and starts generating a CSV to stream that can take up to 30 minutes to generate.)
If my credentials are incorrect, I immediately get back a 401. If the request takes a long time (more than 10 seconds), I know my credentials are correct b/c the external server has started to generate the CSV.
I want my request to timeout after 10 seconds. But I'm having issues w/ HttpWebRequest.Timeout and HttpWebRequest.ReadWriteTimeout... the values don't seem to be acknowledged & my request will wait seemingly forever for the full response.
How do I get the timeouts to actually happen?
Here's what I've got:
public async Task<bool> GetResponse(string url, string username, string secret)
{
HttpWebRequest _request = new HttpWebRequest(url);
NetworkCredential _credential = new NetworkCredential(username, secret);
CredentialCache _credentialCache = new CredentialCache { { url, "Basic", _credential } };
_request.PreAuthenticate = true;
_request.Credentials = _credentialCache;
// I set all 3 of these timeouts while testing but none of them seem to be acknowledged.
_request.ContinueTimeout = 10000;
_request.ReadWriteTimeout = 10000;
_request.Timeout = 10000;
try
{
using (HttpWebResponse _response = await _request.GetResponseAsync())
{
if (_response.StatusCode == HttpStatusCode.OK && _response.ContentType.Contains("text/csv")
{
return true; // successfully retrieved response.
}
return false; // failed to retrieve successful response.
}
}
catch (WebException ex)
{
if (ex.Status = WebExceptionStatus.Timeout)
{
return true; // timeout == success.
}
return false;
}
}

It took me way too long to figure out... HttpWebRequest.GetResponseAsync() does not acknowledge the Timeout. But HttpWebRequest.GetResponse() does. You have to use the synchronous version.
Also, I only needed to set the Timeout, not the ReadWriteTimeout or ContinueTimeout.
EDIT
I ultimately decided to use HttpClient instead of HttpWebRequest. Here's what that looks like:
public sealed class MyClass : IMyClass
{
private static readonly HttpClient _httpClient = new HttpClient();
public async Task<bool> MyMethod(string url, string username, string secret)
{
try
{
CancellationTokenSource _cts = new CancellationTokenSource(Timespan.FromSeconds(10));
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Basic",
Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{secret}"))));
HttpResponseMessage _response = await _httpClient.SendAsync(
new HttpRequestMessage(HttpMethod.Head, model.ApiUrl),
HttpCompletionOption.ResponseHeadersRead,
_cts.Token);
switch (_response.StatusCode)
{
case HttpStatusCode.OK:
return true;
default:
return false;
}
}
catch (TaskCanceledException)
{
return true;
}
}
}

Related

Does these HttpClient call be called in parallel?

I've a .NET Core web app that, once a web method (i.e. Test()) is invoked, call another remote api.
Basically, I do it this way (here's a POST example, called within a web method Test()):
public T PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
T returnValue = default(T);
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
HttpResponseMessage res = null;
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var task = Task.Run(() => client.PostAsync($"{baseAddress}/{url}", body));
task.Wait();
res = task.Result;
succeded = res.IsSuccessStatusCode;
if (succeded)
{
returnValue = JsonConvert.DeserializeObject<T>(res.Content.ReadAsStringAsync().Result);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(res)}");
}
}
}
catch (Exception ex)
{
Log($"PostRead error: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
}
return returnValue;
}
Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
Because if they will be called in serial, the last one will take the time of the previous ones, and that's not what I want.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
They will be called in parallel. Nothing in the code would make it in a blocking way.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
There are at least 2 things that need to be fixed here:
Properly using async/await
Properly using HttpClient
The first one is easy:
public async Task<T> PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var responseMessage = await client.PostAsync($"{baseAddress}/{url}", body);
var responseContent = await responseMessage.Content.ReadAsStringAsync();
if (responseMessage.IsSuccessStatusCode)
{
succeeded = true;
return JsonConvert.DeserializeObject<T>();
}
else
{
// log...
}
}
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}
The second one is a bit more tricky and can quite possibly be the cause of the problems you're seeing. Generally, unless you have some very specific scenario, new HttpClient() is plain wrong. It wastes resources, hides dependencies and prevents mocking. The correct way to do this is by using IHttpClientFactory, which can be very easily abstracted by using the AddHttpClient extension methods when registering the service.
Basically, this would look like this:
services.AddHttpClient<YourClassName>(x =>
{
x.BaseAddress = new Uri(baseAddress);
x.DefaultRequestHeaders.Add("Authorization", MyApiKey);
}
Then it's a matter of getting rid of all using HttpClient in the class and just:
private readonly HttpClient _client;
public MyClassName(HttpClient client) { _client = client; }
public async Task<T> PostRead<T>(string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
string json = JsonConvert.SerializeObject(entity);
var responseMessage = await _client.PostAsync($"{baseAddress}/{url}", body);
//...
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}
[HttpClient call using parallel and solve JSON Parse error]
You can use task. when for parallel call . but when you use N number of calls and try to parse as JSON this may throw an error because result concatenation change the Json format so we can use Jarray. merge for combining the result .
Code
public async Task < string > (string baseAddress, string url, out bool succeded, object entity = null) {
using(HttpClient client = new HttpClient()) {
string responseContent = string.Empty;
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var tasks = new List < Task < string >> ();
var jArrayResponse = new JArray();
int count = 10; // number of call required
for (int i = 0; i <= count; i++) {
async Task < string > RemoteCall() {
var response = await client.PostAsync($"{baseAddress}/{url}", body);
return await response.Content.ReadAsStringAsync();
}
tasks.Add(RemoteCall());
}
await Task.WhenAll(tasks);
foreach(var task in tasks) {
string postResponse = await task;
if (postResponse != "[]") {
var userJArray = JArray.Parse(postResponse);
jArrayResponse.Merge(userJArray);
}
}
responseContent = jArrayResponse.ToString();
return responseContent;
}
}

C# GetResponse() How to timeout dynamically created url?

This is my method, to which I am passing the url to check if it's active.
The link is being activated on the wowza service so it takes some time until it's "alive"
GetResponse is returning the 404 Error because the url is not reached.
Is there a way to get the timeout instead of 404 error if the url is not alive after specified time?
public async Task<IActionResult> GetLinkIsAlive([FromQuery] string url, [FromQuery] int timeout)
{
HttpWebResponse webResponse;
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Timeout = timeout;
webRequest.Method = "GET";
webResponse = webRequest.GetResponse() as HttpWebResponse;
return Ok(webResponse.StatusCode);
}
catch (WebException ex)
{
return BadRequest(ex.Message);
}
}
You can use connection pooling.
It’s using IHttpClientFactory that helps to maintain the pooling and lifetime of clients
In your startup class :
services.AddHttpClient<NameofyourService, NameofyourService>()
.AddTransientHttpErrorPolicy(
p => p.WaitAndRetryAsync(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(10)
}));
It requires Microsoft.Extensions.Http.Polly
You need to use HttpClient for your service. All added configurations will apply automatically.
My solution to this was calling the link in the while loop every 1s and await the whole Task.
private async Task<bool> IsLiveStreamAlive(string streamUrl, int retriesCount = 30)
{
try
{
bool res = false;
HttpClient client = new HttpClient();
Uri uri = new Uri(streamUrl);
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(uri);
var request = new HttpRequestMessage(HttpMethod.Get, uri);
while (retriesCount > 1)
{
await Task.Delay(1000);
retriesCount--;
HttpResponseMessage httpResponse = await client.GetAsync(uri);
res = httpResponse.StatusCode == HttpStatusCode.OK ? true : false;
if (res)
{
Log.Info(string.Format("Stream alive: {0}", streamUrl));
break;
}
}
return res;
}
catch (WebException ex)
{
Log.Error(ex);
}
return false;
}

Azure .NET Core Appservice locks up sending 502.3 because of async calls

I have this method running in a .NET Core 2.X app running in Azure app service. I have a remote server that we use this method to call from button presses in our Angular website. that calls a remote device.
Angular button --> .NET Core app service in Azure --> another app service --> internet\cell connected device. We wait for the response from the device to return a status code.
If I quickly send commands [2 or 3 in a second] to this method it causes the app service to stop responding until I restart it. I read this post and added the [, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false)].
However I can still freeze the entire app and require a restart from quickly sending commands to this method.
private async void SetEndPointValueAsync(string stunnelUrl, string username, string password)
{
try
{
//set the user name and password
var httpClientHandler = new HttpClientHandler()
{
Credentials = new NetworkCredential(username, password)
};
using (var client = new HttpClient(httpClientHandler))
{
using (var response = await client.GetAsync(stunnelUrl**, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false)**)
{
if (response.IsSuccessStatusCode)
{
LogInfo(typeof(IntegrationService), stunnelUrl, LogAction.EndpointUpdate);
}
else
{
//request failed.
LogWarning(typeof(IntegrationService), stunnelUrl, LogAction.DeviceRequest);
}
//using (var content = response.Content)
//{
// //do here your changes when required
//}
}
}
}
catch (Exception e)
{
LogErrorDetailed(e, typeof(IntegrationService), stunnelUrl, LogAction.DeviceRequest);
}
}
Generally, you don't want to create so many instances of HttpClient as you lose a lot of the benefits of the management it provides.
You could reduce some overhead by only having a single instance of it, like so...
private readonly HttpClient _client;
public ClassConstructor(HttpClient client)
{
_client = client ?? new HttpClient();
}
Then you could change your method to look something like this...
private async Task SetEndPointValueAsync(Uri stunnelUri, string username, string password)
{
if (stunnelUri == null) throw new ArgumentNullException(nameof(stunnelUri));
if (String.IsNullOrEmpty(username)) throw new ArgumentNullException(nameof(username));
if (String.IsNullOrEmpty(password)) throw new ArgumentNullException(nameof(password));
byte[] byteArray = Encoding.ASCII.GetBytes($"{username}:{password}");
string scheme = "Basic";
string parameter = Convert.ToBase64String(byteArray);
HttpRequestMessage request = new HttpRequestMessage();
request.Method = HttpMethod.Post;
request.RequestUri = stunnelUri;
request.Headers.Authorization = new AuthenticationHeaderValue(scheme, parameter);
try
{
HttpResponseMessage response = await _client.SendAsync(request);
// This will throw an HttpRequestException if there is a failure above
response.EnsureSuccessStatusCode();
// do your stuff here...
// { ... }
}
catch (HttpRequestException)
{
// { log your connection / bad request issue. }
}
catch (Exception)
{
// I don't recommend this, but since you had it...
// { handle all other exceptions }
}
}

ASP.NET web API - System.Threading.Tasks.TaskCanceledException: A task was canceled dilemma

I have an asp.net web API. (net framework 4.6.1) I am calling a 3rd party rest API (Product) in one of my actions. In functional tests, everything works as expected, no problem. But when I do spike tests (let's say, 100 users, 10 seconds ramp-up time) I am getting System.Threading.Tasks.TaskCanceledException: A task was canceled error after some successful responses. I googled and came across that it might be due to timeout. I added TimeSpan.FromSeconds(600); but still getting the same error.
Here is how I call this 3rd party API. Is there any problem with an async call? Would you please have a look at my code?
public class UtilitiesTest
{
private static readonly HttpClient _httpClient = new HttpClient();
//some not relevant code here
public static async Task<HttpResponseMessage> CallRazer(GameRequest gameRequest, string url)
{
try
{
FormUrlEncodedContent content = null;
if (url == "Product/")
{
try
{
//some code here
//Timeout
_httpClient.Timeout = TimeSpan.FromSeconds(600);
//Call Game
var response = await _httpClient.PostAsync("https://test.com/" + url, content);
return response;
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
finally
{
content.Dispose();
}
}
else
{
try
{
//some code here
//Call Game
var response = await _httpClient.PostAsync("https://test.com/" + url, content);
return response;
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
finally
{
content.Dispose();
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
}
and here is how I call Razer:
private async Task<HttpResponseMessage> CallProducts()
{
//some code here
#region Call Razer for products
//**Call Razer**
var response = await Utilities.CallRazer(products, "Product/");
var htmlResponse = await response.Content.ReadAsStringAsync();
var model = JsonConvert.DeserializeObject<ProductResponseDto>(htmlResponse);
// some code here
return response;
}
Somebody in another forum, suggested me to use HTTPWeb Request instead of Httpclient. So I changed my code like below and all my troubles are gone.
//HTTPWebRequest
var request = (HttpWebRequest) WebRequest.Create("http://test.com" + url);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
var keyValueContent = productRequest.ToKeyValue();
var formUrlEncodedContent = new FormUrlEncodedContent(keyValueContent);
var urlEncodedString = await formUrlEncodedContent.ReadAsStringAsync();
using (var streamWriter = new StreamWriter(await request.GetRequestStreamAsync()))
{
streamWriter.Write(urlEncodedString);
}
HttpWebResponse httpResponse = (HttpWebResponse) (await request.GetResponseAsync());
response = new HttpResponseMessage
{
StatusCode = httpResponse.StatusCode,
Content = new StreamContent(httpResponse.GetResponseStream()),
};
return response;
If it's still timing out after 600 seconds, then your web service just can't keep up.
But yes, a timeout will generate a TaskCanceledException, which is not intuitive at all. This will actually be changing in .NET Core at least (just fixed last week), but that doesn't help you.
Here is code I've used to rethrow a TimeoutException instead. The only other reason a TaskCanceledException could be thrown is if you passed a CancellationToken that ended up being cancelled, but you're not. So it's definitely a timeout.
} catch (TaskCanceledException) {
//Could have been caused by cancellation or timeout if you used one.
//If that was the case, rethrow.
//cancellationToken.ThrowIfCancellationRequested();
//HttpClient throws TaskCanceledException when the request times out. That's dumb.
//Throw TimeoutException instead and say how long we waited.
string time;
if (_httpClient.Timeout.TotalHours > 1) {
time = $"{_httpClient.Timeout.TotalHours:N1} hours";
} else if (_httpClient.Timeout.TotalMinutes > 1) {
time = $"{_httpClient.Timeout.TotalMinutes:N1} minutes";
} else if (_httpClient.Timeout.TotalSeconds > 1) {
time = $"{_httpClient.Timeout.TotalSeconds:N1} seconds";
} else {
time = $"{_httpClient.Timeout.TotalMilliseconds:N0} milliseconds";
}
throw new TimeoutException($"No response after waiting {time}.");
}

Task method returns TaskCanceledException was caught

I am trying to make a call to a service, the service is supposed to return all the records but i am getting a TaskCancelledException with a message A Task was cancelled and then the results are null. I know there is lots of data (tried with small data and works fine). How can I make it so it continues working (keep grabbing records) without throwing the "A Task was cancelled". Here is my method
public async Task<ActionResult<string>> ObtainData(string url, CancellationToken ct = default(CancellationToken))
{
Result<string> result = new Result<string>();
try
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url);
HttpResponseMessage response = await httpClient.SendAsync(request); //HttpClient
string jsonresult = await HandleReceipt(response, type);
result.Result = jsonresult;
}
catch (Exception ex)
{
result.Message = "Problem obtaining data.";
}
return result;
}
I see there is a cancellation token being passed, is there a way to handle the cancellation?. Also, this method is being called from here:
public async void GetJsonData()
{
string json = await ObtainData();
}
I would like it to complete getting the request without cancelling.
It sounds from your question and comments that your request is timing out. Try setting the HttpClient.Timeout property
public async Task<ActionResult<string>> ObtainData(string url, CancellationToken ct = default(CancellationToken))
{
Result<string> result = new Result<string>();
try
{
var request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "GET";
HttpResponseMessage response = await httpClient.SendAsync(request); //HttpClient
httpClient.TimeOut = TimeSpan.MaxValue // change this as you like
string jsonresult = await HandleReceipt(response, type);
result.Result = jsonresult;
}
catch (Exception ex)
{
result.Message = "Problem obtaining data.";
}
return result;
}
Edit
Regarding you comment about the exception, this is from MSDN:
The Timeout property must be set before the GetRequestStream or GetResponse method is called.
Create a new instance of HttpClient before making the request, or set it inside the code where you initialize the client (constructor i assume).
As Yuval Itzchakov mentioned you should set the Timeout property to a larger value (but it is a TimeSpan not an integer) and you should use the cancellation token in the HTTP request call if you are using with the async pattern.
And there is no need to use a global HttpClient (as Yuval pointed out) esspecially in this heavy async method calling nightmare (additionally i don't see the reason why should call anything inside the ObtainData async).
public async Task<ActionResult<string>> ObtainData(string url, CancellationToken ct = default(CancellationToken))
{
try
{
using (var httpClient = new HttpClient
{
Timeout = TimeSpan.MaxValue
})
{
using (var request = new HttpRequestMessage(HttpMethod.Get, url))
{
using (var response = await httpClient.SendAsync(request, ct))
{
return new ActionResult<string>()
{
Result = await HandleReceipt(response, type)
};
}
}
}
}
catch (Exception ex)
{
return new ActionResult<string>
{
Message = "Problem obtaining data. " + ex
};
}
}

Categories