.NET Core 3.1 Console application on Windows, I'm trying to figure out why the httpClient.Timeout does not seem to be working when getting the content after using HttpCompletionOption.ResponseHeadersRead
static async Task Main(string[] args)
{
var httpClient = new HttpClient();
// if using HttpCompletionOption this timeout doesn't work
httpClient.Timeout = TimeSpan.FromSeconds(5);
var uri = new Uri("http://brokenlinkcheckerchecker.com/files/200MB.zip");
// will not timeout
//using var httpResponseMessage = await httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead);
// will timeout after 5s with a TaskCanceledException
var httpResponseMessage = await httpClient.GetAsync(uri);
Console.WriteLine($"Status code is {httpResponseMessage.StatusCode}. Press any key to get content");
Console.ReadLine();
Console.WriteLine("getting content");
var html = await httpResponseMessage.Content.ReadAsStringAsync();
Console.WriteLine($"finished and length is {html.Length}");
}
Have also tried a CancellationToken
// will not timeout
var cts = new CancellationTokenSource(5000);
using var httpResponseMessage = await httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead,
cts.Token);
and ReadAsStreamAsync
// will not timeout
using (Stream streamToReadFrom = await httpResponseMessage.Content.ReadAsStreamAsync())
{
string fileToWriteTo = Path.GetTempFileName();
using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
{
await streamToReadFrom.CopyToAsync(streamToWriteTo);
}
}
I learned about HttpCompletionOption from this great article:
https://www.stevejgordon.co.uk/using-httpcompletionoption-responseheadersread-to-improve-httpclient-performance-dotnet
Update
Using #StephenCleary answer below of passing the cancellationToken into the CopyToAsync method this now works as expected.
I've included the updated code below which shows copying into a MemoryStream then into a string, which I found tricky to find how to do. For my use case this is good.
string html;
await using (var streamToReadFrom = await httpResponseMessage.Content.ReadAsStreamAsync())
await using (var streamToWriteTo = new MemoryStream())
{
await streamToReadFrom.CopyToAsync(streamToWriteTo, cts.Token);
// careful of what encoding - read from incoming MIME
html = Encoding.UTF8.GetString(streamToWriteTo.ToArray());
}
I would expect HttpClient.Timeout to only apply to the GetAsync part of the request. HttpCompletionOption.ResponseHeadersRead means "consider the Get complete when the response headers are read", so it's complete. So the problem is that it just doesn't apply to reading from the stream.
I recommend using Polly's Timeout instead of HttpClient.Timeout; Polly is a generic library that can be used to timeout any operation.
If you don't want to use Polly at this time, you can pass the CancellationToken to Stream.CopyToAsync.
Related
I got a weird situation when using C# HttpClient. I am trying to use the HttpCompletionOption.ResponseHeadersRead option in GetAsync to get response headers without content as quickly as possible. But when downloading files, I am in await GetAsync until the whole content is downloaded over the network (i checked this with Fiddler). I am attaching an example code that downloads a 1Gb test file. The example application will hang in the await client.GetAsync until all file content is received over the network. How do I get control back when the headers have finished receiving and not wait for the complete content transfer over the network?
using System;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
private const int HttpBufferSize = 81920;
private static async Task Main(string[] args)
{
var url = new Uri("http://212.183.159.230/1GB.zip");
await DownloadFileAsync(#"C:\1GB.zip", url, CancellationToken.None).ConfigureAwait(false);
}
private static async Task DownloadFileAsync(string filePath, Uri fileEndpoint,
CancellationToken token)
{
using var client = new HttpClient();
using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);
await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
}
}
You are sending a GET request. If you only require the headers then you can use HEAD request. An example for HttpClient:
client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url))
Caution: Servers can block HEAD requests so make sure to handle gracefully. For example, fallback to GET request if the response fails but it will be at the cost of speed.
I have identified the reason for this behavior. The reason was Fiddler. It acted as a proxy and did not seem to redirect partially received responses. To check this, I've added console output for each of the operations:
Console.WriteLine($"Start GetAsync - {DateTime.Now}");
using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);
Console.WriteLine($"End GetAsync - {DateTime.Now}");
response.EnsureSuccessStatusCode();
await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);
Console.WriteLine($"Start CopyToAsync - {DateTime.Now}");
await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
Console.WriteLine($"End CopyToAsync - {DateTime.Now}");
Results with running program Fiddler:
Start GetAsync - 30.06.2021 17:46:03
End GetAsync - 30.06.2021 17:46:49
Start CopyToAsync - 30.06.2021 17:46:49
End CopyToAsync - 30.06.2021 17:46:51
Results without Fiddler:
Start GetAsync - 30.06.2021 17:38:32
End GetAsync - 30.06.2021 17:38:32
Start CopyToAsync - 30.06.2021 17:38:32
End CopyToAsync - 30.06.2021 17:39:48
Conclusion: be careful with proxies
Good day,
I have an issue with a custom response in API Gateway Ocelot with Middleware.
inside FormatResponse(context.Response) I change response for specific endpoint and I see the new response on debug but I receive the original Response in final result on postman.
ex :
original Response
{
"name":"mo"
}
after a change will be
{
"name":"mo123"
}
my code
// .NET Core 3.1
public async Task InvokeAsync(HttpContext context)
{
context.Request.EnableBuffering();
var builder = new StringBuilder();
var request = await FormatRequest(context.Request);
builder.Append("Request: ").AppendLine(request);
builder.AppendLine("Request headers:");
foreach (var header in context.Request.Headers)
{
builder.Append(header.Key).Append(':').AppendLine(header.Value);
}
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using var responseBody = new MemoryStream();
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response); // here ,i see new respose
builder.Append("Response: ").AppendLine(response);
builder.AppendLine("Response headers: ");
foreach (var header in context.Response.Headers)
{
builder.Append(header.Key).Append(':').AppendLine(header.Value);
}
//Save log to chosen datastore
_logger.LogInformation(builder.ToString());
//Copy the contents of the new memory stream (which contains the response) to the original
// stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
text = CustomRes(text); //************************ here, I change response
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
Reference: https://www.abhith.net/blog/dotnet-core-api-gateway-ocelot-logging-http-requests-response-including-headers-body/
The best answer from Richard Deeming:
https://www.codeproject.com/Questions/5294847/Fix-issue-with-custom-response-NET-core-API-gatewa
After 2-4 downloading of videos data from API using HttpClient suddenly prompt error.
Here's my code:
public async Task<byte[]> GetMedia(string id)
{
var api = $"/api/v1/download/{id}";
var Uri = $"{MccBaseURL}{api}";
byte[] responseBody;
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("No");
try
{
HttpResponseMessage response = await httpClient.GetAsync(Uri);
response.EnsureSuccessStatusCode();
responseBody = await response.Content.ReadAsByteArrayAsync();
if (response.IsSuccessStatusCode)
{
return responseBody;
}
}
catch (Exception ex)
{
Debug.Print(ex.Message);
throw;
}
}
Then below is the error faced:
Additional error info:
Error
Please help me?
First, you should dispose your HttpResponseMessage, as you have in your answer, but not in the original question.
The most likely issue, though, is your use of DefaultRequestHeaders. You should only use this for headers that apply to every request that the HttpClient instance will send, and then you should set them only once, when you create the client, as the documentation implies ("Headers set on this property don't need to be set on request messages again").
While HttpClient is essentially thread-safe, the DefaultRequestHeaders (and BaseAddress) properties are not. You're changing these values while the client instance is potentially busy using them elsewhere. It's not clear whether you're using the singleton HttpClient elsewhere as well, possibly changing the default headers there too, but if so that would significantly increase the chances of issues arising.
Some additional references about the non-thread-safety of these properties:
https://github.com/dotnet/dotnet-api-docs/issues/1085
http://www.michaeltaylorp3.net/httpclient-is-it-really-thread-safe/
https://github.com/MicrosoftDocs/architecture-center/issues/935
I found an answer which is:
public async Task<bool> GetMedia(string saveDir, string id)
{
var api = $"/api/v1/download/{id}";
var Uri = $"{MccBaseURL}{api}";
using (HttpClient client = new HttpClient())
{
using (HttpResponseMessage response = await client.GetAsync(Uri, HttpCompletionOption.ResponseHeadersRead))
using (System.IO.Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
string fileToWriteTo = System.IO.Path.GetTempFileName();
using (System.IO.FileStream streamToWriteTo = new System.IO.FileStream(saveDir, System.IO.FileMode.Create))
{
await streamToReadFrom.CopyToAsync(streamToWriteTo);
return true;
}
}
}
}
It was really memory something problem which continuously using same HttpClient over and over again. So I created a new instance. I'm a super noob! Sorry!
I'm trying to use the Azure media services with REST api, in a xamarin shared project on visual studio 2013. This is the code i use to get the access token.
public HttpFactory()
{
string token = GetToken(serviceURI).Result;
//some more methods also async tasks
}
private async Task<string> GetToken(Uri serviceUri)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serviceURI);
request.Accept = "application/json";
request.Method = "GET";
request.Headers["Host"] = "media.windows.net";
request.Headers["x-ms-version"] = "2.9";
request.Headers["Authorization"] = "Bearer " + token;
var response = (HttpWebResponse) await request.GetResponseAsync();
if (response.StatusCode == HttpStatusCode.MovedPermanently)
{
serviceURI = new Uri(response.Headers["Location"]);
HttpWebRequest req = ( HttpWebRequest)WebRequest.Create(serviceURI);
var res = (HttpWebResponse)await req.GetResponseAsync();
using (Stream responseStream = res.GetResponseStream())
{
using (StreamReader reader = new StreamReader(responseStream))
{
string str = reader.ReadToEnd();
// var test = JsonConvert.DeserializeObject(str);
JToken jtoken = JsonConvert.DeserializeObject<JToken>(str);
return jtoken["access_token"].Value<string>();
}
}
}
return "";
}
But when the compiler reaches -
var response = (HttpWebResponse) await request.GetResponseAsync();
it skips the rest of the code, and i never get the response. I know the code is working - because it works just fine without the task, in a async void method.
Anyone knows how to fix this, or am i doing something wrong? I also tried this in vs2015 but its the same.
You have a deadlock over the UI thread.
You're blocking the thread with Task.Result when it is needed to complete the async method which will complete the task that it's waiting on.
That's why you shouldn't block synchronously on asynchronous code. You should await the task returned from GetToken instead:
string token = await GetToken(serviceURI);
If you can't use async in that method, either move that logic to a different method (e.g. OnLoad event handler).
Another solution would be to use ConfigureAwait on the GetResponseAsync task and so the rest of the method wouldn't run on the UI thread, avoiding the deadlock:
var response = (HttpWebResponse) await request.GetResponseAsync().ConfigureAwait(false);
I am making a lot of web api calls in my c# code. I don't know how to catch errors. Suppose, internet connection isn't working, then my code shows runtime error. How to I properly put them into try catch block, What's the general rule. All the articles I found were on how to throw back the error and error message.
example API calls:
WebResponse webResponse = webRequest.GetResponse();
string res = webResponse.ToString();
Also
using (var client = new HttpClient())
client.BaseAddress = new Uri(CairoBaseUrl);
var getStringTask = client.GetStringAsync(requestUrl);
response = await getStringTask;
And,
HttpResponseMessage response = await client.PostAsync(
url,requestContent);
if you are calling asp.net web api i would suggest you to use HttpClient which is done for that purposes
try
{
HttpResponseMessage response = await client.GetAsync("http://www.contoso.com/");
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
// Above three lines can be replaced with new helper method below
// string responseBody = await client.GetStringAsync(uri);
Console.WriteLine(responseBody);
}
catch(HttpRequestException e)
{
Console.WriteLine("\nException Caught!");
Console.WriteLine("Message :{0} ",e.Message);
}
This is example from MSDN how to deal with exceptions using http client
in you example you have
using (var client = new HttpClient())
client.BaseAddress = new Uri(CairoBaseUrl);
var getStringTask = client.GetStringAsync(requestUrl);
response = await getStringTask;
But it wont work since await operator can be used only with methods marked async so it should be
var getStringTask = await client.GetStringAsync(requestUrl);