We are trying to implement user-determined (on a settings screen) optional gzip compression in our client which uses HttpClient, so we can log and compare performance across a number of different calls over a period of time. Our first attempt was to simply conditionally add the header as follows:
HttpRequestMessage request = new HttpRequestMessage(Method, Uri);
if (AcceptGzipEncoding)
{
_client.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
}
//Send to the server
result = await _client.SendAsync(request);
//Read the content of the result response from the server
content = await result.Content.ReadAsStringAsync();
This created the correct request, but the gzipped response was not decompressed on return, resulting in a garbled response. I found that we had to include the HttpClientHandler when constructing the HttpClient:
HttpClient _client = new HttpClient(new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip
});
This all works well, but we'd like to change whether the client sends the Accept-Encoding: gzip header at runtime, and there doesn't appear to be any way to access or change the HttpClientHandler after it's passed to the HttpClient constructor. In addition, altering the headers of the HttpRequestMessage object doesn't have any effect on the headers of the request if they are defined by the HttpClientHandler.
Is there any way to do this without recreating the HttpClient each time this changes?
Edit: I've also tried to modify a reference to the HttpClientHandler to change AutomaticDecompression at runtime, but that's throwing this exception:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
You're almost there with the first example, you just need to deflate the stream yourself. MS's GZipSteam will help with this:
HttpRequestMessage request = new HttpRequestMessage(Method, Uri);
if (AcceptGzipEncoding)
{
_client.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
}
//Send to the server
result = await _client.SendAsync(request);
//Read the content of the result response from the server
using (Stream stream = await result.Content.ReadAsStreamAsync())
using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
using (StreamReader reader = new StreamReader(decompressed))
{
content = reader.ReadToEnd();
}
If you want to use the same HttpClient and only want to enable compression for some requests, you are not able to use automatic decompression. When automatic decompression is enabled, the framework also resets the Content-Encoding header of the response. This means that you are unable to find out whether the response was really compressed or not. By the way, also the Content-Length header of the response matches the size of the decompressed content if you turn on automatic decompression.
So you need to decompress the content manually. The following sample shows an implementation for gzip-compressed content (as also shown in #ToddMenier's response):
private async Task<string> ReadContentAsString(HttpResponseMessage response)
{
// Check whether response is compressed
if (response.Content.Headers.ContentEncoding.Any(x => x == "gzip"))
{
// Decompress manually
using (var s = await response.Content.ReadAsStreamAsync())
{
using (var decompressed = new GZipStream(s, CompressionMode.Decompress))
{
using (var rdr As New IO.StreamReader(decompressed))
{
return await rdr.ReadToEndAsync();
}
}
}
else
// Use standard implementation if not compressed
return await response.Content.ReadAsStringAsync();
}
As per the comments above, recreating the HttpClient is really the only (robust) way to do this. Manual decompression can be achieved but it seems to be very difficult to reliably/efficiently determine whether the content has been encoded or not, to determine whether to apply decoding.
Related
I'm sending a request with HttpClient to a remote endpoint. I want to download the content and save it to a file as an UTF-8 string.
If the server would respond with the proper Content-Type text/plain; charset=utf-8, then the following code processes it just fine:
HttpClient client = new();
HttpResponseMessage res = await client.GetAsync(url);
string text = await res.Content.ReadAsStringAsync();
File.WriteAllText("file.txt", text);
However, the server always returns the basic Content-Type text/plain and I'm unable to get that as an UTF-8 string.
HttpClient cl = new();
HttpResponseMessage res = await cl.GetAsync(url);
string attempt1 = await res.Content.ReadAsStringAsync();
string attempt2 = Encoding.UTF8.GetString(await res.Content.ReadAsByteArrayAsync());
Stream stream = await res.Content.ReadAsStreamAsync();
byte[] bytes = ((MemoryStream)stream).ToArray();
string attempt3 = Encoding.UTF8.GetString(bytes);
I tried all three of these approaches, all resulted in scrambled characters due to the encoding mismatch. I don't have control over the server, so I can't change the headers.
Is there any way to force HttpClient to parse it as UTF-8? Why are the manual approaches not working?
I've built a Cloudflare worker to demonstrate this behavior and allow you to easily debug:
https://headers.briganreiz.workers.dev/charset-in-header
https://headers.briganreiz.workers.dev/no-charset
Edit: Turns out it was the GZip compression on the main server which I didn't notice. This question solved it for me: Decompressing GZip Stream from HTTPClient Response
I find it works well with these different classes WebRequest and HttpWebResponse. I have not added plumbing for resp.StatusCode etc but obviously presuming all went well is a tad naive.
Give it a go i am sure You'll find the WebRequest and HttpWebResponse more capable for dynamic requests (?)
var req = WebRequest.CreateHttp(url)
var getResponse = req.GetResponseAsync();
getResponse.Wait(ResponseTimeoutMilliseconds);
var resp = (HttpWebResponse)getResponse.Result;
using (Stream responseStream = resp.GetResponseStream())
{
var reader = new StreamReader(responseStream, Encoding.UTF8);
string content = reader.ReadToEnd();
}
Obviously once you have things working, you should absolutely use the ..Async versions but for debugging, since we already waited for response it is more convenient to simply step through i find, feel free to not take that middle step :)
I'm fairly new to .NET's HTTPClient class, hence kindly excuse if I sounded noob. I'm tryin to replicate Postman's POST request in C# .Net and written following code. However I'm not getting any response but StatusCode: 404. Could someone assist understanding where I'm going wrong?
Also I'd like to understand, how do set Body in following code.
var httpClient = new HttpClient
{
BaseAddress = new Uri("https://testURL.com"),
Timeout = TimeSpan.FromMinutes(10)
};
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("audio/wav"));
httpClient.DefaultRequestHeaders.Add("Authorization", "Basic ldjfdljfdlfjdsjfdsl");
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("model", "Test"),
});
var result = httpClient.PostAsync("api/v1/recognize", content).Result;
Here is what I'm doing in Postman and it works:
"Params" in Postman refers to query parameters which are appended to the URL. You'll see that the URL in Postman contains the parameters you added in the "Params" tab:
However, it seems those are just dummy values you've entered so perhaps you don't need them? In any case, the way you add query parameters to the request for HttpClient is a little different as it needs to be added to the URL.
After that you also need to add the audio file as content to your request. At the moment you're setting the "Accept" header to "audio/wav" but you probably want to set the "Content-Type" header instead (or are you expecting a WAV file to be returned in the response too?).
As far as I can see this is what you're missing:
using (var httpClient = new HttpClient())
{
httpClient.Timeout = TimeSpan.FromMinutes(10);
// Set request headers
httpClient.DefaultRequestHeaders.Add("Authorization", "Basic ldjfdljfdlfjdsjfdsl");
// Set query parameters
var uriBuilder = new UriBuilder("https://testURL.com/api/v1/recognize");
uriBuilder.Query = "model=Test";
// Build request body
// Read bytes from the file being uploaded
var fileBytes = File.ReadAllBytes(wavFilePath);
// Create request content with metadata/headers to tell the
// backend which type of data (media type) is being uploaded
var byteArrayContent = new ByteArrayContent(fileBytes);
byteArrayContent.Headers.ContentType = MediaTypeHeaderValue.Parse("audio/wav");
// Wrap/encode the content as "multipart/form-data"
// See example of how the output/request looks here:
// https://dotnetfiddle.net/qDMwFh
var requestContent = new MultipartFormDataContent
{
{byteArrayContent, "audio", "filename.wav"}
};
var response = await httpClient.PostAsync(uriBuilder.Uri, requestContent);
}
I haven't tested this of course against your application, but it should be something along the lines of this. It might be that the backend doesn't expect "multipart/form-data" and just needs the "audio/wav". I can't see the output headers in your Postman screenshots, but if so, you can use byteArrayContent directly instead of wrapping it in MultipartFormDataContent.
Note: Don't use httpClient.PostAsync(...).Result. If you want to use the asynchronous method, you should await it. Depending on your code, using Result might give you problems if you're not careful. And remember to dispose the HttpClient after use (easiest solution is to use a using statement). If you plan on reusing the HttpClient for more requests, you can avoid disposing it until you're done.
I'm trying to create a .net core http client that would stream bytes to the remote server in a multipart/form-data request. The bytes to be sent are generated dynamically by another task. At the moment of sending of the request I don't have the full data to be send in memory or anywhere - this data will be produced during the course of the request by another task. I tried to use the MemoryStream as a source for the StreamContent and feed this memory stream from another task but it's not suited for such case. Is there any stream in the standard library that could be use?
It would be best if I could access the request's stream and write (push) directly to it but I'm not sure how to do that in the case of multipart request.
class Program
{
static async Task Main(string[] args)
{
using var httpClient = new HttpClient();
using var contentData = new MultipartFormDataContent();
Stream sharedStream = SomeKindOfStream(); // <-- Need a stream that another task could write to and close
var streamContent = new StreamContent(sharedStream);
contentData.Add(streamContent, "file", Guid.NewGuid().ToString());
using (var httpResponseMessage =
await httpClient.PostAsync("http://localhost:5000/async/abcd",
contentData))
{
Console.WriteLine($"Got Response: '{await httpResponseMessage.Content.ReadAsStringAsync()}'");
}
}
}
I am developing a .net core middle-ware (api) and thinking to use pipes with following flow, Can someone tell me is this is a good approach and comply best practices or should i use different strategy.
Request comes to api
Authorization pipe validates the request.
Request pipe logs the request into db.
Request goes to api and perform action and return a result.
Response pipe gets the response and logs into db and return the result to client.
I know that we can read stream only time (point 3) but i figured this out already and after reading i have attach it to request stream again.
So, confusion is where to write the response? In api? or in separate pipe.
If i do it in separate pipe then i am handling my response two time (one is creating response in api, second is reading response in separate pipe) which is a performance hit.
Can i pass the data from point number 4 to 5 like from api to my pipe and from there that response should added into response stream and if it is correct then how can i pass the data from api to pipe?
Yes, response stream can only be read once. You can use the MemoryStream to load the response , reference article :
First, read the request and format it into a string.
Next, create a dummy MemoryStream to load the new response into.
Then, wait for the server to return a response.
Finally, copy the dummy MemoryStream (containing the actual response) into the original stream, which gets returned to the client.
Code sample :
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
And register the middleware :
app.UseMiddleware<RequestResponseLoggingMiddleware>();
Our WinRT app is incredibly slow when opening connections to our servers. Requests take ~500ms to run. This is blocking some of our scenarios.
When debugging, we noticed that when Fiddler is active, the requests are much faster - ~100ms per request. Some searches later we understood that was because Fiddler was using Keep-Alive connections when proxying calls, which makes our proxied calls much faster.
We double-checked this in two ways.
We set UseProxy to false and observed that the request went back to being slow.
We turned off Fiddler's "reuse connections" option and observed that the requests went back to being slow.
We tried enabling keep-alive through the Connection header (.Connection.Add("Keep-Alive")) but this does not seem to have any effect - in fact, the header seems to be blatantly ignored by the .NET component and is not being sent on the request (again, by inspecting thru Fiddler).
Does anyone know how to set keep-alive on requests in Windows 8, WinRT, HttpClient class?
The following sets the correct headers to turn on keep-alive for me (client is an HttpClient)
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = false;
// The next line isn't needed in HTTP/1.1
client.DefaultRequestHeaders.Connection.Add("Keep-Alive");
If you want to turn keep-alive off, use
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = true;
Try using the HttpContent class to add the headers - something like this based on (but untested) http://social.msdn.microsoft.com/Forums/en-CA/winappswithcsharp/thread/ce2563d1-cd96-4380-ad41-6b0257164130
Behind the scenes HttpClient uses HttpWebRequest which would give you direct access to KeepAlive but since you are going through HttpClient you can't directly access that property on the HttpWebRequest class.
public static async Task KeepAliveRequest()
{
var handler = new HttpClientHandler();
var client = new HttpClient(handler as HttpMessageHandler);
HttpContent content = new StringContent(post data here if doing a post);
content.Headers.Add("Keep-Alive", "true");
//choose your type depending what you are sending to the server
content.Headers.ContentType = new MediaTypeHeaderValue("application/x-www-form-urlencoded");
HttpResponseMessage response = await client.PostAsync(url, content);
Stream stream = await response.Content.ReadAsStreamAsync();
return new StreamReader(stream).ReadToEnd();
}
EDIT
Since you only want GET, you can do that with:
public static async Task KeepAliveRequest(string url)
{
var client = new HttpClient();
var request = new HttpRequestMessage()
{
RequestUri = new Uri("http://www.bing.com"),
Method = HttpMethod.Get,
};
request.Headers.Add("Connection", new string[] { "Keep-Alive" });
var responseMessage = await client.SendAsync(request);
return await responseMessage.Content.ReadAsStringAsync();
}