We Keep Coding

Streaming bytes to remote in a multipart HTTP request .net core

I'm trying to create a .net core http client that would stream bytes to the remote server in a multipart/form-data request. The bytes to be sent are generated dynamically by another task. At the moment of sending of the request I don't have the full data to be send in memory or anywhere - this data will be produced during the course of the request by another task. I tried to use the MemoryStream as a source for the StreamContent and feed this memory stream from another task but it's not suited for such case. Is there any stream in the standard library that could be use?
It would be best if I could access the request's stream and write (push) directly to it but I'm not sure how to do that in the case of multipart request.
class Program
{
static async Task Main(string[] args)
{
using var httpClient = new HttpClient();
using var contentData = new MultipartFormDataContent();
Stream sharedStream = SomeKindOfStream(); // <-- Need a stream that another task could write to and close
var streamContent = new StreamContent(sharedStream);
contentData.Add(streamContent, "file", Guid.NewGuid().ToString());
using (var httpResponseMessage =
await httpClient.PostAsync("http://localhost:5000/async/abcd",
contentData))
{
Console.WriteLine($"Got Response: '{await httpResponseMessage.Content.ReadAsStringAsync()}'");
}
}
}

Forwarding a stream in c# api

In my service we need to get a zip file created by another service and return it.
This is my code (code has been simplified for the question):
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
}
With the id I can gather the info I need to create the fileIds string and call the other service.
Here's the api on the other service (code has been simplified for the question):
[HttpGet("bulk/{idList}")]
public async Task<IActionResult> DownloadBulk(string idList)
{
var ids = string.IsNullOrEmpty(idList) ? new List<int>() : idList.Split(',').Select(x => Convert.ToInt32(x));
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var index = archive.CreateEntry("hello.txt");
using (var entryStream = index.Open())
using (var streamWriter = new StreamWriter(entryStream))
{
streamWriter.Write("hello");
}
}
var byteArray = memoryStream.ToArray();
return File(byteArray, "application/octet-stream", "media_files.zip");
}
but when the client tries to open the zip we get
Exception has occurred. ArchiveException (FormatException: Could not
find End of Central Directory Record)
I'm absolutely not confident about these two lines of the /mediafiles/{id}
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
And probably the issue might be there.
I just need to forward back the file-service response, but I don't know why

I believe the problem you're experiencing is that in DownloadMediaFiles(int id) you are using an HttpClient that gets disposed when leaving the function scope. The stream you created from the response therefore is closed and disposed of as well, before the response payload has finished writing its contents to the client. The client therefore receives an incomplete zip-file that you can't open. See here for reference.
In this answer there's a simple solution you could use, which is simply to read the response stream (the response stream from $"http://file-service/bulk/{fileIds}") into a byte array and then pass it to the response to the client:
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var byteArr = await response.Content.ReadAsByteArrayAsync();
return File(byteArr, "application/octet-stream", "media_files.zip");
You might realize that this means loading the whole file into memory, which can quickly become an issue if you plan on working with large files or even with medium sized files if the API is supposed to be used by a lot of clients simultaneously. Your web application would most likely run out of memory at some point.
Instead, I came upon this article which shows how you can return the contents of the stream from a request using an HttpClient. You should be able to stick with the first section of that article (all the ZIP-file and callback-based response stuff is unrelated).
To recap on that article all you need is something like this:
// Your ControllerClass.cs
private static HttpClient Client { get; } = new HttpClient();
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
var stream = await Client.GetStreamAsync($"http://file-service/bulk/{fileIds}");
return File(stream, "application/octet-stream", "media_files.zip");
}
You'll notice, that the stream object is not disposed of here but ASP.Net Core does this for you as part of writing the response payload to the client. The Client which is stored in a static global variable is not disposed of either, which means you can reuse it between requests (it's usually recommended not to instantiate a new HttpClient everytime you need it). ASP.Net Core 2.1 and up has special support for dependency injecting the client for you through the IHttpClientFactory interface. I would suggest you do that instead of a static variable. Read here for the most basic usage of injecting the client factory.
Now you should be able to enjoy streaming the file contents directly from your "other service" without loading it into memory in your API web application.

The await httpClient.GetByteArrayAsync() in HttpClient suddenly stop after many videos downloaded?

After 2-4 downloading of videos data from API using HttpClient suddenly prompt error.
Here's my code:
public async Task<byte[]> GetMedia(string id)
{
var api = $"/api/v1/download/{id}";
var Uri = $"{MccBaseURL}{api}";
byte[] responseBody;
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("No");
try
{
HttpResponseMessage response = await httpClient.GetAsync(Uri);
response.EnsureSuccessStatusCode();
responseBody = await response.Content.ReadAsByteArrayAsync();
if (response.IsSuccessStatusCode)
{
return responseBody;
}
}
catch (Exception ex)
{
Debug.Print(ex.Message);
throw;
}
}
Then below is the error faced:
Additional error info:
Error
Please help me?

First, you should dispose your HttpResponseMessage, as you have in your answer, but not in the original question.
The most likely issue, though, is your use of DefaultRequestHeaders. You should only use this for headers that apply to every request that the HttpClient instance will send, and then you should set them only once, when you create the client, as the documentation implies ("Headers set on this property don't need to be set on request messages again").
While HttpClient is essentially thread-safe, the DefaultRequestHeaders (and BaseAddress) properties are not. You're changing these values while the client instance is potentially busy using them elsewhere. It's not clear whether you're using the singleton HttpClient elsewhere as well, possibly changing the default headers there too, but if so that would significantly increase the chances of issues arising.
Some additional references about the non-thread-safety of these properties:
https://github.com/dotnet/dotnet-api-docs/issues/1085
http://www.michaeltaylorp3.net/httpclient-is-it-really-thread-safe/
https://github.com/MicrosoftDocs/architecture-center/issues/935

I found an answer which is:
public async Task<bool> GetMedia(string saveDir, string id)
{
var api = $"/api/v1/download/{id}";
var Uri = $"{MccBaseURL}{api}";
using (HttpClient client = new HttpClient())
{
using (HttpResponseMessage response = await client.GetAsync(Uri, HttpCompletionOption.ResponseHeadersRead))
using (System.IO.Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
string fileToWriteTo = System.IO.Path.GetTempFileName();
using (System.IO.FileStream streamToWriteTo = new System.IO.FileStream(saveDir, System.IO.FileMode.Create))
{
await streamToReadFrom.CopyToAsync(streamToWriteTo);
return true;
}
}
}
}
It was really memory something problem which continuously using same HttpClient over and over again. So I created a new instance. I'm a super noob! Sorry!

c#: How to Post async request and get stream with httpclient?

I need to send async request to the server and get the information from the response stream.
I'm using HttpClient.GetStreamAsync(), but the server response that POST should be used. Is there a similar method like PostStreamAsync()? Thank you.

If you want to use HttpClient for streaming large data then you should not use PostAsync cause message.Content.ReadAsStreamAsync would read the entire stream into memory. Instead you can use the following code block.
var message = new HttpRequestMessage(HttpMethod.Post, "http://localhost:3100/api/test");
var response = await client.SendAsync(message, HttpCompletionOption.ResponseHeadersRead);
var stream = await response.Content.ReadAsStreamAsync();
The key thing here is the HttpCompletionOption.ResponseHeadersRead option which tells the client not to read the entire content into memory.

Use HttpClient.PostAsync and you can get the response stream via HttpResponseMessage.Content.ReadAsStreamAsync() method.
var message = await client.PostAsync(url, content);
var stream = await message.Content.ReadAsStreamAsync();

Instead of HttpClient, maybe you should use HttpWebRequest ?
They offer both async method and the possibility to switch between post and get by setting their method property.
e.g:
var request = (HttpWebRequest) WebRequest.Create(uri);
request.Method = "POST";
var postStream = await request.GetRequestStreamAsync()

HttpClient: Conditionally set AcceptEncoding compression at runtime

We are trying to implement user-determined (on a settings screen) optional gzip compression in our client which uses HttpClient, so we can log and compare performance across a number of different calls over a period of time. Our first attempt was to simply conditionally add the header as follows:
HttpRequestMessage request = new HttpRequestMessage(Method, Uri);
if (AcceptGzipEncoding)
{
_client.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
}
//Send to the server
result = await _client.SendAsync(request);
//Read the content of the result response from the server
content = await result.Content.ReadAsStringAsync();
This created the correct request, but the gzipped response was not decompressed on return, resulting in a garbled response. I found that we had to include the HttpClientHandler when constructing the HttpClient:
HttpClient _client = new HttpClient(new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip
});
This all works well, but we'd like to change whether the client sends the Accept-Encoding: gzip header at runtime, and there doesn't appear to be any way to access or change the HttpClientHandler after it's passed to the HttpClient constructor. In addition, altering the headers of the HttpRequestMessage object doesn't have any effect on the headers of the request if they are defined by the HttpClientHandler.
Is there any way to do this without recreating the HttpClient each time this changes?
Edit: I've also tried to modify a reference to the HttpClientHandler to change AutomaticDecompression at runtime, but that's throwing this exception:
This instance has already started one or more requests. Properties can only be modified before sending the first request.

You're almost there with the first example, you just need to deflate the stream yourself. MS's GZipSteam will help with this:
HttpRequestMessage request = new HttpRequestMessage(Method, Uri);
if (AcceptGzipEncoding)
{
_client.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
}
//Send to the server
result = await _client.SendAsync(request);
//Read the content of the result response from the server
using (Stream stream = await result.Content.ReadAsStreamAsync())
using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
using (StreamReader reader = new StreamReader(decompressed))
{
content = reader.ReadToEnd();
}

If you want to use the same HttpClient and only want to enable compression for some requests, you are not able to use automatic decompression. When automatic decompression is enabled, the framework also resets the Content-Encoding header of the response. This means that you are unable to find out whether the response was really compressed or not. By the way, also the Content-Length header of the response matches the size of the decompressed content if you turn on automatic decompression.
So you need to decompress the content manually. The following sample shows an implementation for gzip-compressed content (as also shown in #ToddMenier's response):
private async Task<string> ReadContentAsString(HttpResponseMessage response)
{
// Check whether response is compressed
if (response.Content.Headers.ContentEncoding.Any(x => x == "gzip"))
{
// Decompress manually
using (var s = await response.Content.ReadAsStreamAsync())
{
using (var decompressed = new GZipStream(s, CompressionMode.Decompress))
{
using (var rdr As New IO.StreamReader(decompressed))
{
return await rdr.ReadToEndAsync();
}
}
}
else
// Use standard implementation if not compressed
return await response.Content.ReadAsStringAsync();
}

As per the comments above, recreating the HttpClient is really the only (robust) way to do this. Manual decompression can be achieved but it seems to be very difficult to reliably/efficiently determine whether the content has been encoded or not, to determine whether to apply decoding.