In my service we need to get a zip file created by another service and return it.
This is my code (code has been simplified for the question):
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
}
With the id I can gather the info I need to create the fileIds string and call the other service.
Here's the api on the other service (code has been simplified for the question):
[HttpGet("bulk/{idList}")]
public async Task<IActionResult> DownloadBulk(string idList)
{
var ids = string.IsNullOrEmpty(idList) ? new List<int>() : idList.Split(',').Select(x => Convert.ToInt32(x));
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var index = archive.CreateEntry("hello.txt");
using (var entryStream = index.Open())
using (var streamWriter = new StreamWriter(entryStream))
{
streamWriter.Write("hello");
}
}
var byteArray = memoryStream.ToArray();
return File(byteArray, "application/octet-stream", "media_files.zip");
}
but when the client tries to open the zip we get
Exception has occurred. ArchiveException (FormatException: Could not
find End of Central Directory Record)
I'm absolutely not confident about these two lines of the /mediafiles/{id}
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
And probably the issue might be there.
I just need to forward back the file-service response, but I don't know why
I believe the problem you're experiencing is that in DownloadMediaFiles(int id) you are using an HttpClient that gets disposed when leaving the function scope. The stream you created from the response therefore is closed and disposed of as well, before the response payload has finished writing its contents to the client. The client therefore receives an incomplete zip-file that you can't open. See here for reference.
In this answer there's a simple solution you could use, which is simply to read the response stream (the response stream from $"http://file-service/bulk/{fileIds}") into a byte array and then pass it to the response to the client:
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var byteArr = await response.Content.ReadAsByteArrayAsync();
return File(byteArr, "application/octet-stream", "media_files.zip");
You might realize that this means loading the whole file into memory, which can quickly become an issue if you plan on working with large files or even with medium sized files if the API is supposed to be used by a lot of clients simultaneously. Your web application would most likely run out of memory at some point.
Instead, I came upon this article which shows how you can return the contents of the stream from a request using an HttpClient. You should be able to stick with the first section of that article (all the ZIP-file and callback-based response stuff is unrelated).
To recap on that article all you need is something like this:
// Your ControllerClass.cs
private static HttpClient Client { get; } = new HttpClient();
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
var stream = await Client.GetStreamAsync($"http://file-service/bulk/{fileIds}");
return File(stream, "application/octet-stream", "media_files.zip");
}
You'll notice, that the stream object is not disposed of here but ASP.Net Core does this for you as part of writing the response payload to the client. The Client which is stored in a static global variable is not disposed of either, which means you can reuse it between requests (it's usually recommended not to instantiate a new HttpClient everytime you need it). ASP.Net Core 2.1 and up has special support for dependency injecting the client for you through the IHttpClientFactory interface. I would suggest you do that instead of a static variable. Read here for the most basic usage of injecting the client factory.
Now you should be able to enjoy streaming the file contents directly from your "other service" without loading it into memory in your API web application.
Related
I'm experiencing a problem when trying to use MultipartFormDataContent with HttpClient with a stream of data.
Context
I'm trying to upload a large file to ASP.NET Core Web API. A client should send the file via POST request form-data to a front-end API, which in turn should forward the file to a back-end API.
Because the file can be large, I followed the Microsoft example, i.e. I don't want to use IFormFile type but instead read the Request.Body using MultipartReader. This is to avoid loading the entire file into memory on the server, or saving it in a temporary file on server's hard drive.
Problem
The back-end API controller action looks as follows (this is almost directly copied from the ASP.NET Core 5.0 sample app with just minor simplifications):
[HttpPost]
[DisableRequestSizeLimit]
public async Task<IActionResult> ReceiveLargeFile()
{
var request = HttpContext.Request;
if (!request.HasFormContentType
|| !MediaTypeHeaderValue.TryParse(request.ContentType, out var mediaTypeHeader)
|| string.IsNullOrEmpty(mediaTypeHeader.Boundary.Value))
{
return new UnsupportedMediaTypeResult();
}
var reader = new MultipartReader(mediaTypeHeader.Boundary.Value, request.Body);
/* This throws an IOException: Unexpected end of Stream, the content may have already been read by another component. */
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition,
out var contentDisposition);
if (hasContentDispositionHeader
&& contentDisposition!.DispositionType.Equals("form-data")
&& !string.IsNullOrEmpty(contentDisposition.FileName.Value))
{
/* Fake copy to nothing since it doesn't even get here */
await section.Body.CopyToAsync(Stream.Null);
return Ok();
}
section = await reader.ReadNextSectionAsync();
}
return BadRequest("No files data in the request.");
}
I managed to reduce the problem slightly by making an integration test using Microsoft.AspNetCore.Mvc.Testing NuGet package. The following test replaces the front-end API, so instead of reading Request.Body stream in a Web API, the test just tries to add StreamContent to MultipartFormDataContent and post it via HttpClient to the back-end API:
[Fact]
public async Task Client_posting_to_Api_returns_Ok()
{
/* Arrange */
await using var stream = new MemoryStream();
await using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("FILE CONTENTS");
await writer.FlushAsync();
stream.Position = 0;
using var client = _factory.CreateDefaultClient();
/* Act */
using var response =
await client.PostAsync(
"Receive",
new MultipartFormDataContent
{
{
new StreamContent(stream),
"file",
"fileName"
}
});
/* Assert */
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
The back-end API controller then throws an IOException at await reader.ReadNextSectionAsync(), saying "Unexpected end of Stream, the content may have already been read by another component".
GitHub Repository (Complete Example)
I uploaded a complete example of the problem (including back-end API and the test) a GitHub repo.
Question
I must be doing something wrong. How can I forward a file received in a request with form-data content type in one service (front-end API) to another service (back-end API) without loading the entire file into memory or hard-drive in the front-end API, i.e. to just forward the stream of data to the back-end API?
Thanks in advance for any help.
I expected the same issue as you and it turned out that the MediaTypeHeaderValue.TryParse method parses the boundary value wrong as it wraps the string with '"' characters, because HttpClient sends the content type header like this:
multipart/form-data; boundary="blablabla"
So for me the solution was to add a Trim() method to boundary like this and pass that to the MultipartReader
var boundary = mediaTypeHeader.Boundary.Value.Trim('"');
var reader = new MultipartReader(boundary, request.Body);
I'm trying to create a .net core http client that would stream bytes to the remote server in a multipart/form-data request. The bytes to be sent are generated dynamically by another task. At the moment of sending of the request I don't have the full data to be send in memory or anywhere - this data will be produced during the course of the request by another task. I tried to use the MemoryStream as a source for the StreamContent and feed this memory stream from another task but it's not suited for such case. Is there any stream in the standard library that could be use?
It would be best if I could access the request's stream and write (push) directly to it but I'm not sure how to do that in the case of multipart request.
class Program
{
static async Task Main(string[] args)
{
using var httpClient = new HttpClient();
using var contentData = new MultipartFormDataContent();
Stream sharedStream = SomeKindOfStream(); // <-- Need a stream that another task could write to and close
var streamContent = new StreamContent(sharedStream);
contentData.Add(streamContent, "file", Guid.NewGuid().ToString());
using (var httpResponseMessage =
await httpClient.PostAsync("http://localhost:5000/async/abcd",
contentData))
{
Console.WriteLine($"Got Response: '{await httpResponseMessage.Content.ReadAsStringAsync()}'");
}
}
}
I am developing a .net core middle-ware (api) and thinking to use pipes with following flow, Can someone tell me is this is a good approach and comply best practices or should i use different strategy.
Request comes to api
Authorization pipe validates the request.
Request pipe logs the request into db.
Request goes to api and perform action and return a result.
Response pipe gets the response and logs into db and return the result to client.
I know that we can read stream only time (point 3) but i figured this out already and after reading i have attach it to request stream again.
So, confusion is where to write the response? In api? or in separate pipe.
If i do it in separate pipe then i am handling my response two time (one is creating response in api, second is reading response in separate pipe) which is a performance hit.
Can i pass the data from point number 4 to 5 like from api to my pipe and from there that response should added into response stream and if it is correct then how can i pass the data from api to pipe?
Yes, response stream can only be read once. You can use the MemoryStream to load the response , reference article :
First, read the request and format it into a string.
Next, create a dummy MemoryStream to load the new response into.
Then, wait for the server to return a response.
Finally, copy the dummy MemoryStream (containing the actual response) into the original stream, which gets returned to the client.
Code sample :
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
And register the middleware :
app.UseMiddleware<RequestResponseLoggingMiddleware>();
I am trying to consume a third party API whose URL looks like this:
https://api.crowdin.com/api/project/{PROJECT_NAME}/download/all.zip?key={MY_KEY}
This api returns a zip file as "all.zip" as response.
When I go to browser and make this request I get a all.zip file downloaded. Now I want to write C# code to get this result. Below is my attempt:
public async Task<ActionResult> Index()
{
var client = new HttpClient();
client.BaseAddress = new Uri("https://api.crowdin.com/");
HttpResponseMessage response = await client.GetAsync("api/project/{MY_PROJECT}/download/all.zip?key={MY_KEY}");
// WHAT TO WRITE HERE
return View();
}
Question 1: I got the successful response and content type is application/zip, but now I don't know how to read this response.
Question 2: I want the response to unzipped and saved to a folder.
P.S: The response .zip file is a collection of .resx File.
This is mostly from memory so I haven't tested the code. It should get you pretty close to what you're looking for:
Saving the response to file:
var response = httpClient.GetAsync("api/project/{MY_PROJECT}/download/all.zip?key={MY_KEY}");
using (var stream = await response.Content.ReadAsStreamAsync())
using (var fs = new FileStream(filename) {
await stream.CopyToAsync(fs);
}
Unzipping the file (you could also do this in memory)
System.IO.Compression.ZipFile.ExtractToDirectory(filename, extractPath);
Basically I want to stream a file to a web API and once inside the web api controller I would like to pass data as it comes in to lower level logic via a stream reader. I tried the code below found from another SO post with some modification, but I get:
An asynchronous module or handler completed while an asynchronous
operation was still pending.
public async void Put(int id, HttpRequestMessage request)
{
if (!Request.Content.IsMimeMultipartContent())
throw new InvalidOperationException();
var provider = new MultipartMemoryStreamProvider();
await Request.Content.ReadAsMultipartAsync(provider);
var file = provider.Contents.First();
var filename = file.Headers.ContentDisposition.FileName.Trim('\"');
var buffer = await file.ReadAsByteArrayAsync();
var stream = new MemoryStream(buffer);
using (var s = new StreamReader(stream))
{
saveFile.Execute(id, s);
}
}
I'm open to other solutions as long as I am streaming the data as it comes in. I'm new to await and async and I'm probably making a basic mistake. Any ideas?
Change async void to async Task