Passing a large file stream to MultipartFormDataContent with HttpClient - c#

I'm experiencing a problem when trying to use MultipartFormDataContent with HttpClient with a stream of data.
Context
I'm trying to upload a large file to ASP.NET Core Web API. A client should send the file via POST request form-data to a front-end API, which in turn should forward the file to a back-end API.
Because the file can be large, I followed the Microsoft example, i.e. I don't want to use IFormFile type but instead read the Request.Body using MultipartReader. This is to avoid loading the entire file into memory on the server, or saving it in a temporary file on server's hard drive.
Problem
The back-end API controller action looks as follows (this is almost directly copied from the ASP.NET Core 5.0 sample app with just minor simplifications):
[HttpPost]
[DisableRequestSizeLimit]
public async Task<IActionResult> ReceiveLargeFile()
{
var request = HttpContext.Request;
if (!request.HasFormContentType
|| !MediaTypeHeaderValue.TryParse(request.ContentType, out var mediaTypeHeader)
|| string.IsNullOrEmpty(mediaTypeHeader.Boundary.Value))
{
return new UnsupportedMediaTypeResult();
}
var reader = new MultipartReader(mediaTypeHeader.Boundary.Value, request.Body);
/* This throws an IOException: Unexpected end of Stream, the content may have already been read by another component. */
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition,
out var contentDisposition);
if (hasContentDispositionHeader
&& contentDisposition!.DispositionType.Equals("form-data")
&& !string.IsNullOrEmpty(contentDisposition.FileName.Value))
{
/* Fake copy to nothing since it doesn't even get here */
await section.Body.CopyToAsync(Stream.Null);
return Ok();
}
section = await reader.ReadNextSectionAsync();
}
return BadRequest("No files data in the request.");
}
I managed to reduce the problem slightly by making an integration test using Microsoft.AspNetCore.Mvc.Testing NuGet package. The following test replaces the front-end API, so instead of reading Request.Body stream in a Web API, the test just tries to add StreamContent to MultipartFormDataContent and post it via HttpClient to the back-end API:
[Fact]
public async Task Client_posting_to_Api_returns_Ok()
{
/* Arrange */
await using var stream = new MemoryStream();
await using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("FILE CONTENTS");
await writer.FlushAsync();
stream.Position = 0;
using var client = _factory.CreateDefaultClient();
/* Act */
using var response =
await client.PostAsync(
"Receive",
new MultipartFormDataContent
{
{
new StreamContent(stream),
"file",
"fileName"
}
});
/* Assert */
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
The back-end API controller then throws an IOException at await reader.ReadNextSectionAsync(), saying "Unexpected end of Stream, the content may have already been read by another component".
GitHub Repository (Complete Example)
I uploaded a complete example of the problem (including back-end API and the test) a GitHub repo.
Question
I must be doing something wrong. How can I forward a file received in a request with form-data content type in one service (front-end API) to another service (back-end API) without loading the entire file into memory or hard-drive in the front-end API, i.e. to just forward the stream of data to the back-end API?
Thanks in advance for any help.

I expected the same issue as you and it turned out that the MediaTypeHeaderValue.TryParse method parses the boundary value wrong as it wraps the string with '"' characters, because HttpClient sends the content type header like this:
multipart/form-data; boundary="blablabla"
So for me the solution was to add a Trim() method to boundary like this and pass that to the MultipartReader
var boundary = mediaTypeHeader.Boundary.Value.Trim('"');
var reader = new MultipartReader(boundary, request.Body);

Related

How to download a dynamically generated XML file with ServiceStack and Angular?

The following code works in theory, but it lacks error handling. The problem I have is that it starts downloading the XML file when a new window opens with the url created by the service stack. But now when an error occurs server side, you are on this new page with only the stack trace.
What is the right way to download a dynamic binary (not stored on disk) with service stack and Angular?
Angular with ServiceStack:
downloadExportXML(){
const request = new GetRatingsExportRequest(this.request);
const url = this.jsonServiceClient.createUrlFromDto("GET", request)
window.open(url);
}
WebAPI (C#) to collect the XML File.
public HttpResult Get(GetRatingsExportRequest request)
{
MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(xmlContent));
var result = new HttpResult(ms.ToArray());
var disposition = $"attachment;filename=myfilename;";
result.Headers.Add(HttpHeaders.ContentDisposition, disposition);
return result;
}
If you return the content with a Content-Disposition: attachment HTTP Header the browser will treat it as a separate File Download.
If you don't want that return it as a normal XML Response. But the way you're returning it is fairly inefficient, i.e. calling ToArray() defeats the purpose of using a Stream since you've forced loaded the entire contents of the Stream in memory, instead you should just return the stream so it gets asynchronously written to the HTTP Response as a Stream, e.g:
return new HttpResult(ms, MimeTypes.Xml);
But if you've already got the XML as a string you don't need the overhead of the MemoryStream wrapper either and can just return the XML string as-is, e.g:
return new HttpResult(xmlContent, MimeTypes.Xml);
I couldnt make it work. So i created this workaround: Generating the URL to download the XML with the ServiceStack method jsonServiceClient.createUrlFromDto() and than downloading the file via Angular HTTPClient.
downloadExportXML(){
const request = new GetRatingsExportRequest(this.request);
const url = this.jsonServiceClient.createUrlFromDto("GET", request)
this.httpClient.get(url, {observe: 'response', responseType: 'arraybuffer'})
.subscribe((response) => {
var contentDisposition = response.headers.get('content-disposition');
this.downloadFile(response.body)
},
error => this.handleError()
);
}
private downloadFile(blobData: ArrayBuffer)
{
let file = new Blob([blobData], { type: 'application/binary' });
let fileURL = URL.createObjectURL(file);
let fileLink = document.createElement('a');
fileLink.href = fileURL;
fileLink.download = "MYFILENAME.pdf";
fileLink.click();
}

Forwarding a stream in c# api

In my service we need to get a zip file created by another service and return it.
This is my code (code has been simplified for the question):
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
}
With the id I can gather the info I need to create the fileIds string and call the other service.
Here's the api on the other service (code has been simplified for the question):
[HttpGet("bulk/{idList}")]
public async Task<IActionResult> DownloadBulk(string idList)
{
var ids = string.IsNullOrEmpty(idList) ? new List<int>() : idList.Split(',').Select(x => Convert.ToInt32(x));
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var index = archive.CreateEntry("hello.txt");
using (var entryStream = index.Open())
using (var streamWriter = new StreamWriter(entryStream))
{
streamWriter.Write("hello");
}
}
var byteArray = memoryStream.ToArray();
return File(byteArray, "application/octet-stream", "media_files.zip");
}
but when the client tries to open the zip we get
Exception has occurred. ArchiveException (FormatException: Could not
find End of Central Directory Record)
I'm absolutely not confident about these two lines of the /mediafiles/{id}
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
And probably the issue might be there.
I just need to forward back the file-service response, but I don't know why
I believe the problem you're experiencing is that in DownloadMediaFiles(int id) you are using an HttpClient that gets disposed when leaving the function scope. The stream you created from the response therefore is closed and disposed of as well, before the response payload has finished writing its contents to the client. The client therefore receives an incomplete zip-file that you can't open. See here for reference.
In this answer there's a simple solution you could use, which is simply to read the response stream (the response stream from $"http://file-service/bulk/{fileIds}") into a byte array and then pass it to the response to the client:
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var byteArr = await response.Content.ReadAsByteArrayAsync();
return File(byteArr, "application/octet-stream", "media_files.zip");
You might realize that this means loading the whole file into memory, which can quickly become an issue if you plan on working with large files or even with medium sized files if the API is supposed to be used by a lot of clients simultaneously. Your web application would most likely run out of memory at some point.
Instead, I came upon this article which shows how you can return the contents of the stream from a request using an HttpClient. You should be able to stick with the first section of that article (all the ZIP-file and callback-based response stuff is unrelated).
To recap on that article all you need is something like this:
// Your ControllerClass.cs
private static HttpClient Client { get; } = new HttpClient();
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
var stream = await Client.GetStreamAsync($"http://file-service/bulk/{fileIds}");
return File(stream, "application/octet-stream", "media_files.zip");
}
You'll notice, that the stream object is not disposed of here but ASP.Net Core does this for you as part of writing the response payload to the client. The Client which is stored in a static global variable is not disposed of either, which means you can reuse it between requests (it's usually recommended not to instantiate a new HttpClient everytime you need it). ASP.Net Core 2.1 and up has special support for dependency injecting the client for you through the IHttpClientFactory interface. I would suggest you do that instead of a static variable. Read here for the most basic usage of injecting the client factory.
Now you should be able to enjoy streaming the file contents directly from your "other service" without loading it into memory in your API web application.

How to consume, read and unzip a third party API response in ASP.Net MVC application

I am trying to consume a third party API whose URL looks like this:
https://api.crowdin.com/api/project/{PROJECT_NAME}/download/all.zip?key={MY_KEY}
This api returns a zip file as "all.zip" as response.
When I go to browser and make this request I get a all.zip file downloaded. Now I want to write C# code to get this result. Below is my attempt:
public async Task<ActionResult> Index()
{
var client = new HttpClient();
client.BaseAddress = new Uri("https://api.crowdin.com/");
HttpResponseMessage response = await client.GetAsync("api/project/{MY_PROJECT}/download/all.zip?key={MY_KEY}");
// WHAT TO WRITE HERE
return View();
}
Question 1: I got the successful response and content type is application/zip, but now I don't know how to read this response.
Question 2: I want the response to unzipped and saved to a folder.
P.S: The response .zip file is a collection of .resx File.
This is mostly from memory so I haven't tested the code. It should get you pretty close to what you're looking for:
Saving the response to file:
var response = httpClient.GetAsync("api/project/{MY_PROJECT}/download/all.zip?key={MY_KEY}");
using (var stream = await response.Content.ReadAsStreamAsync())
using (var fs = new FileStream(filename) {
await stream.CopyToAsync(fs);
}
Unzipping the file (you could also do this in memory)
System.IO.Compression.ZipFile.ExtractToDirectory(filename, extractPath);

.NET Core 2.0 Web API for Video Streaming from FileStream

I have found a bunch of examples that use objects not available to me within my application and don't seem to match up to my version of .NET Core web API. In essence I am working on a project that will have <video> tags on a web page and want to load the videos using a stream from the server rather than directly serving the files via a path. One reason is the source of the files may change and serving them via path isn't what my customer wants. So I need to be able to open a stream and async write the video file.
This for some reason produces JSON data so that's wrong. But I just don't understand what I need to do to send a streamed video file to a <video> tag in HTML.
Current Code:
[HttpGet]
public HttpResponseMessage GetVideoContent()
{
if (Program.TryOpenFile("BigBuckBunny.mp4", FileMode.Open, out FileStream fs))
{
using (var file = fs)
{
var range = Request.Headers.GetCommaSeparatedValues("Range").FirstOrDefault();
if (range != null)
{
var msg = new HttpResponseMessage(HttpStatusCode.PartialContent);
var body = GetRange(file, range);
msg.Content = new StreamContent(body);
msg.Content.Headers.Add("Content-Type", "video/mp4");
//msg.Content.Headers.Add("Content-Range", $"0-0/{fs.Length}");
return msg;
}
else
{
var msg = new HttpResponseMessage(HttpStatusCode.OK);
msg.Content = new StreamContent(file);
msg.Content.Headers.Add("Content-Type", "video/mp4");
return msg;
}
}
}
else
{
return new HttpResponseMessage(HttpStatusCode.BadRequest);
}
}
HttpResponseMessage is not used as a return type in asp.net-core it will read that as an object model and serialize it in the response by design, as you have already observed.
Luckily in ASP.NET Core 2.0, you have
Enhanced HTTP header support
If an application visitor requests content with a Range Request header, ASP.NET will recognize that and handle that header. If the requested content can be partially delivered, ASP.NET will appropriately skip and return just the requested set of bytes. You don't need to write any special handlers into your methods to adapt or handle this feature; it's automatically handled for you.
So now all you have to do is return the file stream
[HttpGet]
public IActionResult GetVideoContent() {
if (Program.TryOpenFile("BigBuckBunny.mp4", FileMode.Open, out FileStream fs)) {
FileStreamResult result = File(
fileStream: fs,
contentType: new MediaTypeHeaderValue("video/mp4").MediaType,
enableRangeProcessing: true //<-- enable range requests processing
);
return result;
}
return BadRequest();
}
Making sure to enable range requests processing. Though, as stated in the documentation, that should be handled based on the request headers and whether that data can be partially delivered.
From there it is now a simple matter of pointing to the endpoint from the video client and let it do its magic

API is getting hit twice while returning audio file

i am returning an audio file from web api. requirement is to play the media file instead of downloading,i am using this code.
[HttpGet]
[Route("audiofile/download", Name = "GetAudioFile")]
public HttpResponseMessage GetAudioFile(string q)
{
if (string.IsNullOrWhiteSpace(q))
{
return new HttpResponseMessage { StatusCode = HttpStatusCode.BadRequest };
}
String path = HttpContext.Current.Server.MapPath("~/AudioUploads/");
string filePath = Path.Combine(path, q);
var response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(File.OpenRead(filePath))
};
var contentType = MimeMapping.GetMimeMapping(Path.GetExtension(filePath));
response.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return response;
}
i noticed that this action method is being hit twice as like
Can anyone suggest why its happening? why my api method is being called twice?
P.S I am using Url.Link in order to make uploaded file url. when i hit that, api method is called twice.
Servers only respond to requests. They can't initiate a communication with a client, without initial request.
That said, your client code is to blame here, as it's sending two requests instead of one, and the server correctly responds to both.

Categories