We Keep Coding

Passing a large file stream to MultipartFormDataContent with HttpClient

I'm experiencing a problem when trying to use MultipartFormDataContent with HttpClient with a stream of data.
Context
I'm trying to upload a large file to ASP.NET Core Web API. A client should send the file via POST request form-data to a front-end API, which in turn should forward the file to a back-end API.
Because the file can be large, I followed the Microsoft example, i.e. I don't want to use IFormFile type but instead read the Request.Body using MultipartReader. This is to avoid loading the entire file into memory on the server, or saving it in a temporary file on server's hard drive.
Problem
The back-end API controller action looks as follows (this is almost directly copied from the ASP.NET Core 5.0 sample app with just minor simplifications):
[HttpPost]
[DisableRequestSizeLimit]
public async Task<IActionResult> ReceiveLargeFile()
{
var request = HttpContext.Request;
if (!request.HasFormContentType
|| !MediaTypeHeaderValue.TryParse(request.ContentType, out var mediaTypeHeader)
|| string.IsNullOrEmpty(mediaTypeHeader.Boundary.Value))
{
return new UnsupportedMediaTypeResult();
}
var reader = new MultipartReader(mediaTypeHeader.Boundary.Value, request.Body);
/* This throws an IOException: Unexpected end of Stream, the content may have already been read by another component. */
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition,
out var contentDisposition);
if (hasContentDispositionHeader
&& contentDisposition!.DispositionType.Equals("form-data")
&& !string.IsNullOrEmpty(contentDisposition.FileName.Value))
{
/* Fake copy to nothing since it doesn't even get here */
await section.Body.CopyToAsync(Stream.Null);
return Ok();
}
section = await reader.ReadNextSectionAsync();
}
return BadRequest("No files data in the request.");
}
I managed to reduce the problem slightly by making an integration test using Microsoft.AspNetCore.Mvc.Testing NuGet package. The following test replaces the front-end API, so instead of reading Request.Body stream in a Web API, the test just tries to add StreamContent to MultipartFormDataContent and post it via HttpClient to the back-end API:
[Fact]
public async Task Client_posting_to_Api_returns_Ok()
{
/* Arrange */
await using var stream = new MemoryStream();
await using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("FILE CONTENTS");
await writer.FlushAsync();
stream.Position = 0;
using var client = _factory.CreateDefaultClient();
/* Act */
using var response =
await client.PostAsync(
"Receive",
new MultipartFormDataContent
{
{
new StreamContent(stream),
"file",
"fileName"
}
});
/* Assert */
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
The back-end API controller then throws an IOException at await reader.ReadNextSectionAsync(), saying "Unexpected end of Stream, the content may have already been read by another component".
GitHub Repository (Complete Example)
I uploaded a complete example of the problem (including back-end API and the test) a GitHub repo.
Question
I must be doing something wrong. How can I forward a file received in a request with form-data content type in one service (front-end API) to another service (back-end API) without loading the entire file into memory or hard-drive in the front-end API, i.e. to just forward the stream of data to the back-end API?
Thanks in advance for any help.

I expected the same issue as you and it turned out that the MediaTypeHeaderValue.TryParse method parses the boundary value wrong as it wraps the string with '"' characters, because HttpClient sends the content type header like this:
multipart/form-data; boundary="blablabla"
So for me the solution was to add a Trim() method to boundary like this and pass that to the MultipartReader
var boundary = mediaTypeHeader.Boundary.Value.Trim('"');
var reader = new MultipartReader(boundary, request.Body);

UI thread waits for "await client.SendAsync(request).ConfigureAwait(false)" to finish before moving to another url

Client starts some action via button. In controller its async method: async public Task<JsonResult> CreatePresentation that contains line using (var response = await client.SendAsync(request).ConfigureAwait(false)). Lets say this line takes 10s to finish. While that is happenning client wants to go to another url inside website and clicks on that url (that view is also async meaning its async public Task<ActionResult> BRDetails(int id)). But asp .net does not serve him view immidiatly. Instead it awaits for await line to finish before serving him the view (10s).
Why does asp framework waits for await line to finish before it starts producing view for next url that client requested?
I would thought that deliving the next url should start immidiatly in second thread and waits in first for await to finishes because of .ConfigureAwait(false).
Environment:
.net framework v4.5,
iiexpress that comes with visual studio
Here is part of code (since it's a software product I am not able to share full code):
async public Task<JsonResult> CreateReportFile(int id, string email)
{
try
{
var fileName = "Report " + DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss") + ".txt";
fileName = fileName.Replace('/', '_');
fileName = fileName.Replace(':', '_');
var fullPath = "some path";
var presentationTask = new Creator().CreateSomeFile("some values to pass", fileName);
var ms = await presentationTask.ConfigureAwait(false);
using (FileStream file = new FileStream(fullPath, FileMode.Create, System.IO.FileAccess.Write))
{
ms.WriteTo(file);
ms.Close();
}
var emailAttachment = BusinessReportsEmail.savePresentation(fullPath, fileName);
await BusinessReportsEmail.sendEmail(emailAttachment, email).ConfigureAwait(false);
return Json(new
{
Status = "ok",
Data = email
});
}
catch (Exception ex)
{
return Json(new
{
Status = "error",
Data = ex.Message
});
}
}
public async Task<MemoryStream> CreateSomeFile(string programData, string outputFileName)
{
// Use this in production
var url = string.Format(System.Configuration.ConfigurationManager.AppSettings["workFileUrl"]);
var appCode = string.Format(System.Configuration.ConfigurationManager.AppSettings["appCode"]);
var appKey = string.Format(System.Configuration.ConfigurationManager.AppSettings["appKey"]);
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromMinutes(20);
using (var request = new HttpRequestMessage(HttpMethod.Post, url))
{
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
request.Headers.Add("AppCode", appCode);
request.Headers.Add("AppKey", appKey);
var jsonString = JsonConvert.SerializeObject(new
{
InputType = "json",
Content = programData
});
request.Content = new StringContent(jsonString, Encoding.UTF8, "application/json");
using (var response = await client.SendAsync(request).ConfigureAwait(false))
{
var responseContent = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
if (response.StatusCode != HttpStatusCode.OK)
throw new Exception($"Failed to create {outputFileName} file.");
var reponseObject = await response.Content.ReadAsAsync<FileResponse>().ConfigureAwait(false);
return new MemoryStream(Convert.FromBase64String(reponseObject.Content));
}
}
}
}
and than client goes to this view but needs to wait for previous call to finish.
async public Task<ActionResult> ReviewDetails(int id)
{
return View();
}

Why does asp framework waits for await line to finish before it starts producing view for next url that client requested?
In ASP.NET, await yields to the thread pool, not the HTTP client.
Remember, your web app is serving HTTP responses to HTTP requests. Generally speaking, a single HTTP request has a single HTTP response. Your web app can't return a response and then sometime later return another response. You can think of an await as an "asynchronous wait", and your method will asynchronously wait there, so it won't even get to the return Json(...) until after the await completes.
If you want to kick off an independent background process, then I recommend using asynchronous messaging, as described on my blog. It's complex, but it's necessary for reliable work. Alternatively, you could use something like a SPA and not return early at all - allow the user to change pages while the HTTP request is still in progress.

Forwarding a stream in c# api

In my service we need to get a zip file created by another service and return it.
This is my code (code has been simplified for the question):
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
}
With the id I can gather the info I need to create the fileIds string and call the other service.
Here's the api on the other service (code has been simplified for the question):
[HttpGet("bulk/{idList}")]
public async Task<IActionResult> DownloadBulk(string idList)
{
var ids = string.IsNullOrEmpty(idList) ? new List<int>() : idList.Split(',').Select(x => Convert.ToInt32(x));
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var index = archive.CreateEntry("hello.txt");
using (var entryStream = index.Open())
using (var streamWriter = new StreamWriter(entryStream))
{
streamWriter.Write("hello");
}
}
var byteArray = memoryStream.ToArray();
return File(byteArray, "application/octet-stream", "media_files.zip");
}
but when the client tries to open the zip we get
Exception has occurred. ArchiveException (FormatException: Could not
find End of Central Directory Record)
I'm absolutely not confident about these two lines of the /mediafiles/{id}
var stream = await response.Content.ReadAsStreamAsync();
return File(stream, "application/octet-stream", "media_files.zip");
And probably the issue might be there.
I just need to forward back the file-service response, but I don't know why

I believe the problem you're experiencing is that in DownloadMediaFiles(int id) you are using an HttpClient that gets disposed when leaving the function scope. The stream you created from the response therefore is closed and disposed of as well, before the response payload has finished writing its contents to the client. The client therefore receives an incomplete zip-file that you can't open. See here for reference.
In this answer there's a simple solution you could use, which is simply to read the response stream (the response stream from $"http://file-service/bulk/{fileIds}") into a byte array and then pass it to the response to the client:
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync($"http://file-service/bulk/{fileIds}");
var byteArr = await response.Content.ReadAsByteArrayAsync();
return File(byteArr, "application/octet-stream", "media_files.zip");
You might realize that this means loading the whole file into memory, which can quickly become an issue if you plan on working with large files or even with medium sized files if the API is supposed to be used by a lot of clients simultaneously. Your web application would most likely run out of memory at some point.
Instead, I came upon this article which shows how you can return the contents of the stream from a request using an HttpClient. You should be able to stick with the first section of that article (all the ZIP-file and callback-based response stuff is unrelated).
To recap on that article all you need is something like this:
// Your ControllerClass.cs
private static HttpClient Client { get; } = new HttpClient();
[HttpGet("mediafiles/{id}")]
public async Task<IActionResult> DownloadMediaFiles(int id)
{
var fileIds = _myProvider.GetFileIdsForEntityId(id); // result be like "1,2,3,4"
var stream = await Client.GetStreamAsync($"http://file-service/bulk/{fileIds}");
return File(stream, "application/octet-stream", "media_files.zip");
}
You'll notice, that the stream object is not disposed of here but ASP.Net Core does this for you as part of writing the response payload to the client. The Client which is stored in a static global variable is not disposed of either, which means you can reuse it between requests (it's usually recommended not to instantiate a new HttpClient everytime you need it). ASP.Net Core 2.1 and up has special support for dependency injecting the client for you through the IHttpClientFactory interface. I would suggest you do that instead of a static variable. Read here for the most basic usage of injecting the client factory.
Now you should be able to enjoy streaming the file contents directly from your "other service" without loading it into memory in your API web application.

Asynchronously Streaming Video with .Net Core API from Azure Blob Storage

I have found a bunch of examples that use objects not available to me within my application and don't seem to match up to my version of .NET Core web API. In essence, I am working on a project that will have tags on a web page and want to load the videos using a stream from the server rather than directly serving the files via a path. One reason is the source of the files may change and serving them via path isn't what my customer wants. So I need to be able to open a stream and async write the video file.
This for some reason produces JSON data so that's wrong. I am downloading the video file from Azure Blob storage and returning as a stream, but I just don't understand what I need to do to send a streamed video file to a tag in HTML.
My API Controller,
[AllowAnonymous]
[HttpGet("getintroductoryvideos")]
public async Task<Stream> GetIntroductoryVideos()
{
try
{
return _documentsService.WriteContentToStream().Result;
}
catch (Exception ex)
{
throw ex;
}
}
My Service class,
public async Task<Stream> WriteContentToStream()
{
var cloudBlob = await _blobService.GetBlobAsync(PlatformServiceConstants._blobIntroductoryVideoContainerPath + PlatformServiceConstants.IntroductoryVideo1, introductoryvideocontainerName);
await cloudBlob.FetchAttributesAsync();
var fileStream = new MemoryStream();
await cloudBlob.DownloadToStreamAsync(fileStream);
return fileStream;
}

You can try the code below:
API Controller:
[AllowAnonymous]
[HttpGet("getintroductoryvideos")]
public async Task<FileContentResult> GetIntroductoryVideos(string videoname)
{
return await _documentsService.WriteContentToStream();
}
Service class:
public async Task<FileContentResult> WriteContentToStream()
{
var cloudBlob = await _blobService.GetBlobAsync(PlatformServiceConstants._blobIntroductoryVideoContainerPath + PlatformServiceConstants.IntroductoryVideo1, introductoryvideocontainerName);
MemoryStream fileStream = new MemoryStream();
await cloudBlob.DownloadToStreamAsync(fileStream);
return new FileContentResult (fileStream.ToArray(), "application/octet-stream");
}
Html:
<div className="xxx">
<video height="auto">
<source src="xx/getintroductoryvideos?videoname=xxx" type="video/mp4" />
</video>
</div>

You probably want to avoid loading the entire video in memory before returning it. You should be able to pass through a stream by using FileStreamResult:
[AllowAnonymous]
[HttpGet("getintroductoryvideos")]
public async Task<IActionResult> GetIntroductoryVideos()
{
var cloudBlob = await _blobService.GetBlobAsync(PlatformServiceConstants._blobIntroductoryVideoContainerPath + PlatformServiceConstants.IntroductoryVideo1, introductoryvideocontainerName);
var stream = await cloudBlob.OpenReadAsync();
return new FileStreamResult(stream, "application/octet-stream");
}

Here is what I have done. The key point is the last line. The last parameter there enables the range request called EnableRangeProcessing. Although this starts supporting from .net core 2.1 and 2.1 plus.
[AllowAnonymous]
[HttpGet("getintroductoryvideos")]
public async Task<IActionResult> GetIntroductoryVideos()
{
var cloudBlob = await _blobService.GetBlobAsync(PlatformServiceConstants._blobIntroductoryVideoContainerPath + PlatformServiceConstants.IntroductoryVideo1, introductoryvideocontainerName);
var stream = await cloudBlob.OpenReadAsync();
return new File(stream, "application/octet-stream",true);
}

Pass files to another asp.net core webapi from another webapi

Existed asp.net core Api is look like below
public async Task<IActionResult> UploadAsync()
{
IFormFile file = null;
var files = Request.Form.Files;
if (files.Count > 0)
{
file = Request.Form.Files[0];
var fileText = new StringBuilder();
using (var reader = new StreamReader(file.OpenReadStream()))
{
while (reader.Peek() >= 0)
fileText.AppendLine(reader.ReadLine());
}
int stagingDetailId = await _stagingMarketProvider.GetV1StagingStatusDetailId();
var result = await SaveStagingMarketsAsync(_fileProvider.ReadImportedMarkets(fileText.ToString()));
return Ok(result);
}
return Ok();
}
Now to consume that api from another asp.net core webapi, I have to pass those files through Request object only, I can't change any existed Api code because of business.

Solution 1: Applicable if you want your client to get redirected to other API
Assuming the API caller understands HTTP 302 and can act accordingly, the 302 redirect should help you.
public IActionResult Post()
{
return Redirect("http://file-handler-api/action");
}
From documentation, Redirect method returns 302 or 301 response to client.
Solution 2: C# Code To Post a File Using HttpClient
Below c# code is from this blog post. This is simple code which creates HttpClient object and tries to send the file to a web API.
As you are doing this from one API to another, you will have to save file first at temporary location. That temporary location will be parameter to this method.
Also, After upload you may want to delete the file if it is not required. This private method you can call after file upload to your first API is complete.
private async Task<string> UploadFile(string filePath)
{
_logger.LogInformation($"Uploading a text file [{filePath}].");
if (string.IsNullOrWhiteSpace(filePath))
{
throw new ArgumentNullException(nameof(filePath));
}
if (!File.Exists(filePath))
{
throw new FileNotFoundException($"File [{filePath}] not found.");
}
using var form = new MultipartFormDataContent();
using var fileContent = new ByteArrayContent(await File.ReadAllBytesAsync(filePath));
fileContent.Headers.ContentType = MediaTypeHeaderValue.Parse("multipart/form-data");
form.Add(fileContent, "file", Path.GetFileName(filePath));
form.Add(new StringContent("789"), "userId");
form.Add(new StringContent("some comments"), "comment");
form.Add(new StringContent("true"), "isPrimary");
var response = await _httpClient.PostAsync($"{_url}/api/files", form);
response.EnsureSuccessStatusCode();
var responseContent = await response.Content.ReadAsStringAsync();
var result = JsonSerializer.Deserialize<FileUploadResult>(responseContent);
_logger.LogInformation("Uploading is complete.");
return result.Guid;
}
Hope this helps you.