We Keep Coding

Passing a large file stream to MultipartFormDataContent with HttpClient

I'm experiencing a problem when trying to use MultipartFormDataContent with HttpClient with a stream of data.
Context
I'm trying to upload a large file to ASP.NET Core Web API. A client should send the file via POST request form-data to a front-end API, which in turn should forward the file to a back-end API.
Because the file can be large, I followed the Microsoft example, i.e. I don't want to use IFormFile type but instead read the Request.Body using MultipartReader. This is to avoid loading the entire file into memory on the server, or saving it in a temporary file on server's hard drive.
Problem
The back-end API controller action looks as follows (this is almost directly copied from the ASP.NET Core 5.0 sample app with just minor simplifications):
[HttpPost]
[DisableRequestSizeLimit]
public async Task<IActionResult> ReceiveLargeFile()
{
var request = HttpContext.Request;
if (!request.HasFormContentType
|| !MediaTypeHeaderValue.TryParse(request.ContentType, out var mediaTypeHeader)
|| string.IsNullOrEmpty(mediaTypeHeader.Boundary.Value))
{
return new UnsupportedMediaTypeResult();
}
var reader = new MultipartReader(mediaTypeHeader.Boundary.Value, request.Body);
/* This throws an IOException: Unexpected end of Stream, the content may have already been read by another component. */
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition,
out var contentDisposition);
if (hasContentDispositionHeader
&& contentDisposition!.DispositionType.Equals("form-data")
&& !string.IsNullOrEmpty(contentDisposition.FileName.Value))
{
/* Fake copy to nothing since it doesn't even get here */
await section.Body.CopyToAsync(Stream.Null);
return Ok();
}
section = await reader.ReadNextSectionAsync();
}
return BadRequest("No files data in the request.");
}
I managed to reduce the problem slightly by making an integration test using Microsoft.AspNetCore.Mvc.Testing NuGet package. The following test replaces the front-end API, so instead of reading Request.Body stream in a Web API, the test just tries to add StreamContent to MultipartFormDataContent and post it via HttpClient to the back-end API:
[Fact]
public async Task Client_posting_to_Api_returns_Ok()
{
/* Arrange */
await using var stream = new MemoryStream();
await using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("FILE CONTENTS");
await writer.FlushAsync();
stream.Position = 0;
using var client = _factory.CreateDefaultClient();
/* Act */
using var response =
await client.PostAsync(
"Receive",
new MultipartFormDataContent
{
{
new StreamContent(stream),
"file",
"fileName"
}
});
/* Assert */
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
The back-end API controller then throws an IOException at await reader.ReadNextSectionAsync(), saying "Unexpected end of Stream, the content may have already been read by another component".
GitHub Repository (Complete Example)
I uploaded a complete example of the problem (including back-end API and the test) a GitHub repo.
Question
I must be doing something wrong. How can I forward a file received in a request with form-data content type in one service (front-end API) to another service (back-end API) without loading the entire file into memory or hard-drive in the front-end API, i.e. to just forward the stream of data to the back-end API?
Thanks in advance for any help.

I expected the same issue as you and it turned out that the MediaTypeHeaderValue.TryParse method parses the boundary value wrong as it wraps the string with '"' characters, because HttpClient sends the content type header like this:
multipart/form-data; boundary="blablabla"
So for me the solution was to add a Trim() method to boundary like this and pass that to the MultipartReader
var boundary = mediaTypeHeader.Boundary.Value.Trim('"');
var reader = new MultipartReader(boundary, request.Body);

How to download a dynamically generated XML file with ServiceStack and Angular?

The following code works in theory, but it lacks error handling. The problem I have is that it starts downloading the XML file when a new window opens with the url created by the service stack. But now when an error occurs server side, you are on this new page with only the stack trace.
What is the right way to download a dynamic binary (not stored on disk) with service stack and Angular?
Angular with ServiceStack:
downloadExportXML(){
const request = new GetRatingsExportRequest(this.request);
const url = this.jsonServiceClient.createUrlFromDto("GET", request)
window.open(url);
}
WebAPI (C#) to collect the XML File.
public HttpResult Get(GetRatingsExportRequest request)
{
MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(xmlContent));
var result = new HttpResult(ms.ToArray());
var disposition = $"attachment;filename=myfilename;";
result.Headers.Add(HttpHeaders.ContentDisposition, disposition);
return result;
}

If you return the content with a Content-Disposition: attachment HTTP Header the browser will treat it as a separate File Download.
If you don't want that return it as a normal XML Response. But the way you're returning it is fairly inefficient, i.e. calling ToArray() defeats the purpose of using a Stream since you've forced loaded the entire contents of the Stream in memory, instead you should just return the stream so it gets asynchronously written to the HTTP Response as a Stream, e.g:
return new HttpResult(ms, MimeTypes.Xml);
But if you've already got the XML as a string you don't need the overhead of the MemoryStream wrapper either and can just return the XML string as-is, e.g:
return new HttpResult(xmlContent, MimeTypes.Xml);

I couldnt make it work. So i created this workaround: Generating the URL to download the XML with the ServiceStack method jsonServiceClient.createUrlFromDto() and than downloading the file via Angular HTTPClient.
downloadExportXML(){
const request = new GetRatingsExportRequest(this.request);
const url = this.jsonServiceClient.createUrlFromDto("GET", request)
this.httpClient.get(url, {observe: 'response', responseType: 'arraybuffer'})
.subscribe((response) => {
var contentDisposition = response.headers.get('content-disposition');
this.downloadFile(response.body)
},
error => this.handleError()
);
}
private downloadFile(blobData: ArrayBuffer)
{
let file = new Blob([blobData], { type: 'application/binary' });
let fileURL = URL.createObjectURL(file);
let fileLink = document.createElement('a');
fileLink.href = fileURL;
fileLink.download = "MYFILENAME.pdf";
fileLink.click();
}

Reading HTTP File AND Form data without writing to disk using Web API

My question is similar to this one:
webapi-file-uploading-without-writing-files-to-disk
However, this and most of the other similar questions stop at reading the file data. I have made that bit work fine, but I am also wanting to read the rest of the form data in the form which has other elements for the upload such as 'Title' etc. This comes from the solution in the aforementioned question:
var filesToReadProvider = await Request.Content.ReadAsMultipartAsync();
The filesToReadProvider is a collection of HTTPContent objects, so I tried:
List<HttpContent> uploadedstuff = filesToReadProvider.Contents.ToList();
Image image = new Image(); ; // The image object we will create
Stream filestream; // The file stream object to use with the image
foreach (var thing in uploadedstuff)
{
try
{
string name = thing.Headers.ContentDisposition.Name.Replace("\"", ""); // String is quoted "\""namestring"\"" so need it stripped out
List<NameValueHeaderValue> parameters = thing.Headers.ContentDisposition.Parameters.ToList();
if (name == "file")
{
image.LocalFileName = thing.Headers.ContentDisposition.FileName;
filestream = await thing.ReadAsStreamAsync();
}
if (name == "Title")
{
// vvv- this line causes an exception.
NameValueCollection titleData = await thing.ReadAsFormDataAsync();
}
}
catch (System.Exception e)
{
var message = "Something went wrong";
HttpResponseMessage err = new HttpResponseMessage() { StatusCode = HttpStatusCode.ExpectationFailed, ReasonPhrase = message };
return ResponseMessage(err);
}
}
Any ideas what I should be doing to get to eg: the 'Title' form data? I feel I am close, but may be taking the wrong approach?
Many thanks.

Now that it is sorted out in the comments I shall post this answer to help others maybe.
form multipart content sends a data back to server like this"
---------------------------acebdf13572468
Content-Disposition: form-data; name="file";
Content-Type: image/*
<#INCLUDE *App.jpg*#>
---------------------------acebdf13572468
Content-Disposition: form-data; name="TextField";
Content-Type: application/octet-stream
Text Value
---------------------------acebdf13572468
Content-Disposition: form-data; name="JsonField";
Content-Type: application/json
{
"Json" : "Object"
}
---------------------------acebdf13572468--
Each part that is separated with the separator line (i.e. ---------------------------acebdf13572468) is a content of it's own, hence, multipart.
You can send data as json in one field as shown above or send text or anything. Usually browsers send each control in a single part of it's own. You can read this kind of data via ASP.Net model binder by specifying [FromForm] in controller arguments.
Or in this particular case you might read it with thing.ReadAsStreamAsync(); and thing.ReadAsStringAsync();
--Edit--
So the aforementioned method is useful when directly reading data or otherwise not MVC projects. If you are using MVC you can easily read the like this.
Let's imagine you have a model like
public class Model
{
public string Title { get; set; }
}
then you would create a controller and action like this:
public class MainController : Controller
{
[HttpPost]
public async Task<IActionResult> UploadImage([FromForm]Model model)
{
var files = Request.Files;
var title = model.Title
//And you can save or use files and content at the same time.
}
}
This is usually what you would do in an MVC scenario and if you use Razor or Pages in the client side you wouldn't even need to specify the [FromForm] attribute.
I hope this helps you.
Disclaimer. I have written all this in browser. May contain syntax and other errors.

.NET Core 2.0 Web API for Video Streaming from FileStream

I have found a bunch of examples that use objects not available to me within my application and don't seem to match up to my version of .NET Core web API. In essence I am working on a project that will have <video> tags on a web page and want to load the videos using a stream from the server rather than directly serving the files via a path. One reason is the source of the files may change and serving them via path isn't what my customer wants. So I need to be able to open a stream and async write the video file.
This for some reason produces JSON data so that's wrong. But I just don't understand what I need to do to send a streamed video file to a <video> tag in HTML.
Current Code:
[HttpGet]
public HttpResponseMessage GetVideoContent()
{
if (Program.TryOpenFile("BigBuckBunny.mp4", FileMode.Open, out FileStream fs))
{
using (var file = fs)
{
var range = Request.Headers.GetCommaSeparatedValues("Range").FirstOrDefault();
if (range != null)
{
var msg = new HttpResponseMessage(HttpStatusCode.PartialContent);
var body = GetRange(file, range);
msg.Content = new StreamContent(body);
msg.Content.Headers.Add("Content-Type", "video/mp4");
//msg.Content.Headers.Add("Content-Range", $"0-0/{fs.Length}");
return msg;
}
else
{
var msg = new HttpResponseMessage(HttpStatusCode.OK);
msg.Content = new StreamContent(file);
msg.Content.Headers.Add("Content-Type", "video/mp4");
return msg;
}
}
}
else
{
return new HttpResponseMessage(HttpStatusCode.BadRequest);
}
}

HttpResponseMessage is not used as a return type in asp.net-core it will read that as an object model and serialize it in the response by design, as you have already observed.
Luckily in ASP.NET Core 2.0, you have
Enhanced HTTP header support
If an application visitor requests content with a Range Request header, ASP.NET will recognize that and handle that header. If the requested content can be partially delivered, ASP.NET will appropriately skip and return just the requested set of bytes. You don't need to write any special handlers into your methods to adapt or handle this feature; it's automatically handled for you.
So now all you have to do is return the file stream
[HttpGet]
public IActionResult GetVideoContent() {
if (Program.TryOpenFile("BigBuckBunny.mp4", FileMode.Open, out FileStream fs)) {
FileStreamResult result = File(
fileStream: fs,
contentType: new MediaTypeHeaderValue("video/mp4").MediaType,
enableRangeProcessing: true //<-- enable range requests processing
);
return result;
}
return BadRequest();
}
Making sure to enable range requests processing. Though, as stated in the documentation, that should be handled based on the request headers and whether that data can be partially delivered.
From there it is now a simple matter of pointing to the endpoint from the video client and let it do its magic

Uploading file from Flex to WCF REST Stream issues (how to decode multipart form post in REST WS)

The system is a Flex application communicating with a WCF REST web service. I am trying to upload a file from the Flex application to the server and am running into some issues I'm hoping someone here will be able to help out with.
I'm using a FileReference in the Flex app to browse and upload the file as defined here:
http://blog.flexexamples.com/2007/09/21/uploading-files-in-flex-using-the-filereference-class/
I am then receiving the file as a Stream (shows as System.ServiceModel.Dispatcher.StreamFormatter.MessageBodyStream in the debugger) in the WCF REST web service (using project type of WCF 4 REST Service)
[WebInvoke(Method = "POST", UriTemplate = "_test/upload")]
public void UploadImage(Stream data)
{
// TODO: just hardcode filename for now
var filepath = HttpContext.Current.Server.MapPath(#"~\_test\testfile.txt");
using (Stream file = File.OpenWrite(filepath))
{
CopyStream(data, file);
}
}
private static void CopyStream(Stream input, Stream output)
{
var buffer = new byte[8 * 1024];
int len;
while ((len = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, len);
}
}
Note: CopyStream method used from this post: How do I save a stream to a file in C#?
The file saves without any issues. The issue I have is that the file contains more information than i would like. Here are the contents of the saved file (where the source file only contained "THIS IS THE CONTENT OF THE FILE"):
------------ae0ae0Ef1ae0Ef1ae0gL6gL6Ij5cH2
Content-Disposition: form-data; name="Filename"
testfile.txt
------------ae0ae0Ef1ae0Ef1ae0gL6gL6Ij5cH2
Content-Disposition: form-data; name="Filedata"; filename="testfile.txt"
Content-Type: application/octet-stream
THIS IS THE CONTENT OF THE FILE
------------ae0ae0Ef1ae0Ef1ae0gL6gL6Ij5cH2
Content-Disposition: form-data; name="Upload"
Submit Query
------------ae0ae0Ef1ae0Ef1ae0gL6gL6Ij5cH2--
The contents look exactly like they are described in the Adobe documentation:
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/net/FileReference.html
Are there any facilities in C# to get the file contents from the Stream?
EDIT (3/24 8:15 pm): What the Flex app is sending is a Multipart form POST. How can I decode the multipart body data as represented by the Stream parameter and strip out the pieces of the multipart body?
EDIT (3/25 10 am): some more Stack Overflow posts that are related:
WCF service to accept a post encoded multipart/form-data
POSTing multipart/form-data to a WCF REST service: the action changes
EDIT (3/25 10:45 am): Found a multipart parser that works very well:
http://antscode.blogspot.com/2009/11/parsing-multipart-form-data-in-wcf.html
Thanks in advance.

I open-sourced a C# Http form parser here.
This is slightly more flexible than the other one mentioned which is on CodePlex, since you can use it for both Multipart and non-Multipart form-data, and also it gives you other form parameters formatted in a Dictionary object.
This can be used as follows:
non-multipart
public void Login(Stream stream)
{
string username = null;
string password = null;
HttpContentParser parser = new HttpContentParser(data);
if (parser.Success)
{
username = HttpUtility.UrlDecode(parser.Parameters["username"]);
password = HttpUtility.UrlDecode(parser.Parameters["password"]);
}
}
multipart
public void Upload(Stream stream)
{
HttpMultipartParser parser = new HttpMultipartParser(data, "image");
if (parser.Success)
{
string user = HttpUtility.UrlDecode(parser.Parameters["user"]);
string title = HttpUtility.UrlDecode(parser.Parameters["title"]);
// Save the file somewhere
File.WriteAllBytes(FILE_PATH + title + FILE_EXT, parser.FileContents);
}
}

Thanks to Anthony over at http://antscode.blogspot.com/ for the multipart parser that works great (for images, txt files, etc).
http://antscode.blogspot.com/2009/11/parsing-multipart-form-data-in-wcf.html

I've had some issues with parser that are based on string parsing particularly with large files I found it would run out of memory and fail to parse binary data.
To cope with these issues I've open sourced my own attempt at a C# multipart/form-data parser here
See my answer here for more information.