This is a follow up to a question I had asked previously that was closed for being too broad.Previous Question
In that question I explained that I needed upload a large file (1-3GB) to the database by storing chunks as individual rows. I did this by overriding the MultipartFormDataStreamProvider.GetStream method. That method returned a custom stream that wrote the buffered chunks to the database.
The problem is that the overriden GetStream method is writing the entire request to the database (including the headers). It is successfully writing that data while keeping the Memory levels flat but when I download the file, in addition to the file contents, it's returning all the header information in the downloaded file contents so the file can't be opened.
Is there a way to, in the overriden GetStream method, write just the contents of the file to the database without writing the headers?
API
[HttpPost]
[Route("file")]
[ValidateMimeMultipartContentFilter]
public Task<HttpResponseMessage> PostFormData()
{
var provider = new CustomMultipartFormDataStreamProvider();
// Read the form data and return an async task.
var task = Request.Content.ReadAsMultipartAsync(provider).ContinueWith<HttpResponseMessage>(t =>
{
if (t.IsFaulted || t.IsCanceled)
{
Request.CreateErrorResponse(HttpStatusCode.InternalServerError, t.Exception);
}
return Request.CreateResponse(HttpStatusCode.OK);
});
return task;
}
[HttpGet]
[Route("file/{id}")]
public async Task<HttpResponseMessage> GetFile(string id)
{
var result = new HttpResponseMessage()
{
Content = new PushStreamContent(async (outputStream, httpContent, transportContext) =>
{
await WriteDataChunksFromDBToStream(outputStream, httpContent, transportContext, id);
}),
StatusCode = HttpStatusCode.OK
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zipx");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "test response.zipx" };
return result;
}
return new HttpResponseMessage(HttpStatusCode.BadRequest);
}
private async Task WriteDataChunksFromDBToStream(Stream responseStream, HttpContent httpContent, TransportContext transportContext, string fileIdentifier)
{
// PushStreamContent requires the responseStream to be closed
// for signaling it that you have finished writing the response.
using (responseStream)
{
using (var myConn = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["TestDB"].ConnectionString))
{
await myConn.OpenAsync();
using (var myCmd = new SqlCommand("ReadAttachmentChunks", myConn))
{
myCmd.CommandType = System.Data.CommandType.StoredProcedure;
var fileName = new SqlParameter("#Identifier", fileIdentifier);
myCmd.Parameters.Add(fileName);
// Read data back from db in async call to avoid OutOfMemoryException when sending file back to user
using (var reader = await myCmd.ExecuteReaderAsync(CommandBehavior.SequentialAccess))
{
while (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(3)))
{
using (var data = reader.GetStream(3))
{
// Asynchronously copy the stream from the server to the response stream
await data.CopyToAsync(responseStream);
}
}
}
}
}
}
}// close response stream
}
Custom MultipartFormDataStreamProvider GetStream method implementation
public override Stream GetStream(HttpContent parent, HttpContentHeaders headers)
{
// For form data, Content-Disposition header is a requirement
ContentDispositionHeaderValue contentDisposition = headers.ContentDisposition;
if (contentDisposition != null)
{
// If we have a file name then write contents out to AWS stream. Otherwise just write to MemoryStream
if (!String.IsNullOrEmpty(contentDisposition.FileName))
{
var identifier = Guid.NewGuid().ToString();
var fileName = contentDisposition.FileName;// GetLocalFileName(headers);
if (fileName.Contains("\\"))
{
fileName = fileName.Substring(fileName.LastIndexOf("\\") + 1).Replace("\"", "");
}
// We won't post process files as form data
_isFormData.Add(false);
var stream = new CustomSqlStream();
stream.Filename = fileName;
stream.Identifier = identifier;
stream.ContentType = headers.ContentType.MediaType;
stream.Description = (_formData.AllKeys.Count() > 0 && _formData["description"] != null) ? _formData["description"] : "";
return stream;
//return new CustomSqlStream(contentDisposition.Name);
}
// We will post process this as form data
_isFormData.Add(true);
// If no filename parameter was found in the Content-Disposition header then return a memory stream.
return new MemoryStream();
}
throw new InvalidOperationException("Did not find required 'Content-Disposition' header field in MIME multipart body part..");
#endregion
}
Implemented Write method of Stream called by CustomSqlStream
public override void Write(byte[] buffer, int offset, int count)
{
//write buffer to database
using (var myConn = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["TestDB"].ConnectionString)) {
using (var myCmd = new SqlCommand("WriteAttachmentChunk", myConn)) {
myCmd.CommandType = System.Data.CommandType.StoredProcedure;
var pContent = new SqlParameter("#Content", buffer);
myCmd.Parameters.Add(pContent);
myConn.Open();
myCmd.ExecuteNonQuery();
if (myConn.State == System.Data.ConnectionState.Open)
{
myConn.Close();
}
}
}
((ManualResetEvent)_dataAddedEvent).Set();
}
The "ReadAttachmentChunks" stored procedure gets the rows respective to the file from the db ordered by the time they are inserted into the database. So, the way the code works is it pulls those chunks back and then async writes it back to the PushStreamContent to go back to the user.
So my question is:
Is there a way to write ONLY the content of the file being uploaded as opposed to the headers in addition to the content?
Any help would be greatly appreciated. Thank you.
I finally figured it out. I over-complicated the write process which brought about most of the struggle. Here is my solution to my initial issue:
To keep .net from buffering the file in memory (so that you can handle large file uploads), you first need to override the WebHostBufferPolicySelector so that it doesnt buffer the input stream for your controller and then replace the BufferPolicy Selector.
public class NoBufferPolicySelector : WebHostBufferPolicySelector
{
public override bool UseBufferedInputStream(object hostContext)
{
var context = hostContext as HttpContextBase;
if (context != null)
{
if (context.Request.RequestContext.RouteData.Values["controller"] != null)
{
if (string.Equals(context.Request.RequestContext.RouteData.Values["controller"].ToString(), "upload", StringComparison.InvariantCultureIgnoreCase))
return false;
}
}
return true;
}
public override bool UseBufferedOutputStream(HttpResponseMessage response)
{
return base.UseBufferedOutputStream(response);
}
}
then for replacing the BufferPolicy Selector
GlobalConfiguration.Configuration.Services.Replace(typeof(IHostBufferPolicySelector), new NoBufferPolicySelector());
Then to avoid the default behavior of having the file stream written to disk, you need to provide a stream provider that will write to the database instead. To do this you inherit MultipartStreamProvider and override the GetStream method to return the stream that will write to your database.
public override Stream GetStream(HttpContent parent, HttpContentHeaders headers)
{
// For form data, Content-Disposition header is a requirement
ContentDispositionHeaderValue contentDisposition = headers.ContentDisposition;
if (contentDisposition != null && !String.IsNullOrEmpty(contentDisposition.FileName))
{
// We won't post process files as form data
_isFormData.Add(false);
//create unique identifier for this file upload
var identifier = Guid.NewGuid();
var fileName = contentDisposition.FileName;
var boundaryObj = parent.Headers.ContentType.Parameters.SingleOrDefault(a => a.Name == "boundary");
var boundary = (boundaryObj != null) ? boundaryObj.Value : "";
if (fileName.Contains("\\"))
{
fileName = fileName.Substring(fileName.LastIndexOf("\\") + 1).Replace("\"", "");
}
//write parent container for the file chunks that are being stored
WriteLargeFileContainer(fileName, identifier, headers.ContentType.MediaType, boundary);
//create an instance of the custom stream that will write the chunks to the database
var stream = new CustomSqlStream();
stream.Filename = fileName;
stream.FullFilename = contentDisposition.FileName.Replace("\"", "");
stream.Identifier = identifier.ToString();
stream.ContentType = headers.ContentType.MediaType;
stream.Boundary = (!string.IsNullOrEmpty(boundary)) ? boundary : "";
return stream;
}
else
{
// We will post process this as form data
_isFormData.Add(true);
// If no filename parameter was found in the Content-Disposition header then return a memory stream.
return new MemoryStream();
}
}
The custom stream you create needs to inherit Stream and override the Write method. This is where I overthought the problem and thought I needed to parse out the boundary headers that were passed via the buffer parameter. But this is actually done for you by leveraging the offset and count parameters.
public override void Write(byte[] buffer, int offset, int count)
{
//no boundary is inluded in buffer
byte[] fileData = new byte[count];
Buffer.BlockCopy(buffer, offset, fileData, 0, count);
WriteData(fileData);
}
From there, it's just plugging in the api methods for upload and download.
For upload:
public Task<HttpResponseMessage> PostFormData()
{
var provider = new CustomMultipartLargeFileStreamProvider();
// Read the form data and return an async task.
var task = Request.Content.ReadAsMultipartAsync(provider).ContinueWith<HttpResponseMessage>(t =>
{
if (t.IsFaulted || t.IsCanceled)
{
Request.CreateErrorResponse(HttpStatusCode.InternalServerError, t.Exception);
}
return Request.CreateResponse(HttpStatusCode.OK);
});
return task;
}
For download, and in order to keep the memory footprint low, I leveraged the PushStreamContent to push the chunks back to the user:
[HttpGet]
[Route("file/{id}")]
public async Task<HttpResponseMessage> GetFile(string id)
{
string mimeType = string.Empty;
string filename = string.Empty;
if (!string.IsNullOrEmpty(id))
{
//get the headers for the file being sent back to the user
using (var myConn = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["PortalBetaConnectionString"].ConnectionString))
{
using (var myCmd = new SqlCommand("ReadLargeFileInfo", myConn))
{
myCmd.CommandType = System.Data.CommandType.StoredProcedure;
var pIdentifier = new SqlParameter("#Identifier", id);
myCmd.Parameters.Add(pIdentifier);
myConn.Open();
var dataReader = myCmd.ExecuteReader();
if (dataReader.HasRows)
{
while (dataReader.Read())
{
mimeType = dataReader.GetString(0);
filename = dataReader.GetString(1);
}
}
}
}
var result = new HttpResponseMessage()
{
Content = new PushStreamContent(async (outputStream, httpContent, transportContext) =>
{
//pull the data back from the db and stream the data back to the user
await WriteDataChunksFromDBToStream(outputStream, httpContent, transportContext, id);
}),
StatusCode = HttpStatusCode.OK
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue(mimeType);// "application/octet-stream");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = filename };
return result;
}
return new HttpResponseMessage(HttpStatusCode.BadRequest);
}
private async Task WriteDataChunksFromDBToStream(Stream responseStream, HttpContent httpContent, TransportContext transportContext, string fileIdentifier)
{
// PushStreamContent requires the responseStream to be closed
// for signaling it that you have finished writing the response.
using (responseStream)
{
using (var myConn = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["PortalBetaConnectionString"].ConnectionString))
{
await myConn.OpenAsync();
//stored proc to pull the data back from the db
using (var myCmd = new SqlCommand("ReadAttachmentChunks", myConn))
{
myCmd.CommandType = System.Data.CommandType.StoredProcedure;
var fileName = new SqlParameter("#Identifier", fileIdentifier);
myCmd.Parameters.Add(fileName);
// The reader needs to be executed with the SequentialAccess behavior to enable network streaming
// Otherwise ReadAsync will buffer the entire BLOB into memory which can cause scalability issues or even OutOfMemoryExceptions
using (var reader = await myCmd.ExecuteReaderAsync(CommandBehavior.SequentialAccess))
{
while (await reader.ReadAsync())
{
//confirm the column that has the binary data of the file returned is not null
if (!(await reader.IsDBNullAsync(0)))
{
//read the binary data of the file into a stream
using (var data = reader.GetStream(0))
{
// Asynchronously copy the stream from the server to the response stream
await data.CopyToAsync(responseStream);
await data.FlushAsync();
}
}
}
}
}
}
}// close response stream
}
Ugh. This is nasty. With the upload, you have to make sure to
separate the headers from the content portion - you must follow the requirements RFC documents for HTTP.
Allow for chunked transfers
Of course, the content portion (unless you are transmitting text) will be binary encoded into strings.
Allow for transfers that are compressed, i.e. GZIP or DEFLATE.
Maybe - just maybe - take the encoding into account (ASCII, Unicode, UTF8, etc).
You can't really ensure that you're persisting the right info to the database without looking at all of these. For the latter items, all of your metadata as to what to do will be somewhere in the header, so it's not just a throwaway.
Related
Over the last few weeks I have been working on an ASP.NET WebAPI that was designed to stream video from one of the company servers and play it on an HTML5 <video> element. Following a guide on C# Corner, we got the API published and now when the link for one of our videos is pasted into a browser, it starts to download (which, by the way, I'm not sure if it's supposed to do that when all we're trying to do is stream).
The files we need to stream are mp4 and are going to be used largely on iOS devices through Safari. And before anyone asks: srcVid is programmed and confirmed to be able to encode .mp4 files successfully, as we have managed to hard-code videos into this element with no issue. With that said, this is how the page handles its HTML5 elements:
<video autoplay muted id="trainVid" style="width: 75%; height: auto;" controls>
<source id="srcVid" runat="server"
type='video/mp4; codecs*="avc1.424085, mp4a.40.2"' />
</video>
On the API side, here are how the videos are processed, largely following the example set by the C# article, as well as some help from a Stephen Cleary article:
public class VidService
{
public async void WriteVidBytes(Stream outputStream,
HttpContent content, TransportContext tc)
{
try
{
var filePath = "\\\\server\\link\\to\\file.mp4";
int bufferSize = 1000;
byte[] buffer = new byte[bufferSize];
using (var fileStream = new FileStream(
filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
int fileSize = (int)fileStream.Length;
while (fileSize > 0)
{
int cnt = fileSize > bufferSize ? bufferSize : fileSize,
readBufferSize = fileStream.Read(buffer, 0, cnt);
await outputStream.WriteAsync(buffer, 0, readBufferSize);
fileSize -= readBufferSize;
}
}
}
catch (HttpException ex) { return; }
finally { outputStream.Close(); }
}
public HttpResponseMessage GetVidContent()
{
// NOTE: please see the Edit on 6/10
var httpResponse = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new PushStreamContent(
(Action<Stream, HttpContent, TransportContext>)WriteVidBytes
)
};
return httpResponse;
}
}
public class VidController : ApiController
{
private static readonly VidService vs = new VidService();
[HttpGet]
public HttpResponseMessage GetVid(int id)
{
return vs.GetVidContent(id);
}
}
*Note that the actual program dynamically fetches video links through a Video.cs object
And finally on the C# side:
protected void LoadVideo(int vidId)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(
string.Format("http://mobileAPI.website.com/Vid/GetVid/" + vidId.ToString()));
req.Method = "GET";
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
string jsonString;
using (Stream stream = resp.GetResponseStream())
{
StreamReader r = new StreamReader(stream, System.Text.Encoding.UTF8);
jsonString = r.ReadToEnd();
}
srcVid.Src = jsonString;
}
When opening the page, LoadVideo() seems to execute with no errors -- but after this, the page goes blank and hangs forever. I'm thinking this may be because I'm putting the wrong value into srcVid.Src, but if I don't put in the jsonString, then what do I put in for the source?
As always, any help would be greatly appreciated! If I missed anything obvious, please let me know, as this is the first time I have worked with WebAPI.
UPDATE 1 (6/10)
I made a secondary method that took WriteVidBytes and turned it into a Task -- and other than turning it into a Task, the code inside is exactly the same. Another difference, also, is how GetVidContent fetches the data:
public HttpResponseMessage GetVidContent(int vId)
{
var httpResponse = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new PushStreamContent(async
(outputStream, httpContext, transportContext) =>
{
await WriteVidTask(outputStream, httpContext, transportContext);
}),
};
return httpResponse;
}
However, the page still hangs even though there is no problem getting the file through Postman or Fiddler.
Following an example from Robert Huang on CodeProject, I was able to get the video to stream through API without any issue.
The first thing I changed was the way that the <video> reads the source. Rather than a JSON string, the video loads the API link.
srcVid.Src = "http://api.website.com/Vid/GetVid?id=" + vidId.ToString();
Following the CodeProject link, I created an HttpResponseMessage, very similar to the one provided -- only this one supports asynchronous loading:
public HttpResponseMessage GetVidContent(RangeHeaderValue rh, FileInfo fi)
{
HttpResponseMessage response = new HttpResponseMessage();
response.Headers.AcceptRanges.Add("bytes");
long totalLength = fi.Length;
if (rh == null || !rh.Ranges.Any())
{ // treat request normally if there is no range header
response.StatusCode = HttpStatusCode.OK;
response.Content = new PushStreamContent(async (outputStream,
httpContent, transpContext) =>
{
using (outputStream) // copy file to output stream straightforward
using (Stream inputStream = fi.OpenRead())
{
try
{
await inputStream.CopyToAsync(outputStream, ReadStreamBufferSize);
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}
}
}, GetMimeNameFromExt(fi.Extension));
return response;
}
long start = 0, end = 0;
if (rh.Unit != "bytes" || rh.Ranges.Count > 1 || TryReadRangeItem(rh.Ranges.First(),
totalLength, out start, out end))
{
response.StatusCode = HttpStatusCode.RequestedRangeNotSatisfiable;
response.Content = new StreamContent(Stream.Null);
response.Content.Headers.ContentRange = new ContentRangeHeaderValue(totalLength);
response.Content.Headers.ContentType = GetMimeNameFromExt(fi.Extension);
return response;
}
var contentRange = new ContentRangeHeaderValue(start, end, totalLength);
response.StatusCode = HttpStatusCode.PartialContent;
response.Content = new PushStreamContent(async (outputStream, httpContent, transpContext) =>
{
using (outputStream)
using (Stream inputStream = fi.OpenRead())
await CreatePartialContent(inputStream, outputStream, start, end);
}, GetMimeNameFromExt(fi.Extension));
response.Content.Headers.ContentLength = end - start + 1;
response.Content.Headers.ContentRange = contentRange;
return response;
}
Any methods from the CodeProject article used in await areas were marked with the async expression.
From my MVC app I am trying to do a POST request to my Web API app. After receiving the data inside Web API app, when I try to read it like HttpContext.Current.Request.Form["SenderAddress"]; I get an error saying Either BinaryRead, Form, Files, or InputStream was accessed before the internal storage was filled by the caller of HttpRequest.GetBufferedInputStream. Below is the code for both sending and receiving:
POST request (from MVC):
var form = new MultipartFormDataContent();
var uri = _domain + "/api/email/send";
//adding all properties to the form
if (model.SenderAddress != "" && model.SenderAddress != null)
form.Add(new StringContent(model.SenderAddress), "SenderAddress");
if (model.Recipients.FirstOrDefault() != null)
{
foreach (var recipient in model.Recipients)
form.Add(new StringContent(recipient), "Recipients");
}
if (model.Attachments.FirstOrDefault() != null)
{
foreach (var attachment in model.Attachments)
{
byte[] fileData = null;
using (var binaryReader = new BinaryReader(attachment.InputStream))
{
fileData = binaryReader.ReadBytes(attachment.ContentLength);
}
form.Add(new ByteArrayContent(fileData, 0, fileData.Length), "Attachments", attachment.FileName);
}
}
HttpResponseMessage response = await client.PostAsync(uri, form);
response.EnsureSuccessStatusCode();
string sd = response.Content.ReadAsStringAsync().Result;
Receiving data (at Web API):
var temp3 = HttpContext.Current.Request.Form["SenderAddress"];
Here I get that error. How do I solve it?
i don't understand what this has to do with end of multipart stream but here is the working code:
public async Task<HttpResponseMessage> PostFormData()
{
// Check if the request contains multipart/form-data.
if (!Request.Content.IsMimeMultipartContent())
{
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
string root = HttpContext.Current.Server.MapPath("~/App_Data");
var provider = new MultipartFormDataStreamProvider(root);
try
{
// Read the form data.
await Request.Content.ReadAsMultipartAsync(provider);
// This illustrates how to get the file names.
foreach (MultipartFileData file in provider.FileData)
{
Trace.WriteLine(file.Headers.ContentDisposition.FileName);
Trace.WriteLine("Server file path: " + file.LocalFileName);
}
return Request.CreateResponse(HttpStatusCode.OK);
}
catch (System.Exception e)
{
return Request.CreateErrorResponse(HttpStatusCode.InternalServerError, e);
}
}
also checked the tutorial code in asp.net site as well:
https://learn.microsoft.com/en-us/aspnet/web-api/overview/advanced/sending-html-form-data-part-2
I am struggling with being able to create a file with its data based on the byte array returned from the WebAPI. The following is my code for making the call to the web api
using (var http = new WebClient())
{
string url = string.Format("{0}api/FileUpload/FileServe?FileID=" + fileID, webApiUrl);
http.Headers[HttpRequestHeader.ContentType] = "application/octet-stream";
http.Headers[HttpRequestHeader.Authorization] = "Bearer " + authCookie.Value;
http.DownloadDataCompleted += Http_DownloadDataCompleted;
byte[] json = await http.DownloadDataTaskAsync(url);
}
The api code is
[HttpGet]
[Route("FileServe")]
[Authorize(Roles = "Admin,SuperAdmin,Contractor")]
public async Task<HttpResponseMessage> GetFile(int FileID)
{
using (var repo = new MBHDocRepository())
{
var file = await repo.GetSpecificFile(FileID);
if (file == null)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
var stream = File.Open(file.PathLocator, FileMode.Open);
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(file.FileType);
return response;
}
}
I receive a byte array as a response however am unable to create the corresponding file from that byte array. I have no idea how to convert the byte array into the relevant file type (such as jpg, or pdf based on file type in the web api). any help will be appreciated.
Alright so there are a few ways of solving your problem firstly, on the server side of things you can either simply send the content type and leave it at that or you can also send the complete filename which helps you even further.
I have removed the code that is specific to your stuff with basic test code, please just ignore that stuff and use it in terms of your code.
Some design notes here:
[HttpGet]
[Route("FileServe")]
[Authorize(Roles = "Admin,SuperAdmin,Contractor")]
public async Task<HttpResponseMessage> GetFileAsync(int FileID) //<-- If your method returns Task have it be named with Async in it
{
using (var repo = new MBHDocRepository())
{
var file = await repo.GetSpecificFile(FileID);
if (file == null)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
var stream = File.Open(file.PathLocator, FileMode.Open);
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(file.FileType);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment") { FileName=Path.GetFileName(file.PathLocator)};
return response;
}
}
Your client side code has two options here:
static void Main(string[] args)
{
using (var http = new WebClient())
{
string url = string.Format("{0}api/FileUpload/FileServe?FileID={1}",webApiUrl, fileId);
http.Headers[HttpRequestHeader.ContentType] = "application/octet-stream";
http.Headers[HttpRequestHeader.Authorization] = "Bearer " + authCookie.Value;
var response = http.OpenRead(url);
var fs = new FileStream(String.Format(#"C:\Users\Bailey Miller\Downloads\{0}", GetName(http.ResponseHeaders)), FileMode.Create);
response.CopyTo(fs); <-- how to move the stream to the actual file, this is not perfect and there are a lot of better examples
fs.Flush();
fs.Close();
}
}
private static object GetName(WebHeaderCollection responseHeaders)
{
var c_type = responseHeaders.GetValues("Content-Type"); //<-- do a switch on this and return a really weird file name with the correct extension for the mime type.
var cd = responseHeaders.GetValues("Content-Disposition")[0].Replace("\"", ""); <-- this gets the attachment type and filename param, also removes illegal character " from filename if present
return cd.Substring(cd.IndexOf("=")+1); <-- extracts the file name
}
I am trying to send large file (GB) from one WebAPI (.NET Core) to another WebApi (.Net Core).
I already managed to send smaller file as part of Multipart Request like in last post here: link
To send bigger file I need (I think) send this file as StreamContent, however i am getting Content length = 0 in API which receives request.
Problem occurs even when I am sending (for test) smaller files (10 Mb).
Clientside code:
[HttpPost("UploadFiles")]
public async Task<IActionResult> Post(IFormFile file)
{
var filePath = Path.GetTempFileName();
using (var stream = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
await file.CopyToAsync(stream);
using (var formDataContent = new MultipartFormDataContent())
{
using (var httpClient = new HttpClient())
{
formDataContent.Add(CreateFileContent(stream, "myfile.test", "application/octet-stream"));
var response = await httpClient.PostAsync(
"http://localhost:56595/home/upload",
formDataContent);
return Json(response);
}
}
}
}
internal static StreamContent CreateFileContent(Stream stream, string fileName, string contentType)
{
var fileContent = new StreamContent(stream);
fileContent.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("form-data")
{
Name = "\"file\"",
FileName = "\"" + fileName + "\"",
};
fileContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(contentType);
return fileContent;
}
Serverside code:
[HttpPost]
public ActionResult Upload()
{
IFormFile fileFromRequest = Request.Form.Files.First();
string myFileName = fileFromRequest.Name;
// some code
return Ok();
}
Where is the problem?
To create Multipart request I used advices from:
HttpClient StreamContent append filename twice
POST StreamContent with Multiple Files
Finally I figured it out:
There were 2 problems:
1. Stream pointer position
In client side code, change this:
await file.CopyToAsync(stream);
to that:
await file.CopyToAsync(stream);
stream.Position = 0;
Problem was that file from request was copied to stream and left position of pointer in the end of the stream. That is why request send from client had stream with proper length, but actually when it started to read it, it could not (read 0 bytes).
2. Wrong way of handling request on server.
I used code from dotnetcoretutorials.com
Working code below:
Client side:
[HttpPost("UploadFiles")]
public async Task<IActionResult> Post(IFormFile file)
{
var filePath = Path.GetTempFileName();
using (var stream = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
await file.CopyToAsync(stream);
stream.Position = 0;
using (var formDataContent = new MultipartFormDataContent())
{
using (var httpClient = new HttpClient())
{
formDataContent.Add(CreateFileContent(stream, "myfile.test", "application/octet-stream"));
var response = await httpClient.PostAsync(
"http://localhost:56595/home/upload",
formDataContent);
return Json(response);
}
}
}
}
internal static StreamContent CreateFileContent(Stream stream, string fileName, string contentType)
{
var fileContent = new StreamContent(stream);
fileContent.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("form-data")
{
Name = "\"file\"",
FileName = "\"" + fileName + "\"",
};
fileContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(contentType);
return fileContent;
}
Server side:
Controller:
[HttpPost]
[DisableFormValueModelBinding]
public async Task<IActionResult> Upload()
{
var viewModel = new MyViewModel();
try
{
FormValueProvider formModel;
using (var stream = System.IO.File.Create("c:\\temp\\myfile.temp"))
{
formModel = await Request.StreamFile(stream);
}
var bindingSuccessful = await TryUpdateModelAsync(viewModel, prefix: "",
valueProvider: formModel);
if (!bindingSuccessful)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
}
}
catch(Exception exception)
{
throw;
}
return Ok(viewModel);
}
Helper classes for methods from controller:
public static class MultipartRequestHelper
{
// Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
// The spec says 70 characters is a reasonable limit.
public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
{
var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary);
if (string.IsNullOrWhiteSpace(boundary.ToString()))
{
throw new InvalidDataException("Missing content-type boundary.");
}
if (boundary.Length > lengthLimit)
{
throw new InvalidDataException(
$"Multipart boundary length limit {lengthLimit} exceeded.");
}
return boundary.ToString();
}
public static bool IsMultipartContentType(string contentType)
{
return !string.IsNullOrEmpty(contentType)
&& contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
}
public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="key";
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& string.IsNullOrEmpty(contentDisposition.FileName.ToString())
&& string.IsNullOrEmpty(contentDisposition.FileNameStar.ToString());
}
public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& (!string.IsNullOrEmpty(contentDisposition.FileName.ToString())
|| !string.IsNullOrEmpty(contentDisposition.FileNameStar.ToString()));
}
}
public static class FileStreamingHelper
{
private static readonly FormOptions _defaultFormOptions = new FormOptions();
public static async Task<FormValueProvider> StreamFile(this HttpRequest request, Stream targetStream)
{
if (!MultipartRequestHelper.IsMultipartContentType(request.ContentType))
{
throw new Exception($"Expected a multipart request, but got {request.ContentType}");
}
// Used to accumulate all the form url encoded key value pairs in the
// request.
var formAccumulator = new KeyValueAccumulator();
string targetFilePath = null;
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
ContentDispositionHeaderValue contentDisposition;
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out contentDisposition);
if (hasContentDispositionHeader)
{
if (MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
{
await section.Body.CopyToAsync(targetStream);
}
else if (MultipartRequestHelper.HasFormDataContentDisposition(contentDisposition))
{
// Content-Disposition: form-data; name="key"
//
// value
// Do not limit the key name length here because the
// multipart headers length limit is already in effect.
var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name);
var encoding = GetEncoding(section);
using (var streamReader = new StreamReader(
section.Body,
encoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1024,
leaveOpen: true))
{
// The value length limit is enforced by MultipartBodyLengthLimit
var value = await streamReader.ReadToEndAsync();
if (String.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase))
{
value = String.Empty;
}
formAccumulator.Append(key.ToString(), value);
if (formAccumulator.ValueCount > _defaultFormOptions.ValueCountLimit)
{
throw new InvalidDataException($"Form key count limit {_defaultFormOptions.ValueCountLimit} exceeded.");
}
}
}
}
// Drains any remaining section body that has not been consumed and
// reads the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
// Bind form data to a model
var formValueProvider = new FormValueProvider(
BindingSource.Form,
new FormCollection(formAccumulator.GetResults()),
CultureInfo.CurrentCulture);
return formValueProvider;
}
private static Encoding GetEncoding(MultipartSection section)
{
MediaTypeHeaderValue mediaType;
var hasMediaTypeHeader = MediaTypeHeaderValue.TryParse(section.ContentType, out mediaType);
// UTF-7 is insecure and should not be honored. UTF-8 will succeed in
// most cases.
if (!hasMediaTypeHeader || Encoding.UTF7.Equals(mediaType.Encoding))
{
return Encoding.UTF8;
}
return mediaType.Encoding;
}
}
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var formValueProviderFactory = context.ValueProviderFactories
.OfType<FormValueProviderFactory>()
.FirstOrDefault();
if (formValueProviderFactory != null)
{
context.ValueProviderFactories.Remove(formValueProviderFactory);
}
var jqueryFormValueProviderFactory = context.ValueProviderFactories
.OfType<JQueryFormValueProviderFactory>()
.FirstOrDefault();
if (jqueryFormValueProviderFactory != null)
{
context.ValueProviderFactories.Remove(jqueryFormValueProviderFactory);
}
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}
Additional thoughts:
(on clientside) line:
fileContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(contentType);
is not necessary to send the file.
(on clientside) file is send when MediaTypeHeaderValue is one of these:
application/x-msdownload
application/json
application/octet-stream
(on serverside) to use lines with contentDisposition.FileNameStar on serverside you need to change them to contentDisposition.FileNameStar.ToString()
(on serverside) code used in question for serverside will work with smaller files (Mb's) but to send GB file we need code which is pasted in the answer.
some parts of code are taken from aspnet core docs
I have a self-hosted app that's using OWIN to provide a basic web server. The key part of the configuration is the following line:
appBuilder.UseFileServer(new FileServerOptions {
FileSystem = new PhysicalFileSystem(filePath)
});
This provides the static files listed in the filePath for browsing, and this much is working as expected.
However I've run into a case where I want to slightly modify one of the files on a request-by-request basis. In particular, I want to load the "normal" version of the file from the filesystem, alter it slightly based on the incoming web request's headers, and then return the altered version to the client instead of the original. All other files should remain unmodified.
How do I go about doing this?
Well, I don't know if this is a good way to do this, but it seems to work:
internal class FileReplacementMiddleware : OwinMiddleware
{
public FileReplacementMiddleware(OwinMiddleware next) : base(next) {}
public override async Task Invoke(IOwinContext context)
{
MemoryStream memStream = null;
Stream httpStream = null;
if (ShouldAmendResponse(context))
{
memStream = new MemoryStream();
httpStream = context.Response.Body;
context.Response.Body = memStream;
}
await Next.Invoke(context);
if (memStream != null)
{
var content = await ReadStreamAsync(memStream);
if (context.Response.StatusCode == 200)
{
content = AmendContent(context, content);
}
var contentBytes = Encoding.UTF8.GetBytes(content);
context.Response.Body = httpStream;
context.Response.ETag = null;
context.Response.ContentLength = contentBytes.Length;
await context.Response.WriteAsync(contentBytes, context.Request.CallCancelled);
}
}
private static async Task<string> ReadStreamAsync(MemoryStream stream)
{
stream.Seek(0, SeekOrigin.Begin);
using (var reader = new StreamReader(stream, Encoding.UTF8))
{
return await reader.ReadToEndAsync();
}
}
private bool ShouldAmendResponse(IOwinContext context)
{
// logic
}
private string AmendContent(IOwinContext context, string content)
{
// logic
}
}
Add this to the pipeline somewhere before the static files middleware.