We Keep Coding

Appending to a Response Body in C# .NET Core

I am working on a project where I am populating some pdfs on the back-end, then I convert those pdfs into a List of byte[] which gets merged into one very large array and finally, send back via the response body as a Memory Stream.
My issue is that this is a large amount of data and during the process of getting the list of byte arrays to merge I am using a lot of memory.
I am wondering if instead of converting the final merged byte[] into a Memory Stream and adding that to the response body; could I create several Memory Stream objects that I an append to the Response.Body as they are created? Alternatively, I wondered if there was a way to use the one Memory Stream and just keep adding to it as a create each new byte[] for each pdf document?
Edit: This is probably a little long winded but I was too vague with my original post. At the core of what I am trying to do I have several pdf documents, they are each several pages long. Each of them is represented in the code below as one of the byte[] items in the filesToMerge List. Ideally, I would like to go through these one by one and convert them into a memory stream and send them to the client one right after the other in a loop. However, when I try to do this I get errors that the Response body has already been sent. Is there a way to append something to the response body so it is updated each time through the loop?
[HttpGet("template/{formId}/fillforms")]
public void FillForms(string formId/*, [FromBody] IList<IDictionary<string, string>> fieldDictionaries*/)
{
List<byte[]> filesToMerge = new List<byte[]>();
// For testing
var mockData = new MockData();
IList<IDictionary<string, string>> fieldDictionaries = mockData.GetMock1095Dictionaries();
foreach(IDictionary<string, string> dictionary in fieldDictionaries)
{
var populatedForm = this.dataRepo.PopulateForm(formId, dictionary);
// write to rb
filesToMerge.Add(populatedForm);
}
byte[] mergedFilesAsByteArray = this.dataRepo.GetMergedByteArray(filesToMerge);
this.SendResponse(formId + "_filled.pdf", new MemoryStream(mergedFilesAsByteArray));
}
private void SendResponse(string formName, MemoryStream ms, IDictionary<string, string> fieldData = null)
{
Response.Clear();
Response.ContentType = "application/pdf";
Response.Headers.Add("content-disposition", $"attachment;filename={formName}.pdf");
ms.WriteTo(Response.Body);
}

Memory streams are really just byte arrays with a bunch of nice methods on top. So switching to byte arrays won't help that much. A problem that a log of people run into when dealing with byte arrays and memory streams is not releasing the memory when you are done with the data since they occupy the memory of the machine you are running on so you can easily run out of memory. So you should be disposing of data as soon as you don't need it anymore with "using statements" as an example. Memory streams has a method called Dispose that will release all resources used by the stream
If you wanted to transfer the data from your application as quickly as possible the best approach would be to cut the stream into smaller parts and re-assemble them in the correct order at the destination. You could cut them to 1mb or 126kb really whatever you want. When you send the data to the destination you need to also pass what the order number of this part is because this method allows you to POSt the data in parallel and there is no guarantee of order.
To split a stream into multiple streams
private static List<MemoryStream> CreateChunks(Stream stream)
{
byte[] buffer = new byte[4000000]; //set the size of your buffer (chunk)
var returnStreams = new List<MemoryStream>();
using (MemoryStream ms = new MemoryStream())
{
while (true) //loop to the end of the file
{
var returnStream = new MemoryStream();
int read = stream.Read(buffer, 0, buffer.Length); //read each chunk
returnStream.Write(buffer, 0, read); //write chunk to [wherever];
if (read <= 0)
{ //check for end of file
return returnStreams;
}
else
{
returnStream.Position = 0;
returnStreams.Add(returnStream);
}
}
}
}
I then looped through the streams that were created to create tasks to post to the service and each task would post to the server. I would await all of the tasks to finish then call my server again to tell it I had finished uploading and it could combine all of the data into one in the correct order. My service has the concept of an upload session to keep track of all of the parts and which order they would go in. It would also save each part to the database as they came in; in my case Azure Blob storage.

It's not clear why you would be getting errors copying the contents of multiple MemoryStreams to the Response.Body. You should certainly be able to do this, although you'll need to be sure not to try and change response headers or the status code after you begin writing data (also don't try to call Response.Clear() after you begin writing data).
Here is a simple example of starting a response and then writing data:
[ApiController]
[Route("[controller]")]
public class RandomDataController : ControllerBase {
private readonly ILogger<RandomDataController> logger;
private const String CharacterData = "abcdefghijklmnopqrstuvwxyz0123456789 ";
public RandomDataController(ILogger<RandomDataController> logger) {
this.logger = logger;
}
[HttpGet]
public async Task Get(CancellationToken cancellationToken) {
this.Response.ContentType = "text/plain";
this.Response.ContentLength = 1000;
await this.Response.StartAsync(cancellationToken);
logger.LogInformation("Response Started");
var rand = new Random();
for (var i = 0; i < 1000; i++) {
// You should be able to copy the contents of a MemoryStream or other buffer here instead of sending random data like this does.
await this.Response.Body.WriteAsync(Encoding.UTF8.GetBytes(CharacterData[rand.Next(0, CharacterData.Length)].ToString()), cancellationToken);
Thread.Sleep(50); // This is just to demonstrate that data is being sent to the client as it is written
cancellationToken.ThrowIfCancellationRequested();
if (i % 100 == 0 && i > 0) {
logger.LogInformation("Response In Flight {PercentComplete}", (Double)i / 1000);
}
}
logger.LogInformation("Response Complete");
}
}
You can verify that this streams data back to the client using netcat:
% nc -nc 127.0.0.1 5000
GET /randomdata HTTP/1.1
Host: localhost:5000
Connection: Close
(Enter an extra blank line after Connection: Close to begin the request). You should see data appear in netcat as it is written to Response.Body on the server.
One thing to note is that this approach involves calculating the length of the data to be sent up front. If you are unable to calculate the size of the response up front, or prefer not to, you can look into Chunked Transfer Encoding, which ASP.Net should automatically use if you start writing data to the Response.Body without specifying the Content-Length.

How to check file content is missing in File Upload Web API

I have Web API to upload a single file which is sent along the request body. I have the below code to read file binaries from the stream
Task<HttpResponseMessage> task = Request.Content.ReadAsStreamAsync().ContinueWith<HttpResponseMessage>(t =>
{
if (t.IsFaulted || t.IsCanceled)
throw new HttpResponseException(HttpStatusCode.InternalServerError);
try
{
using (Stream stream = t.Result)
{
using (MemoryStream ms = new MemoryStream())
{
stream.Seek(0, SeekOrigin.Begin);
stream.CopyTo(ms);
byte[] fileBinaries = ms.ToArray();
//logic to process the file
}
}
}
catch (Exception e)
{
//exception handling logic
}
return Request.CreateResponse(HttpStatusCode.Created);
});
return task;
The API works fine when called with file being uploaded; and returns Http status code 201. But if I didn't attached file to the API call still it returns the same code as there is no check on binary data received. I want to add that check to return appropriate error message to user.
I tried to perform this check by evaluating the length of fileBinaries byte array read from Request.Content. But the array has few bytes in it which represents text [object FileList] (don't know how this bytes are filled in the array as i haven't attached any file with the API call). So this won't work for me.
I also tried using the HttpContext.Current.Request.Files.Count but it always returns 0 (Probably due to the file binaries sent in request body) so not suitable for my check.
I can't rely on any headers like File Name as those are not sent in request.
Any help how to perform this?

Try using MultipartMemoryStreamProvider which is ideal for file uploads using webapi
public async Task<IHttpActionResult> UploadFile()
{
var filesReadToProvider = await Request.Content.ReadAsMultipartAsync();
foreach (var stream in filesReadToProvider.Contents)
{
var fileBytes = await stream.ReadAsByteArrayAsync();
}
}
Here fileBytes may not have those 17 bytes.

How use the Byte Array of a image?

So, i am getting the byte array of a LongRaw image from Oracle...
I am using a webapi to this. After get the array, how i use it on the Client-side ?
Do Its better i convert to base64string and pass this value converting just at the client side ?
cmd.InitialLONGFetchSize = -1;
var reader = cmd.ExecuteReader();
if (reader.Read())
{
// Fetch the LONG RAW
OracleBinary imgBinary = reader.GetOracleBinary(0);
// Get the bytes from the binary obj
byte[] imgBytes = imgBinary.IsNull ? null : imgBinary.Value;
//var imgString = Uri.EscapeDataString(Convert.ToBase64String(imgBytes));
}
//CRIO A LISTA
lretorno.Load(reader, LoadOption.OverwriteChanges, "BUSCAFOTO");
reader.Close();
connection.Close();
connection.Dispose();
var teste = lretorno.Tables[0].AsEnumerable().Select(row => new FotoEnvolvido
{
FOTO = (byte[])(row["FOTO"]),
//FOTO = Convert.ToString(row["FOTO"]),
});
return teste;

You can write a Web API Controller that returns the binary data of an image. Base64 strings impose a overhead of the amount of bytes that have to be transmitted. Please avoid this.
A sample controller can look like this example:
public class WebApiController : ApiController
{
public async Task<HttpResponseMessage> Get(string id)
{
var bytes = await GetBytesFromDataLayerAsync(id);
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new MemoryStream(bytes);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("image/jpeg");
return result;
}
private async Task<byte[]> GetBytesFromDataLayerAsync(string id)
{
// put your Oracle logic here
return ...
}
}

Depending on what your doing as rboe said writing the bytes directly to the client will save some data size(approx. 37%) and computing overhead. If your not only displaying jpeg images you should also set the mime-type to the correct value... take a look at this source for a rather complete set of extension to mime-type mappings. If you do not know the mime-type you can try "application/octet-stream" as that is the general mime-type for binary data.
If your displaying your content via web browser you could just use an <img> tag something like <img src="view_image.aspx?id=5"> you can even create the dynamically with javascript/jQuery.
If you really do want the image data embedded in a json request which might be useful if you have a lot of little icons and don't want a ton of requests (with http/2 I don't think this will matter) or another reason, then yes first encode the binary data using...
string base64EncodedData = Convert.ToBase64String(bytes);
If the client is javascript you can decode using the latest browsers native functions
var decodedImageData = window.atob(base64EncodedData);
See:
mozilla.org docs
This answer
This answer
If you are however just sending it to another c# endpoint you can use...
byte[] decodedImageData = Convert.FromBase64String(base64EncodedData);
And like I mentioned in the comment to ensure it's encrypted just make the site only support https:// and if you don't have a SSL cert you can get one for free from http://startssl.com

Can I directly stream from HttpResponseMessage to file without going through memory?

My program uses HttpClient to send a GET request to a Web API, and this returns a file.
I now use this code (simplified) to store the file to disc:
public async Task<bool> DownloadFile()
{
var client = new HttpClient();
var uri = new Uri("http://somedomain.com/path");
var response = await client.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
var fileName = response.Content.Headers.ContentDisposition.FileName;
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.Content.CopyToAsync(fs);
return true;
}
}
return false;
}
Now, when this code runs, the process loads all of the file into memory. I actually would rather expect the stream gets streamed from the HttpResponseMessage.Content to the FileStream, so that only a small portion of it is held in memory.
We are planning to use that on large files (> 1GB), so is there a way to achieve that without having all of the file in memory?
Ideally without manually looping through reading a portion to a byte[] and writing that portion to the file stream until all of the content is written?

It looks like this is by-design - if you check the documentation for HttpClient.GetAsync() you'll see it says:
The returned task object will complete after the whole response
(including content) is read
You can instead use HttpClient.GetStreamAsync() which specifically states:
This method does not buffer the stream.
However you don't then get access to the headers in the response as far as I can see. Since that's presumably a requirement (as you're getting the file name from the headers), then you may want to use HttpWebRequest instead which allows you you to get the response details (headers etc.) without reading the whole response into memory. Something like:
public async Task<bool> DownloadFile()
{
var uri = new Uri("http://somedomain.com/path");
var request = WebRequest.CreateHttp(uri);
var response = await request.GetResponseAsync();
ContentDispositionHeaderValue contentDisposition;
var fileName = ContentDispositionHeaderValue.TryParse(response.Headers["Content-Disposition"], out contentDisposition)
? contentDisposition.FileName
: "noname.dat";
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.GetResponseStream().CopyToAsync(fs);
}
return true
}
Note that if the request returns an unsuccessful response code an exception will be thrown, so you may wish to wrap in a try..catch and return false in this case as in your original example.

Instead of GetAsync(Uri) use the the GetAsync(Uri, HttpCompletionOption) overload with the HttpCompletionOption.ResponseHeadersRead value.
The same applies to SendAsync and other methods of HttpClient
Sources:
docs (see remarks)
https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.getasync?view=netcore-1.1#System_Net_Http_HttpClient_GetAsync_System_Uri_System_Net_Http_HttpCompletionOption_
The returned Task object will complete based on the completionOption parameter after the part or all of the response (including content) is read.
.NET Core implementation of GetStreamAsync that uses HttpCompletionOption.ResponseHeadersRead https://github.com/dotnet/corefx/blob/release/1.1.0/src/System.Net.Http/src/System/Net/Http/HttpClient.cs#L163-L168
HttpClient spike in memory usage with large response
HttpClient.GetStreamAsync() with custom request? (don't mind the comment on response, the ResponseHeadersRead is what does the trick)

Another simple and quick way to do it is:
public async Task<bool> DownloadFile(string url)
{
using (MemoryStream ms = new MemoryStream()) {
new HttpClient().GetStreamAsync(webPath).Result.CopyTo(ms);
... // use ms in what you want
}
}
now you have the file downloaded as stream in ms.

How to use TransmitFile with HttpListener

I have HTTP server written using HttpListener and want zero-copy technology for sending files to clients.
Is there are any option to use TransmitFile to respond?

I assume you're referring to HttpResponse.TransmitFile? HttpListener doesn't buffer the response content, so you just need to write directly to the output stream.
You can use an extension method like this to mimic the ASP.NET behavior:
public static void TransmitFile(this HttpListenerResponse response, string fileName)
{
using (var fileStream = File.OpenRead(filename))
{
response.ContentLength64 = fileStream.Length;
fileStream.CopyTo(response.OutputStream);
}
}