Using ASP.Net Web API, I am developing a service which (amongst other things) retrieves data from Azure, and returns it to the client.
One way of doing this would be to read the netire blob into a buffer, and then write that buffer to the response. However, I'd rather stream the contents, for better performance.
This is simple with the Azure API:
CloudBlobContainer container = BlobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
using (var buffer = new MemoryStream())
{
await blob.DownloadToStreamAsync(buffer);
}
And elsewhere in the code, this is returned to the client:
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(buffer);
But can I be certain that the MemoryStream won't be closed/disposed before the client finishes reading?
As long as you don't wrap your memory stream in a "using" statement you will be fine. If you do use "using" you end up with a weird race condition where it works sometimes and fails at other times.
I have code like yours in production and it works fine.
Only thing to be mindful of is that the whole blob is copied into memory before anything is sent to the client. This may cause memory pressures on your server and initial lag, depending on the size of the file.
If that is a concern, you have a couple of options.
One is to create a "lease" on the blob and give the user a URL to read it direct from blob storage for a limited time. That only works for low security scenarios though.
Alternatively you can use chunked transfer encoding. Basically, you read the file from blob storage in chunks and sends it to the client in those chunks. That saves memory - but I have not been able to make it work async, so you are trading memory for threads. Which is the right solution for you will depend in your specific circumstances.
(I have not got the code to hand, post a comment if you want it and I'll try to dig it out, even if it's a bit old).
Related
I'm storing large media files in Azure Blob Storage (audio, images, videos) that need to be previewed on my web client application.
Currently the client requests a media file and my server downloads the entire blob to memory, then returns the file to the client.
Controller Action
[HttpGet("[action]/{blobName}")]
public async Task<IActionResult> Audio(string blobName)
{
byte[] byteArray = await _blobService.GetAudioAsync(blobName);
return File(byteArray, AVHelper.GetContentType(blobName));
}
Download Service Method
private async Task<byte[]> GetAudioAsync(CloudBlobContainer container, string blobName)
{
using (MemoryStream stream = new MemoryStream())
{
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
await blob.DownloadToStreamAsync(stream);
return stream.ToArray();
}
}
I'm concerned that this is not good design as the file is being downloaded twice in serial which would cause slower downloads and heightened server memory usage. File sizes can be several hundred MB.
Is there some recommended method for doing this? Maybe something where the server is downloading from blob storage and streaming the file to the client pseudo simultaneously? So the client doesn't have to wait for the server to completely download the file to start its download, and the server can remove already transmitted file contents from memory.
To make the answer visible to others, I'm summarizing the answer shared in comment:
The suggestion is to redirect to the Blob Url directly so that the file download can start to client machine directly and the web application don't need to download it to stream or file on the server. Steps:
1.When client clicks on Download, an AJAX request comes to the server.
2.the server code performs necessary verification and returns the file URL of Azure Storage.
3.The AJAX code get the URL returned from the server and opens up a new browser window and redirects it to the URL.
I have two cloud provider with their client SDKs say SDK1 and SDK2. And I wanted to copy one file from one cloud to another cloud storage. These SDKs have upload and download APIs like this:
Response uploadAsync(Uri uploadLocation, Stream fileStream);
Stream downloadAsync(Uri downloadLocation);
Earlier I was copying downloaded Stream to MemoryStream and passing it to upload API. And it was working but obviously, it will load entire file to memory which is not good.
I cannot directly pass downloaded Stream to upload API as somewhere it's checking Length of Stream and System.Net.ConnectStream being non-seekable throws Exception.
Any pointer on how can we use the downloaded Stream (which is of Type System.Net.ConnectStream) in upload API without actually storing entire file?
I'm writing an application similar to HFS which is a HTTP File Server with customization themes in HTML/CSS/JS and i would like to be able to serve my files in multiple parts, because most of download managers connect to the server through multiple connections and download the file as 8 pieces, and that feature ultimately boosts the download speed, and it makes the download to have the capability to resumed and paused.
As far as i know the HTTP's Partial Content makes this possible, i've looked around the web but couldn't find any good example of how to implement it in my code where i use HttpListener to serve webpages and files.
I've seen somewhere that someone suggested to use TcpListener instead but as my whole app works on HttpListener and haven't really find any good examples of Serving Partial Content with TcpListener to switch.
The webserver is multi-threaded and doesn't have any problem handling many requests through different connection simultaneously.
But whenever i download a huge file with IDM it just serves the content though a single connection and IDM shows that the server isn't capable of serving "206" (HTTP Partial Content)
Here's the code that i'm currently using to serve files:
context.Response.ContentType = GetMeme(filename);
Stream input = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
context.Response.ContentLength64 = input.Length;
byte[] buffer = new byte[1024 * 16];
int nbytes;
while ((nbytes = input.Read(buffer, 0, buffer.Length)) > 0)
context.Response.OutputStream.Write(buffer, 0, nbytes);
input.Close();
context.Response.StatusCode = (int)HttpStatusCode.OK;
context.Response.OutputStream.Flush();
context.Response.OutputStream.Close();
I tried to get the buffer offset from the HTTP heads but it fails to close the stream due to the offset and says Cannot close stream until all bytes are written.
Is there any better alternative?
Can even HttpListener handle HTTP: 206 correctly?
How would partial content work on TcpListener?
Any useful links and information would be much appreciated.
Disclaimer: this answer is not pretending to be complete, it's just so many things to talk about in this context. But as a beginning...
The listener you use has no relation to it. Your code should be aware of RANGE HTTP header. When RANGE is supplied, read and serve the file as specified in this header + send 206. Otherwise, serve the entire file and send 200.
It's not the buffer offset you get, but the file offset.
First set response code and other metadata (headers), and write to the stream as the last step.
And you'll probably have to completely change the way you actually serve files. For instance, call CopyToAsync() on FileStream you have.
And it's not Meme, it's MIME.
I have a Windows Service will be reading from local disk (video files) and post them to remote service via API.
Video files over 2gb size and I need to transfer them to another location through HttpClient/POST request.
There is no limitation on API call, so even if file is 10gb I can still post file to destination.
Right now I am reading entire video file as byte[] fileContent and pass it to function as
ByteArrayContent contentBody = new ByteArrayContent(fileContent);
It works for now, but since this is not scalable. (If multiple files will be transferred at the same time, it can fill up memory) I am seeking for a solution that transfer will happen in chunks.
Question: Can I read big files in buffer and transfer over HTTP as I am reading from local disk?
You can use the PostAsync(Uri, HttpContent) method of HttpClient. In order to stream the contents of your local file, use the StreamContent subclass of HttpContent and supply a file reader stream. A brief example:
async Task PostBigFileAsync(Uri uri, string filename)
{
using (var fileStream = File.OpenRead(filename))
{
var client = new HttpClient();
var response = await client.PostAsync(uri, new StreamContent(fileStream));
}
}
I have a WCF Service that returns a byte array with a Zip file (50MB) to any client that requests it. If the Zip is very small (say 1MB), the SOAP response is coming from WCF with the byte array embedded in it. But the response size is very huge even for a 1MB file. If I try to transfer the 50MB file the service hangs and throws an out of memory exception, because the SOAP response becomes huge in size.
What is the best option available with WCF / web service to transfer large files (mainly ZIP format) as I am sending back a byte array. Is there any good approach instead of that for sending back the file?
Whether WCF / web service is best way to transfer large files to any client or is there any other better option/technology available so that interoperability and scalability for 10,000 users can be achieved?
My Ccode is below:
String pathfordownload = #"D:\New Folder.zip";
FileStream F2D = new FileStream(pathfordownload, FileMode.Open,FileAccess.Read);
BinaryReader binReader = new BinaryReader(F2D);
binReader.BaseStream.Position = 0;
byte[] binFile = binReader.ReadBytes(Convert.ToInt32 (binReader.BaseStream.Length));
binReader.Close();
return binFile;
A working piece/real piece of information will be really helpful as I am struggling with all the data available in Google and have had no good results for last week.
You can transfer a Stream through WCF and then you can send (almost) limitless length files.
I've faced the exact same problem. The out of memory is inevitable because you are using Byte arrays.
What we did is to flush the data on the hard drive, so in stead of being limited by your virtual memory your capacity for concurrent transactions is the HD space.
Then for transfer, we jut placed the file on the other computer. Of course in our case it was a server to server file transfer. If you want to de decoupled form the peer, you can use a file download in http.
So instead than responding with a file, your service could respond with a http url to the file location. Then when the client has successfully downloaded form the server with a standard HttpRequest or WebClient it calls a method to delete the file. In SOAP that could be Delete(string url), in REST that would be delete method on the resource.
I hope this makes sense to you. The most importnat part of this is to understand that in a scalable software especially if you are looking at 10000 clients (concurrent?) is that you may not use resources that are limited, like memory streams or byte arrays. But rather rely on large and easily expandable resources like a hard drive partition that coule eventually be on a SAN and IT could grow the partition as needed.