I have a Windows Service will be reading from local disk (video files) and post them to remote service via API.
Video files over 2gb size and I need to transfer them to another location through HttpClient/POST request.
There is no limitation on API call, so even if file is 10gb I can still post file to destination.
Right now I am reading entire video file as byte[] fileContent and pass it to function as
ByteArrayContent contentBody = new ByteArrayContent(fileContent);
It works for now, but since this is not scalable. (If multiple files will be transferred at the same time, it can fill up memory) I am seeking for a solution that transfer will happen in chunks.
Question: Can I read big files in buffer and transfer over HTTP as I am reading from local disk?
You can use the PostAsync(Uri, HttpContent) method of HttpClient. In order to stream the contents of your local file, use the StreamContent subclass of HttpContent and supply a file reader stream. A brief example:
async Task PostBigFileAsync(Uri uri, string filename)
{
using (var fileStream = File.OpenRead(filename))
{
var client = new HttpClient();
var response = await client.PostAsync(uri, new StreamContent(fileStream));
}
}
Related
I'm storing large media files in Azure Blob Storage (audio, images, videos) that need to be previewed on my web client application.
Currently the client requests a media file and my server downloads the entire blob to memory, then returns the file to the client.
Controller Action
[HttpGet("[action]/{blobName}")]
public async Task<IActionResult> Audio(string blobName)
{
byte[] byteArray = await _blobService.GetAudioAsync(blobName);
return File(byteArray, AVHelper.GetContentType(blobName));
}
Download Service Method
private async Task<byte[]> GetAudioAsync(CloudBlobContainer container, string blobName)
{
using (MemoryStream stream = new MemoryStream())
{
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
await blob.DownloadToStreamAsync(stream);
return stream.ToArray();
}
}
I'm concerned that this is not good design as the file is being downloaded twice in serial which would cause slower downloads and heightened server memory usage. File sizes can be several hundred MB.
Is there some recommended method for doing this? Maybe something where the server is downloading from blob storage and streaming the file to the client pseudo simultaneously? So the client doesn't have to wait for the server to completely download the file to start its download, and the server can remove already transmitted file contents from memory.
To make the answer visible to others, I'm summarizing the answer shared in comment:
The suggestion is to redirect to the Blob Url directly so that the file download can start to client machine directly and the web application don't need to download it to stream or file on the server. Steps:
1.When client clicks on Download, an AJAX request comes to the server.
2.the server code performs necessary verification and returns the file URL of Azure Storage.
3.The AJAX code get the URL returned from the server and opens up a new browser window and redirects it to the URL.
I have two cloud provider with their client SDKs say SDK1 and SDK2. And I wanted to copy one file from one cloud to another cloud storage. These SDKs have upload and download APIs like this:
Response uploadAsync(Uri uploadLocation, Stream fileStream);
Stream downloadAsync(Uri downloadLocation);
Earlier I was copying downloaded Stream to MemoryStream and passing it to upload API. And it was working but obviously, it will load entire file to memory which is not good.
I cannot directly pass downloaded Stream to upload API as somewhere it's checking Length of Stream and System.Net.ConnectStream being non-seekable throws Exception.
Any pointer on how can we use the downloaded Stream (which is of Type System.Net.ConnectStream) in upload API without actually storing entire file?
I was pleasantly surprised to discover in .net 4.5 I can connect an HTTP client stream to a Zip archive and get it unzipped on the local file system.
Here is the code that does it:
System.Net.WebClient wc = new System.Net.WebClient();
string url = "http://www.example.com/abc.zip";
Stream zipReadingStream = wc.OpenRead(url);
System.IO.Compression.ZipArchive zip = new System.IO.Compression.ZipArchive(zipReadingStream);
System.IO.Compression.ZipFileExtensions.ExtractToDirectory(zip, "C:\\ExtractHere");
It works fine. But I want to keep track of the progress. I tried to check the Position property of the stream but I got Not Supported error message. Is there other way to do it?
Using ASP.Net Web API, I am developing a service which (amongst other things) retrieves data from Azure, and returns it to the client.
One way of doing this would be to read the netire blob into a buffer, and then write that buffer to the response. However, I'd rather stream the contents, for better performance.
This is simple with the Azure API:
CloudBlobContainer container = BlobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
using (var buffer = new MemoryStream())
{
await blob.DownloadToStreamAsync(buffer);
}
And elsewhere in the code, this is returned to the client:
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(buffer);
But can I be certain that the MemoryStream won't be closed/disposed before the client finishes reading?
As long as you don't wrap your memory stream in a "using" statement you will be fine. If you do use "using" you end up with a weird race condition where it works sometimes and fails at other times.
I have code like yours in production and it works fine.
Only thing to be mindful of is that the whole blob is copied into memory before anything is sent to the client. This may cause memory pressures on your server and initial lag, depending on the size of the file.
If that is a concern, you have a couple of options.
One is to create a "lease" on the blob and give the user a URL to read it direct from blob storage for a limited time. That only works for low security scenarios though.
Alternatively you can use chunked transfer encoding. Basically, you read the file from blob storage in chunks and sends it to the client in those chunks. That saves memory - but I have not been able to make it work async, so you are trading memory for threads. Which is the right solution for you will depend in your specific circumstances.
(I have not got the code to hand, post a comment if you want it and I'll try to dig it out, even if it's a bit old).
I have a fairly complex scenario that I try to port to Windows 8 from Windows Phone 7.
I need to
Download s Zip file from the internet
Unzip it to the isolated storage
Read the unzipped xml files and images
Problems
In Windows Phone 7 I use WebClient that is no longer available in Windows 8. I tried HttpClientHandler but I am only able to download the ZIP file as a string and I do no know how to save it to isolated storage.
I found ZipArchive class but it takes a IO.Stream and I am not really sure how to use it (if I had the file save somewehre - point 1)
I'm just starting out with the new API's as well (so this might be off a bit), but based on the documentation:
HttpClient (and it's default handler HttpClientHandler) return a Task<HttpResponseMessage> from SendAsync.
The HttpResponseMessage has a property, Content which is of type HttpContent.
HttpContent in turn has a method, ReadAsStreamAsync, which returns Task<Stream> which you should be able to use (albeit indirectly) to pass to ZipArchive.
Or you can just use the HttpClient.GetStreamAsync method to get the stream (much simpler):
HttpClient client = new HttpClient();
Stream stream = await client.GetStreamAsync(uri);
If that doesn't work then you could also just wrap the string you get now in a MemoryStream and pass it to ZipArchive but that sounds kind of unsafe because of possible encoding problems.