I have around 1GB of data which my client want to download from my portal as a zip file, but in the backend the files are served from AWS S3.
Currently i am downloading all files to memory stream and zipping it, which is taking a lot of time and at times it timeout and client is not sure if request is processing as I dont have download progress in the browser.
So is there a best solution to download huge data zipped from S3
Thanks
You could run some code on an Amazon EC2 instance that downloads the data from Amazon S3 (very quick if in the same region), zips it, then puts the zip back into S3.
The user can then download it directly from S3.
If you want to get fancy, they could download via a Pre-Signed URL and you could have a lifecycle rule that deletes it after a day or two.
OK, so I have little experience with S3 myself, but in general cases like this call for asynchronous processing. As in - when your user clicks "download", you initiate a background process that downloads and zips the files in some temporary location. During this, your client sees something like "preparing download, please wait". Ideally with a progress bar so that he can see that the process isn't stalled. When it's done, the download starts for real and without any timeouts since you already have the full ZIP file in a temp location.
Alternatively, see if you can streamline the whole process. Right now it sounds like you're downloading all the files to memory, creating the ZIP file in memory, and only then you're starting to output the first bytes to the client. You can do better. There are libraries out there that allow zipping "on-the-fly". While you're still downloading the files from S3 on one end, the other end is already spitting out ZIP file to the client. This way you don't need to keep everything in memory either.
Related
I want to download a file from a direct link. Those files are a between 900mb and 30GB. That's prettly large so I don't want to download them to a temp folder and then upload them. I want to use something like Azure Functions to do this every x hours and the temp storage then becomes pretty limited.
Is there a way to download / download stream and upload simultaneously to blobstorage? I don't want to save it first.
Hope you can help me out
Is there a way to download / download stream and upload simultaneously
to blobstorage? I don't want to save it first.
You don't really need to do it. Azure Storage can do this for you.
You will need to use Copy Blob functionality and provide the URL of the file you wish to transfer and Azure Storage will asynchronously copy the file into blob storage. Please do note that this is an asynchronous operation and you do not have control over when the blob gets copied.
If you want synchronous copy operation, you can take a look at Put Block From URLenter link description here operation. This is where you control how many bytes of data you want to transfer from source to blob storage.
I have an C# console app where I want to upload multiple files(roughly ~20k files in single run with each file less than 5 mb) (not multi-part) to S3 bucket . One way is to call PutObjectRequest in for each loop but i don`t think so its most effiecient way of doing so.
Is there any better way of uploading multiple files to S3?
I have all files in a local hard disk , where I have to change file name before uploading
You cannot upload multiple files in a single request, however, you can easily upload multiple files in parallel.
The simplest way would be use the Task Parallel Library
I want to download a single file in a remote Zip file that is in the cloud. The zip file is too large for me to download as a whole therefore I have decided to look for a way to download only a single file(XML) that I need within the archive. I have tried and Tested a webclient and web request but it downloads the whole zip file(also too large file for these technuques usually fails). I'm eyeing the SharpZipLib but I dont know how to use it. Is it the right library I should use or there are other available ones I can get and test. Thank you so much.
Suppose, I have a list of MP3 files on my server. And I want a user to download multiple files (which he wants through any means). For, this what i want is to create the zip file dynamically and while saving it into the Output Stream using the dotnetzip or ioniczip libraries.
Well, that's not the perfect solution if the zip file got heavy in size. As, in that scenario the server doesn't support resumable downloads. So, to overcome this approach, I need to handle the zip file structure internally and provide the resume support.
So, is there any library (open source) which i can use to provide resumable dyanamic zip files stream directly to the Output Stream. Or, if possible I will be happy if someone let me know the structure of zip file specially the header content + data content.
Once a download has started, you should not alter the ZIP file anymore. Because then a resume will just result in a broken ZIP file. So make sure your dynamically created ZIP file stays available!
The issue of providing resume-functionality was solved in this article for .NET 1.1, and it is still valid and functional.
I've got a project which requires a fairly complicated process and I want to make sure I know the best way to do this. I'm using ASP.net C# with Adobe Flex 3. The app server is Mosso (cloud server) and the file storage server is Amazon S3. The existing site can be viewed at NoiseTrade.com
I need to do this:
Allow users to upload MP3 files to
an album "widget"
After the user has uploaded their
album/widget, I need to
automatically zip the mp3 (for other
users to download) and upload the
zip along with the mp3 tracks to
Amazon S3
I actually have this working already (using client side processing in Flex) but this no longer works because of Adobe's flash 10 "security" update. So now I need to implement this server-side.
The way I am thinking of doing this is:
Store the mp3 in a temporary folder
on the app server
When the artist "publishes" create a
zip of the files in that folder
using a c# library
Start the amazon S3 upload process (zip and mp3s)
and email the user when it is
finished (as well as deleting the
temporary folder)
The major problem I see with this approach is that if a user deletes or adds a track later on I'll have to update the zip file but the temporary files will not longer exist.
I'm at a loss at the best way to do this and would appreciate any advice you might have.
Thanks!
The bit about updating the zip but not having the temporary files if the user adds or removes a track leads me to suspect that you want to build zips containing multiple tracks, possibly complete albums. If this is incorrect and you're just putting a single mp3 into each zip, then StingyJack is right and you'll probably end up making the file (slightly) larger rather than smaller by zipping it.
If my interpretation is correct, then you're in luck. Command-line zip tools frequently have flags which can be used to add files to or delete files from an existing zip archive. You have not stated which library or other method you're using to do the zipping, but I expect that it probably has this capability as well.
MP3's are compressed. Why bother zipping them?
I would say it is not necessary to zip a compressed file format, you are only gong to get a five percent reduction in filesize, give or take a little. Mp3's dont really zip up by their nature the have compressed most of the possible data already.
DotNetZip can zip up files from C#/ASP.NET. I concur with the prior posters regarding compressibility of MP3s. DotNetZip will automatically skip compression on MP3, and just store the file, just for this reason. It still may be interesting to use a zip as a packaging/archive container, aside from the compression.
If you change the zip file later (user adds a track), you could grab the .zip file from S3, and just update it. DotNetZip can update zip files, too. But in this case you would have to pay for the transfer cost into and out of S3.
DotNetZip can do all of this with in-memory handling of the zips - though that may not be feasible for large archives with lots of MP3s and lots of concurrent users.