We Keep Coding

Stream videos from Azure blob storage and ASP.NET Core 3

I'm using latest and recommended Azure.Storage.Blobs package. I'm uploading the video file as chunks, which works fine. The problem is now returning back the video to the web client, which is videojs. The player is using Range request.
My endpoint:
[HttpGet]
[Route("video/{id}")]
[AllowAnonymous]
public async Task<IActionResult> GetVideoStreamAsync(string id)
{
var stream = await GetVideoFile(id);
return File(stream, "video/mp4", true); // true is for enableRangeProcessing
}
And my GetVideoFile method
var ms = new MemoryStream();
await blobClient.DownloadToAsync(ms, null, new StorageTransferOptions
{
InitialTransferLength = 1024 * 1024,
MaximumConcurrency = 20,
MaximumTransferLength = 4 * 1024 * 1024
});
ms.Position = 0;
return ms;
The video gets downloaded and streamed just fine. But it downloads the whole video and not respecting Range at all. I've also tried with DownloadTo(HttpRange)
var ms = new MemoryStream();
// parse range header...
var range = new HttpRange(from, to);
BlobDownloadInfo info = await blobClient.DownloadAsync(range);
await info.Content.CopyToAsync(ms);
return ms;
But nothing gets displayed in the browser. What is the best way to achieve that?

Answering my own question if someone comes across.
CloudBlockBlob (version I'm using: 11.2.2) now has OpenReadAsync() method which returns stream. In my case I'm returning this stream to videojs which handles the Range header on its own.

Please try by resetting the memory stream's position to 0 before returning:
var ms = new MemoryStream();
// parse range header...
var range = new HttpRange(from, to);
BlobDownloadInfo info = await blobClient.DownloadAsync(range);
await info.Content.CopyToAsync(ms);
ms.Position = 0;//ms is positioned at the end of the stream so we need to reset that.
return ms;

I believe it's not possible to achieve it only using Azure Blob. More info in here: https://stackoverflow.com/a/26053910/1384539
but in summary, you can use a CDN that offers the Seek Start / End position : https://docs.vdms.com/cdn/re3/Content/Streaming/HPD/Seeking_Within_a_Video.htm
Another possibility is to use Azure Media Services that supports streamming. Your approach is actually a progressive download which is not exactly the same idea, and you'd probably spend a lot with network out. (assuming you have many access to the same file)

VirusTotal Uploaded File is Zero Bytes

I'm trying to upload a file to VirusTotal using .Net Core.But the uploaded file size is Zero Bytes.Why does this happen?
[Route("api/[controller]")]
public class ScannerController : Controller
{ [HttpGet]
public async Task<VirusTotalNet.Results.FileReport> ScanAsync(string file_id)
{
file_id = "./wwwroot/Upload/node-v12.14.1-x64.msi";
VirusTotal virusTotal = new VirusTotal("");
// virusTotal.UseTLS = true;
FileStream stream = System.IO.File.OpenRead(file_id);
byte[] fileBytes = new byte[stream.Length];
stream.Read(fileBytes, 0, fileBytes.Length);
VirusTotalNet.Results.FileReport report = await virusTotal.GetFileReportAsync(stream);
return report;
}
}

You've read the entire file into a byte[] and there's an overload of GetFileReportAsync that will take that, so change the parameter from stream to fileBytes:
VirusTotalNet.Results.FileReport report = await virusTotal.GetFileReportAsync(fileBytes);
Derviş Kayımbaşıoğlu suggested resetting the stream's position but it turns out that the location mentioned was incorrect. Either of these:
stream.Seek(0L, SeekOrigin.Begin);
// or
stream.Position = 0L;
Needed to be done immediately before calling GetFileReportAsync, after the file had been read, not before. That would've worked.
But wait, there's more!
There's no need to read the file into fileBytes, which means there's no need to reset the position. The stream can be opened and passed directly to GetFileReportAsync. Including proper resource disposal, the entire method becomes this:
[HttpGet]
public async Task<VirusTotalNet.Results.FileReport> ScanAsync(string file_id)
{
file_id = "./wwwroot/Upload/node-v12.14.1-x64.msi";
VirusTotal virusTotal = new VirusTotal("");
// virusTotal.UseTLS = true;
using (FileStream stream = System.IO.File.OpenRead(file_id))
{
VirusTotalNet.Results.FileReport report = await virusTotal.GetFileReportAsync(stream);
return report;
}
}
This allows both the file to be read and the socket to be written asynchronously, and the data can be buffered in small amounts so that large files don't have to be loaded entirely into memory.

.NET Core 2.0 MVC file upload progress

So I've had a hell of a time trying to display a progress bar for my .Net Core MVC app and the official documentation has not been very helpful.
Docs here: https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.0#uploading-large-files-with-streaming
I also want to upload the file to Azure blob storage when it gets to my controllers.
The user can upload as many files as they like.
Here is my code for uploading:
for (int i = 0; i < videoFile.Count; i++)
{
long totalBytes = videoFile[i].Length;
byte[] buffer = new byte[16 * 1024];
using (Stream input = videoFile[i].OpenReadStream())
{
long totalReadBytes = 0;
int readBytes;
while ((readBytes = input.Read(buffer, 0, buffer.Length)) > 0)
{
totalReadBytes += readBytes;
var progress = (int)((float)totalReadBytes / (float)totalBytes * 100.0);
}
}
String videoPath = videoFile[i].FileName;
await sc.UploadBlobAsync(groupContainer, videoPath, videoFile[i]);
}
And here is my UploadBlobAsync method:
public async Task<bool> UploadBlobAsync(string blobContainer, string blobName, IFormFile file) {
CloudBlobContainer container = await GetContainerAsync(blobContainer);
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
CancellationToken cancellationToken = new CancellationToken();
IProgress<StorageProgress> progressHandler = new Progress<StorageProgress>(
progress => Console.WriteLine("Progress: {0} bytes transferred", progress.BytesTransferred)
);
using (var filestream = file.OpenReadStream()) {
await blob.UploadFromStreamAsync(filestream,
default(AccessCondition),
default(BlobRequestOptions),
default(OperationContext),
progressHandler,
cancellationToken);
}
return true;
}
What I'd want to know is:
To my understanding, I would have to do 2 progress bars, 1 for the client machine to my server, than another from my server to azure. Is this correct?
How do I display the progress for each of my files to the frontend? I'd imagine it would be an ajax request to a List[i] I set up in my Controller?
Am I reading the bytes in the while loop when the file is already buffered? If I can access the file stream, isn't the file already buffered on the server?
How can I make use of Azure's IProgress implementation to return the result to me when it changes? Console.Writeline does not seem to work.

From another angle, you should try using the checking mechanism of a blob upload
progress, a sample is bellow:
var speedSummary = blobService.createBlockBlobFromStream('mycontainer', file.name, fileStream, file.size, function(error, result, response) {...}
setTimeout(function() {
if (!finishedOrError) {
var process = speedSummary.getCompletePercent();
displayProcess(process);
refreshProgress();
}
}, 200);
You can find more details here: https://github.com/Azure/azure-storage-node/issues/310
1- You can only use one progress bar
2- You could use the progress mechanism recently published in github based on a blob upload.

You are close, you need to provide a StorageProgress instance to your handler before calling UploadFromStreamAsync:
long byteLength = 1048576L; // 1MB at time
StorageProgress storageProgress = new StorageProgress(byteLength);
progressHandler.Report(storageProgress);

How to return a FileResult using an Amazon S3 ResponseStream?

Using the AWSSDK.S3 nuget package, I'm trying to return a file that has been retrieved from an S3 bucket. My starting point is based on the example given in this SO answer.
Example controller code:
public FileResult GetFile(Guid id)
{
// no using block as it will be disposed of by the File method
var amazonResponse = _foo.GetAmazonResponseWrapper(_user, id);
// set Response content-length header
// set Response content-type header
var bufferSize = 1024;
var buffer = new byte[bufferSize];
int bytesRead;
while ((bytesRead = amazonResponse.ResponseStream.Read(buffer, 0, buffer.Length)) > 0 && Response.IsClientConnected)
{
Response.OutputStream.Write(buffer, 0, bytesRead);
Response.OutputStream.Flush();
buffer = new byte[bufferSize];
}
// this will not work (can't read from this stream)
return File(Response.OutputStream, "text/plain", "bar.txt");
}
If I write to a MemoryStream I create and use in the while loop, I will get a file, but there won't be any content.
The only way I've found to get content in the file is to call .ToArray() on the stream like so:
return File(memStream.ToArray(), "text/plain", "foo.txt");
Is there a way to actually stream a file to the browser without loading it into the memory of the web server?

Just pass the stream forward
public FileResult GetFile(Guid id) {
// no using block as it will be disposed of by the File method
var amazonResponse = _foo.GetAmazonResponseWrapper(_user, id);
return File(amazonResponse.ResponseStream, "text/plain", "bar.txt");
}
You have already shown that you can read from the response stream. Then just pass the response stream on and the File result will read it and return the response.

Asynchronous streaming of large files using ASP.Net Framework 2.0

I am working on an ASP.NET framework 2.0 application. On a particular page I am providing a link to user. By clicking on this link a window opens with another aspx page. This page actually sends http request to a third-party url which points to a file(like - mirror urls to download file from cloud). The http response is sent back to user on the very first page using response.write from where user click the link.
Now, the problem I am facing is if the file size is low then it works fine. But, if the file is large (i.e., more than 1 GB), then my application waits until whole file is downloaded from the URL. I have tried using response.flush() to send chunk by chunk data to user, but still user is unable to use application because the worker process is busy getting streams of data from third party URL.
Is there any way by which large files can be downloaded asynchronously so that my pop-up window finishes its execution(download will be in progress) and also user can do other activities on application parallely.
Thanks,
Suvodeep

Use WebClient to read the remote file. Instead of downloading you can take the Stream from the WebClient. Put that in while() loop and push the bytes from the WebClient stream in the Response stream. On this way, you will be async downloading and uploading at the same time.
HttpRequest example:
private void WriteFileInDownloadDirectly()
{
//Create a stream for the file
Stream stream = null;
//This controls how many bytes to read at a time and send to the client
int bytesToRead = 10000;
// Buffer to read bytes in chunk size specified above
byte[] buffer = new byte[bytesToRead];
// The number of bytes read
try
{
//Create a WebRequest to get the file
HttpWebRequest fileReq = (HttpWebRequest)HttpWebRequest.Create("Remote File URL");
//Create a response for this request
HttpWebResponse fileResp = (HttpWebResponse)fileReq.GetResponse();
if (fileReq.ContentLength > 0)
fileResp.ContentLength = fileReq.ContentLength;
//Get the Stream returned from the response
stream = fileResp.GetResponseStream();
// prepare the response to the client. resp is the client Response
var resp = HttpContext.Current.Response;
//Indicate the type of data being sent
resp.ContentType = "application/octet-stream";
//Name the file
resp.AddHeader("Content-Disposition", $"attachment; filename=\"{ Path.GetFileName("Local File Path - can be fake") }\"");
resp.AddHeader("Content-Length", fileResp.ContentLength.ToString());
int length;
do
{
// Verify that the client is connected.
if (resp.IsClientConnected)
{
// Read data into the buffer.
length = stream.Read(buffer, 0, bytesToRead);
// and write it out to the response's output stream
resp.OutputStream.Write(buffer, 0, length);
// Flush the data
resp.Flush();
//Clear the buffer
buffer = new byte[bytesToRead];
}
else
{
// cancel the download if client has disconnected
length = -1;
}
} while (length > 0); //Repeat until no data is read
}
finally
{
if (stream != null)
{
//Close the input stream
stream.Close();
}
}
}
WebClient Stream reading:
using (WebClient client = new WebClient())
{
Stream largeFileStream = client.OpenRead("My Address");
}