Problem Statement:
I'm trying to iterate over a Streamed file upload in a HttpPut request using the Request.Body stream and I'm having a real hard time and my google-fu has turned up little. The situation is that I expect something like this to work and it doesn't:
[HttpPut("{accountName}/{subAccount}/{revisionId}/{randomNumber}")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status500InternalServerError)]
public async Task<IActionResult> PutTest()
{
var memStream = new MemoryStream();
var b = new Memory<byte>();
int totalBytes = 0;
int bytesRead = 0;
byte[] buffer = new byte[1024];
do
{
bytesRead = await Request.Body.ReadAsync(new Memory<byte>(buffer), CancellationToken.None);
totalBytes += bytesRead;
await memStream.WriteAsync(buffer, 0, bytesRead);
} while (bytesRead > 0);
return Ok(memStream);
}
In the debugger, I can examine the Request.Body and look at it's internal _buffer. It contains the desired data. When the above code runs, the MemoryStream is full of zeros. During "Read", the buffer is also full of zeros. The Request.Body also has a length of 0.
The Goal:
Use a HttpPut request to upload a file via streaming, iterate over it in chunks, do some processing, and stream those chunks using gRPC to another endpoint. I want to avoid reading the entire file into memory.
What I've tried:
This works:
using (var sr = new StreamReader(Request.Body))
{
var body = await sr.ReadToEndAsync();
return Ok(body);
}
That code will read all of the Stream into memory as a string which is quite undesirable, but it proves to me that the Request.Body data can be read in some fashion in the method I'm working on.
In the configure method of the Startup.cs class, I have included the following to ensure that buffering is enabled:
app.Use(async (context, next) => {
context.Request.EnableBuffering();
await next();
});
I have tried encapsulating the Request.Body in another stream like BufferedStream and FileBufferingReadStream and those don't make a difference.
I've tried:
var reader = new BinaryReader(Request.Body, Encoding.Default);
do
{
bytesRead = reader.Read(buffer, 0, buffer.Length);
await memStream.WriteAsync(buffer);
} while (bytesRead > 0);
This, as well, turns up a MemoryStream with all zeros.
I use to do this kind of request body stream a lot in my current project.
This works perfectly fine for me:
[HttpPut("{accountName}/{subAccount}/{revisionId}/{randomNumber}")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status500InternalServerError)]
public async Task<IActionResult> PutTest(CancellationToken cancel) {
using (var to = new MemoryStream()) {
var from = HttpContext.Request.Body;
var buffer = new byte[8 * 1024];
long totalBytes = 0;
int bytesRead;
while ((bytesRead = await from.ReadAsync(buffer, 0, buffer.Length, cancel)) > 0) {
await to.WriteAsync(buffer, 0, bytesRead, cancel);
totalBytes += bytesRead;
}
return Ok(to);
}
}
The only things I am doing different are:
I am creating the MemoryStream in a scoped context (using).
I am using a slightly bigger buffer (some trial and error led me to this specific size)
I am using a different overload of Stream.ReadAsync, where I pass the bytes[] buffer, the reading length and the reading start position as 0.
I am trying to get progress of an api along with the response. ResponseHeadersRead works fine to get the progress but I can't figure out why it doesn't return the response.
Download part
public async Task StartDownload()
{
_httpClient = new HttpClient { Timeout = TimeSpan.FromDays(1), };
using (var response = await _httpClient.GetAsync(_downloadUrl, HttpCompletionOption.ResponseHeadersRead))
{
await DownloadFileFromHttpResponseMessage(response);
string strResp = await response.Content.ReadAsStringAsync();
Debug.WriteLine(strResp); // Doesn't print anything
}
}
Reading Stream part
private async Task DownloadFileFromHttpResponseMessage(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
{
await ProcessContentStream(totalBytes, contentStream);
}
}
The code is actually from another answer.
I am just not getting the response. If I use ResponseContentRead I get response but it defeats the purpose of progress.
EDIT
ProcessContentStream code - This part read the response as it comes bit by bit and posts the progress in TriggerProgressChanged
private async Task ProcessContentStream(long? totalDownloadSize, Stream contentStream)
{
var totalBytesRead = 0L;
var readCount = 0L;
var buffer = new byte[8192];
var isMoreToRead = true;
using (var fileStream = new FileStream(DestinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
{
do
{
var bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
isMoreToRead = false;
//TriggerProgressChanged(totalDownloadSize, totalBytesRead);
continue;
}
await fileStream.WriteAsync(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
readCount += 1;
if (readCount % 100 == 0)
{
//TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
}
while (isMoreToRead);
}
}
Post the progress
private void TriggerProgressChanged(long? totalDownloadSize, long totalBytesRead)
{
if (ProgressChanged == null)
return;
double? progressPercentage = null;
if (totalDownloadSize.HasValue)
progressPercentage = Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2);
ProgressChanged(totalDownloadSize, totalBytesRead, progressPercentage);
}
ProgressChanged is a delegate method.
Project link
Ok so I found the solution. I needed to read the file on which I wrote the bytes as it was coming in with a StreamReader. Reading file with StreamReader has been explained here.
private async Task DownloadFileFromHttpResponseMessage(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
{
await ProcessContentStream(totalBytes, contentStream);
// Added code
char[] buffer;
using (StreamReader sr = new StreamReader(DestinationFilePath))
{
buffer = new char[(int)sr.BaseStream.Length];
await sr.ReadAsync(buffer, 0, (int)sr.BaseStream.Length);
}
Debug.WriteLine(new string(buffer)); // Now it prints the response
}
}
I'm using this method for asynchronously file copy with notifications and cancellation. It is good for copy files locally. But in case of cross drive copy its performance reduces, because at every moment only one drive works. The worst situation occurs when I copy large files from SSD to slow flash or vice versa.
Can any body advice me better solution? Maybe something based on producer-consumer pattern or there are some libraries? (I have searched, but without result)
P.S.: It is not method for direct use - it is wrapped in some others, which prepares files list and choose bufferSize
private static async Task<long> CopyFileAsync(
[NotNull]string sourcePath,
[NotNull]string destPath,
[NotNull]IProgress<FileCopyProgress> progress,
CancellationToken cancellationToken,
long bufferSize = 1024 * 1024 * 10
)
{
if (bufferSize <= 0)
{
throw new ArgumentException(nameof(bufferSize));
}
long totalRead = 0;
long fileSize;
var buffer = new byte[bufferSize];
using (var reader = File.Open(sourcePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
fileSize = reader.Length;
using (var writer = File.Create(destPath, Convert.ToInt32(bufferSize), FileOptions.Asynchronous))
{
while (totalRead < fileSize)
{
var readCount = await reader.ReadAsync(buffer, 0, Convert.ToInt32(bufferSize), cancellationToken).ConfigureAwait(false);
await writer.WriteAsync(buffer, 0, readCount, cancellationToken).ConfigureAwait(false);
totalRead += readCount;
progress.Report(new FileCopyProgress(totalRead, fileSize, null));
cancellationToken.ThrowIfCancellationRequested();
}
}
}
progress.Report(new FileCopyProgress(fileSize, fileSize, null));
return fileSize;
}
i have a file downloader function:
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(url);
InMemoryRandomAccessStream randomAccessStream = new InMemoryRandomAccessStream();
// To save downloaded image to local storage
var imageFile = await ApplicationData.Current.LocalFolder.CreateFileAsync(
filename, CreationCollisionOption.ReplaceExisting);
var fs = await imageFile.OpenAsync(FileAccessMode.ReadWrite);
DataWriter writer = new DataWriter(fs.GetOutputStreamAt(0));
writer.WriteBytes(await response.Content.ReadAsByteArrayAsync());
await writer.StoreAsync();
//current.image.SetSource(randomAccessStream);
writer.DetachStream();
await fs.FlushAsync();
How can i realize progress bar functionality?
Maybe i can get the writers bytes written so far? Or something?
P.S. I cant use DownloadOperation(Background transferring) because data from server requests certificate - and this functionality doesn't exist in DownloadOperations.
From .Net 4.5 onwards: Use IProgress<T>
Since .Net 4.5 you can handle asynchronous progress reporting with the IProgress<T> interface. You can write an extension method for downloading files using the HttpClient that can be called like this where progress is the implementation of IProgress<float> for your progress bar or other UI stuff:
// Seting up the http client used to download the data
using (var client = new HttpClient()) {
client.Timeout = TimeSpan.FromMinutes(5);
// Create a file stream to store the downloaded data.
// This really can be any type of writeable stream.
using (var file = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None)) {
// Use the custom extension method below to download the data.
// The passed progress-instance will receive the download status updates.
await client.DownloadAsync(DownloadUrl, file, progress, cancellationToken);
}
}
Implementation
The code for this extension method looks like this. Note that this extension depends on another extension for handling asynchronous stream copying with progress reporting.
public static class HttpClientExtensions
{
public static async Task DownloadAsync(this HttpClient client, string requestUri, Stream destination, IProgress<float> progress = null, CancellationToken cancellationToken = default) {
// Get the http headers first to examine the content length
using (var response = await client.GetAsync(requestUri, HttpCompletionOption.ResponseHeadersRead)) {
var contentLength = response.Content.Headers.ContentLength;
using (var download = await response.Content.ReadAsStreamAsync()) {
// Ignore progress reporting when no progress reporter was
// passed or when the content length is unknown
if (progress == null || !contentLength.HasValue) {
await download.CopyToAsync(destination);
return;
}
// Convert absolute progress (bytes downloaded) into relative progress (0% - 100%)
var relativeProgress = new Progress<long>(totalBytes => progress.Report((float)totalBytes / contentLength.Value));
// Use extension method to report progress while downloading
await download.CopyToAsync(destination, 81920, relativeProgress, cancellationToken);
progress.Report(1);
}
}
}
}
With stream extension for the real progress reporting:
public static class StreamExtensions
{
public static async Task CopyToAsync(this Stream source, Stream destination, int bufferSize, IProgress<long> progress = null, CancellationToken cancellationToken = default) {
if (source == null)
throw new ArgumentNullException(nameof(source));
if (!source.CanRead)
throw new ArgumentException("Has to be readable", nameof(source));
if (destination == null)
throw new ArgumentNullException(nameof(destination));
if (!destination.CanWrite)
throw new ArgumentException("Has to be writable", nameof(destination));
if (bufferSize < 0)
throw new ArgumentOutOfRangeException(nameof(bufferSize));
var buffer = new byte[bufferSize];
long totalBytesRead = 0;
int bytesRead;
while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) != 0) {
await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken).ConfigureAwait(false);
totalBytesRead += bytesRead;
progress?.Report(totalBytesRead);
}
}
}
Here's a self-contained class that'll do the download, and report back the progress percentage, based on code from TheBlueSky on this SO answer, and eriksendc on this GitHub comment.
public class HttpClientDownloadWithProgress : IDisposable
{
private readonly string _downloadUrl;
private readonly string _destinationFilePath;
private HttpClient _httpClient;
public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public event ProgressChangedHandler ProgressChanged;
public HttpClientDownloadWithProgress(string downloadUrl, string destinationFilePath)
{
_downloadUrl = downloadUrl;
_destinationFilePath = destinationFilePath;
}
public async Task StartDownload()
{
_httpClient = new HttpClient { Timeout = TimeSpan.FromDays(1) };
using (var response = await _httpClient.GetAsync(_downloadUrl, HttpCompletionOption.ResponseHeadersRead))
await DownloadFileFromHttpResponseMessage(response);
}
private async Task DownloadFileFromHttpResponseMessage(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
await ProcessContentStream(totalBytes, contentStream);
}
private async Task ProcessContentStream(long? totalDownloadSize, Stream contentStream)
{
var totalBytesRead = 0L;
var readCount = 0L;
var buffer = new byte[8192];
var isMoreToRead = true;
using (var fileStream = new FileStream(_destinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
{
do
{
var bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
isMoreToRead = false;
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
continue;
}
await fileStream.WriteAsync(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
readCount += 1;
if (readCount % 100 == 0)
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
while (isMoreToRead);
}
}
private void TriggerProgressChanged(long? totalDownloadSize, long totalBytesRead)
{
if (ProgressChanged == null)
return;
double? progressPercentage = null;
if (totalDownloadSize.HasValue)
progressPercentage = Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2);
ProgressChanged(totalDownloadSize, totalBytesRead, progressPercentage);
}
public void Dispose()
{
_httpClient?.Dispose();
}
}
Usage:
var downloadFileUrl = "http://example.com/file.zip";
var destinationFilePath = Path.GetFullPath("file.zip");
using (var client = new HttpClientDownloadWithProgress(downloadFileUrl, destinationFilePath))
{
client.ProgressChanged += (totalFileSize, totalBytesDownloaded, progressPercentage) => {
Console.WriteLine($"{progressPercentage}% ({totalBytesDownloaded}/{totalFileSize})");
};
await client.StartDownload();
}
Result:
7.81% (26722304/342028776)
8.05% (27535016/342028776)
8.28% (28307984/342028776)
8.5% (29086548/342028776)
8.74% (29898692/342028776)
8.98% (30704184/342028776)
9.22% (31522816/342028776)
The best way to go is using Windows.Web.Http.HttpClient instead of System.Net.Http.HttpClient. The first one supports progress.
But if for some reason you want to stick to the System.Net one, you will need to implement your own progress.
Remove the DataWriter, remove the InMemoryRandomAccessStream and add HttpCompletionOption.ResponseHeadersRead to GetAsync call so it returns as soon as headers are received, not when the whole response is received. I.e.:
// Your original code.
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(
url,
HttpCompletionOption.ResponseHeadersRead); // Important! ResponseHeadersRead.
// To save downloaded image to local storage
var imageFile = await ApplicationData.Current.LocalFolder.CreateFileAsync(
filename,
CreationCollisionOption.ReplaceExisting);
var fs = await imageFile.OpenAsync(FileAccessMode.ReadWrite);
// New code.
Stream stream = await response.Content.ReadAsStreamAsync();
IInputStream inputStream = stream.AsInputStream();
ulong totalBytesRead = 0;
while (true)
{
// Read from the web.
IBuffer buffer = new Windows.Storage.Streams.Buffer(1024);
buffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.None);
if (buffer.Length == 0)
{
// There is nothing else to read.
break;
}
// Report progress.
totalBytesRead += buffer.Length;
System.Diagnostics.Debug.WriteLine("Bytes read: {0}", totalBytesRead);
// Write to file.
await fs.WriteAsync(buffer);
}
inputStream.Dispose();
fs.Dispose();
The simplest way to implement progress tracking for both uploading and downloading is to use ProgressMessageHandler from the Microsoft.AspNet.WebApi.Client nuget package.
Note: this library was originally named System.Net.Http.Formatting, and was renamed to Microsoft.AspNet.WebApi.Client. However, this library is not related to ASP.Net and can be used by any project looking for official Microsoft extensions to HttpClient. The source code is available here.
Example:
var handler = new HttpClientHandler() { AllowAutoRedirect = true };
var ph = new ProgressMessageHandler(handler);
ph.HttpSendProgress += (_, args) =>
{
Console.WriteLine($"upload progress: {(double)args.BytesTransferred / args.TotalBytes}");
};
ph.HttpReceiveProgress += (_, args) =>
{
Console.WriteLine($"download progress: {(double)args.BytesTransferred / args.TotalBytes}");
};
var client = new HttpClient(ph);
await client.SendAsync(...);
Note that this will not report progress if uploading a byte array. The request message content must be a stream.
The following code shows a minimal example of what must be done against the HttpClient api to get download progress.
HttpClient client = //...
// Must use ResponseHeadersRead to avoid buffering of the content
using (var response = await client.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead)){
// You must use as stream to have control over buffering and number of bytes read/received
using (var stream = await response.Content.ReadAsStreamAsync())
{
// Read/process bytes from stream as appropriate
// Calculated by you based on how many bytes you have read. Likely incremented within a loop.
long bytesRecieved = //...
long? totalBytes = response.Content.Headers.ContentLength;
double? percentComplete = (double)bytesRecieved / totalBytes;
// Do what you want with `percentComplete`
}
}
The above does not tell you how to process the stream, how to report the process, or try to provide a direct solution to the code in the original question. However, this answer may be more accessible to future readers who wish to apply having progress to in their code.
same as #René Sackers solution above, but added the ability to cancel the download
class HttpClientDownloadWithProgress : IDisposable
{
private readonly string _downloadUrl;
private readonly string _destinationFilePath;
private readonly CancellationToken? _cancellationToken;
private HttpClient _httpClient;
public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public event ProgressChangedHandler ProgressChanged;
public HttpClientDownloadWithProgress(string downloadUrl, string destinationFilePath, CancellationToken? cancellationToken = null)
{
_downloadUrl = downloadUrl;
_destinationFilePath = destinationFilePath;
_cancellationToken = cancellationToken;
}
public async Task StartDownload()
{
_httpClient = new HttpClient { Timeout = TimeSpan.FromDays(1) };
using (var response = await _httpClient.GetAsync(_downloadUrl, HttpCompletionOption.ResponseHeadersRead))
await DownloadFileFromHttpResponseMessage(response);
}
private async Task DownloadFileFromHttpResponseMessage(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
await ProcessContentStream(totalBytes, contentStream);
}
private async Task ProcessContentStream(long? totalDownloadSize, Stream contentStream)
{
var totalBytesRead = 0L;
var readCount = 0L;
var buffer = new byte[8192];
var isMoreToRead = true;
using (var fileStream = new FileStream(_destinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
{
do
{
int bytesRead;
if (_cancellationToken.HasValue)
{
bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length, _cancellationToken.Value);
}
else
{
bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
}
if (bytesRead == 0)
{
isMoreToRead = false;
continue;
}
await fileStream.WriteAsync(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
readCount += 1;
if (readCount % 10 == 0)
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
while (isMoreToRead);
}
//the last progress trigger should occur after the file handle has been released or you may get file locked error
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
private void TriggerProgressChanged(long? totalDownloadSize, long totalBytesRead)
{
if (ProgressChanged == null)
return;
double? progressPercentage = null;
if (totalDownloadSize.HasValue)
progressPercentage = Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2);
ProgressChanged(totalDownloadSize, totalBytesRead, progressPercentage);
}
public void Dispose()
{
_httpClient?.Dispose();
}
}
René Sackers version is excellent but it could be better. Specifically, it has a subtle race condition caused by TriggerProgressChanged firing before the stream closes. The fix is to fire the event after the stream is explicitly disposed. The version below includes the above change, inherits from HttpClient and adds support for cancellation tokens.
public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public class HttpClientWithProgress : HttpClient
{
private readonly string _DownloadUrl;
private readonly string _DestinationFilePath;
public event ProgressChangedHandler ProgressChanged;
public HttpClientWithProgress(string downloadUrl, string destinationFilePath)
{
_DownloadUrl = downloadUrl;
_DestinationFilePath = destinationFilePath;
}
public async Task StartDownload()
{
using (var response = await GetAsync(_DownloadUrl, HttpCompletionOption.ResponseHeadersRead))
await DownloadFileFromHttpResponseMessage(response);
}
public async Task StartDownload(CancellationToken cancellationToken)
{
using (var response = await GetAsync(_DownloadUrl, HttpCompletionOption.ResponseHeadersRead, cancellationToken))
await DownloadFileFromHttpResponseMessage(response);
}
private async Task DownloadFileFromHttpResponseMessage(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
long? totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
await ProcessContentStream(totalBytes, contentStream);
}
private async Task ProcessContentStream(long? totalDownloadSize, Stream contentStream)
{
long totalBytesRead = 0L;
long readCount = 0L;
byte[] buffer = new byte[8192];
bool isMoreToRead = true;
using (FileStream fileStream = new FileStream(_DestinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
{
do
{
int bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
isMoreToRead = false;
continue;
}
await fileStream.WriteAsync(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
readCount += 1;
if (readCount % 10 == 0)
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
while (isMoreToRead);
}
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
}
private void TriggerProgressChanged(long? totalDownloadSize, long totalBytesRead)
{
if (ProgressChanged == null)
return;
double? progressPercentage = null;
if (totalDownloadSize.HasValue)
progressPercentage = Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2);
ProgressChanged(totalDownloadSize, totalBytesRead, progressPercentage);
}
}
This is my variation on the answer of René Sackers. Main differences:
A more functional style.
Only one method instead of a whole object.
Can cancel the download
public async static Task Download(
string downloadUrl,
string destinationFilePath,
Func<long?, long, double?, bool> progressChanged)
{
using var httpClient = new HttpClient { Timeout = TimeSpan.FromDays(1) };
using var response = await httpClient.GetAsync(downloadUrl, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using var contentStream = await response.Content.ReadAsStreamAsync();
var totalBytesRead = 0L;
var readCount = 0L;
var buffer = new byte[8192];
var isMoreToRead = true;
static double? calculatePercentage(long? totalDownloadSize, long totalBytesRead) => totalDownloadSize.HasValue ? Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2) : null;
using var fileStream = new FileStream(destinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true);
do
{
var bytesRead = await contentStream.ReadAsync(buffer);
if (bytesRead == 0)
{
isMoreToRead = false;
if (progressChanged(totalBytes, totalBytesRead, calculatePercentage(totalBytes, totalBytesRead)))
{
throw new OperationCanceledException();
}
continue;
}
await fileStream.WriteAsync(buffer.AsMemory(0, bytesRead));
totalBytesRead += bytesRead;
readCount++;
if (readCount % 100 == 0)
{
if (progressChanged(totalBytes, totalBytesRead, calculatePercentage(totalBytes, totalBytesRead)))
{
throw new OperationCanceledException();
}
}
}
while (isMoreToRead);
}
It can be called this way:
// Change this variable to stop the download
// You can use a global variable or some kind of state management
var mustStop = false;
var downloadProgress = (long? _, long __, double? progressPercentage) =>
{
if (progressPercentage.HasValue)
progressBar.Value = progressPercentage.Value;
// In this example only the variable is checked
// You could write other code that evaluates other conditions
return mustStop;
};
SomeClass.Download("https://example.com/bigfile.zip", "c:\downloads\file.zip", downloadProgress);
Hm, you could have another thread check the current size of the stream being written (you'd also pass the expected file size to it) and then update the progress bar accordingly.
This is a modified version of René Sackers answer with the following functional changes:
http client not disposed (because it should not be disposed)
better progress handling
callback to create httpRequest (custom header support)
utilizes ArrayPool to reduce memory footprint
automatic event subscribe+unsubscribe to prevent memory leaks by event handlers
You can also use this nuget package https://www.nuget.org/packages/Amusoft.Toolkit.Http to gain all benefits. Since it supports net462 and above that is probably the easiest way.
Usage:
await DownloadWithProgress.ExecuteAsync(HttpClients.General, assetUrl, downloadFilePath, progressHandler, () =>
{
var requestMessage = new HttpRequestMessage(HttpMethod.Get, assetUrl);
requestMessage.Headers.Accept.TryParseAdd("application/octet-stream");
return requestMessage;
});
I guess i am not the only one who needs custom headers so i figured i would share this rewrite
Implementation:
public delegate void DownloadProgressHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public static class DownloadWithProgress
{
public static async Task ExecuteAsync(HttpClient httpClient, string downloadPath, string destinationPath, DownloadProgressHandler progress, Func<HttpRequestMessage> requestMessageBuilder = null)
{
requestMessageBuilder ??= GetDefaultRequestBuilder(downloadPath);
var download = new HttpClientDownloadWithProgress(httpClient, destinationPath, requestMessageBuilder);
download.ProgressChanged += progress;
await download.StartDownload();
download.ProgressChanged -= progress;
}
private static Func<HttpRequestMessage> GetDefaultRequestBuilder(string downloadPath)
{
return () => new HttpRequestMessage(HttpMethod.Get, downloadPath);
}
}
internal class HttpClientDownloadWithProgress
{
private readonly HttpClient _httpClient;
private readonly string _destinationFilePath;
private readonly Func<HttpRequestMessage> _requestMessageBuilder;
private int _bufferSize = 8192;
public event DownloadProgressHandler ProgressChanged;
public HttpClientDownloadWithProgress(HttpClient httpClient, string destinationFilePath, Func<HttpRequestMessage> requestMessageBuilder)
{
_httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
_destinationFilePath = destinationFilePath ?? throw new ArgumentNullException(nameof(destinationFilePath));
_requestMessageBuilder = requestMessageBuilder ?? throw new ArgumentNullException(nameof(requestMessageBuilder));
}
public async Task StartDownload()
{
using var requestMessage = _requestMessageBuilder.Invoke();
using var response = await _httpClient.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead);
await DownloadAsync(response);
}
private async Task DownloadAsync(HttpResponseMessage response)
{
response.EnsureSuccessStatusCode();
var totalBytes = response.Content.Headers.ContentLength;
using (var contentStream = await response.Content.ReadAsStreamAsync())
await ProcessContentStream(totalBytes, contentStream);
}
private async Task ProcessContentStream(long? totalDownloadSize, Stream contentStream)
{
var totalBytesRead = 0L;
var readCount = 0L;
var buffer = ArrayPool<byte>.Shared.Rent(_bufferSize);
var isMoreToRead = true;
using (var fileStream = new FileStream(_destinationFilePath, FileMode.Create, FileAccess.Write, FileShare.None, _bufferSize, true))
{
do
{
var bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
isMoreToRead = false;
ReportProgress(totalDownloadSize, totalBytesRead);
continue;
}
await fileStream.WriteAsync(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
readCount += 1;
if (readCount % 100 == 0)
ReportProgress(totalDownloadSize, totalBytesRead);
}
while (isMoreToRead);
}
ArrayPool<byte>.Shared.Return(buffer);
}
private void ReportProgress(long? totalDownloadSize, long totalBytesRead)
{
double? progressPercentage = null;
if (totalDownloadSize.HasValue)
progressPercentage = Math.Round((double)totalBytesRead / totalDownloadSize.Value * 100, 2);
ProgressChanged?.Invoke(totalDownloadSize, totalBytesRead, progressPercentage);
}
}
Im not really sure how to measure how the completion logic, but for now this seems to do it.
public event ProgressChangedHandler ProgressChanged;
public event ProgressCompleteHandler DownloadComplete;
...
TriggerProgressChanged(totalDownloadSize, totalBytesRead);
TriggerDownloadComplete(totalBytesRead == totalDownloadSize);
private void TriggerDownloadComplete(bool status)
{
DownloadComplete(status);
}
client.DownloadComplete += (status) =>
{
if (status)
{
// success
}
};
I am trying to create an asynchronous TPL file server using sockets and NetworkStream. When testing it, my browser small HTML file (1.9 KB) sends just fine, and sometimes even Javascript or CSS files that it links send to, but it won't download much more from the HTML page, including flash, images, etc. I receive no errors, including no connection errors. I can download a 96K image but that's about the limit. I set Connection: Keep-Alive in all response headers.
Does anyone know why my output streaming seems to be stalling?
async Task<> WriteToStream(NetworkStream _networkStream, string filePath, int startingPoint = 0)
{
using (FileStream sourceStream = new FileStream(filePath,
FileMode.Open, FileAccess.Read, FileShare.Read,
bufferSize: 4096, useAsync: true))
{
byte[] buffer = new byte[4096];
int numRead;
while ((numRead = await sourceStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
_networkStream.Write(buffer, 0, numRead);
}
}
}
I also tried replacing this:
_networkStream.Write(buffer, 0, numRead);
with this:
await _networkStream.WriteAsync(buffer, 0, numRead);
and I still have the same problem.
I'm using sockets because I can't use HttpListener or TcpListener classes since I need to access incoming UDP and TCP requests.
I can call WriteToStream() with this simplified method:
private async void SendFileExample()
{
//This method is only for demonstration, so parameters are hardcoded.
// Get info and assemble header
string file = #"C:\www\webpage.html";
byte[] data = null;
string responseCode = "200 OK";
string contentType = "text/html";
long dataLength = 1901;
string serverName = "my Stack Overflow server is overflowing with...";
string header = string.Format("HTTP/1.1 {0}\r\n"
+ "Server: {1}\r\n"
+ "Content-Length: {2}\r\n"
+ "Content-Type: {3}\r\n"
+ "Connection: Keep-Alive\r\n"
+ "\r\n",
responseCode, serverName, dataLength, contentType);
var headerBytes = Encoding.ASCII.GetBytes(header);
//send header
await _networkStream.WriteAsync(headerBytes, 0, headerBytes.Length);
//send payload
await WriteToStream(_networkStream, file, 0);
//flush networkstream
await _networkStream.FlushAsync();
}
EDIT:
Here what calls the listen loop:
_listenTask = Task.Factory.StartNew(() => ListenLoop());
Here is the loop that spools the requests, spawning a client each time:
private async void ListenLoop()
{
for (; ; )
{
// Wait for connection
var socket = await _tcpListener.AcceptSocketAsync();
if (socket == null)
break;
// Got new connection, create a client handler for it
var client = new Client(socket,dbInfo,frmClient);
// Create a task to handle new connection
Task.Factory.StartNew(client.Do);
}
}
Connections are handled by this method:
public async void Do()
{
byte[] buffer = new byte[4096];
for (; ; )
{
// Read a chunk of data
int bytesRead = await _networkStream.ReadAsync(buffer, 0, buffer.Length);
// If Read returns with no data then the connection is closed.
if (bytesRead == 0)
return;
// Write to buffer and process request
_memoryStream.Seek(0, SeekOrigin.End);
_memoryStream.Write(buffer, 0, bytesRead);
bool done = ProcessHeader();
if (done)
break;
}
}
ProcessHeader() mostly just gets meta data like MIME types then passes the stream to the WriteToStream() method at the top of this post.