I am writing a class to handle file downloads and i am using this code [simplified]:
var webRequest = (HttpWebRequest)WebRequest.Create(downloadOperation.Link);
webRequest.Proxy = null;
using (var webResponse = await webRequest.GetResponseAsync())
{
using (var downloadStream = webResponse.GetResponseStream())
{
using (var outputFileWriteStream = await outputFile.OpenStreamForWriteAsync())
{
var buffer = new byte[4096];
var downloadedBytes = 0;
var totalBytes = webResponse.ContentLength;
while (downloadedBytes < totalBytes)
{
//*************************THIS LINE TAKES ABOUT 32 SECONDS TO EXECUTE ON FIRST INVOKE, ALL NEXT INVOKES TAKE ABOUT 120MS***************************
var currentRead = await downloadStream.ReadAsync(buffer, 0, buffer.Length);
//*******************************************************************************************************************************************************************
await outputFileWriteStream.WriteAsync(buffer, 0, currentRead);
}
}
}
}
Can you please explain to me why is it taking that long on first invoke and not on the next ones? I am worried that it is downloading the entire file on the first read.
Note that the files are usually between 3~15MB.
I am worried that it is downloading the entire file on the first read.
That's precisely what's happening. You can change that by setting webRequest.AllowReadStreamBuffering to false.
So i found a way to fix this problem, but it doesn't use WebRequest class.
I am now using the HttpClient found in (Windows.Web.Http).
Here is the fixed code:
var client = new Windows.Web.Http.HttpClient(); // prepare the http client
//get the response from the server
using (var webResponse = await client.GetAsync(downloadOperation.Link, HttpCompletionOption.ResponseHeadersRead)) //***********Node the HttpCompletionOption.ResponseHeaderRead, this means that the operation completes as soon as the client receives the http headers instead of waiting for the entire response content to be read
{
using (var downloadStream = (await webResponse.Content.ReadAsInputStreamAsync()).AsStreamForRead() )
{
using (var outputFileWriteStream = await outputFile.OpenStreamForWriteAsync())
{
var buffer = new byte[4096];
var downloadedBytes = 0;
var totalBytes = webResponse.ContentLength;
while (downloadedBytes < totalBytes)
{
//*************************THIS LINE NO LONGER TAKES A LONG TIME TO PERFORM FIRST READ***************************
var currentRead = await downloadStream.ReadAsync(buffer, 0, buffer.Length);
//*******************************************************************************************************************************************************************
await outputFileWriteStream.WriteAsync(buffer, 0, currentRead);
}
}
}
}
Hope this will help someone out there ;)
thank you all
Related
I've written a code that set a downloaded HTML-document into the variable. But i don't need all the HTML-document, only first 200 bytes of it. I need to cancel the method 'System.Net.WebClient.DownloadString' when it saves the document enough.
try
{
WebClient webClient = new WebClient();
html = webClient.DownloadString("https://example.com/index.html");
} catch(Exception e) {
MessageBox.Show(e.Message);
}
Try the sample given below.It uses the more modern HttpClient instead of a WebClient. I'm not sure if it actually limits the amount of bytes to 200 (see also How to retrieve partial response with System.Net.HttpClient) but you can give it a try.
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
static async Task Main()
{
var client = new HttpClient();
using (var response = await client.GetAsync("https://example.com/index.html"))
using (var stream = await response.Content.ReadAsStreamAsync())
{
var buffer = new byte[200];
var count = await stream.ReadAsync(buffer, 0, buffer.Length);
var result = Encoding.UTF8.GetString(buffer);
}
}
}
}
You can't read first N chars using WebClient since it reads response till the end.
Assuming that for some reason you can't use HttpClient, use WebReposense and GetResponseStream in particular to read a part of response.
Note, that "first N bytes" != "first N chars". You need to try to convert bytes to string using appropriate encoding, and use string only if conversion was successful.
As an option:
public async Task<string> GetPartialResponseAsync(string url, int length)
{
var request = System.Net.WebRequest.Create(url);
request.Method = "GET";
using (var response = await request.GetResponseAsync())
using (var responseStream = response.GetResponseStream())
{
byte[] buffer = new byte[length];
await responseStream.ReadAsync(buffer, 0, length);
return System.Text.Encoding.Default.GetString(buffer);
}
}
Description: I am modifying the ASP.NET Core Web API service (hosted in Windows Service) that supports resumable file uploads. This works fine and resumes file uploads in many failure conditions except one described below.
Problem: When the service is on ther other computer and the client is on mine and I unplug the cable on my computer, the client detects the absence of network while the service hangs on fileSection.FileStream.Read(). Sometimes the service detects the failure in 8 min, sometimes in 20, sometimes never.
I also noticed that after I unplug cable and stop the client, the service becomes stuck at Read() function and the file size is x KB, but when the service finally detects the exception some time later, it writes additional 4 KB to the file. This is weird because I turned off buffering and the buffer size is 2 KB.
Question: How to properly detect the absence of network on the service, or timeout properly, or cancel the request
The service code:
public static async Task<List<(Guid, string)>> StreamFileAsync(
this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = fileSection.FileStream.Read(buffer, 0, buffer.Length)) > 0) // HANGS HERE
{
outfile.Write(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
}
}
catch (Exception e)
{
...
}
}
}
The client code:
var request = CreateRequest(fileTransferId, boundary, header, footer, filePath, offset, headers, null);
using (Stream formDataStream = request.GetRequestStream())
{
formDataStream.ReadTimeout = 60000;
formDataStream.Write(Encoding.UTF8.GetBytes(header), 0, header.Length);
byte[] buffer = new byte[2048];
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
fs.Seek(offset, SeekOrigin.Begin);
for (int i = 0; i < fs.Length - offset;)
{
int k = await fs.ReadAsync(buffer, 0, buffer.Length);
if (k > 0)
{
await Task.Delay(100);
await formDataStream.WriteAsync(buffer, 0, k);
}
i = i + k;
}
}
formDataStream.Write(footer, 0, footer.Length);
}
var uploadingResult = request.GetResponse() as HttpWebResponse;
private static HttpWebRequest CreateRequest(
Guid fileTransferId,
string boundary,
string header,
byte[] footer,
string filePath,
long offset,
NameValueCollection headers,
Dictionary<string, string> postParameters)
{
var url = $"{_BaseAddress}v1/ResumableUpload?fileTransferId={fileTransferId}";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "multipart/form-data; boundary=\"" + boundary + "\"";
request.UserAgent = "Agent 1.0";
request.Headers.Add(headers); // custom headers
request.Timeout = 120000;
request.KeepAlive = true;
request.AllowReadStreamBuffering = false;
request.ReadWriteTimeout = 120000;
request.AllowWriteStreamBuffering = false;
request.ContentLength = CalculateContentLength(filePath, offset, header, footer, postParameters, boundary);
return request;
}
What I tried:
I added these in into config files:
Tried to set timeout on the server
var host = new WebHostBuilder().UseKestrel(o => { o.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(2);})
Used async and non-async Read()
Tried with keep alive and without
Tried to abort the request when network was restored: request?.Abort();
Tried to set formDataStream.ReadTimeout = 60000;
Since I did not find a better way, I decided to add a timeout to the reading stream and saving it to the file. The good example was posted here: https://blogs.msdn.microsoft.com/pfxteam/2012/10/05/how-do-i-cancel-non-cancelable-async-operations/
public static async Task<List<(Guid, string)>> StreamFileAsync(this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
await SaveMyFile(...);
}
catch (OperationCanceledException){...}
catch (Exception){...}
}
}
private static async Task SaveMyFile(...)
{
var cts = CancellationTokenSource.CreateLinkedTokenSource(myOtherCancellationToken);
cts.CancelAfter(streamReadTimeoutInMs);
var myReadTask = StreamFile(transferInfo, fileSection, cts.Token);
await ExecuteMyTaskWithCancellation(myReadTask, cts.Token);
}
private static async Task<T> ExecuteMyTaskWithCancellation<T>(Task<T> task, CancellationToken cancellationToken)
{
var tcs = new TaskCompletionSource<bool>();
using (cancellationToken.Register(s => ((TaskCompletionSource<bool>) s).TrySetResult(true), tcs))
{
if (task != await Task.WhenAny(task, tcs.Task))
{
throw new OperationCanceledException(cancellationToken);
}
}
return await task;
}
private static async Task<bool> StreamFile(...)
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = await fileSection.FileStream.ReadAsync(buffer, 0, buffer.Length, token)) > 0)
{
if (token.IsCancellationRequested)
{
break;
}
await outfile.WriteAsync(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
return true;
}
}
My HttpClient uses digest authentication to connect to the server and expects search queries in response. These search queries can come in any time so the client is expected to leave the connection open at all times.
The connection is made using the following code:
public static async void ListenForSearchQueries(int resourceId)
{
var url = $"xxx/yyy/{resourceId}/waitForSearchRequest?token=abc";
var httpHandler = new HttpClientHandler { PreAuthenticate = true };
using (var digestAuthMessageHandler = new DigestAuthMessageHandler(httpHandler, "user", "password"))
using (var client = new HttpClient(digestAuthMessageHandler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
var tokenSource = new CancellationTokenSource();
tokenSource.CancelAfter(TimeSpan.FromMilliseconds(Timeout.Infinite));
using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, tokenSource.Token))
{
Console.WriteLine("\nResponse code: " + response.StatusCode);
using (var body = await response.Content.ReadAsStreamAsync())
using (var reader = new StreamReader(body))
while (!reader.EndOfStream)
Console.WriteLine(reader.ReadLine());
}
}
}
This is how the method is being used in the Main method of a console application.
private static void Main(string[] args)
{
const int serviceId = 128;
.
.
.
ListenForSearchQueries(resourceId);
Console.ReadKey();
}
This is what the output on the console window looks like:
Response code: OK
--searchRequestBoundary
Even though the timeout for the client is set to infinity, the connection times out after roughly five minutes (which is not the default timeout of the HttpClient) after the first output, throwing the following exception.
System.IO.IOException occurred
HResult=0x80131620
Message=The read operation failed, see inner exception.
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Http.DelegatingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.get_EndOfStream()
at ConsoleTester.Program.<ListenSearchQueriesDigestAuthMessageHandler>d__10.MoveNext() in C:\Users\xyz\ProjName\ConsoleTester\Program.cs:line 270
Inner Exception 1:
WebException: The operation has timed out.
The DelegateHandler used for the authentication is a a rough adaption of this code (see the source section).
Why is the client timing out and how can I prevent this?
My ultimate goal is to make a call and wait indefinitely for a response. When a response does come, I don't want the connection to close because more responses might come in the future. Unfortunately, I can't change anything at the server end.
Although the default value for Stream.CanTimeout is false, returning a stream via the response.Content.ReadAsStreamAsync() gives a stream where the CanTimeout property returns true.
The default read and write time out for this stream is 5 minutes. That is after five minutes of inactivity, the stream will throw an exception. Much similar to the exception shown in the question.
To change this behavior, ReadTimeout and/or the WriteTimeout property of the stream can be adjusted.
Below is the modified version of the ListenForSearchQueries method that changes the ReadTimeout to Infinite.
public static async void ListenForSearchQueries(int resourceId)
{
var url = $"xxx/yyy/{resourceId}/waitForSearchRequest?token=abc";
var httpHandler = new HttpClientHandler { PreAuthenticate = true };
using (var digestAuthMessageHandler = new DigestAuthMessageHandler(httpHandler, "user", "password"))
using (var client = new HttpClient(digestAuthMessageHandler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
var tokenSource = new CancellationTokenSource();
tokenSource.CancelAfter(TimeSpan.FromMilliseconds(Timeout.Infinite));
using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, tokenSource.Token))
{
Console.WriteLine("\nResponse code: " + response.StatusCode);
using (var body = await response.Content.ReadAsStreamAsync())
{
body.ReadTimeout = Timeout.Infinite;
using (var reader = new StreamReader(body))
while (!reader.EndOfStream)
Console.WriteLine(reader.ReadLine());
}
}
}
}
This fixed the exception which was actually being thrown by the stream but seemed like was being thrown by the HttpClient.
Make the method return a Task
public static async Task ListenForSearchQueries(int resourceId) {
//...code removed for brevity
}
Update the console's main method to Wait on the Task to complete.
public static void Main(string[] args) {
const int serviceId = 128;
.
.
.
ListenForSearchQueries(resourceId).Wait();
Console.ReadKey();
}
I solved this problem in the following way:
var stream = await response.Content.ReadAsStreamAsync();
while (b == 1)
{
var bytes = new byte[1];
try
{
var bytesread = await stream.ReadAsync(bytes, 0, 1);
if (bytesread > 0)
{
text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
using (System.IO.StreamWriter escritor = new System.IO.StreamWriter(#"C:\orden\ConSegu.txt", true))
{
if (ctext == 100)
{
escritor.WriteLine(text);
ctext = 0;
}
escritor.Write(text);
}
}
}
catch (Exception ex)
{
Console.WriteLine("error");
Console.WriteLine(ex.Message);
}
}
in this way I get byte to byte the answer and I save it in a txt
later I read the txt and I'm erasing it again. for the moment it is the solution I found to receive the notifications sent to me by the server from the persistent HTTP connection.
My code looks like:
_req = new HttpRequestMessage(HttpMethod.Get, _uri);
_response = await _httpClient.SendAsync(_req, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
var rawResponse = new byte[_maxContentLength];
using (var responseStream = await _response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
int read;
int offset = 0;
do
{
read = await responseStream.ReadAsync(rawResponse, 0, rawResponse.Length).ConfigureAwait(false);
html += Encoding.UTF8.GetString(rawResponse, 0, read);
offset += read;
if (offset > _maxContentLength)
{
return Error("Too big");
}
} while (read != 0);
}
and is "return Error()" enough to free stream and sockets to get next url?
//edit:
For some pages await responseStream.ReadAsync hangs and never returns
Try creating a CancellationTokenSource and calling Cancel on that when you want to stop. I'm assuming that you can't read the Content-Length header at the beginning because the response is chunk encoded?
I am taking a string message and breaking it up into chunks so that I can send it to an sms service (that consequently doesn't break it up for you). I after I do my work for break those messages up, I try loop through the resulting array and execute a web request. The problem is that it only works for the first message, and then hangs after that. After a short time, I get an error message saying "The connection was closed unexpectedly." This occurs at the second time it attempts GetResponse(); I've seen a few other posts on here that were simply saying to close and dispose the response and request streams. This isn't working for me at all. This is where my code is currently:
private static void Main(string[] args)
{
var oldMessage = GetFileString();
Console.WriteLine(string.Format("Old message: {0}", oldMessage.Length));
var newMessage = UrlPathEncodeString(oldMessage);
Console.WriteLine(string.Format("New message: {0}", newMessage.Length));
var brokenUp = SplitByLength(newMessage, 145).ToArray();
for(var i = 0; i < brokenUp.Count(); i++)
{
brokenUp[i] = brokenUp[i].Insert(0, UrlPathEncodeString(string.Format("({0:D2} of {1:D2})", i + 1, brokenUp.Count())));
Console.WriteLine(string.Format("Appended length: {0}", brokenUp[i].Length));
}
System.Net.ServicePointManager.DefaultConnectionLimit = 100;
foreach (var block in brokenUp)
{
Thread.Sleep(1500);
SendSms((HttpWebRequest)WebRequest.Create("http://172.20.5.214:90/method/sendsms"), block);
}
Console.ReadKey();
}
public static void SendSms(HttpWebRequest request, string message)
{
//build the request
var url = "http://ipaddress/method/sendsms";
//var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var fields = "CellNumber={0}&Message={1}";
fields = string.Format(fields, "16021234567", message);
var fieldsBytes = Encoding.UTF8.GetBytes(fields);
request.ContentLength = fieldsBytes.Length;
var length = fieldsBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
responseStream.Close();
}
}
requestStream.Close();
}
}
public static byte[] ReadFully(Stream stream)
{
var buffer = new byte[32768];
using (var ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
I've run into cases where response.Close appears to hang if you fail to download the entire contents of the response. Why this is, I don't know, but placing a call to request.Abort before calling Close() solves the problem. I wouldn't expect you to be seeing this problem, though, unless the response could potentially be many megabytes in size.
Also, failing to close the request stream before calling GetResponse might prevent all of the data from being sent. I would suggest calling requestStream.Close before making the request. But again, it seems odd that your code would work the first time but not on subsequent requests.
Your modified code, taking into account the changes I suggested, would be:
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
}
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
// read the response here.
request.Abort();
responseStream.Close();
}
}