The following code gets a Stream from a URI and will read in in chunks using a loop. Note that behind the specified URI is an online radio stream, which means there is no known size.
var uri = new Uri("http://*******", UriKind.Absolute);
var http = new HttpClient();
var stream = await http.GetStreamAsync(uri);
var buffer = new byte[65536];
while (true)
{
var read = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
Debug.WriteLine("Read: {0}", read);
}
Now while this works perfectly fine in a .NET 4.5 console app, this exactly same code does not work as expected in WinRT - it will read the first chunk, and when calling ReadAsync for the second time, it will just get stuck and never continue.
If I switch the URI to a file (of known size) everything works fine in both projects.
Any tips?
EDIT> note that this behaviour happens only on WP8.1. I just searched some more on SO and found that my question might be a duplicate of this one: WP8.1 HttpClient Stream got only 65536 bytes data If that is true, I will close my question
Use HttpClient.GetAsync() with HttpCompletionOption.ResponseHeadersRead. That returns when the headers are received, then do HttpResponse.Content.ReadAsInputStreamAsync().
Looks like your while loop is infinite. How are you ensuring that it's finite? Instead of while(true) make it something like this
var uri = new Uri("http://*******", UriKind.Absolute);
var http = new HttpClient();
var stream = http.GetStreamAsync(uri).Result;
using (var reader = new StreamReader(stream))
{
while (!reader.EndOfStream)
{
var response = reader.ReadToEnd();
}
}
Related
I try to download and parallel upload streams (.ts, .mkv, .avi) from one source (locally limited) to an ASP.NET MVC response, so I can access the stream without any manipulation from outside.
I have this code so far.
using (HttpClient client = new HttpClient())
{
using (HttpResponseMessage response = client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead).Result)
using (Stream streamToReadFrom = response.Content.ReadAsStreamAsync().Result)
{
using (var returnStream = new MemoryStream())
{
streamToReadFrom.CopyToAsync(returnStream);
return new FileStreamResult(returnStream, "video/mp2t");
}
}
}
I thought I can just pass the download stream onto the response stream, but I got stuck with the asynchronous part. The system is responding with the following:
ObjectDisposedException: Cannot access a closed Stream.
Does anyone have an idea what I have to change, to get this running?
I found a second solution for my problem which seems to work sort of. At least the code wont crash. But when I try to save the file, it wont have any data (0kB)
var client = new HttpClient();
var result = await client.GetAsync(url);
var stream = await result.Content.ReadAsStreamAsync();
return new FileStreamResult(stream, "video/mp2t");
But now I can't access the stream. I am not sure why that is.
I have set the action to asynchronous, to use the code.
I am not sure if I have to change the result stream or add some header information or what to do, to get it work.
I am trying to retrieve content from a URL, I have tried .NET Core's HttpClient and WebClient, both of which take ~10 seconds to load this specific website.
However, when I use Python's urllib.request it loads within the same second. I have tried pretty much all the different combinations including: DownloadString, GetStringAsync, GetStreamAsync, GetAsync, OpenRead, etc.
I can provide the specific URL if needed. Any possible ideas?
Attempt #1
WebClient client = new WebClient();
Stream data = client.OpenRead("https://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/dtpp/search/");
StreamReader reader = new StreamReader(data);
string s = await reader.ReadToEndAsync();
data.Close();
reader.Close();
return s;
Attempt #2
using (var wc = new HttpClient())
{
var test = wc.GetAsync("https://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/dtpp/search/").Result;
var contents = test.Content.ReadAsStringAsync().Result;
return contents;
}
Attempt #3
using (var wc = new HttpClient())
{
HTML = await wc.GetStringAsync("https://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/dtpp/search/");
return HTML;
}
All three attempts work, just take ~10 seconds everytime. If I run this the same sort of thing in Python it returns within the same second.
Python Version
with urllib.request.urlopen('https://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/dtpp/search/') as response:
html = response.read()
Getting hit with a OutOfMemoryException unhandled.
using(var httpclient = new HttpClient(httpClientHandler))
{
httpclient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
httpclient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
var request = new HttpRequestMessage(HttpMethod.Post, url);
request.Content = new FormUrlEncodedContent(parameters);
var response = await httpclient.SendAsync(request);
var contents = await response.Content.ReadAsStringAsync();
var source = contents.ToString();
return source;
}
I'm not really sure what to do, or what is the specific cause, I believe it has something to do with " await response.Content.ReadAsStringAsync();
someone suggested to use
ReadAsStreamAsync();
instead and output to a file, however I need to output as a string to " source " so I can analyse the data in another function..
I would also like to add I'm running threads..
Is it possible the
Response.Content
is being stored in the memory even after it has finished that specific function? Do i need to dispose/clear memory or contents after I've returned it to source?
The advised solution is correct. it seems that the OS memory which your application is hosted on does not have enough memory. In order to workaround this It is wise to write stream as file to disk instead of memory. TransferEncodingChunked may also help but sender need to support it.
using (FileStream fs = File.Create("fileName.ext")
{
....
var stream = await response.Content.ReadAsStreamAsync();
await fs.CopyToAsync(stream);
fs.Close();
}
My program uses HttpClient to send a GET request to a Web API, and this returns a file.
I now use this code (simplified) to store the file to disc:
public async Task<bool> DownloadFile()
{
var client = new HttpClient();
var uri = new Uri("http://somedomain.com/path");
var response = await client.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
var fileName = response.Content.Headers.ContentDisposition.FileName;
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.Content.CopyToAsync(fs);
return true;
}
}
return false;
}
Now, when this code runs, the process loads all of the file into memory. I actually would rather expect the stream gets streamed from the HttpResponseMessage.Content to the FileStream, so that only a small portion of it is held in memory.
We are planning to use that on large files (> 1GB), so is there a way to achieve that without having all of the file in memory?
Ideally without manually looping through reading a portion to a byte[] and writing that portion to the file stream until all of the content is written?
It looks like this is by-design - if you check the documentation for HttpClient.GetAsync() you'll see it says:
The returned task object will complete after the whole response
(including content) is read
You can instead use HttpClient.GetStreamAsync() which specifically states:
This method does not buffer the stream.
However you don't then get access to the headers in the response as far as I can see. Since that's presumably a requirement (as you're getting the file name from the headers), then you may want to use HttpWebRequest instead which allows you you to get the response details (headers etc.) without reading the whole response into memory. Something like:
public async Task<bool> DownloadFile()
{
var uri = new Uri("http://somedomain.com/path");
var request = WebRequest.CreateHttp(uri);
var response = await request.GetResponseAsync();
ContentDispositionHeaderValue contentDisposition;
var fileName = ContentDispositionHeaderValue.TryParse(response.Headers["Content-Disposition"], out contentDisposition)
? contentDisposition.FileName
: "noname.dat";
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.GetResponseStream().CopyToAsync(fs);
}
return true
}
Note that if the request returns an unsuccessful response code an exception will be thrown, so you may wish to wrap in a try..catch and return false in this case as in your original example.
Instead of GetAsync(Uri) use the the GetAsync(Uri, HttpCompletionOption) overload with the HttpCompletionOption.ResponseHeadersRead value.
The same applies to SendAsync and other methods of HttpClient
Sources:
docs (see remarks)
https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.getasync?view=netcore-1.1#System_Net_Http_HttpClient_GetAsync_System_Uri_System_Net_Http_HttpCompletionOption_
The returned Task object will complete based on the completionOption parameter after the part or all of the response (including content) is read.
.NET Core implementation of GetStreamAsync that uses HttpCompletionOption.ResponseHeadersRead https://github.com/dotnet/corefx/blob/release/1.1.0/src/System.Net.Http/src/System/Net/Http/HttpClient.cs#L163-L168
HttpClient spike in memory usage with large response
HttpClient.GetStreamAsync() with custom request? (don't mind the comment on response, the ResponseHeadersRead is what does the trick)
Another simple and quick way to do it is:
public async Task<bool> DownloadFile(string url)
{
using (MemoryStream ms = new MemoryStream()) {
new HttpClient().GetStreamAsync(webPath).Result.CopyTo(ms);
... // use ms in what you want
}
}
now you have the file downloaded as stream in ms.
I'm trying to download files from my FTP server - multiples at the same time. When i use the DownloadFileAsync .. random files are returned with a byte[] Length of 0. I can 100% confirm the file exists on the server and has content AND there FTP server (running Filezilla Server) isn't erroring and say's the file has been transferred.
private async Task<IList<FtpDataResult>> DownloadFileAsync(FtpFileName ftpFileName)
{
var address = new Uri(string.Format("ftp://{0}{1}", _server, ftpFileName.FullName));
var webClient = new WebClient
{
Credentials = new NetworkCredential(_username, _password)
};
var bytes = await webClient.DownloadDataTaskAsync(address);
using (var stream = new MemoryStream(bytes))
{
// extract the stream data (either files in a zip OR a file);
return result;
}
}
When I try this code, it's slower (of course) but all the files have content.
private async Task<IList<FtpDataResult>> DownloadFileAsync(FtpFileName ftpFileName)
{
var address = new Uri(string.Format("ftp://{0}{1}", _server, ftpFileName.FullName));
var webClient = new WebClient
{
Credentials = new NetworkCredential(_username, _password)
};
// NOTICE: I've removed the AWAIT and a different method.
var bytes = webClient.DownloadData(address);
using (var stream = new MemoryStream(bytes))
{
// extract the stream data (either files in a zip OR a file);
return result;
}
}
Can anyone see what I'm doing wrong, please? Why would the DownloadFileAsync be randomly returning zero bytes?
Try out FtpWebRequest/FtpWebResponse classes. You have more available to you for debugging purposes.
FtpWebRequest - http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest(v=vs.110).aspx
FtpWebResponse - http://msdn.microsoft.com/en-us/library/system.net.ftpwebresponse(v=vs.110).aspx
Take a look at http://netftp.codeplex.com/. It appears as though almost all methods implement IAsyncResult. There isn't much documentation on how to get started, but I would assume that it is similar to the synchronous FTP classes from the .NET framework. You can install the nuget package here: https://www.nuget.org/packages/System.Net.FtpClient/