I need to download a text file from the internet using C#. The file size can be quite large and the information I need is always within the first 1000 bytes. Is this possible?
Stolen from here.
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
// the following code is alternative, you may implement the function after your needs
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
byte[] buffer = new byte[1024];
int read = stream.Read(buffer, 0, 1000);
Array.Resize(ref buffer, read);
return Encoding.ASCII.GetString(buffer);
}
}
}
(Edited as requested in the comments... ;) )
I did this as an answer to your newer question. You could put the range header in too if you want, but I excluded it.
string GetWebPageContent(string url)
{
//string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
var buffer = new char[bytesToGet];
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
sr.Read(buffer, 0, bytesToGet);
}
}
return new string(buffer);
}
Related
I've recently written a C# function that does a multi part form post for uploading files. To track the progress, I'd write the form data to the request stream at 4096 bytes at a time and call back with each write. However, it seems that the request does not even get sent until GetResponseAsync() is called.
If this is the case, is the reporting of every 4096 bytes written to the request stream an accurate reporting of upload progress?
If not, how can I accurately report progress? WebClient is out of the question for me, this is in a PCL Xamarin project.
private async Task<string> PostFormAsync (string postUrl, string contentType, byte[] formData)
{
try {
HttpWebRequest request = WebRequest.Create (postUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = contentType;
request.Headers ["Cookie"] = Constants.Cookie;
byte[] buffer = new byte[4096];
int count = 0;
int length = 0;
using (Stream requestStream = await request.GetRequestStreamAsync ()) {
using (Stream inputStream = new MemoryStream (formData)) {
while ((count = await inputStream.ReadAsync (buffer, 0, buffer.Length)) > 0) {
await requestStream.WriteAsync (buffer, 0, count);
length += count;
Device.BeginInvokeOnMainThread (() => {
_progressBar.Progress = length / formData.Length;
});
}
}
}
_progressBar.Progress = 0;
WebResponse resp = await request.GetResponseAsync ();
using (Stream stream = resp.GetResponseStream ()) {
StreamReader respReader = new StreamReader (stream);
return respReader.ReadToEnd ();
}
} catch (Exception e) {
Debug.WriteLine (e.ToString ());
return String.Empty;
}
}
Please note that I am asking about monitoring progress of an upload at 4096 bytes at a time, not a download
I ended up accomplishing this by setting the AllowWriteStreamBuffering property of the WebRequest equal to false and the SendChunked property to true.
HOWEVER Xamarin.PCL (Profile 78) does not allow you to access these properties of the HttpWebRequest, so I had to instantiate my HttpWebRequest and return it from a dependency service in my platform specific project (only tested in iOS).
public class WebDependency : IWebDependency
{
public HttpWebRequest GetWebRequest(string uri)
{
var request = WebRequest.Create (uri) as HttpWebRequest;
request.SendChunked = true;
request.AllowWriteStreamBuffering = false;
return request;
}
}
And then to instantiate my web request -
HttpWebRequest request = DependencyService.Get<IWebDependency>().GetWebRequest(uri);
I'm passing the stream this way:
StreamReader sr = new StreamReader(openFileDialog1.FileName);
byte[] fileStream = Utility.ReadFully(sr.BaseStream);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(baseAddress));
request.Method = "POST";
request.ContentType = "application/octet-stream";
Stream serverStream = request.GetRequestStream();
serverStream.Write(fileStream, 0, fileStream.Length);
serverStream.Close();
HttpWebResponse response2 = (HttpWebResponse)request.GetResponse();
if (response2.StatusCode == HttpStatusCode.OK)
{
MessageBox.Show(Utility.ReadResponse(response2));
}
-------------------------------------------------------------------------
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
if (input != null)
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
}
return ms.ToArray();
}
}
Then handling it on the server:
public bool UploadPhotoStream(string someStringParam, Stream fileData)
{
string filePath = string.Format("{0}/{1}", 'sdfgsdf87s7df8sd', '24asd54s4454d5f4g');
ProductPhoto newphoto = new ProductPhoto();
newphoto.FileSizeBytes = fileData.Length / 1024 / 1024;
newphoto.FileLocation = filePath;
...
}
Now I'm getting NotSupportedException when calling fileData.Length. I know it happens because the stream is closed. But how can I re-open it? Or what should I do so that when I pass the stream to the service I can still get its length?
Why don't you pass content-length header? Your server can check the header and know exactly how many bytes is the content being sent. How you read the header depends on which http framework you are using, ASP.NET Web Api, classic WCF Web Api, HttpListener, etc.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(baseAddress));
request.Method = "POST";
request.ContentType = "application/octet-stream";
request.ContentLength = new FileInfo(openFileDialog1.FileName).Length
Without a Content-Length header, an http server can never know how many bytes are left to read. All it knows is there is a Stream and will read it till there is no more data. This is also how your browser can display a progress bar when downloading something. It takes bytesDownloaded / Content-Length.
According to this post: https://stackoverflow.com/a/8239268/1160036
You can access the header like this from your web method.
long dataLength = long.Parse(HttpContext.Current.Request.Headers["Content-Length"]);
how can i read some bytes and disconnect? i use such code
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
using (StreamReader sr = new StreamReader(sm, Encoding.Default))
{
sr.Read();
sr.Close();
}
}
}
but it wait for end of stream
You probably don't want to use a StreamReader to read a WebResonse stream unless you know for sure that the stream contains newlines. StreamReader likes to think in terms of lines, and if there aren't any newlines in the stream, it's going to hang.
Your best bet is to read as many bytes as you want into a byte[] buffer, and then convert that to text. For example:
int BYTES_TO_READ = 1000;
var buffer = new byte[BYTES_TO_READ];
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
int totalBytesRead = 0;
int bytesRead;
do
{
// You have to do this in a loop because there's no guarantee that
// all the bytes you need will be ready when you call.
bytesRead = sm.Read(buffer, totalBytesRead, BYTES_TO_READ-totalBytesRead);
totalBytesRead += bytesRead;
} while (totalBytesRead < BYTES_TO_READ);
// Sometimes WebResponse will hang if you try to close before
// you've read the entire stream. So you can abort the request.
request.Abort();
}
}
At this point, the buffer has the first BYTES_TO_READ bytes from the buffer. You can then convert that to a string, like this:
string s = Encoding.Default.GetString(buffer);
Or you can open a MemoryStream on the buffer if you want to use StreamReader.
I have run into WebResponse hanging sometimes if you don't read everything. I don't know why it does that, and I can't reliably reproduce it, but I've found that if I do request.Abort() before closing the stream, everything works. See
On a side note, the word you want is "unresponsive" rather than "unresponsible."
Could you do something like this?
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
}
return result;
}
The key is using AddRange in your request.
It is beacuse http 1.1 connection is persistent connection default and the tcp connection has not been closed,so the stream dose not receive the end.
you can use myHttpWebRequest1.KeepAlive=false;
so the tcp connection will close after the http reponse.
https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.connection(v=vs.110).aspx#
If you talking winforms or webforms I would put the request into a threadpool (or Task if you are using .net 4). Streams, even with a good handling, are too easy to put GUI in a wait-state that dislikes by most users.
I had similar never-ending request parsing with examples listed here. The following is what I came up with to eliminate those issues:
// url is a string where the request is going.
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
// Do what you have to do with the Request.
StringBuilder builder = new StringBuilder();
int toRead = 1000;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
HttpStatusCode status = response.StatusCode;
using (Stream receiveStream = response.GetResponseStream())
{
using (StreamReader readStream = new StreamReader(receiveStream, Encoding.GetEncoding("utf-8")))
{
Char[] read = new Char[toRead];
int count = readStream.Read(read, 0, toRead);
while (count > 0)
{
string str = new String(read, 0, count);
builder.Append(str);
count = readStream.Read(read, 0, toRead);
}
readStream.Close();
}
}
response.Close();
}
return builder.ToString();
I'm just curious about whether this is possible - I know how to download a file, but how can I download only the first 100KB of a file?
Try this:
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
// the following code is alternative, you may implement the function after your needs
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
}
return result;
}
Stolen from here.
In C# rather than having to dowloading a file from the web using httpwebrequest, save this to file somewhere, and then upload to a webservice using a POST with the file as one of the parameters...
Can I instead somehow open a reader stream from httpwebresponse and then stream this into the http POST? Any code someone could post to show how?
In other words I'm trying to avoid haing to save to disk first.
Thanks
Something like that should do the trick :
HttpWebRequest downloadRequest = WebRequest.Create(downloadUri) as HttpWebRequest;
using(HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
{
HttpWebRequest uploadRequest = new HttpWebRequest(uploadUri);
uploadRequest.Method = "POST";
uploadRequest.ContentLength = downloadResponse.ContentLength;
using (Stream downloadStream = downloadResponse.GetResponseStream())
using (Stream uploadStream = uploadRequest.GetRequestStream())
{
byte[] buffer = new byte[4096];
int totalBytes = 0;
while(totalBytes < downloadResponse.ContentLength)
{
int nBytes = downloadStream.Read(buffer, 0, buffer.Length);
uploadStream.Write(buffer, 0, nBytes);
totalBytes += nRead;
}
}
HttpWebResponse uploadResponse = uploadRequest.GetResponse() as HttpWebResponse;
uploadResponse.Close();
}
(untested code)