how can i read some bytes and disconnect? i use such code
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
using (StreamReader sr = new StreamReader(sm, Encoding.Default))
{
sr.Read();
sr.Close();
}
}
}
but it wait for end of stream
You probably don't want to use a StreamReader to read a WebResonse stream unless you know for sure that the stream contains newlines. StreamReader likes to think in terms of lines, and if there aren't any newlines in the stream, it's going to hang.
Your best bet is to read as many bytes as you want into a byte[] buffer, and then convert that to text. For example:
int BYTES_TO_READ = 1000;
var buffer = new byte[BYTES_TO_READ];
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
int totalBytesRead = 0;
int bytesRead;
do
{
// You have to do this in a loop because there's no guarantee that
// all the bytes you need will be ready when you call.
bytesRead = sm.Read(buffer, totalBytesRead, BYTES_TO_READ-totalBytesRead);
totalBytesRead += bytesRead;
} while (totalBytesRead < BYTES_TO_READ);
// Sometimes WebResponse will hang if you try to close before
// you've read the entire stream. So you can abort the request.
request.Abort();
}
}
At this point, the buffer has the first BYTES_TO_READ bytes from the buffer. You can then convert that to a string, like this:
string s = Encoding.Default.GetString(buffer);
Or you can open a MemoryStream on the buffer if you want to use StreamReader.
I have run into WebResponse hanging sometimes if you don't read everything. I don't know why it does that, and I can't reliably reproduce it, but I've found that if I do request.Abort() before closing the stream, everything works. See
On a side note, the word you want is "unresponsive" rather than "unresponsible."
Could you do something like this?
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
}
return result;
}
The key is using AddRange in your request.
It is beacuse http 1.1 connection is persistent connection default and the tcp connection has not been closed,so the stream dose not receive the end.
you can use myHttpWebRequest1.KeepAlive=false;
so the tcp connection will close after the http reponse.
https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.connection(v=vs.110).aspx#
If you talking winforms or webforms I would put the request into a threadpool (or Task if you are using .net 4). Streams, even with a good handling, are too easy to put GUI in a wait-state that dislikes by most users.
I had similar never-ending request parsing with examples listed here. The following is what I came up with to eliminate those issues:
// url is a string where the request is going.
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
// Do what you have to do with the Request.
StringBuilder builder = new StringBuilder();
int toRead = 1000;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
HttpStatusCode status = response.StatusCode;
using (Stream receiveStream = response.GetResponseStream())
{
using (StreamReader readStream = new StreamReader(receiveStream, Encoding.GetEncoding("utf-8")))
{
Char[] read = new Char[toRead];
int count = readStream.Read(read, 0, toRead);
while (count > 0)
{
string str = new String(read, 0, count);
builder.Append(str);
count = readStream.Read(read, 0, toRead);
}
readStream.Close();
}
}
response.Close();
}
return builder.ToString();
Related
Struggling to find anyone experiencing a similar issue or anything similar.
I'm currently consuming a stream over http (json) which has a GZip requirement, and I am experiencing a delay from when the data is sent, to when reader.ReadLine() reads it. It has been suggested to me that this could be related to the decoding keeping back data in a buffer?
This is what I have currently, it works fine apart from the delay.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(username, password);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Timeout = 30;
request.BeginGetResponse(AsyncCallback, request);
Then inside the AsyncCallback method I have:
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
{
while (!reader.EndOfStream)
{
string line = reader.ReadLine();
if (string.IsNullOrWhiteSpace(line)) continue;
Console.WriteLine(line);
}
}
It just sits on reader.Readline() until more data is received, and then even holds back some of that. There are also keep-alive newlines received, these are often are read out all at once when it does decide to read something.
I have tested the stream running side by side with a curl command running, the curl command receives and decompresses the data perfectly fine.
Any insight would be terrific.
Thanks,
Dan
EDIT
Had no luck using the buffer size on streamreader.
new StreamReader(stream, Encoding.UTF8, true, 1)
EDIT
Also had no luck updating to .NET 4.5 and using
request.AllowReadStreamBuffering = false;
Update: This seems to have issues over long periods of time with higher rates of volume, and should only be used on small volume where the buffer is impacting the application's functionality. I have since switched back to a StreamReader.
So this is what I ended up coming up with. This works, without the delay. This does not get buffered by automated GZip decompression.
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (MemoryStream memory = new MemoryStream())
using (GZipStream gzip = new GZipStream(memory, CompressionMode.Decompress))
{
byte[] compressedBuffer = new byte[8192];
byte[] uncompressedBuffer = new byte[8192];
List<byte> output = new List<byte>();
while (stream.CanRead)
{
int readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
memory.Write(compressedBuffer.Take(readCount).ToArray(), 0, readCount);
memory.Position = 0;
int uncompressedLength = gzip.Read(uncompressedBuffer, 0, uncompressedBuffer.Length);
output.AddRange(uncompressedBuffer.Take(uncompressedLength));
if (!output.Contains(0x0A)) continue;
byte[] bytesToDecode = output.Take(output.LastIndexOf(0x0A) + 1).ToArray();
string outputString = Encoding.UTF8.GetString(bytesToDecode);
output.RemoveRange(0, bytesToDecode.Length);
string[] lines = outputString.Split(new[] { Environment.NewLine }, new StringSplitOptions());
for (int i = 0; i < (lines.Length - 1); i++)
{
Console.WriteLine(lines[i]);
}
memory.SetLength(0);
}
}
There may be something to the Delayed ACK C.Evenhuis discusses, but I've got a weird gut feeling it's the StreamReader that's causing you headaches...you might try something like this:
public void AsyncCallback(IAsyncResult result)
{
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
{
var buffer = new byte[2048];
while(stream.CanRead)
{
var readCount = stream.Read(buffer, 0, buffer.Length);
var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray());
Console.WriteLine(line);
}
}
}
EDIT: Here's the full harness I used to test this theory (maybe the difference from your situation will jump out at you)
(LINQPad-ready)
void Main()
{
Task.Factory.StartNew(() => Listener());
_blocker.WaitOne();
Request();
}
public bool _running;
public ManualResetEvent _blocker = new ManualResetEvent(false);
public void Listener()
{
var listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8080/");
listener.Start();
"Listener is listening...".Dump();;
_running = true;
_blocker.Set();
var ctx = listener.GetContext();
"Listener got context".Dump();
ctx.Response.KeepAlive = true;
ctx.Response.ContentType = "application/json";
var outputStream = ctx.Response.OutputStream;
using(var zipStream = new GZipStream(outputStream, CompressionMode.Compress))
using(var writer = new StreamWriter(outputStream))
{
var lineCount = 0;
while(_running && lineCount++ < 10)
{
writer.WriteLine("{ \"foo\": \"bar\"}");
"Listener wrote line, taking a nap...".Dump();
writer.Flush();
Thread.Sleep(1000);
}
}
listener.Stop();
}
public void Request()
{
var endPoint = "http://localhost:8080";
var username = "";
var password = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(username, password);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Timeout = 30;
request.BeginGetResponse(AsyncCallback, request);
}
public void AsyncCallback(IAsyncResult result)
{
Console.WriteLine("In AsyncCallback");
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
{
while(stream.CanRead)
{
var buffer = new byte[2048];
var readCount = stream.Read(buffer, 0, buffer.Length);
var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray());
Console.WriteLine("Reader got:" + line);
}
}
}
Output:
Listener is listening...
Listener got context
Listener wrote line, taking a nap...
In AsyncCallback
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
This may have to do with Delayed ACK in combination with Nagle's algorithm. It occurs when the server sends multiple small responses in a row.
On the server side, the first response is sent, but subsequent response data chunks are only sent when the server has received an ACK from the client, or until there is enough data for a big packet to send (Nagle's algorithm).
On the client side, the first bit of response is received, but the ACK is not sent immediately - since traditional applications have a request-response-request-response behavior, it assumes it can send the ACK along with the next request - which in your case does not happen.
After a fixed amount of time (500ms?) it decides to send the ACK anyway, causing the server to send the next packages it has accumulated sofar.
The problem (if this is indeed the problem you're experiencing) can be fixed on the server side at the socket level by setting the NoDelay property, disabling Nagle's algorithm. I think you can also disable it operating system wide.
You could also temporarily disable Delayed ACK (I know windows has a registry entry for it) on the client side to see if this is indeed the problem, without having to change anything on your server. Delayed ACK prevents DDOS attacks, so make sure you restore the setting afterwards.
Sending keepalives less frequently may also help, but you'll still have a chance for the problem to occur.
I am taking a string message and breaking it up into chunks so that I can send it to an sms service (that consequently doesn't break it up for you). I after I do my work for break those messages up, I try loop through the resulting array and execute a web request. The problem is that it only works for the first message, and then hangs after that. After a short time, I get an error message saying "The connection was closed unexpectedly." This occurs at the second time it attempts GetResponse(); I've seen a few other posts on here that were simply saying to close and dispose the response and request streams. This isn't working for me at all. This is where my code is currently:
private static void Main(string[] args)
{
var oldMessage = GetFileString();
Console.WriteLine(string.Format("Old message: {0}", oldMessage.Length));
var newMessage = UrlPathEncodeString(oldMessage);
Console.WriteLine(string.Format("New message: {0}", newMessage.Length));
var brokenUp = SplitByLength(newMessage, 145).ToArray();
for(var i = 0; i < brokenUp.Count(); i++)
{
brokenUp[i] = brokenUp[i].Insert(0, UrlPathEncodeString(string.Format("({0:D2} of {1:D2})", i + 1, brokenUp.Count())));
Console.WriteLine(string.Format("Appended length: {0}", brokenUp[i].Length));
}
System.Net.ServicePointManager.DefaultConnectionLimit = 100;
foreach (var block in brokenUp)
{
Thread.Sleep(1500);
SendSms((HttpWebRequest)WebRequest.Create("http://172.20.5.214:90/method/sendsms"), block);
}
Console.ReadKey();
}
public static void SendSms(HttpWebRequest request, string message)
{
//build the request
var url = "http://ipaddress/method/sendsms";
//var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var fields = "CellNumber={0}&Message={1}";
fields = string.Format(fields, "16021234567", message);
var fieldsBytes = Encoding.UTF8.GetBytes(fields);
request.ContentLength = fieldsBytes.Length;
var length = fieldsBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
responseStream.Close();
}
}
requestStream.Close();
}
}
public static byte[] ReadFully(Stream stream)
{
var buffer = new byte[32768];
using (var ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
I've run into cases where response.Close appears to hang if you fail to download the entire contents of the response. Why this is, I don't know, but placing a call to request.Abort before calling Close() solves the problem. I wouldn't expect you to be seeing this problem, though, unless the response could potentially be many megabytes in size.
Also, failing to close the request stream before calling GetResponse might prevent all of the data from being sent. I would suggest calling requestStream.Close before making the request. But again, it seems odd that your code would work the first time but not on subsequent requests.
Your modified code, taking into account the changes I suggested, would be:
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
}
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
// read the response here.
request.Abort();
responseStream.Close();
}
}
I am trying to make an HttpWebRequest to send data via POST to a REST web service, and then follow-up with an additional HttpWebRequest to download a response after processing.
The POST is sending binary data up to the web service, as follows:
HttpWebRequest uploadRequest = (HttpWebRequest)WebRequest.Create(baseAddress + uploadURIRequest);
uploadRequest.Timeout = Timeout.Infinite;
uploadRequest.ReadWriteTimeout = Timeout.Infinite;
uploadRequest.Method = "POST";
uploadRequest.ContentLength = fileInfo.Length;
uploadRequest.ContentType = "application/octet-stream";
using (Stream writeStream = uploadRequest.GetRequestStream())
{
using (FileStream readStream = new FileStream(fileStreamFileName, FileMode.Open, FileAccess.Read))
{
byte[] data= new byte[readStream.Length];
int bytesRead = readStream.Read(data, 0, (int)readStream.Length);
writeStream.Write(data, 0, bytesRead);
readStream.Close();
}
writeStream.Close();
}
Then the next request is made to tell the web service to process data on the server and return the status response.
HttpWebRequest processRequest = (HttpWebRequest)WebRequest.Create(baseAddress + processURIRequest);
processRequest.Timeout = 10000;
processRequest.ReadWriteTimeout = 10000;
processRequest.ContentType = "GET";
HttpWebResponse processRequestResponse = (HttpWebResponse)processRequest.GetResponse();
using (Stream processRequestResponseStream = processRequestResponse.GetResponseStream())
{
//Do stuff...
}
When I change content-type, it works for XML data. However, when I keep it as shown above with binary data, the operation always times out...even if I increase the timeout to longer than 10 seconds. The processing that it is doing should not take this long to return.
When I debug, it always hangs on the line with GetResponse:
HttpWebResponse processRequestResponse = (HttpWebResponse)processRequest.GetResponse();
Adding
HttpWebResponse uploadRequestResponse = (HttpWebResponse)uploadRequest.GetResponse();
after writing to the stream resolved this issue.
I need to download a text file from the internet using C#. The file size can be quite large and the information I need is always within the first 1000 bytes. Is this possible?
Stolen from here.
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
// the following code is alternative, you may implement the function after your needs
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
byte[] buffer = new byte[1024];
int read = stream.Read(buffer, 0, 1000);
Array.Resize(ref buffer, read);
return Encoding.ASCII.GetString(buffer);
}
}
}
(Edited as requested in the comments... ;) )
I did this as an answer to your newer question. You could put the range header in too if you want, but I excluded it.
string GetWebPageContent(string url)
{
//string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
var buffer = new char[bytesToGet];
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
sr.Read(buffer, 0, bytesToGet);
}
}
return new string(buffer);
}
In C# rather than having to dowloading a file from the web using httpwebrequest, save this to file somewhere, and then upload to a webservice using a POST with the file as one of the parameters...
Can I instead somehow open a reader stream from httpwebresponse and then stream this into the http POST? Any code someone could post to show how?
In other words I'm trying to avoid haing to save to disk first.
Thanks
Something like that should do the trick :
HttpWebRequest downloadRequest = WebRequest.Create(downloadUri) as HttpWebRequest;
using(HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
{
HttpWebRequest uploadRequest = new HttpWebRequest(uploadUri);
uploadRequest.Method = "POST";
uploadRequest.ContentLength = downloadResponse.ContentLength;
using (Stream downloadStream = downloadResponse.GetResponseStream())
using (Stream uploadStream = uploadRequest.GetRequestStream())
{
byte[] buffer = new byte[4096];
int totalBytes = 0;
while(totalBytes < downloadResponse.ContentLength)
{
int nBytes = downloadStream.Read(buffer, 0, buffer.Length);
uploadStream.Write(buffer, 0, nBytes);
totalBytes += nRead;
}
}
HttpWebResponse uploadResponse = uploadRequest.GetResponse() as HttpWebResponse;
uploadResponse.Close();
}
(untested code)