We Keep Coding

Passing data from WebResponse to different method

I am trying to pass HttpWebResponse data from a method which checks whether web address written by user exists to another method which will use a StreamReader to get the html sourcecode and later working with it but even though it doesn't show any error I am not getting the sourcode written in prepared listbox. There is as well a button click event which I am not including and shouldn't have any impact on the problem.
protected bool ZkontrolujExistenciStranky(string WebovaStranka)
{
try
{
var pozadavek = WebRequest.Create(WebovaStranka) as HttpWebRequest;
pozadavek.Method = "HEAD";
using (var odezva = (HttpWebResponse)pozadavek.GetResponse())
{
GetData(odezva);
return odezva.StatusCode == HttpStatusCode.OK;
}
}
catch
{
return false;
}
}
protected void GetData(HttpWebResponse ziskanaOdezva)
{
using (Stream strm = ziskanaOdezva.GetResponseStream())
{
StreamReader reader = new StreamReader(strm);
string prochazec;
while ((prochazec = reader.ReadLine()) != null)
{
listBox1.Items.Add(prochazec);
}
}
}

You are using the HEAD method, whose whole point is not to return a body; only headers are returned. Use GET if you want the body.
HTTP HEAD method:
The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request.

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

WebRequest extremely slow

My method looks like this:
public string Request(string action, NameValueCollection parameters, uint? timeoutInSeconds = null)
{
parameters = parameters ?? new NameValueCollection();
ProvideCredentialsFor(ref parameters);
var data = parameters.ToUrlParams(); // my extension method converts the collection to a string, works well
byte[] dataStream = Encoding.UTF8.GetBytes(data);
string request = ServiceUrl + action;
var webRequest = (HttpWebRequest)WebRequest.Create(request);
webRequest.AllowAutoRedirect = false;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = dataStream.Length;
webRequest.Timeout = (int)(timeoutInSeconds == null ? DefaultTimeoutMs : timeoutInSeconds * 1000);
webRequest.Proxy = null; // should make it faster...
using (var newStream = webRequest.GetRequestStream())
{
newStream.Write(dataStream, 0, dataStream.Length);
}
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string uri = webResponse.Headers["Location"];
string result;
using (var sr = new StreamReader(webResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
}
The server sends JSON in response. It works fine for small JSON, but when I request a large one - something goes wrong. By large one I mean something that takes 1-2 minutes to appear in a browser (google chrome, including server side generation time). It's actually 412KB of text. When I try to ask for the same JSON with the method above I get a web exception (timeout). I changed the timeout to 10 minutes (at least 5 times longer than chrome). Still the same.
Any ideas?
EDIT
This seems to have something to do with MS technologies. On IE this JSON also won't load.

Make sure you close your request. Otherwise, once you hit the maximum number of allowed connections (as low as four, for me, on one occasion), you have to wait for the earlier ones to time out. This is best done using
using (var response = webRequest.GetResponse()) {...

GetRequestStream() is throwing time out exception when posting data to HTTPS url

I'm calling an API hosted on Apache server to post data. I'm using HttpWebRequest to perform POST in C#.
API has both normal HTTP and secure layer (HTTPS) PORT on the server. When I call HTTP URL it works perfectly fine. However, when I call HTTPS it gives me time-out exception (at GetRequestStream() function). Any insights? I'm using VS 2010, .Net framework 3.5 and C#. Here is the code block:
string json_value = jsonSerializer.Serialize(data);
HttpWebRequest request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;
System.IO.Stream reqStream = request.GetRequestStream();
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
EDIT:
The console program suggested by Peter works fine. But when I add data (in JSON format) that needs to be posted to the API, it throws out operation timed out exception. Here is the code that I add to console based application and it throws error.
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;

I ran into the same issue. It seems like it is solved for me. I went through all my code making sure to invoke webResponse.Close() and/or responseStream.Close() for all my HttpWebResponse objects. The documentation indicates that you can close the stream or the HttpWebResponse object. Calling both is not harmful, so I did. Not closing the responses may cause the application to run out of connections for reuse, and this seems to affect the HttpWebRequest.GetRequestStream as far as I can observe in my code.

I don't know if this will help you with your specific problem but you should consider Disposing some of those objects when you are finished with them. I was doing something like this recently and wrapping stuff up in using statements seems to clean up a bunch of timeout exceptions for me.
using (var reqStream = request.GetRequestStream())
{
if (reqStream == null)
{
return;
}
//do whatever
}
also check these things
Is the server serving https in your local dev environment?
Have you set up your bindings *.443 (https) properly?
Do you need to set credentials on the request?
Is it your application pool account accessing the https resources or is it your account being passed through?
Have you thought about using WebClient instead?
using (WebClient client = new WebClient())
{
using (Stream stream = client.OpenRead("https://server-url-xxxx.com"))
using (StreamReader reader = new StreamReader(stream))
{
MessageBox.Show(reader.ReadToEnd());
}
}
EDIT:
make a request from console.
internal class Program
{
private static void Main(string[] args)
{
new Program().Run();
Console.ReadLine();
}
public void Run()
{
var request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
using (var reqStream = request.GetRequestStream())
{
using(var response = new StreamReader(reqStream )
{
Console.WriteLine(response.ReadToEnd());
}
}
}
}

Try this:
WebRequest req = WebRequest.Create("https://server-url-xxxx.com");
req.Method = "POST";
string json_value = jsonSerializer.Serialize(data); //Body data
ServicePointManager.Expect100Continue = false;
using (var streamWriter = new StreamWriter(req.GetRequestStream()))
{
streamWriter.Write(json_value);
streamWriter.Flush();
streamWriter.Close();
}
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
Stream GETResponseStream = resp.GetResponseStream();
StreamReader sr = new StreamReader(GETResponseStream);
var response = sr.ReadToEnd(); //Response
resp.Close(); //Close response
sr.Close(); //Close StreamReader
And review the URI:
Reserved characters. Send reserved characters by the URI can bring
problems ! * ' ( ) ; : # & = + $ , / ? # [ ]
URI Length: You should not exceed 2000 characters

I ran into this, too. I wanted to simulate hundreds of users with a Console app. When simulating only one user, everything was fine. But with more users came the Timeout exception all the time.
Timeout occurs because by default the ConnectionLimit=2 to a ServicePoint (aka website).
Very good article to read: https://venkateshnarayanan.wordpress.com/2013/04/17/httpwebrequest-reuse-of-tcp-connections/
What you can do is:
1) make more ConnectionGroups within a servicePoint, because ConnectionLimit is per ConnectionGroups.
2) or you just simply increase the connection limit.
See my solution:
private HttpWebRequest CreateHttpWebRequest<U>(string userSessionID, string method, string fullUrl, U uploadData)
{
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(fullUrl);
req.Method = method; // GET PUT POST DELETE
req.ConnectionGroupName = userSessionID; // We make separate connection-groups for each user session. Within a group connections can be reused.
req.ServicePoint.ConnectionLimit = 10; // The default value of 2 within a ConnectionGroup caused me always a "Timeout exception" because a user's 1-3 concurrent WebRequests within a second.
req.ServicePoint.MaxIdleTime = 5 * 1000; // (5 sec) default was 100000 (100 sec). Max idle time for a connection within a ConnectionGroup for reuse before closing
Log("Statistics: The sum of connections of all connectiongroups within the ServicePoint: " + req.ServicePoint.CurrentConnections; // just for statistics
if (uploadData != null)
{
req.ContentType = "application/json";
SerializeToJson(uploadData, req.GetRequestStream());
}
return req;
}
/// <summary>Serializes and writes obj to the requestStream and closes the stream. Uses JSON serialization from System.Runtime.Serialization.</summary>
public void SerializeToJson(object obj, Stream requestStream)
{
DataContractJsonSerializer json = new DataContractJsonSerializer(obj.GetType());
json.WriteObject(requestStream, obj);
requestStream.Close();
}

You may want to set timeout property, check it here http://www.codeproject.com/Tips/69637/Setting-timeout-property-for-System-Net-WebClient

Canonical HTTP POST code?

I've seen so many implementations of sending an http post, and admittedly I don't fully understand the underlying details to know what's required.
What is the succinct/correct/canonical code to send an HTTP POST in C# .NET 3.5?
I want a generic method like
public string SendPost(string url, string data)
that can be added to a library and always used for posting data and will return the server response.

I believe that the simple version of this would be
var client = new WebClient();
return client.UploadString(url, data);
The System.Net.WebClient class has other useful methods that let you download or upload strings or a file, or bytes.
Unfortunately there are (quite often) situations where you have to do more work. The above for example doesn't take care of situations where you need to authenticate against a proxy server (although it will use the default proxy configuration for IE).
Also the WebClient doesn't support uploading of multiple files or setting (some specific) headers and sometimes you will have to go deeper and use the
System.Web.HttpWebRequest and System.Net.HttpWebResponse instead.

As others have said, WebClient.UploadString (or UploadData) is the way to go.
However the built-in WebClient has a major drawback : you have almost no control over the WebRequest that is used behind the scene (cookies, authentication, custom headers...). A simple way to solve that issue is to create your custom WebClient and override the GetWebRequest method. You can then customize the request before it is sent (you can do the same for the response by overridingGetWebResponse). Here is an example of a cookie-aware WebClient. It's so simple it makes me wonder why the built-in WebClient doesn't handle it out-of-the-box...

Compare:
// create a client object
using(System.Net.WebClient client = new System.Net.WebClient()) {
// performs an HTTP POST
client.UploadString(url, xml);
}
to
string HttpPost (string uri, string parameters)
{
// parameters: name1=value1&name2=value2
WebRequest webRequest = WebRequest.Create (uri);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes (parameters);
Stream os = null;
try
{ // send the Post
webRequest.ContentLength = bytes.Length; //Count bytes to send
os = webRequest.GetRequestStream();
os.Write (bytes, 0, bytes.Length); //Send it
}
catch (WebException ex)
{
MessageBox.Show ( ex.Message, "HttpPost: Request error",
MessageBoxButtons.OK, MessageBoxIcon.Error );
}
finally
{
if (os != null)
{
os.Close();
}
}
try
{ // get the response
WebResponse webResponse = webRequest.GetResponse();
if (webResponse == null)
{ return null; }
StreamReader sr = new StreamReader (webResponse.GetResponseStream());
return sr.ReadToEnd ().Trim ();
}
catch (WebException ex)
{
MessageBox.Show ( ex.Message, "HttpPost: Response error",
MessageBoxButtons.OK, MessageBoxIcon.Error );
}
return null;
} // end HttpPost
Why are people using/writing the latter?