We Keep Coding

Clear context after HttpWebRequest

We use an HttpWebRequest to send query to a webservice with NetworkCredential like this :
Byte[] bytes = Encoding.UTF8.GetBytes("...");
HttpWebRequest request = (HttpWebRequest)WebRequest.CreateHttp(strURL);
request.Method = "POST";
request.ContentLength = bytes.Length;
request.ContentType = "application/xml";
request.Timeout = 10000;
request.KeepAlive = false;
NetworkCredential cred = new NetworkCredential
{
UserName = "...",
Password = "..."
};
request.Credentials = cred;
WebRequest.DefaultWebProxy.Credentials = CredentialCache.DefaultCredentials;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = request.GetResponse() as HttpWebResponse;
To test it, we created a small console project with just this request and constant information (always the same).
After two successful requests, the third one systematically stops and ends in timeout.
But if we restart our application, we can immediately reissue two requests before the third one freezes again.
Do you have any idea of the cause? It looks like some information needs to be purged before we can continue.
-- EDIT --
I tried to start each request on a separate thread (Using Thread or Task) with the same result.

I change everything to use HttpClient and, now, everything works fine.

How can I send a login request to a website?

so I want to send a request to log in to a website, how can I do that
I have tried the code bellow:
string formUrl = "https://account.mojang.com/login";
string formParams = string.Format("email_address={0}&password={1}", "your email", "your password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "GET";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
I get an error at line 9 : "Can not send a body of content with this type of verb."

I will answer this question in two parts: first the error message that you have received, and secondly how you should solve the problem.
Your error message resulted from line 9, evidently req.GetRequestStream() has failed for some reason:
using (Stream os = req.GetRequestStream())
Looking closer at the text of the message, it is evident what the problem is - you are trying to send a x-form-urlencoded message body with a GET request when this is not supported by the WebRequest implementation.
Can not send a body of content with this type of verb.
The function of GetRequestStream() is to get a stream to which you can write a request body (a "body of content"), in your case the login parameters. You are sending a GET request (GET and POST are referred to as "verbs" in the HTTP specification), and WebRequest does not support this.
Going back to the actual problem here, I believe that you did not mean to send a GET request and need to send a POST request intead. If that is the case then you simply need to change line 6 to read:
req.Method = "POST";
Most login services expect POST requests and GET requests do not work in any case. However, if you wanted to do a GET request with data using WebRequest you need to do it in a different way. The data must be encoded into the initial URL and there is no need to get a request stream or write to it. The fixed code in that case would read:
string formURL = string.Format("https://account.mojang.com/login?email_address={0}&password={1}", "your email", "your password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.Method = "GET";
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];

Do I have to update every cookies everytime?

I am using my c# app to send request to the webpage. Before I send any request I have to be logged in this webpage. I want to avoid logging everytime I want to do some work, so I am storing cookies in sql server database in VARBINARY column.
Lets say I am sending 50 POST requests every day:
private string getRequest(string url, string postData = "")
{
try
{
System.Net.ServicePointManager.Expect100Continue = true;
StreamReader reader;
var request = WebRequest.Create(url) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14";
request.CookieContainer = Cookie;
request.AllowAutoRedirect = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
postData = Uri.EscapeUriString(postData);
request.Method = "POST";
request.ContentLength = postData.Length;
Stream requestStream = request.GetRequestStream();
Byte[] postBytes = Encoding.ASCII.GetBytes(postData);
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
foreach (Cookie tempCookie in response.Cookies)
{
Cookie.Add(tempCookie);
}
reader = new StreamReader(response.GetResponseStream());
string readerReadToEnd = reader.ReadToEnd();
response.Close();
database.updateCookie(AccountId, Cookie); //here I am updating the cookies
return readerReadToEnd;
}
catch { return ""; }
Do I really need to update Cookies after every request? Or maybe could I update my cookies only once, after sending 50 POST requests?
I am asking, because sometimes my cookies lasts couple days, and sometimes they died after 1 minute. I don't really know why and how to avoid that.
Do I have to store the newest verion of cookies or maybe can I use the same everytime?

Each site decides how long particular cookies are valid. Hence you need to get that information for each site individually to do it correctly. Paying attention to "expiration" on the cookie (in "set-cookie" headers of response) may give you initial guidance on when cookie is guaranteed to expire.
Common expiration/validity ranges:
authentication cookies - from 10 minutes to 1 day
ASP.Net session state - from 20 minutes
CSRF protection - possibly updated on every response

HTTP Post with JSON

I can't seem to get the hang of my HTTP POST methods. I have just learned how to do GET methods to retrieve webpages but now i'm trying to fill in information on the webpage and can't seem to get it working. The source code that comes back is always an invalid page (full of broken images/not the right information)
public static void jsonPOST(string url)
{
url = "http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx/GetTaxReceipt";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(new Uri(url));
httpWebRequest.ContentType = "application/json; charset=utf-8";
httpWebRequest.Accept = "application/json, text/javascript, */*; q=0.01";
httpWebRequest.Headers.Add("Accept-Encoding: gzip, deflate");
httpWebRequest.CookieContainer = cookieJar;
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("Accept-Language: en-US,en;q=0.5");
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW65; Trident/7.0; MAM5; rv:11.0) like Gecko";
httpWebRequest.Referer = "http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx";
string postData = "{\"startDate\":\"1/1/2013\",\"parcelNumber\":\"17609419\"}";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData);
httpWebRequest.ContentLength = bytes.Length;
System.IO.Stream os = httpWebRequest.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
System.Net.WebResponse resp = httpWebRequest.GetResponse();
if (resp == null)
{
Console.WriteLine("null");
}
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
string source = sr.ReadToEnd().Trim();
}
EDIT: I updated the code to reflect my new problem. The problem i have now is that the source code is not what is coming back to me. I am getting just the raw JSON information in the source. Which i can use to deserialize the information i need to obtain, but i'm curious why the actual source code isn't coming back to me

The source code that comes back is always an invalid page (full of broken images/not the right information)
It sounds like you just get the Source code without thinking of relative paths. As long as there are relative paths on the site it will not show correctly at your copy. You have to replace all the relative paths before it is useful.
http://webdesign.about.com/od/beginningtutorials/a/aa040502a.htm
Remember crossdomain ajax can be a problem in that situation.

HttpWebRequest to https server works in fiddler but NOT from Visual Studio

Here is the scenario. I have written code use a digital certificate to GET cookies from a secured url, to in turn POST data back to another url using the retrieved cookies and same digital certificate. The GET works and cookies are retrieved, the POST comes back with an error 500. I installed fiddler to see what was going on...POST looks fine...cookies are present. I used the feature in fiddler that allows creating a request via drag and drop. POST the exact same POST recorded from C# code that was recorded in fiddler and it works!
What is Fiddler doing that Visual Studio is not? It must be doing something if fiddler can POST the data but Visual Studio returns an error 500. Here is the code below:
X509Certificate cert = new X509Certificate("mycert.pfx", "certpassword");
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
req.CookieContainer = new CookieContainer();
req.Method = "GET";
req.ClientCertificates.Add(cert);
HttpWebResponse Response = (HttpWebResponse)req.GetResponse();
CookieCollection ck = req.CookieContainer.GetCookies(req.RequestUri);
string strcookie = ck[0].Value;
string strcookie2 = ck[1].Value;
Response.Close();
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
req.CookieContainer = new CookieContainer();
Cookie auth = new Cookie("_wl_authcookie_", strcookie2);
Cookie jsess = new Cookie("JSESSIONID", strcookie);
auth.Domain = "server";
jsess.Domain = "server";
req.CookieContainer.Add(auth);
req.CookieContainer.Add(jsess);
req.ClientCertificates.Add(cert);
req.Method = "POST";
Byte[] data = ReadByteArrayFromFile("filewithdatatoPOST.txt");
req.ContentLength = data.Length;
Stream myStream = req.GetRequestStream();
myStream.Write(data, 0, data.Length);
myStream.Close();
HttpWebResponse Response2 = (HttpWebResponse)req.GetResponse();
Stream strm = Response2.GetResponseStream();
StreamReader sr2 = new StreamReader(strm);
Response2.Close();

Does your code work if you set
req.ServicePoint.Expect100Continue = false;
on all your WebRequests?

Solved!!! It wasn't related to the Expect100Continue.
So after weeks of troubleshooting....6 different programmers....I figured it out. I'm not sure if this is always true, but in this scenario the problem was that the url we were getting cookies from:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
was not the same as the url that we were posting the data back to:
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
Getting the cookies and posting back to the same url fixed the issue:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");
req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");