We Keep Coding

CookieContainer does not store cookies for internationalized domain names

I'm trying to perform authorization on a cyrillic domain using WebClient. Authorization goes through few stages with redirects between normal and punicode domains. The problem is HttpWebRequest can not store cookies in assigned CookieContaier if it was set by punycode domain.
For example, this code will throw CookieException.
var cookie = new Cookie("test_cookie", "test_value", "/", ".xn----7sbcca6bi0ak9b0a6f.xn--p1ai");
var container = new CookieContainer().Add(cookie);
The problem is exacerbated by the fact that response that sets cookie redirects to another page, i.e. after WebClient.UploadValues(...) have been executed there's no cookie information in WebClient.ResponseHeaders.
Below is normal authorization process (using browser)
Method Result Received Type URL RedirectURL Set-Cookie
POST 302 1,18 K text/html http://xn----7sbcca6bi0ak9b0a6f.xn--p1ai/admin/login http://xn----7sbcca6bi0ak9b0a6f.xn--p1ai/admin sess_id=.......; expires=Mon, 06-Jun-2016 07:20:57 GMT; Max-Age=31536000; path=/; domain=.xn----7sbcca6bi0ak9b0a6f.xn--p1ai; httponly
GET 302 722 text/html http://xn----7sbcca6bi0ak9b0a6f.xn--p1ai/admin /admin/orders
GET 200 200,00 K text/html http://xn----7sbcca6bi0ak9b0a6f.xn--p1ai/admin/orders
Is there any workaround?

Are you sure the IDN is the problem?
The following code snippet (which is the same as yours, but with the second line split up to make it compile)
var cookie = new Cookie("test_cookie", "test_value", "/", ".xn----7sbcca6bi0ak9b0a6f.xn--p1ai");
var container = new CookieContainer();
container.Add(cookie);
container.GetCookies(new Uri("http://test.xn----7sbcca6bi0ak9b0a6f.xn--p1ai")).Dump();
does not throw a CookieException at all (run from LINQPad). Could the problem perhaps be in the name and/or value you are trying to set for the cookie? What is the exact message you get from the CookieException?

Using c#/ASP.NET to programmatically fake a login to a website

So I'm in the process of attempting to simulate multiple logins all generating exceptions at the same time on our corporate website for the purpose of testing our logging framework (which we think there may be issues with thread synchronization). Anyway, so I need to log in to our website programatically. Here's what I have so far:
// Block 1
Uri url = new Uri("http://withheld");
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string viewState = string.Empty;
string previousPage = string.Empty;
string eventValidation = string.Empty;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string strResponse = reader.ReadToEnd();
viewState = HttpUtility.UrlEncode(GetTagValue(strResponse, "__VIEWSTATE"));
previousPage = HttpUtility.UrlEncode(GetTagValue(strResponse, "__PREVIOUSPAGE"));
eventValidation = HttpUtility.UrlEncode(GetTagValue(strResponse, "__EVENTVALIDATION"));
}
// Block 2
string username = "user01";
string password = "password99";
HttpWebRequest request2 = WebRequest.Create(url) as HttpWebRequest;
request2.KeepAlive = true;
request2.Method = "POST";
request2.ContentType = "application/x-www-form-urlencoded";
string postData = string.Format("__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE={0}&__PREVIOUSPAGE={1}&__EVENTVALIDATION={2}&UserName={3}&Password={4}&LoginButton=Log+in", new string[] { viewState, previousPage, eventValidation, username, password});
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request2.ContentLength = dataBytes.Length;
using (Stream postStream = request2.GetRequestStream())
{
// Here's the problem
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
// At this point httpResponse.Cookies is null
// I believe it's because the line above has actually initiated another
// request/response which DOES NOT include the authentication cookie.
// See fiddler output below to understand why I think that.
// Block 3
//Uri url2 = new Uri("http://Withheld/GenerateException.aspx");
//http = WebRequest.Create(url2) as HttpWebRequest;
//http.CookieContainer = new CookieContainer();
//http.CookieContainer.Add(httpResponse.Cookies);
//HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Looks reasonably straightforward right? Well it doesn't work. I don't know whether I need the viewState and whatnot or not, but I figured I'd mimic what a regular browser does as closely as possible.
Near as I can tell what's happening is this:
We hit the page with a simple GET. That gives us the viewstate, etc which is parsed out to be included in the following request.
We now post the viewstate, username, password, etc to the server using postStream.Write(). The server at this point responds with an authentication cookie, and forwards us off to /Default.aspx.
Now we execute reqest2.GetResponse(), but instead of getting the response that forwarded us to /default.aspx and had he authentication cookie, it looks like this line is actually causing ANOTHER request to get us the resource at /default.aspx. The problem is httpWebResponse DOES NOT include the authentication cookie we need for the next request (which is commented out currently).
Why? Why is this behaving in this manor, and how do I handle it properly. Here's the output from fiddler to further explain what's going on:
Block 1 generates this request/response
Request Header:
GET http://withheld/Login.aspx HTTP/1.1
Host: withheld
Connection: Keep-Alive
Response Header:
HTTP/1.1 200 OK
Connection: close
Date: Mon, 04 Feb 2013 16:37:37 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: .EXTASPXAUTH=; expires=Tue, 12-Oct-1999 04:00:00 GMT; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 16975
Response is the actual login webpage. Omitted for obvious reasons.
Stepping through the code, this request/response is generated immediately following the call to postStream.Write:
Request:
POST http://Withheld/Login.aspx HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Content-Length: 2109
Expect: 100-continue
__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2fwEPDwUJNTE1NTIxNjkxD2QWAgIDD2QWBgIDDxYCHgVjbGFzcwUQaGVhZGVyX2Jhbm5lcl9lbhYCAgEPFgQeB29uY2xpY2sFI3dpbmRvdy5sb2NhdGlvbj0naHR0cDovL3d3dy5tZG0uY2EnHwAFFmhlYWRlcl9iYW5uZXJfZW5fc21hbGxkAgUPZBYQAgcPDxYEHhdQYXJ0aWFsUmVuZGVyaW5nQ2hlY2tlZGceGUlzUGFydGlhbFJlbmRlcmluZ0VuYWJsZWRoZGQCCQ8PFgQfAmcfA2hkZAIPDw8WBB8CZx8DaGRkAhEPDxYCHg1PbkNsaWVudENsaWNrBcEBamF2YXNjcmlwdDpQYWdlX0NsaWVudFZhbGlkYXRlKCdMb2dpbkN0cmwnKTsgaWYgKFBhZ2VfSXNWYWxpZCkgeyBMb2dpbkJ1dHRvbi5zdHlsZS52aXNpYmlsaXR5ID0gJ2hpZGRlbic7IExvZ2luQnV0dG9uLnN0eWxlLmRpc3BsYXkgPSAnbm9uZSc7IExvZ2luQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJzsgfWRkAhkPDxYCHgRUZXh0Bc0BSWYgeW91IHJlcXVpcmUgYXNzaXN0YW5jZSwgb3VyIE1EIE9ubGluZSBTdXBwb3J0IFNwZWNpYWxpc3RzIGNhbiBiZSByZWFjaGVkIGF0ICg4NzcpIDQzMS0wMzMwIG9yIGJ5IGUtbWFpbCBhdCA8YSBocmVmPSJtYWlsdG86d2Vic3VwcG9ydEBjbWEuY2EiPndlYnN1cHBvcnRAY21hLmNhPC9hPi48YnI%2bPGJyPldlIGVuY291cmFnZSB5b3UgdG8gcmV2aWV3IHRoZWRkAhsPDxYCHwQFOXdpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvbGVnYWwvJyk7cmV0dXJuIGZhbHNlO2RkAh8PDxYCHwQFRndpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvc3lzdGVtLWVuaGFuY2VtZW50LycpO3JldHVybiBmYWxzZTtkZAIhDw8WAh8EBUZ3aW5kb3cub3BlbignaHR0cDovLzE5Mi4xNjguMTY1LjMyL3N5c3RlbS1lbmhhbmNlbWVudC8nKTtyZXR1cm4gZmFsc2U7ZGQCCQ9kFgICAQ9kFgICAg9kFgJmD2QWAgIBD2QWAgIdDw8WAh8EBfsBamF2YXNjcmlwdDpjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uLnN0eWxlLnZpc2liaWxpdHkgPSAnaGlkZGVuJzsgY3RybENyZWF0ZU5ld1Bhc3N3b3JkX0NyZWF0ZU5ld1Bhc3N3b3JkUHVzaEJ1dHRvbi5zdHlsZS5kaXNwbGF5ID0gJ25vbmUnOyBjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJztkZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUMaWJ0bk1vcmVJbmZvdZmFXkMbPfVWPQYtreXvFt8Bck8%3d&__PREVIOUSPAGE=1aYW5DqTKrT4ieGPkHcnrQLIq8lEcSIVkql1EugwSQNV_5102t5D7QDmOnuQFA4Tz9Mh5-CEYpkRngMROFFeeAG12Ss1&__EVENTVALIDATION=%2fwEWCQKKr%2bXcBgKvruq2CALSxeCRDwL%2bjNCfDwKH8YSKBgKN6O7XCwKz9P38DALl3I74DwLWxI74D6Nz%2f2bCBFC%2bM9glZmEyM%2byOCTZg&UserName=user01&Password=password99&LoginButton=Log+in
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:36:55 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Default.aspx?locale=en
Set-Cookie: .EXTASPXAUTH=65BB5BFDD274F730E26CAEAAEB417792A764E7B8E8C6C9AC8C47FA97EF35DFACF551A53EAA6EA67D868C8A9BF55EBA758A5E724C58269028EE48F56268A204CBED19B60FC1AF58892989D9546202C037E97BF0EEE6A6281FF5EEA461BC30C5C7A71DFD64027AEB796D3FD21AE97ECFB16FF0F95C; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 140
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Note that the above response includes an authenication cookie.
Now we run the following line of code with the intention of getting that cookie:
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
But instead the following request/response is generated in fiddler:
Request:
GET http://withtheld/Default.aspx?locale=en HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:37:38 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Login.aspx?ReturnUrl=%2fDefault.aspx%3flocale%3den&locale=en
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 182
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
I believe this is the response that httpResponse now contains. How can I actually get the cookie to request up another protected page after the login is done?
Thanks!

Well it turns out I had two problems here. One is that I needed to call this:
request.AllowAutoRedirect = false
This keeps the framework from just skipping the response that had the authentication cookie in it, and actually gives back the response we're interested in.
The other issue was that you have to create a new instance of a CookieContainer and assign it to the request. Without having done that Response.Cookies contains no cookies. As soon as you've assigned your own container it's populated after the response is made. I don't know why.

When you login, the authentication token will be sent as a cookie from the server to your client. You will need to store this cookie, and then re-send it with every future request. Re-sending the cookie tells the server that you have been authenticated as a certain user.
To get the cookies that were sent in response after you logged in:
HttpCookieCollection loginResponseCookies = Response.Cookies;
This collection will include the auth cookie, any session cookies, etc.
Then, just re-send these cookies with every subsequent request, and the server will authenticate you as a result.
foreach(HttpCookie loginResponseCookie in loginResponseCookies)
{
Response.Cookies.Add(loginResponseCookie);
}

HTTPS problems with cookies C#

OK, now that I have investigated the code, there are no cookies to speak of when it comes to storing the cookie collection in a CookieContainer, so I would like to divert and use the headers. The only problem is I do not understand how to use them so that I may download the file form the website.
Could someone give me an example of how I would potentially use headers?
Also the code I used for the cookies is as follows, maybe there is a mistake:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
CookieContainer newCookies = new CookieContainer();
newCookies.Add(response.Cookies);
All I get is this header Headers = {Content-Length: 6292
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sun, 22 Jul 2012 03:12:59 GMT
Set-Cookie: ASP.NET_SessionId=dpub2i550ynjfonuv0o0n4nb; domain=website.co.nz; path=/; HttpOnly
Server: Microsoft-IIS/6.0
X-AspNet-Vers...
The code does not throw an exception. Just as a side not I am using request.Method = "GET";.

That's the result of the .ToString() method which is called when viewing the variable in the debugger. You need to access the members of CookieContainer.

c# HttpWebRequest headers

Why the location is not listed on the response Headers?
My code:
string url = "http://hehe.freevar.com/files.php";
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "HEAD";
Console.WriteLine(req.GetResponse().Headers);

From Wikipedia:
The HTTP Location header is returned
in responses from an HTTP server under
two circumstances:
To force a web browser to load a different web page. It is passed as
part of the response by a web server
when the requested URI has:
Moved temporarily, or
Moved permanently
The HttpWebRequest class has a property AllowAutoRedirect which defaults to true:
Set AllowAutoRedirect to true if you
want the request to automatically
follow HTTP redirection headers to the
new location of the resource.
This means you will never see the redirect request unless you set AllowAutoRedirect to false before making the request:
string url = "http://hehe.freevar.com/files.php";
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.AllowAutoRedirect = false;
req.Method = "HEAD";
Console.WriteLine(req.GetResponse().Headers);
Then you get the following response which does include Location:
Keep-Alive: timeout=1, max=10000
Connection: Keep-Alive
Content-Type: text/html
Date: Wed, 01 Jun 2011 01:32:18 GMT
Location: http://www.160by2.com/post-registration.aspx
Server: Apache
X-Powered-By: PHP/5.2.13

I think you are looking for ResponseUri property.
var responseUri = req.GetResponse().ResponseUri;

Get the .ASPXAUTH cookie value programmatically

Is there a way to get the .ASPXAUTH value programmatically.
Example I login to a website with my own credentials (POST) and then read the response...it does not return the .APSXAUTH in the CookieContainer that I use to track the session.
Anyone has a clue how can I get it and send it with the subsequent gets and posts?
[EDIT] Here's what I do to be more specific:
send a HTTP GET to a page. read values like _VIEWSTATE etc.
send a HTTP POST to the Login page. It includes the login information.
The server sends a 302 response (redirect) to some Default page. The forms authentication cookie is supposed to be included but it's not.
So I was thinking that there might be a better way than this to track session:
CookieContainer _cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);
request.CookieContainer = _cookieJar;

So the summarize the answer:
If you're trying to login programatically on a Forms based authentication website trough your own application make sure you follow the steps you take that track the cookies.
First create a initial GET request, and then do the subsequential POST requests that will do the postback.The request and the responses should be formulated in this way:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);
request.CookieContainer = _cookieJar;
HttpWebResponse httpsResponse = (HttpWebResponse)request.GetResponse();
The CookieContainer class handles the cookies as expected.
And if your response is encoded with Gzip just include the following line:
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
before you call request.GetResponse()
Hope this helps someone out there.