OK, now that I have investigated the code, there are no cookies to speak of when it comes to storing the cookie collection in a CookieContainer, so I would like to divert and use the headers. The only problem is I do not understand how to use them so that I may download the file form the website.
Could someone give me an example of how I would potentially use headers?
Also the code I used for the cookies is as follows, maybe there is a mistake:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
CookieContainer newCookies = new CookieContainer();
newCookies.Add(response.Cookies);
All I get is this header Headers = {Content-Length: 6292
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sun, 22 Jul 2012 03:12:59 GMT
Set-Cookie: ASP.NET_SessionId=dpub2i550ynjfonuv0o0n4nb; domain=website.co.nz; path=/; HttpOnly
Server: Microsoft-IIS/6.0
X-AspNet-Vers...
The code does not throw an exception. Just as a side not I am using request.Method = "GET";.
That's the result of the .ToString() method which is called when viewing the variable in the debugger. You need to access the members of CookieContainer.
Related
I tried to send a cookie to Server. But when it does, it throws a weird exception I can not trace:
Message "The operation identifier is not valid. (Exception from HRESULT: 0x800710DD)" string
StackTrace " at Windows.Web.Http.Filters.HttpBaseProtocolFilter.SendRequestAsync(HttpRequestMessage request)\r\n at System.Net.Http.HttpHandlerToFilter.<SendAsync>d__1.MoveNext()" string
The line where the exception is thrown:
Request.BeginGetResponse( new AsyncCallback( GetResponseCallback ) , Request );
At the beginning I've never thought it was a cookie issue until I commented out this line:
WCRequest.CookieContainer = WBackgroundTransfer.Cookies;
The exception is gone if the line above is commented. So I tried another approach:
CookieContainer CC = new CookieContainer();
CC.Add( new Cookie( "aa", "bb" ) );
WCRequest.CookieContainer = CC;
The above line does not throw any exception. Then I proceed to modify my code:
CookieContainer CC = new CookieContainer();
foreach( Cookie C in WBackgroundTransfer.Cookies.GetCookies( ReqURI ) )
{
CC.Add( new Cookie( C.Name, C.Value, C.Path, C.Domain ) );
}
WCRequest.CookieContainer = CC;
And the exception is back!
Then I tried to modify the code again:
WCRequest.Headers[ HttpRequestHeader.Cookie ] = WBackgroundTransfer.Cookies.GetCookieHeader( ReqUri );
The exception still persist. The I modify the code to:
WCRequest.Headers[ HttpRequestHeader.Cookie ] = "ab=cd";
This works!
So I think some of the cookie is causing this issue. I proceed to compare the server response.
The exact cookie I am sending to server is:
PHPSESSID=j0g94fgvum0flhlop5868gjkklc5bh1v
When I send the above cookie to server, the response is:
HTTP/1.1 200 OK
Date: Fri, 06 Nov 2015 16:51:13 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dd4e3badd19f3f67e86fe1431b7fe895a1446828673; expires=Sat, 05-Nov-16 16:51:13 GMT; path=/; domain=.example.com; HttpOnly
Set-Cookie: PHPSESSID=j0g94fgvum0flhlop5868gjkklc5bh1v; path=/; HttpOnly
Vary: Accept-Encoding
Server: nginx
CF-RAY: 24124d464d493439-HKG
When I send a random, irrelevant cookie, or not give any cookie to the sever:
HTTP/1.1 200 OK
Date: Fri, 06 Nov 2015 16:53:24 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d0a29df98a2984c4f5c4b2ddbc900fde01446828804; expires=Sat, 05-Nov-16 16:53:24 GMT; path=/; domain=.example.com; HttpOnly
Vary: Accept-Encoding
Server: nginx
CF-RAY: 2412507ad4c6330d-HKG
The only difference of the two request is this line:
Set-Cookie: PHPSESSID=j0g94fgvum0flhlop5868gjkklc5bh1v; path=/; HttpOnly
Which tells me to set the cookie where I sent to it.
This is very frustrating. Is this an API bug? Or am I missing something? I cannot modify the sever behaviour so I am not 100% sure this is the cause.
Any ideas?
Update: This only happens in x64 platform! I am starting to think maybe this is a heisengbug...
Update2: No, this happens on all platforms. I just didn't test it thoroughly:(
Update3: This is hopeless. I am going to try switching to socket to see if it helps
Have you checked this solution: https://stackoverflow.com/a/33863365
You can remove the extra cookies you don't need.
HttpBaseProtocolFilter filter = new HttpBaseProtocolFilter();
HttpCookieManager cookieManager = filter.CookieManager;
foreach (HttpCookie cookie in cookieManager.GetCookies(uri))
{
cookieManager.DeleteCookie();
}
If anyone comes into this issue. For me it is magically resolved by clearing the generated libraries inside the project. i.e. Remove the following directories:
obj/*
bin/*
Debug/*
ARM/*
Release/*
x64/*
Update 1: I just noticed that the HttpWebRequest is Setting the Cookies internally the sending them automatically. Was it supposed to do that?
Okay, so I have a web api controller with put action and a return type of void. When I run it using VS's builtin iisepxress and call it, I get back a 204 as expected. The here are the headers:
Cache-Control no-cache
Connection close
Content-Type text/html
Date Thu, 10 Oct 2013 19:33:43 GMT
Expires -1
Pragma no-cache
Server Microsoft-IIS/8.0
X-AspNet-Version 4.0.30319
X-Powered-By ASP.NET
When I put the exact same code in our sbx environment, I get a 204, but with the following headers:
Cache-Control no-cache
Date Thu, 10 Oct 2013 19:39:59 GMT
Expires -1
Pragma no-cache
Server Microsoft-IIS/7.5
X-AspNet-Version 4.0.30319
X-Identifier 17253
X-Powered-By ASP.NET
The pertinent difference being the lack of contentType in the second one.
The problem this creates is that in firefox (and IE I think) it defaults to xml, tries to parse it and fails.
I know how to fix this by setting my contentType in my web api controller, but that doesn't seem like the best fix to me.
So, what I'm asking is what setting difference in IIS might be causing this?
Thanks
Note:
My url looks like this /foo/bar/2 so it isn't mimetype.
If your service is responding with a 204, the response should not contain a message-body. This is by spec. I can only assume you are responding with something in your message body.
Your response from the API method should be like this:
return new HttpResponseMessage { StatusCode = System.Net.HttpStatusCode.NoContent }
Edit. I noticed you mentioned you return "void". Your method should return HttpResponseMessage with the StatusCode I noted above.
That will solve the isssue:
protected internal virtual IHttpActionResult NoContent()
{
HttpResponseMessage responseMsg = new HttpResponseMessage(HttpStatusCode.NoContent) {Content = new StringContent(string.Empty, Encoding.UTF8)};
return this.ResponseMessage(responseMsg);
}
But still doesn't explain why IIS is adding by default:
Content-Type text/html
Or even better how to remove it using web.config or IIS config.
I use this:
Return StatusCode(HttpStatusCode.NoContent);
I was hoping the following code would yield a non-zero value:
var webRequest = (HttpWebRequest) WebRequest.Create(loginUrl);
var webResponse = (HttpWebResponse)webRequest.GetResponse();
Console.WriteLine(webResponse.Cookies.Count);
yet no cookies seem to show up in webRespone.Cookies. I'm positive the cookies are there, as I'm sniffing the data with Fiddler. This is the response I'm getting:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
Set-Cookie: __Abc=Def; path=/; HttpOnly
PS-ResponseTime: 00:00:00.0624001
PS-Build: 2013-03-19-11-36-59
PS-Node: 02
Date: Tue, 19 Mar 2013 21:14:51 GMT
Content-Length: 57872
Does it have anything to do with the fact the cookie is HttpOnly?
Edit
It's seems I can get them through HttpWebRequest's CookieContainer which is certainly useful if I intend to proceed to a sequence of requests/responses. But why can't I access them the same through the HttpWebResponse.Cookies field, anyway?
Thanks
You're right, you'll need to hand the request an instance of CookieContainer and then reference that instance to see the cookies. Basically, HttpWebResponse does not directly expose cookies on the response that are marked as HttpOnly.
So I'm in the process of attempting to simulate multiple logins all generating exceptions at the same time on our corporate website for the purpose of testing our logging framework (which we think there may be issues with thread synchronization). Anyway, so I need to log in to our website programatically. Here's what I have so far:
// Block 1
Uri url = new Uri("http://withheld");
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string viewState = string.Empty;
string previousPage = string.Empty;
string eventValidation = string.Empty;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string strResponse = reader.ReadToEnd();
viewState = HttpUtility.UrlEncode(GetTagValue(strResponse, "__VIEWSTATE"));
previousPage = HttpUtility.UrlEncode(GetTagValue(strResponse, "__PREVIOUSPAGE"));
eventValidation = HttpUtility.UrlEncode(GetTagValue(strResponse, "__EVENTVALIDATION"));
}
// Block 2
string username = "user01";
string password = "password99";
HttpWebRequest request2 = WebRequest.Create(url) as HttpWebRequest;
request2.KeepAlive = true;
request2.Method = "POST";
request2.ContentType = "application/x-www-form-urlencoded";
string postData = string.Format("__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE={0}&__PREVIOUSPAGE={1}&__EVENTVALIDATION={2}&UserName={3}&Password={4}&LoginButton=Log+in", new string[] { viewState, previousPage, eventValidation, username, password});
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request2.ContentLength = dataBytes.Length;
using (Stream postStream = request2.GetRequestStream())
{
// Here's the problem
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
// At this point httpResponse.Cookies is null
// I believe it's because the line above has actually initiated another
// request/response which DOES NOT include the authentication cookie.
// See fiddler output below to understand why I think that.
// Block 3
//Uri url2 = new Uri("http://Withheld/GenerateException.aspx");
//http = WebRequest.Create(url2) as HttpWebRequest;
//http.CookieContainer = new CookieContainer();
//http.CookieContainer.Add(httpResponse.Cookies);
//HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Looks reasonably straightforward right? Well it doesn't work. I don't know whether I need the viewState and whatnot or not, but I figured I'd mimic what a regular browser does as closely as possible.
Near as I can tell what's happening is this:
We hit the page with a simple GET. That gives us the viewstate, etc which is parsed out to be included in the following request.
We now post the viewstate, username, password, etc to the server using postStream.Write(). The server at this point responds with an authentication cookie, and forwards us off to /Default.aspx.
Now we execute reqest2.GetResponse(), but instead of getting the response that forwarded us to /default.aspx and had he authentication cookie, it looks like this line is actually causing ANOTHER request to get us the resource at /default.aspx. The problem is httpWebResponse DOES NOT include the authentication cookie we need for the next request (which is commented out currently).
Why? Why is this behaving in this manor, and how do I handle it properly. Here's the output from fiddler to further explain what's going on:
Block 1 generates this request/response
Request Header:
GET http://withheld/Login.aspx HTTP/1.1
Host: withheld
Connection: Keep-Alive
Response Header:
HTTP/1.1 200 OK
Connection: close
Date: Mon, 04 Feb 2013 16:37:37 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: .EXTASPXAUTH=; expires=Tue, 12-Oct-1999 04:00:00 GMT; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 16975
Response is the actual login webpage. Omitted for obvious reasons.
Stepping through the code, this request/response is generated immediately following the call to postStream.Write:
Request:
POST http://Withheld/Login.aspx HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Content-Length: 2109
Expect: 100-continue
__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2fwEPDwUJNTE1NTIxNjkxD2QWAgIDD2QWBgIDDxYCHgVjbGFzcwUQaGVhZGVyX2Jhbm5lcl9lbhYCAgEPFgQeB29uY2xpY2sFI3dpbmRvdy5sb2NhdGlvbj0naHR0cDovL3d3dy5tZG0uY2EnHwAFFmhlYWRlcl9iYW5uZXJfZW5fc21hbGxkAgUPZBYQAgcPDxYEHhdQYXJ0aWFsUmVuZGVyaW5nQ2hlY2tlZGceGUlzUGFydGlhbFJlbmRlcmluZ0VuYWJsZWRoZGQCCQ8PFgQfAmcfA2hkZAIPDw8WBB8CZx8DaGRkAhEPDxYCHg1PbkNsaWVudENsaWNrBcEBamF2YXNjcmlwdDpQYWdlX0NsaWVudFZhbGlkYXRlKCdMb2dpbkN0cmwnKTsgaWYgKFBhZ2VfSXNWYWxpZCkgeyBMb2dpbkJ1dHRvbi5zdHlsZS52aXNpYmlsaXR5ID0gJ2hpZGRlbic7IExvZ2luQnV0dG9uLnN0eWxlLmRpc3BsYXkgPSAnbm9uZSc7IExvZ2luQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJzsgfWRkAhkPDxYCHgRUZXh0Bc0BSWYgeW91IHJlcXVpcmUgYXNzaXN0YW5jZSwgb3VyIE1EIE9ubGluZSBTdXBwb3J0IFNwZWNpYWxpc3RzIGNhbiBiZSByZWFjaGVkIGF0ICg4NzcpIDQzMS0wMzMwIG9yIGJ5IGUtbWFpbCBhdCA8YSBocmVmPSJtYWlsdG86d2Vic3VwcG9ydEBjbWEuY2EiPndlYnN1cHBvcnRAY21hLmNhPC9hPi48YnI%2bPGJyPldlIGVuY291cmFnZSB5b3UgdG8gcmV2aWV3IHRoZWRkAhsPDxYCHwQFOXdpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvbGVnYWwvJyk7cmV0dXJuIGZhbHNlO2RkAh8PDxYCHwQFRndpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvc3lzdGVtLWVuaGFuY2VtZW50LycpO3JldHVybiBmYWxzZTtkZAIhDw8WAh8EBUZ3aW5kb3cub3BlbignaHR0cDovLzE5Mi4xNjguMTY1LjMyL3N5c3RlbS1lbmhhbmNlbWVudC8nKTtyZXR1cm4gZmFsc2U7ZGQCCQ9kFgICAQ9kFgICAg9kFgJmD2QWAgIBD2QWAgIdDw8WAh8EBfsBamF2YXNjcmlwdDpjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uLnN0eWxlLnZpc2liaWxpdHkgPSAnaGlkZGVuJzsgY3RybENyZWF0ZU5ld1Bhc3N3b3JkX0NyZWF0ZU5ld1Bhc3N3b3JkUHVzaEJ1dHRvbi5zdHlsZS5kaXNwbGF5ID0gJ25vbmUnOyBjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJztkZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUMaWJ0bk1vcmVJbmZvdZmFXkMbPfVWPQYtreXvFt8Bck8%3d&__PREVIOUSPAGE=1aYW5DqTKrT4ieGPkHcnrQLIq8lEcSIVkql1EugwSQNV_5102t5D7QDmOnuQFA4Tz9Mh5-CEYpkRngMROFFeeAG12Ss1&__EVENTVALIDATION=%2fwEWCQKKr%2bXcBgKvruq2CALSxeCRDwL%2bjNCfDwKH8YSKBgKN6O7XCwKz9P38DALl3I74DwLWxI74D6Nz%2f2bCBFC%2bM9glZmEyM%2byOCTZg&UserName=user01&Password=password99&LoginButton=Log+in
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:36:55 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Default.aspx?locale=en
Set-Cookie: .EXTASPXAUTH=65BB5BFDD274F730E26CAEAAEB417792A764E7B8E8C6C9AC8C47FA97EF35DFACF551A53EAA6EA67D868C8A9BF55EBA758A5E724C58269028EE48F56268A204CBED19B60FC1AF58892989D9546202C037E97BF0EEE6A6281FF5EEA461BC30C5C7A71DFD64027AEB796D3FD21AE97ECFB16FF0F95C; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 140
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Note that the above response includes an authenication cookie.
Now we run the following line of code with the intention of getting that cookie:
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
But instead the following request/response is generated in fiddler:
Request:
GET http://withtheld/Default.aspx?locale=en HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:37:38 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Login.aspx?ReturnUrl=%2fDefault.aspx%3flocale%3den&locale=en
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 182
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
I believe this is the response that httpResponse now contains. How can I actually get the cookie to request up another protected page after the login is done?
Thanks!
Well it turns out I had two problems here. One is that I needed to call this:
request.AllowAutoRedirect = false
This keeps the framework from just skipping the response that had the authentication cookie in it, and actually gives back the response we're interested in.
The other issue was that you have to create a new instance of a CookieContainer and assign it to the request. Without having done that Response.Cookies contains no cookies. As soon as you've assigned your own container it's populated after the response is made. I don't know why.
When you login, the authentication token will be sent as a cookie from the server to your client. You will need to store this cookie, and then re-send it with every future request. Re-sending the cookie tells the server that you have been authenticated as a certain user.
To get the cookies that were sent in response after you logged in:
HttpCookieCollection loginResponseCookies = Response.Cookies;
This collection will include the auth cookie, any session cookies, etc.
Then, just re-send these cookies with every subsequent request, and the server will authenticate you as a result.
foreach(HttpCookie loginResponseCookie in loginResponseCookies)
{
Response.Cookies.Add(loginResponseCookie);
}
I'm scrapping a page which is the result of a redirect: I visit page1 first, then it redirects to page2 via http-equiv="refresh". I'm scrapping page2. Content on page2 is based on some cookies page1 sets. I see page1 returns 2 cookies, but when I request page 2 (sending the same CookieContainer, one cookie is missing. What's wrong in my code?
Thank you:
First :
I create a CookieContainer and an HttpWebRequest and request for page1.
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(eQuery);
req.AllowAutoRedirect = true; // but it doesn't autoredirects the meta-refresh
req.CookieContainer = cookiesContainer;
This is the result I get this from visiting page1
HTTP/1.1 200 OK
Date: Tue, 12 Apr 2011 19:14:06 GMT
Server: (...)
Set-Cookie: NAME1=VALUE1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: NAME2=VALUE2; expires=Wed, 13-Apr-2011 19:14:06 GMT
Content-Length: 174
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
(...)
Everything is fine so far, I get two cookies are there and I get two cookie objects within the container.
Then I parse the "content" value of the meta http-equiv for the next url. And request it using a similar code and using the same container. But only one cookie is sent. Here is the HTTP sent:
GET DETECTED_URL_IN_HTTP_EQUIV_REFRESH HTTP/1.1
User-Agent: (...)
Host: example.com
Cookie: NAME1=VALUE1
As you see cookie NAME2 is missing. Why is that happening? is something related differences in the two cookies (one has path and other has expiration date)?
Any idea how can I pass the two cookies?
PS: I don't have access to page1, so I can't set path, or expiration for cookies. I'm scrapping those pages.
Thank you.
If you don't specify a path on your cookie it will default to the path it was requested on. So if you received a cookie on this request with no path declaration:
http://contoso.com/subfolder/test.aspx
The browser would only send back that cookie for more requests in the /subfolder/ directory. To have the browser send it back for all paths you need to include path=/ when setting the cookie.