Why the location is not listed on the response Headers?
My code:
string url = "http://hehe.freevar.com/files.php";
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "HEAD";
Console.WriteLine(req.GetResponse().Headers);
From Wikipedia:
The HTTP Location header is returned
in responses from an HTTP server under
two circumstances:
To force a web browser to load a different web page. It is passed as
part of the response by a web server
when the requested URI has:
Moved temporarily, or
Moved permanently
The HttpWebRequest class has a property AllowAutoRedirect which defaults to true:
Set AllowAutoRedirect to true if you
want the request to automatically
follow HTTP redirection headers to the
new location of the resource.
This means you will never see the redirect request unless you set AllowAutoRedirect to false before making the request:
string url = "http://hehe.freevar.com/files.php";
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.AllowAutoRedirect = false;
req.Method = "HEAD";
Console.WriteLine(req.GetResponse().Headers);
Then you get the following response which does include Location:
Keep-Alive: timeout=1, max=10000
Connection: Keep-Alive
Content-Type: text/html
Date: Wed, 01 Jun 2011 01:32:18 GMT
Location: http://www.160by2.com/post-registration.aspx
Server: Apache
X-Powered-By: PHP/5.2.13
I think you are looking for ResponseUri property.
var responseUri = req.GetResponse().ResponseUri;
Related
I am trying to call a REST service from a C# ASP.NET 4.0 application using RestSharp.
It's a fairly straightforward POST call to a https:// address; my code is something like this (CheckStatusRequest is a plain simple DTO with about four or five string and int properties - nothing fancy):
public CheckStatusResponse CheckStatus(CheckStatusRequest request) {
// set up RestClient
RestClient client = new RestClient();
string uri = "https://.......";
// create the request (see below)
IRestRequest restRequest = CreateRequestWithHeaders(url, Method.POST);
// add the body to the request
restRequest.AddBody(request);
// execute call
var restResponse = _restClient.Execute<CheckStatusResponse>(restRequest);
}
// set up request
private IRestRequest CreateRequestWithHeaders(string uri, Method method) {
// define request
RestRequest request = new RestRequest(uri, method);
// add two required HTTP headers
request.AddHeader("Accept", "application/json");
request.AddHeader("Content-Type", "application/json");
// define JSON as my format
request.RequestFormat = DataFormat.Json;
// attach the JSON.NET serializer for RestSharp
request.JsonSerializer = new RestSharpJsonNetSerializer();
return request;
}
The problem I'm having when I send these requests through Fiddler to see what's going on is that my request suddenly gets a third and unwanted HTTP header:
POST https://-some-url- HTTP/1.1
Accept: application/json
User-Agent: RestSharp/104.4.0.0
Content-Type: application/json
Host: **********.com
Content-Length: 226
Accept-Encoding: gzip, deflate <<<=== This one here is UNWANTED!
Connection: Keep-Alive
I suddenly have that Accept-Encoding HTTP header, which I never specified (and which I don't want to have in there). And now my response is no longer proper JSON (which I'm able to parse), but suddenly I get back gzipped binary data instead (which doesn't do real well when trying to JSON-deserialize)....
How can I get rid of that third unwanted HTTP header?
I tried to set it to something else - whatever I enter just gets appended to those settings
I tried to somehow "clear" that HTTP header - without any success
I tried finding a property on the RestClient or the RestRequest classes to specify "do not use GZip"
Looking at the sources (Http.Sync.cs and Http.Async.cs) of RestSharp you can see that these values are hardcoded:
webRequest.AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
There is also an open issue that describes this problem. It was opened August 2014 but still not solved. I think you can leave a comment there and maybe they will pay attention.
From Fiddler , I can see the Cookie in Request Header
However, when I tried to parse the Cookie in C# by webresponse HTTP,
var webResponse = (HttpWebResponse)WebRequestBuilder
.GetRequest(new Uri(url))
.AllowAutoRedirect(false)
.GetResponse();
var test = webResponse.Headers;
in the test, which is returned the headers containing the Set-Cookie, what I get is just :
ASP.NET_SessionId=nod1bi3znubfdf55fzre2hye; path=/;
HttpOnly,host=SCANDICWEB203; path=/; HttpOnly,ieAlertDisplay=true;
path=/,scandic_lb_cookie=1733559050.0.0000; path=/
Not all of the Cookies were returned like evar58,s_sv_sid, scandic_lb_cookie.... I need all of them in order to push those cookies to execute another url.
So what I have to do to get all the cookie from request header like Fiddler in C#?
I was hoping the following code would yield a non-zero value:
var webRequest = (HttpWebRequest) WebRequest.Create(loginUrl);
var webResponse = (HttpWebResponse)webRequest.GetResponse();
Console.WriteLine(webResponse.Cookies.Count);
yet no cookies seem to show up in webRespone.Cookies. I'm positive the cookies are there, as I'm sniffing the data with Fiddler. This is the response I'm getting:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
Set-Cookie: __Abc=Def; path=/; HttpOnly
PS-ResponseTime: 00:00:00.0624001
PS-Build: 2013-03-19-11-36-59
PS-Node: 02
Date: Tue, 19 Mar 2013 21:14:51 GMT
Content-Length: 57872
Does it have anything to do with the fact the cookie is HttpOnly?
Edit
It's seems I can get them through HttpWebRequest's CookieContainer which is certainly useful if I intend to proceed to a sequence of requests/responses. But why can't I access them the same through the HttpWebResponse.Cookies field, anyway?
Thanks
You're right, you'll need to hand the request an instance of CookieContainer and then reference that instance to see the cookies. Basically, HttpWebResponse does not directly expose cookies on the response that are marked as HttpOnly.
So I'm in the process of attempting to simulate multiple logins all generating exceptions at the same time on our corporate website for the purpose of testing our logging framework (which we think there may be issues with thread synchronization). Anyway, so I need to log in to our website programatically. Here's what I have so far:
// Block 1
Uri url = new Uri("http://withheld");
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string viewState = string.Empty;
string previousPage = string.Empty;
string eventValidation = string.Empty;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string strResponse = reader.ReadToEnd();
viewState = HttpUtility.UrlEncode(GetTagValue(strResponse, "__VIEWSTATE"));
previousPage = HttpUtility.UrlEncode(GetTagValue(strResponse, "__PREVIOUSPAGE"));
eventValidation = HttpUtility.UrlEncode(GetTagValue(strResponse, "__EVENTVALIDATION"));
}
// Block 2
string username = "user01";
string password = "password99";
HttpWebRequest request2 = WebRequest.Create(url) as HttpWebRequest;
request2.KeepAlive = true;
request2.Method = "POST";
request2.ContentType = "application/x-www-form-urlencoded";
string postData = string.Format("__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE={0}&__PREVIOUSPAGE={1}&__EVENTVALIDATION={2}&UserName={3}&Password={4}&LoginButton=Log+in", new string[] { viewState, previousPage, eventValidation, username, password});
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request2.ContentLength = dataBytes.Length;
using (Stream postStream = request2.GetRequestStream())
{
// Here's the problem
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
// At this point httpResponse.Cookies is null
// I believe it's because the line above has actually initiated another
// request/response which DOES NOT include the authentication cookie.
// See fiddler output below to understand why I think that.
// Block 3
//Uri url2 = new Uri("http://Withheld/GenerateException.aspx");
//http = WebRequest.Create(url2) as HttpWebRequest;
//http.CookieContainer = new CookieContainer();
//http.CookieContainer.Add(httpResponse.Cookies);
//HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Looks reasonably straightforward right? Well it doesn't work. I don't know whether I need the viewState and whatnot or not, but I figured I'd mimic what a regular browser does as closely as possible.
Near as I can tell what's happening is this:
We hit the page with a simple GET. That gives us the viewstate, etc which is parsed out to be included in the following request.
We now post the viewstate, username, password, etc to the server using postStream.Write(). The server at this point responds with an authentication cookie, and forwards us off to /Default.aspx.
Now we execute reqest2.GetResponse(), but instead of getting the response that forwarded us to /default.aspx and had he authentication cookie, it looks like this line is actually causing ANOTHER request to get us the resource at /default.aspx. The problem is httpWebResponse DOES NOT include the authentication cookie we need for the next request (which is commented out currently).
Why? Why is this behaving in this manor, and how do I handle it properly. Here's the output from fiddler to further explain what's going on:
Block 1 generates this request/response
Request Header:
GET http://withheld/Login.aspx HTTP/1.1
Host: withheld
Connection: Keep-Alive
Response Header:
HTTP/1.1 200 OK
Connection: close
Date: Mon, 04 Feb 2013 16:37:37 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: .EXTASPXAUTH=; expires=Tue, 12-Oct-1999 04:00:00 GMT; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 16975
Response is the actual login webpage. Omitted for obvious reasons.
Stepping through the code, this request/response is generated immediately following the call to postStream.Write:
Request:
POST http://Withheld/Login.aspx HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Content-Length: 2109
Expect: 100-continue
__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2fwEPDwUJNTE1NTIxNjkxD2QWAgIDD2QWBgIDDxYCHgVjbGFzcwUQaGVhZGVyX2Jhbm5lcl9lbhYCAgEPFgQeB29uY2xpY2sFI3dpbmRvdy5sb2NhdGlvbj0naHR0cDovL3d3dy5tZG0uY2EnHwAFFmhlYWRlcl9iYW5uZXJfZW5fc21hbGxkAgUPZBYQAgcPDxYEHhdQYXJ0aWFsUmVuZGVyaW5nQ2hlY2tlZGceGUlzUGFydGlhbFJlbmRlcmluZ0VuYWJsZWRoZGQCCQ8PFgQfAmcfA2hkZAIPDw8WBB8CZx8DaGRkAhEPDxYCHg1PbkNsaWVudENsaWNrBcEBamF2YXNjcmlwdDpQYWdlX0NsaWVudFZhbGlkYXRlKCdMb2dpbkN0cmwnKTsgaWYgKFBhZ2VfSXNWYWxpZCkgeyBMb2dpbkJ1dHRvbi5zdHlsZS52aXNpYmlsaXR5ID0gJ2hpZGRlbic7IExvZ2luQnV0dG9uLnN0eWxlLmRpc3BsYXkgPSAnbm9uZSc7IExvZ2luQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJzsgfWRkAhkPDxYCHgRUZXh0Bc0BSWYgeW91IHJlcXVpcmUgYXNzaXN0YW5jZSwgb3VyIE1EIE9ubGluZSBTdXBwb3J0IFNwZWNpYWxpc3RzIGNhbiBiZSByZWFjaGVkIGF0ICg4NzcpIDQzMS0wMzMwIG9yIGJ5IGUtbWFpbCBhdCA8YSBocmVmPSJtYWlsdG86d2Vic3VwcG9ydEBjbWEuY2EiPndlYnN1cHBvcnRAY21hLmNhPC9hPi48YnI%2bPGJyPldlIGVuY291cmFnZSB5b3UgdG8gcmV2aWV3IHRoZWRkAhsPDxYCHwQFOXdpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvbGVnYWwvJyk7cmV0dXJuIGZhbHNlO2RkAh8PDxYCHwQFRndpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvc3lzdGVtLWVuaGFuY2VtZW50LycpO3JldHVybiBmYWxzZTtkZAIhDw8WAh8EBUZ3aW5kb3cub3BlbignaHR0cDovLzE5Mi4xNjguMTY1LjMyL3N5c3RlbS1lbmhhbmNlbWVudC8nKTtyZXR1cm4gZmFsc2U7ZGQCCQ9kFgICAQ9kFgICAg9kFgJmD2QWAgIBD2QWAgIdDw8WAh8EBfsBamF2YXNjcmlwdDpjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uLnN0eWxlLnZpc2liaWxpdHkgPSAnaGlkZGVuJzsgY3RybENyZWF0ZU5ld1Bhc3N3b3JkX0NyZWF0ZU5ld1Bhc3N3b3JkUHVzaEJ1dHRvbi5zdHlsZS5kaXNwbGF5ID0gJ25vbmUnOyBjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJztkZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUMaWJ0bk1vcmVJbmZvdZmFXkMbPfVWPQYtreXvFt8Bck8%3d&__PREVIOUSPAGE=1aYW5DqTKrT4ieGPkHcnrQLIq8lEcSIVkql1EugwSQNV_5102t5D7QDmOnuQFA4Tz9Mh5-CEYpkRngMROFFeeAG12Ss1&__EVENTVALIDATION=%2fwEWCQKKr%2bXcBgKvruq2CALSxeCRDwL%2bjNCfDwKH8YSKBgKN6O7XCwKz9P38DALl3I74DwLWxI74D6Nz%2f2bCBFC%2bM9glZmEyM%2byOCTZg&UserName=user01&Password=password99&LoginButton=Log+in
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:36:55 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Default.aspx?locale=en
Set-Cookie: .EXTASPXAUTH=65BB5BFDD274F730E26CAEAAEB417792A764E7B8E8C6C9AC8C47FA97EF35DFACF551A53EAA6EA67D868C8A9BF55EBA758A5E724C58269028EE48F56268A204CBED19B60FC1AF58892989D9546202C037E97BF0EEE6A6281FF5EEA461BC30C5C7A71DFD64027AEB796D3FD21AE97ECFB16FF0F95C; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 140
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Note that the above response includes an authenication cookie.
Now we run the following line of code with the intention of getting that cookie:
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
But instead the following request/response is generated in fiddler:
Request:
GET http://withtheld/Default.aspx?locale=en HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:37:38 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Login.aspx?ReturnUrl=%2fDefault.aspx%3flocale%3den&locale=en
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 182
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
I believe this is the response that httpResponse now contains. How can I actually get the cookie to request up another protected page after the login is done?
Thanks!
Well it turns out I had two problems here. One is that I needed to call this:
request.AllowAutoRedirect = false
This keeps the framework from just skipping the response that had the authentication cookie in it, and actually gives back the response we're interested in.
The other issue was that you have to create a new instance of a CookieContainer and assign it to the request. Without having done that Response.Cookies contains no cookies. As soon as you've assigned your own container it's populated after the response is made. I don't know why.
When you login, the authentication token will be sent as a cookie from the server to your client. You will need to store this cookie, and then re-send it with every future request. Re-sending the cookie tells the server that you have been authenticated as a certain user.
To get the cookies that were sent in response after you logged in:
HttpCookieCollection loginResponseCookies = Response.Cookies;
This collection will include the auth cookie, any session cookies, etc.
Then, just re-send these cookies with every subsequent request, and the server will authenticate you as a result.
foreach(HttpCookie loginResponseCookie in loginResponseCookies)
{
Response.Cookies.Add(loginResponseCookie);
}
OK, now that I have investigated the code, there are no cookies to speak of when it comes to storing the cookie collection in a CookieContainer, so I would like to divert and use the headers. The only problem is I do not understand how to use them so that I may download the file form the website.
Could someone give me an example of how I would potentially use headers?
Also the code I used for the cookies is as follows, maybe there is a mistake:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
CookieContainer newCookies = new CookieContainer();
newCookies.Add(response.Cookies);
All I get is this header Headers = {Content-Length: 6292
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sun, 22 Jul 2012 03:12:59 GMT
Set-Cookie: ASP.NET_SessionId=dpub2i550ynjfonuv0o0n4nb; domain=website.co.nz; path=/; HttpOnly
Server: Microsoft-IIS/6.0
X-AspNet-Vers...
The code does not throw an exception. Just as a side not I am using request.Method = "GET";.
That's the result of the .ToString() method which is called when viewing the variable in the debugger. You need to access the members of CookieContainer.