I'm using the HttpClient (System.Net.Http.HttpCient) to send some requests and I'm also using a CookieContainer to hande Cookies. For some Webpages everything works fine, but on some other pages no cookies are stored, although my browser saves the Cookies when I visit the webpage.
Can someone here explain what's the problem.
ceddy
Maybe these pages redirect to an other url? Cookies are stored per url and hence it's possible you "loose" a cookie.
To verify the behavior you may set
request.AllowAutoRedirect = false;
and look at the response object about what's going on. If this is really the issue in your case, you can copy the cookies from one url to the other via the CookieContainer.
Related
I've used several HttpWebRequest's in the past but they've all been used to login into a site.
I was wondering how does one make the WebRequest mimic a WebBrowser as in once you're logged in, navigate to a new page, maybe perform an action there, then go to a different page?
I've researched a little about this before and I think it might involve using the prior request's cookies or something.
My question is how do I (I'm assuming) get the cookies from the previous session, then navigate to a page, or complete an action as if we were still on the last request if that makes sense.
the HttpWebRequest has a Cookies property and HttpWebResponse has a CookieContainer property.
you record the cookies from the container, and add them to the next request.
you may also need to set the HTTP referrer header field on the request object.
EDIT :
this will still not get you mimicking a web browser. things like JavaScript will not work/run. and you won't have a DOM to work against.
I'm trying to scrape a web app that uses a few redirects and logouts/logins in between requests. I believe I'll have to set AllowAutoRedirect to false so I can capture the redirect requests and manually redirect while watching for new cookies. My only experience with cookies is to just set the container and forget about it... do I have to parse the response headers to decide actions to take with cookies? Can someone lay out a general approach?
Update
It turns out that Chris was right. The redirects and cookies were working just fine... the application I was hitting did not like that I was not setting all the right headers (content type, user agent). After adding those in, I'm getting the response I expect.
You got the CookieContainer part right which most people miss so kudos to you!
AllowAutoRedirect should pick up cookies as redirects happen. Is there a reason that you need to manually process things?
I set a cookie like this in one page:
Request.Cookies["lang"].Value = "en-US";
Request.Cookies["lang"].Expires = DateTime.Now.AddDays(50);
On another page I try and read the cookie:
string lang = Server.HtmlEncode(Request.Cookies["lang"].Value);
The cookie is not null but the value is an empty string. What am I doing wrong?
You should be using Response.Cookies to set the cookie, and Request.Cookies to read any cookies sent back from the client.
The code in your question is setting the cookie in the Request object, not the Response.
Are cookies enabled on the client? The fact that you set a cookie doesn't mean that the client supports them and will send them back.
Remember, you're dealing with two disconnected systems; your server doesn't keep state and you know little about the client.
If I remember correctly I think you should be using response instead of request as request is what is being sent to you. Response is when you want to set something back to the client browser.
EDIT: What you are doing is modifying the cookies in that particular request which would make sense why you are not seeing on subsequent pages. That is not saving them back to the client.
i have created an application with the help of webrequest and webresponse . when i try to log in to yahoo i succeed. after that i open the next page and i get the login page again by the response. How can I fix this?
If I'd have to guess (which I do, given the current state of the question), I'd say you have neglected to support cookies in your request.
I think you already answered your own question, almost anyway (you guessed this was session-related). You need to keep track of the cookies that the initial login request sets, and send those cookies back on subsequent requests.
I need to login to a website and perform an action. The website is REST based so I can easily login by doing this (the login info is included as a querystring on the URL, so I dont't need to set the credentials):
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest firstRequest = (HttpWebRequest) WebRequest.Create(loginUrl);
firstRequest.CookieContainer = cookieJar;
firstRequest.KeepAlive = true;
firstRequest.Method = "POST";
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
That works and logs me in. I get a cookie back to maintain the session and it's stored in the cookieJar shown above. Then I do a second request such as this:
HttpWebRequest secondRequest = (HttpWebRequest) WebRequest.Create(actionUrl);
secondRequest.Method = "POST";
secondRequest.KeepAlive = true;
secondRequest.CookieContainer = cookieJar;
WebResponse secondResponse = secondRequest.GetResponse();
And I ensure I assign the cookies to the new request. But for some reason this doesn't appear to work. I get back an error telling me "my session has timed out or expired", and this is done one right after the other so its not a timing issue.
I've used Fiddler to examine the HTTP headers but I'm finding that difficult since this is HTTPS. (I know i can decrypt it but doesn't seem to work well.)
I can take my URL's for this rest service and paste them into firefox and it all works fine, so it must be something I'm doing wrong and not the other end of the connection.
I'm not very familiar with HTTPS. Do I need to do something else to maintain my session? I thought the cookie would be it, but perhaps there is something else I need to maintain across the two requests?
Here are the headers returned when I send in the first request (except I changed the cookie to protect the innocent!):
X-DB-Content-length=19
Keep-Alive=timeout=15, max=50
Connection=Keep-Alive
Transfer-Encoding=chunked
Content-Type=text/html; charset=WINDOWS-1252
Date=Mon, 16 Nov 2009 15:26:34 GMT
Set-Cookie:MyCookie stuff goes here
Server=Oracle-Application-Server-10g
Any help would be appreciated, I'm running out of ideas.
I finally got it working after decrypting the HTTP traffic from my program.
The cookie I'm getting back doesn't list the Path variable. So .NET takes the current path and assigns that as the path on the cookie including the current page. ie: If it was at http://mysite/somepath/somepage.htm it would set the cookie path=/somepath/somepage.htm. This is a bug as it should be assigned to "/" which is what all web browsers do. (hope they fix this.)
After noticing this I grabbed the cookie and modified the path property and everything works fine now.
Anyone else with a problem like this check out Fiddler. .NET uses the windows certificate store so to decrypt http traffic from your program you will need to follow the instructions here: http://www.fiddler2.com/Fiddler/help/httpsdecryption.asp . You will also need to turn on decryption under the Options\HTTPS tab of Fiddler.
From MSDN:
When a user moves back and forth between secure and public areas, the ASP.NET-generated session cookie (or URL if you have enabled cookie-less session state) moves with them in plaintext, but the authentication cookie is never passed over unencrypted HTTP connections as long as the Secure cookie property is set.
So basically, the cookie can be passed over both HTTP and HTTPS if the 'Secure' property is set to 'false'.
see also how can I share an asp.net session between http and https