i have created an application with the help of webrequest and webresponse . when i try to log in to yahoo i succeed. after that i open the next page and i get the login page again by the response. How can I fix this?
If I'd have to guess (which I do, given the current state of the question), I'd say you have neglected to support cookies in your request.
I think you already answered your own question, almost anyway (you guessed this was session-related). You need to keep track of the cookies that the initial login request sets, and send those cookies back on subsequent requests.
Related
I want to get HTML code of B page. Unfortunately site requires to open A page first to get session_id, after it I can finally open webpage I wanted. What is solution to get html code of B page? I try do it with WebClient, but session_id is probably not saved.
var client = new WebClient();
client.DownloadString("http://moria.umcs.lublin.pl/link/");
client.DownloadString("http://moria.umcs.lublin.pl/link/grid/1/810");
It depends on how the server tracks that you have already visited page A when you visit page B.
Most likely it uses some kind of session ID, which is probably saved in cookies. Examining HTTP request and response headers in any browser's developer tools can get you an idea of what this website does to track the user.
If you need to be able to store session ID in cookies, cookies-aware web-client sample is given here
I would use HttpWebRequest instead of WebClient. I did not see any method in WebClient where you can get or set cookies. Take a look at this MSDN link. Your code for the initial request would be something like in the link. For the next request to another page, set the CookieContainers with the cookies from the response that you got from the initial request; before you request for the response.
https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer(v=vs.110).aspx
I'm using the HttpClient (System.Net.Http.HttpCient) to send some requests and I'm also using a CookieContainer to hande Cookies. For some Webpages everything works fine, but on some other pages no cookies are stored, although my browser saves the Cookies when I visit the webpage.
Can someone here explain what's the problem.
ceddy
Maybe these pages redirect to an other url? Cookies are stored per url and hence it's possible you "loose" a cookie.
To verify the behavior you may set
request.AllowAutoRedirect = false;
and look at the response object about what's going on. If this is really the issue in your case, you can copy the cookies from one url to the other via the CookieContainer.
I've used several HttpWebRequest's in the past but they've all been used to login into a site.
I was wondering how does one make the WebRequest mimic a WebBrowser as in once you're logged in, navigate to a new page, maybe perform an action there, then go to a different page?
I've researched a little about this before and I think it might involve using the prior request's cookies or something.
My question is how do I (I'm assuming) get the cookies from the previous session, then navigate to a page, or complete an action as if we were still on the last request if that makes sense.
the HttpWebRequest has a Cookies property and HttpWebResponse has a CookieContainer property.
you record the cookies from the container, and add them to the next request.
you may also need to set the HTTP referrer header field on the request object.
EDIT :
this will still not get you mimicking a web browser. things like JavaScript will not work/run. and you won't have a DOM to work against.
I'm trying to scrape a web app that uses a few redirects and logouts/logins in between requests. I believe I'll have to set AllowAutoRedirect to false so I can capture the redirect requests and manually redirect while watching for new cookies. My only experience with cookies is to just set the container and forget about it... do I have to parse the response headers to decide actions to take with cookies? Can someone lay out a general approach?
Update
It turns out that Chris was right. The redirects and cookies were working just fine... the application I was hitting did not like that I was not setting all the right headers (content type, user agent). After adding those in, I'm getting the response I expect.
You got the CookieContainer part right which most people miss so kudos to you!
AllowAutoRedirect should pick up cookies as redirects happen. Is there a reason that you need to manually process things?
I set a cookie like this in one page:
Request.Cookies["lang"].Value = "en-US";
Request.Cookies["lang"].Expires = DateTime.Now.AddDays(50);
On another page I try and read the cookie:
string lang = Server.HtmlEncode(Request.Cookies["lang"].Value);
The cookie is not null but the value is an empty string. What am I doing wrong?
You should be using Response.Cookies to set the cookie, and Request.Cookies to read any cookies sent back from the client.
The code in your question is setting the cookie in the Request object, not the Response.
Are cookies enabled on the client? The fact that you set a cookie doesn't mean that the client supports them and will send them back.
Remember, you're dealing with two disconnected systems; your server doesn't keep state and you know little about the client.
If I remember correctly I think you should be using response instead of request as request is what is being sent to you. Response is when you want to set something back to the client browser.
EDIT: What you are doing is modifying the cookies in that particular request which would make sense why you are not seeing on subsequent pages. That is not saving them back to the client.