I let my program (c#) log in to website, i get correct buffered cookie information. Yet, i get a 401, session timed out, when i want to retrieve the correct data behind my login.
So i figured, the website must not be able to retrieve that cookie info. Can't figure out how to store it so that the website can retrieve it.
WebRequest req = WebRequest.Create(Url);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(Gegevens);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
cookieHeader, contains the correct info. Thanks in advance.
You need to assign a CookieContainer to your web request and use this very same CookieContainer for the following requests, too.
See MSDN for reference.
You could (if you want to persist the cookies upon closing your application) get the list of Cookies from the CookieContainer and serialize these. Upon opening the application you could deserialize and rebuild the CookieContainer.
From the comment you provided, I'm going to hazard a guess and say you aren't properly adding your login cookies to your next WebRequest. Cookie handling with a WebRequest object is a bit difficult, so I recommend using HttpWebRequest and HttpWebResponse which has built-in cookie parsing. You only have to change a couple lines of code here and there:
Building a request (using the same example in your question)
CookieContainer cookies = new CookieContainer();
// When using HttpWebRequest or HttpWebResponse, you need to cast for it to work properly
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.CookieContainer = cookies;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(Gegevens);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
// Cast here as well
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
{
// Code related to the web response goes in here
}
Right now, your cookie information is saved in the CookieContainer object. It can be reused later in your code to validate your login. You don't need to write this cookie information to the disk if you don't need to.
Building a request with cookie information
(Pretty much the same as above, but you're not adding POST data and you're using a GET request)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.CookieContainer = cookies; // This is where you add your login cookies to the current request
req.Method = "GET";
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
{
// Code related to the web response goes here
}
Hopefully this will get you on the right track :)
Related
I'm trying to write a script in c# capable to retrieve some information from a website. These information are protected so I need to login before I can read them. That's what I think should be my procedure:
First of all use a POST request to login into the website. Here my first problem: the page where I find the login form is this https://idp.kk-abcdefg.com/idp/Authn/UserPassword . Should I submit the POST request to this page or should I use a different address?
I've tested the headers using some tools of Firefox or Chrome but I can't understand which is the right procedure. I have noticed that If I open this login page I receive some cookies. If I delete these cookies and try to login by inserting user and password (via browser) I get an error as a response from the website .. it says that I need to activate cookies to be able to login. So it seems like when I open the login page for the first time I receive some cookies and then I need to send them together with the first POST request for login. Does it make sense for any of you?
That's the code I'm using right now:
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword”);
string cookieHeader;
string pageSource;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
And I believe this is working because the result of pageSource changes if I use correct user / pass or I write them wrong. But when user / pass are correct, I'm still not able to login because I get the following error message: "This application requires cookies. Please make sure cookies are enabled in the settings of your browser. Please reload the login page and try logging in again".
This is the same error that I get if I disable cookies in my browser or if I delete cookies that I get when I load the login page for the first time.
Can you help me with all that? My idea is that I need to save the cookies received when I open the login page for the first time and then send them together with following requests but I don't know how to do ..
Thanks a lot!
In Web Application, once the user is logged in successfully a cookie is sent back to the browser to track the user session and to determine if the user is logged in or not during further requests. Furthermore login process of your application requires cookies to be sent from client along with username and password. So when you are trying to perform login without browser it complains about missing cookies.
If you know what cookies and their values need to be sent along with username and password for login, you can send them using cookieContainer in WebRequest as following.
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword");
string cookieHeader;
string pageSource;
CookieContainer cookieContainer = new CookieContainer();
Cookie cookie1 = new Cookie("<<cookiename>>", "<<cookievalue>>","/", "<<yourdomainname>>");
Cookie cookie2 = new Cookie("<<cookiename>>", "<<cookievalue>>", "/", "<<yourdomainname>>");
cookieContainer.Add(cookie1);
cookieContainer.Add(cookie2);
// You can keep adding all required cookies this way.
var req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookieContainer;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
// You can access the cookies coming as part of response as following.
HttpWebResponse response = resp as HttpWebResponse;
if(response != null)
{
var cookiesCollections = response.Cookies;
}
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
If you don't know the cookies and you need to get the cookies first by requesting login page before posting username and password then use following.
var loginPageUrl = "<<Your Login Page url>>";
CookieContainer cookieContainer = new CookieContainer();
var req = (HttpWebRequest)WebRequest.Create(loginPageUrl);
req.CookieContainer = cookieContainer;
req.Method = "GET";
WebResponse resp = req.GetResponse();
HttpWebResponse response = resp as HttpWebResponse;
CookieCollection cookies;
if (response != null)
{
cookies = response.Cookies; //Use this cookies in above code to send with username and password.
}
You send the request login twice and use single CookieContainer object!
I am using my c# app to send request to the webpage. Before I send any request I have to be logged in this webpage. I want to avoid logging everytime I want to do some work, so I am storing cookies in sql server database in VARBINARY column.
Lets say I am sending 50 POST requests every day:
private string getRequest(string url, string postData = "")
{
try
{
System.Net.ServicePointManager.Expect100Continue = true;
StreamReader reader;
var request = WebRequest.Create(url) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14";
request.CookieContainer = Cookie;
request.AllowAutoRedirect = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
postData = Uri.EscapeUriString(postData);
request.Method = "POST";
request.ContentLength = postData.Length;
Stream requestStream = request.GetRequestStream();
Byte[] postBytes = Encoding.ASCII.GetBytes(postData);
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
foreach (Cookie tempCookie in response.Cookies)
{
Cookie.Add(tempCookie);
}
reader = new StreamReader(response.GetResponseStream());
string readerReadToEnd = reader.ReadToEnd();
response.Close();
database.updateCookie(AccountId, Cookie); //here I am updating the cookies
return readerReadToEnd;
}
catch { return ""; }
Do I really need to update Cookies after every request? Or maybe could I update my cookies only once, after sending 50 POST requests?
I am asking, because sometimes my cookies lasts couple days, and sometimes they died after 1 minute. I don't really know why and how to avoid that.
Do I have to store the newest verion of cookies or maybe can I use the same everytime?
Each site decides how long particular cookies are valid. Hence you need to get that information for each site individually to do it correctly. Paying attention to "expiration" on the cookie (in "set-cookie" headers of response) may give you initial guidance on when cookie is guaranteed to expire.
Common expiration/validity ranges:
authentication cookies - from 10 minutes to 1 day
ASP.Net session state - from 20 minutes
CSRF protection - possibly updated on every response
private void button1_Click(object sender, EventArgs e)
{
string userName = textBox1.Text;
string password = textBox2.Text;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.youmint.com/LoginVerification.php?name="+userName+"&pass="+password+"&agreement=true&checkvalue=true");
request.Method = "GET";
request.KeepAlive = true;
request.Headers.Add("Keep-Alive: 300");
WebResponse response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
if (responseFromServer.Equals(""))
MessageBox.Show("Successfully logged in!!");
else if (responseFromServer.Equals("1"))
MessageBox.Show("Login failed!");
request = (HttpWebRequest)WebRequest.Create("http://www.youmint.com/FreeSms.html");
response = request.GetResponse();
dataStream = response.GetResponseStream();
reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
/*secret code :P */
reader.Close();
dataStream.Close();
response.Close();
}
So, that's my code... The first webrequest logs into the website. It works fine in the browser and returns 1 if the login is not correct. Then the second one is a normal webrequest to a webpage of the same website. But the login is already gone and the response I get is what I get if I'm not logged in! Can someone please tell me what I'm doing wrong? How do I keep it alive? Do I have to use an invisible webbroswer control or something like that?
This has nothing to do with "keep alive".
You need to preserve session cookie between requests. For that you first need to enable cookies for your login request (read HttpWebRequest docs - it is a bit unobvious). Then you need to pass that cookie with all the following reuests.
Also please make use of the using()
#liho1eye is correct. Here's some more info from the HttpWebRequest page:
For security reasons, cookies are
disabled by default. If you want to
use cookies, use the CookieContainer
property to enable cookies.
You'll need to reference the HttpWebResponse.Cookies property to get the initial session token cookie.
Edit:
Here's a quick and dirty sample of making a request to a page, and transferring response cookies to the next request. Didn't do much testing or validation (so beware!) - just to give you the idea of the approach.
//this only has login/password info, you may need other parameters to trigger the appropriate action:
const string Parameters = "Login1$username=pfadmin&Login1$Password=password";
System.Net.HttpWebRequest req = (HttpWebRequest)System.Net.WebRequest.Create("http://[WebApp]/Login.aspx");
req.Method = "GET";
req.CookieContainer = new CookieContainer();
System.Net.HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
//Create POST request and transfer session cookies from initial request
req = (HttpWebRequest)System.Net.WebRequest.Create("http://localhost/AdminWeb/Login.aspx");
req.CookieContainer = new CookieContainer();
foreach (Cookie c in resp.Cookies)
{
req.CookieContainer.Add(c);
}
req.ContentType = "application/x-www-form-urlencoded";
//...continue on with your form POST
Here is the scenario. I have written code use a digital certificate to GET cookies from a secured url, to in turn POST data back to another url using the retrieved cookies and same digital certificate. The GET works and cookies are retrieved, the POST comes back with an error 500. I installed fiddler to see what was going on...POST looks fine...cookies are present. I used the feature in fiddler that allows creating a request via drag and drop. POST the exact same POST recorded from C# code that was recorded in fiddler and it works!
What is Fiddler doing that Visual Studio is not? It must be doing something if fiddler can POST the data but Visual Studio returns an error 500. Here is the code below:
X509Certificate cert = new X509Certificate("mycert.pfx", "certpassword");
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
req.CookieContainer = new CookieContainer();
req.Method = "GET";
req.ClientCertificates.Add(cert);
HttpWebResponse Response = (HttpWebResponse)req.GetResponse();
CookieCollection ck = req.CookieContainer.GetCookies(req.RequestUri);
string strcookie = ck[0].Value;
string strcookie2 = ck[1].Value;
Response.Close();
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
req.CookieContainer = new CookieContainer();
Cookie auth = new Cookie("_wl_authcookie_", strcookie2);
Cookie jsess = new Cookie("JSESSIONID", strcookie);
auth.Domain = "server";
jsess.Domain = "server";
req.CookieContainer.Add(auth);
req.CookieContainer.Add(jsess);
req.ClientCertificates.Add(cert);
req.Method = "POST";
Byte[] data = ReadByteArrayFromFile("filewithdatatoPOST.txt");
req.ContentLength = data.Length;
Stream myStream = req.GetRequestStream();
myStream.Write(data, 0, data.Length);
myStream.Close();
HttpWebResponse Response2 = (HttpWebResponse)req.GetResponse();
Stream strm = Response2.GetResponseStream();
StreamReader sr2 = new StreamReader(strm);
Response2.Close();
Does your code work if you set
req.ServicePoint.Expect100Continue = false;
on all your WebRequests?
Solved!!! It wasn't related to the Expect100Continue.
So after weeks of troubleshooting....6 different programmers....I figured it out. I'm not sure if this is always true, but in this scenario the problem was that the url we were getting cookies from:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
was not the same as the url that we were posting the data back to:
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
Getting the cookies and posting back to the same url fixed the issue:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");
req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");
I have done this code to login,to retrieve and show a webpage :
// login info array
string postData = "user_name=tler";
postData += "&user_password=lodvader";
byte[] data = Encoding.ASCII.GetBytes(postData);
// web request
WebRequest req = WebRequest.Create("http://www.lol.com/login.php");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = data.Length;
// stream response to string
Stream newStream = req.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), Encoding.GetEncoding("iso-8859-1"));
string responseString = reader.ReadToEnd();
// retrieve text within title
Regex rx = new Regex(#"(?<=<title>).+?(?=</title>)");
var variable = rx.Matches(responseString);
// output
Console.WriteLine(variable[0]);
Console.ReadLine();
But, the following page after login is an html redirect like :
<meta http-equiv="refresh" content="3; URL="bb.php">
How to follow this link and retrieve next page ?
Just send a new WebRequest to the bb.php file. Make sure that you use the same CookieContainer since I presume that login.php uses cookie-based sessions to remember you. Check out the HttpWebRequest.CookieContainer property. This requires you to cast your WebRequest to a HttpWebRequest.
Added: (Couldn't write example code in the comment.)
I'm just making code up without proofing now...
var cookies = new CookieContainer();
var firstReq = (HttpWebRequest)WebRequest.Create(".../login.php");
firstReq.CookieContainer = cookies;
var secondReq = (HttpWebRequest)WebRequest.Create(".../bb.php");
secondReq.CookieContainer = cookies
I have found the time to finish it, here the response ( i tried to be as clear as possible ) :
// Cookie for our session
var cookieContainer = new CookieContainer();
// Encode post variables
ASCIIEncoding encoding=new ASCIIEncoding();
byte[] loginDataBytes = encoding.GetBytes("user_name=belaz&user_password=123");
// Prepare our login HttpWebRequest
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://blabla.fr/verify.php");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = cookieContainer;
request.ContentLength = loginDataBytes.Length;
// Write encoded post variable to the stream
Stream newStream = request.GetRequestStream();
newStream.Write(loginDataBytes, 0, loginDataBytes.Length);
newStream.Close();
// Retrieve HttpWebResponse
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Link the response cookie to the domain
cookieContainer.Add(new Uri("http://blabla.fr/"),response.Cookies);
// Prepare our navigate HttpWebRequest, and set his cookie.
HttpWebRequest requestProfile = (HttpWebRequest)WebRequest.Create("http://blabla.fr/bb.php");
requestProfile.CookieContainer = cookieContainer;
// Retrieve HttpWebResponse
HttpWebResponse responseProfile = (HttpWebResponse)requestProfile.GetResponse();
// Retrieve stream response and read it to end
Stream st = responseProfile.GetResponseStream();
StreamReader sr = new StreamReader(st);
string buffer = sr.ReadToEnd();
There is a property of HttpWebRequest called AllowAutoRedirects. Set this to true. Also there is a property called MaximumAutomaticRedirections. Set that to some allowable value to make sure all of them are followed.
You can't do it a easy way, since the meta tag is read by the client and executed.
In this case, when you're using HttpWebRequest, the request doesn't care about the functions the text may have.
So you need to do another request to the page in the URL attribute (bb.php).
-
If the server did the redirection you wouldn't have the problem.