We Keep Coding

Cookies and C# HttpWebRequest

I have been trying to log in to a server to grab the authentication cookie (a session cookie), which I can then use for further calls to the server. Every example I have read follows the same pattern:
HttpWebRequest request = WebRequest.Create(loginURL) as HttpWebRequest;
var response = request.GetResponse() as HttpWebResponse;
var cookies = response.Cookies;
This didn't work for me, as the cookies variable ended up empty, and a debug analysis showed response.Cookies was empty. The server is mine, and I can see, through debugging, the cookie is being set. I can also see the cookie in Firefox if I log in to my site with it. So I know the cookie is being set.
After some messing around, I discovered the cookie was being set in the request, not the response. So the code below worked. My question is: Why? Why is the request being populated, but not the response? Is it something to do with being a post, not a get? I am totally baffled.
private void Login()
{
string userName = UserNameText.Text;
string password = PasswordText.Password;
string baseURL = URLText.Text;
string loginURL = baseURL + "/Authentication/LoginAction";
HttpWebRequest request = WebRequest.Create(loginURL) as HttpWebRequest;
request.Method = "POST";
string formContent =
"UserName=" + userName +
"&Password=" + password;
var byteArray = Encoding.UTF8.GetBytes(formContent);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
request.CookieContainer = new CookieContainer();
try
{
using (var dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
using (var response = request.GetResponse() as HttpWebResponse)
{
var cookies = request.CookieContainer;
if (cookies.Count != 0)
{
cookies_ = cookies;
}
}
}
}
catch(Exception ex)
{
// don't bother too much
Debug.WriteLine(ex.Message);
}
}

The CookieContainer should be considered similar to a browser's cookie cache for a particular site. The idea is that you supply the container as part of the request, and then it's populated by the cookies you receive and you can reuse that container for subsequent requests. When you make a request, the cookies in the container are sent with the request (just like the browser would with stored cookies).
So, for example, if you have a page that uses cookies to store an authentication token, you can pass the cookie container with the login request, and then pass it with subsequent requests which require an authenticated cookie.
As to why you can't simply extract it from the request, I guess Microsoft just didn't want to duplicate things when you can pass in a reference to a mutable cookie container in the request.

Login to website which requires Cookies using c#

I'm trying to write a script in c# capable to retrieve some information from a website. These information are protected so I need to login before I can read them. That's what I think should be my procedure:
First of all use a POST request to login into the website. Here my first problem: the page where I find the login form is this https://idp.kk-abcdefg.com/idp/Authn/UserPassword . Should I submit the POST request to this page or should I use a different address?
I've tested the headers using some tools of Firefox or Chrome but I can't understand which is the right procedure. I have noticed that If I open this login page I receive some cookies. If I delete these cookies and try to login by inserting user and password (via browser) I get an error as a response from the website .. it says that I need to activate cookies to be able to login. So it seems like when I open the login page for the first time I receive some cookies and then I need to send them together with the first POST request for login. Does it make sense for any of you?
That's the code I'm using right now:
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword”);
string cookieHeader;
string pageSource;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
And I believe this is working because the result of pageSource changes if I use correct user / pass or I write them wrong. But when user / pass are correct, I'm still not able to login because I get the following error message: "This application requires cookies. Please make sure cookies are enabled in the settings of your browser. Please reload the login page and try logging in again".
This is the same error that I get if I disable cookies in my browser or if I delete cookies that I get when I load the login page for the first time.
Can you help me with all that? My idea is that I need to save the cookies received when I open the login page for the first time and then send them together with following requests but I don't know how to do ..
Thanks a lot!

In Web Application, once the user is logged in successfully a cookie is sent back to the browser to track the user session and to determine if the user is logged in or not during further requests. Furthermore login process of your application requires cookies to be sent from client along with username and password. So when you are trying to perform login without browser it complains about missing cookies.
If you know what cookies and their values need to be sent along with username and password for login, you can send them using cookieContainer in WebRequest as following.
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword");
string cookieHeader;
string pageSource;
CookieContainer cookieContainer = new CookieContainer();
Cookie cookie1 = new Cookie("<<cookiename>>", "<<cookievalue>>","/", "<<yourdomainname>>");
Cookie cookie2 = new Cookie("<<cookiename>>", "<<cookievalue>>", "/", "<<yourdomainname>>");
cookieContainer.Add(cookie1);
cookieContainer.Add(cookie2);
// You can keep adding all required cookies this way.
var req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookieContainer;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
// You can access the cookies coming as part of response as following.
HttpWebResponse response = resp as HttpWebResponse;
if(response != null)
{
var cookiesCollections = response.Cookies;
}
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
If you don't know the cookies and you need to get the cookies first by requesting login page before posting username and password then use following.
var loginPageUrl = "<<Your Login Page url>>";
CookieContainer cookieContainer = new CookieContainer();
var req = (HttpWebRequest)WebRequest.Create(loginPageUrl);
req.CookieContainer = cookieContainer;
req.Method = "GET";
WebResponse resp = req.GetResponse();
HttpWebResponse response = resp as HttpWebResponse;
CookieCollection cookies;
if (response != null)
{
cookies = response.Cookies; //Use this cookies in above code to send with username and password.
}

You send the request login twice and use single CookieContainer object!

How to programmatically log in to microsoft online? [duplicate]

I'm trying to write a utility which will attempt to login to the Microsoft Online Admin website and report back as to whether it is reachable.
Using code mainly from this article, http://odetocode.com/articles/162.aspx and some screen scraping I have pieced together the following. Unfortunately it doesn't work, the final response shows that I am still looking at the login page rather than the target page.
Any help would be terrific. Thanks in advance.
private void LoginToSite()
{
const string LOGIN_URL = "https://admin.microsoftonline.com/Login.aspx";
const string USERNAME = "<username>";
const string PASSWORD = "<password>";
const string TARGET_PAGE_URL = "https://admin.noam.microsoftonline.com/Home/Home.aspx";
// first, request the login form to get the viewstate value
HttpWebRequest webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
StreamReader responseReader = new StreamReader(
webRequest.GetResponse().GetResponseStream()
);
string responseData = responseReader.ReadToEnd();
responseReader.Close();
// extract the viewstate value and build out POST data
string viewState = ExtractViewState(responseData);
string postData =
String.Format(
"__VIEWSTATE={0}&AdminCenterLoginControl$UserNameTextBox={1}&AdminCenterLoginControl$PasswordTextbox={2}&__EVENTTARGET=AdminCenterLoginControl_ActionButton",
viewState, USERNAME, PASSWORD
);
// have a cookie container ready to receive the forms auth cookie
CookieContainer cookies = new CookieContainer();
// now post to the login form
webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
// we don't need the contents of the response, just the cookie it issues
webRequest.GetResponse().Close();
// now we can send out cookie along with a request for the protected page
webRequest = WebRequest.Create(TARGET_PAGE_URL) as HttpWebRequest;
webRequest.CookieContainer = cookies;
responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
// and read the response
responseData = responseReader.ReadToEnd();
responseReader.Close();
MessageBox.Show(responseData);
}
private string ExtractViewState(string s)
{
string viewStateNameDelimiter = "__VIEWSTATE";
string valueDelimiter = "value=\"";
int viewStateNamePosition = s.IndexOf(viewStateNameDelimiter);
int viewStateValuePosition = s.IndexOf(
valueDelimiter, viewStateNamePosition
);
int viewStateStartPosition = viewStateValuePosition +
valueDelimiter.Length;
int viewStateEndPosition = s.IndexOf("\"", viewStateStartPosition);
return HttpUtility.UrlEncodeUnicode(
s.Substring(
viewStateStartPosition,
viewStateEndPosition - viewStateStartPosition
)
);
}
edit
private void LoginToSite()
{
const string LOGIN_URL = "https://admin.microsoftonline.com/login.aspx?ReturnUrl=%2fDefault.aspx";
const string USERNAME = "<username>";
const string PASSWORD = "<password>";
// Request the login form to get the viewstate value
HttpWebRequest webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
string response1 = new StreamReader(webRequest.GetResponse().GetResponseStream()).ReadToEnd();
// Extract the viewstate value and build our POST data
string viewState = ExtractViewState(response1);
string postData = String.Format(
"__VIEWSTATE={0}&AdminCenterLoginControl$UserNameTextBox={1}&AdminCenterLoginControl$PasswordTextbox={2}&__EVENTTARGET=AdminCenterLoginControl_ActionButton",
viewState, USERNAME, PASSWORD);
// Set up the Request properties
webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
CookieContainer cookies = new CookieContainer();
webRequest.CookieContainer = cookies;
// Post back to the form
using (StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream()))
{
requestWriter.Write(postData);
}
// Read response
string response2 = new StreamReader(webRequest.GetResponse().GetResponseStream()).ReadToEnd();
MessageBox.Show(response2);
}

It would appear that MicrosoftOnline.com does not use Windows Live IDs (aka Passport) for login. This is a shame, since there are libraries available that make logging into LiveID pretty simple for client apps.
Your code hits the login page first, scraps cookies from the response, then attempts to navigate to the target page. This doesn't match the normal flow of user behavior. Normally, the user clicks on a link to go to a target page and the web site redirects the request to the login page if the user is not logged in. After logging in, the login page redirects back to the originally requested target page.
You can see this by looking at the login URL when you visit admin.microsoftonline.com in the browser. You are immediately redirected to the login page, but the full URL on the login page is: https://admin.microsoftonline.com/login.aspx?ReturnUrl=%2fDefault.aspx
Note the ReturnUrl query param at the end. This tells the login page what page to redirect back to when the login is completed.
I don't know if redirect is required by the login page, but since this is the primary path for actual end user interaction (that works) and not the path your code is taking, it's something to consider. Among other things, the redirect to login/ redirect back to target technique will take care of setting the browser cookies for the target domain automatically.
p.s. I notice also that the email administration portion of Microsoft Online services uses a different login URL. From this page (http://www.microsoft.com/online/signin.aspx) clicking on the Exchange Hosted Services Administrative Center link takes you to http:admin.messaging.microsoft.com, which immediately redirects to a login url of https://sts.messaging.microsoft.com/login.aspx?ReturnUrl=%2fDefault.aspx%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252fadmin.messaging.microsoft.com%26wctx%3drm%253d0%2526id%253dpassive%2526ru%253d%25252f%26wct%3d2010-10-27T17%253a11%253a50Z&wa=wsignin1.0&wtrealm=https%3a%2f%2fadmin.messaging.microsoft.com&wctx=rm%3d0%26id%3dpassive%26ru%3d%252f&wct=2010-10-27T17%3a11%3a50Z
The domain name sts.messaging.microsoft.com suggests that the Exchange Hosted Services portion of Microsoft Online Services is using a Security Token Service, which suggests that this login system is capable of federated single sign on between different services. You might be able to connect to this using something like the Windows Identity Foundation (WIF) client components. Will that work with the rest of Microsoft Online Services? I don't know.

> // now we can send out cookie along with a request for the protected page
> webRequest = WebRequest.Create(TARGET_PAGE_URL) as
> HttpWebRequest;
> webRequest.CookieContainer = cookies;
> responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
Aren't you setting WebRequest.CookieContainer equal to the blank cookie container that you generated earlier?
Shouldn't you be doing something like:
// we don't need the contents of the response, just the cookie it issues
WebResponse response = webRequest.GetResponse();
cookies = response.cookies;
response.Close();

C#: Post login data to form and process website's response

I am trying to write a C# program to prolong the deadline for my books in my university's library. What I want to do is the following:
1.) Login to library website via WebRequest & POST method, with username & password entered in C# program
2.) Get the url to "View borrowed books" site containing the encrypted password & plain text username as GET parameters
3.) Download content of named page to display to user in the C# program
4.) If the user presses the corresponding button in the program, submit the prolongation form to the website to prolong all media at once.
Right now I'm stuck between 1 and 2, I seem to be able to connect to the website and to enter the userdata, but the WebResponse I get is again the login page (which is not the case if you login manually on the website).
This is the method I wrote to connect to the website:
// Login function, logs the user in, uses passed user number & password
public static Boolean userLogin(String unr, String pass)
{
// Login
// Cookie needed for maintaining php session
CookieContainer cContainer = new CookieContainer();
Console.WriteLine(unr+","+pass);
String postUrl = "https://universitylibrary.com/loan/DB=4/LNG=DU/USERINFO_LOGIN";
String formParams = String.Format("ACT={0}&HOST_NAME={1}&HOST_PORT={2}&HOST_SCRIPT={3}&LOGIN={4}&STATUS={5}&BOR_U={6}&BOR_PW={7}","UI_DATA","","","","KNOWNUSER","HML_OK", unr, pass);
String cookieHeader;
WebRequest wreq = WebRequest.Create(postUrl);
wreq.ContentType = "application/x-www-form-urlencoded";
wreq.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
wreq.ContentLength = bytes.Length;
using (Stream os = wreq.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = wreq.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
//Authentication trial
String PageSource;
String getUrl = "https://universitylibrary.com:443/loan/DB=4/USERINFO";
WebRequest getReq = WebRequest.Create(getUrl);
getReq.Headers.Add("Cookie",cookieHeader);
WebResponse getResp = getReq.GetResponse();
using (StreamReader sr = new StreamReader(getResp.GetResponseStream()))
{
PageSource = sr.ReadToEnd();
}
Console.Write(PageSource);
return true;
}
Can you see my mistake? I get the sourcecode and the params (username, password) output on the console, but the output is again the login page.
I would just look at the php page, but I don't have access to any of the internal system data, all I have is the HTML page.
Any suggestions would be highly appreciated!
EDIT:
I have rethought the whole thing, and rebuilt the HTTP request header completely as recorded by fiddler. That part of the function looks like this now:
// Login
// Cookie needed for maintaining php session
CookieContainer cContainer = new CookieContainer();
HttpCookie cookie = new HttpCookie("cookie", "PSC_4='xxxxxxx'; DB='n'");
CookieCollection cookieCol = new CookieCollection();
cookieCol.Add(cookieCol);
cContainer.Add(cookieCol);
Console.WriteLine(unr+","+pass);
String postUrl = "https://universitylibrary.com:443/loan/DB=4/USERINFO";
String formParams = String.Format("ACT={0}&HOST_NAME={1}&HOST_PORT={2}&HOST_SCRIPT={3}&LOGIN={4}&STATUS={5}&BOR_U={6}&BOR_PW={7}","UI_DATA","","","","KNOWNUSER","HML_OK", unr, pass);
String cookieHeader;
HttpWebRequest wreq = (HttpWebRequest) WebRequest.Create(postUrl);
wreq.Referer = "https://universitylibrary.com/loan/DB=4/LNG=DU/USERINFO_LOGIN";
wreq.KeepAlive = true;
wreq.ContentLength = 119;
wreq.Host = "universitylibrary.com";
wreq.ContentType = "application/x-www-form-urlencoded";
wreq.Method = "POST";
wreq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";
wreq.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
wreq.SendChunked = true;
wreq.TransferEncoding = "gzip, deflate";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
wreq.ContentLength = bytes.Length;
using (Stream os = wreq.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
cookieHeader = "";
try
{
HttpWebResponse resp = (HttpWebResponse) wreq.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
}
catch (WebException ex)
{
Console.WriteLine(ex.Status);
Console.WriteLine(ex.Response);
}
The whole thing still doesn't work though, same issue as before.
Is it possible that HttpWebRequest can't handle https or that something else is missing for https to work? (HTTP & HTTPS seem to be syntactically identical and the port is correctly set to 443, the real difference seems to lay in the additional SSL/TLS layer, maybe I need to add this somewhere?)

When you are returned the login page it probably means that you have not been correctly authenticated. There could be several reasons for this, but in the end it is because you not mirroring the HTTP communication from the manual login on the website correctly.
What I usually do is to use a monitor such as Fiddler to capture the full request/response pattern from the manual login, which I can then subsequently mirror in my code.

You can just modify your login page after checking login data and make the only output is "SuccessfullySignIn", for example.
If this is the data your receive,
$if (getReq == "SuccessfullySignIn")
{
//Do something
}
And try to use Redirect features in your webpage

Asp.Net c# logging in to another website

I know this question has been asked quite a lot of times which is how I have got to where I am at with the code below however I just can't get it to work on the particular website I am trying to access. At the site I am trying to access I need to retrieve certain values from the page however things like price and availability only come up after logging in so I am trying to submit my login information and then go to the product page to get the information I need using HTML Agility Pack.
At the moment it seems to attempt the login however the website is either not accepting it or the cookies are not present on the next page load to actually keep me logged in.
If someone could help me with this I would be very grateful as I am not a programmer but have been assigned this task as part of a software installation.
protected void Button5_Click(object sender, System.EventArgs e)
{
string LOGIN_URL = "http://www.videor.com/quicklogin/1/0/0/0/index.html";
string SECRET_PAGE_URL = "http://www.videor.com/item/47/32/0/703/index.html?scriptMode=&CUSTOMERNO=xxx&USERNAME=xxx&activeTabId=0";
// have a cookie container ready to receive the forms auth cookie
CookieContainer cookies = new CookieContainer();
// first, request the login form to get the viewstate value
HttpWebRequest webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.CookieContainer = cookies;
StreamReader responseReader = new StreamReader(
webRequest.GetResponse().GetResponseStream()
);
string responseData = responseReader.ReadToEnd();
responseReader.Close();
string postData = "CUSTOMERNO=xxxx&USERNAME=xxxxx&PASSWORD=xxxxx";
// now post to the login form
webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
// we don't need the contents of the response, just the cookie it issues
webRequest.GetResponse().Close();
// now we can send out cookie along with a request for the protected page
webRequest = WebRequest.Create(SECRET_PAGE_URL) as HttpWebRequest;
webRequest.CookieContainer = cookies;
responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
// and read the response
responseData = responseReader.ReadToEnd();
responseReader.Close();
Response.Write(responseData);
}

This isn't a direct answer since I'm not sure what's wrong with your code (from a cursory glance it looks ok), but another approach is to use browser automation using Selenium . The following code will actually load the page using Chrome (you can swap out Firefox or IE) and is simpler to code against. It also won't break if they add javascript or something.
var driver = new ChromeDriver();
driver.Navigate().GoToUrl(LOGON_URL);
driver.FindElement(By.Id("UserName")).SendKeys("myuser");
driver.FindElement(By.Id("Password")).SendKeys("mypassword");
driver.FindElement(By.TagName("Form")).Submit();
driver.Navigate().GoToUrl(SECRET_PAGE_URL);
// And now the html can be found as driver.PageSource. You can also look for
// different elements and get their inner text and stuff as well.