We Keep Coding

C#: Post login data to form and process website's response

I am trying to write a C# program to prolong the deadline for my books in my university's library. What I want to do is the following:
1.) Login to library website via WebRequest & POST method, with username & password entered in C# program
2.) Get the url to "View borrowed books" site containing the encrypted password & plain text username as GET parameters
3.) Download content of named page to display to user in the C# program
4.) If the user presses the corresponding button in the program, submit the prolongation form to the website to prolong all media at once.
Right now I'm stuck between 1 and 2, I seem to be able to connect to the website and to enter the userdata, but the WebResponse I get is again the login page (which is not the case if you login manually on the website).
This is the method I wrote to connect to the website:
// Login function, logs the user in, uses passed user number & password
public static Boolean userLogin(String unr, String pass)
{
// Login
// Cookie needed for maintaining php session
CookieContainer cContainer = new CookieContainer();
Console.WriteLine(unr+","+pass);
String postUrl = "https://universitylibrary.com/loan/DB=4/LNG=DU/USERINFO_LOGIN";
String formParams = String.Format("ACT={0}&HOST_NAME={1}&HOST_PORT={2}&HOST_SCRIPT={3}&LOGIN={4}&STATUS={5}&BOR_U={6}&BOR_PW={7}","UI_DATA","","","","KNOWNUSER","HML_OK", unr, pass);
String cookieHeader;
WebRequest wreq = WebRequest.Create(postUrl);
wreq.ContentType = "application/x-www-form-urlencoded";
wreq.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
wreq.ContentLength = bytes.Length;
using (Stream os = wreq.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = wreq.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
//Authentication trial
String PageSource;
String getUrl = "https://universitylibrary.com:443/loan/DB=4/USERINFO";
WebRequest getReq = WebRequest.Create(getUrl);
getReq.Headers.Add("Cookie",cookieHeader);
WebResponse getResp = getReq.GetResponse();
using (StreamReader sr = new StreamReader(getResp.GetResponseStream()))
{
PageSource = sr.ReadToEnd();
}
Console.Write(PageSource);
return true;
}
Can you see my mistake? I get the sourcecode and the params (username, password) output on the console, but the output is again the login page.
I would just look at the php page, but I don't have access to any of the internal system data, all I have is the HTML page.
Any suggestions would be highly appreciated!
EDIT:
I have rethought the whole thing, and rebuilt the HTTP request header completely as recorded by fiddler. That part of the function looks like this now:
// Login
// Cookie needed for maintaining php session
CookieContainer cContainer = new CookieContainer();
HttpCookie cookie = new HttpCookie("cookie", "PSC_4='xxxxxxx'; DB='n'");
CookieCollection cookieCol = new CookieCollection();
cookieCol.Add(cookieCol);
cContainer.Add(cookieCol);
Console.WriteLine(unr+","+pass);
String postUrl = "https://universitylibrary.com:443/loan/DB=4/USERINFO";
String formParams = String.Format("ACT={0}&HOST_NAME={1}&HOST_PORT={2}&HOST_SCRIPT={3}&LOGIN={4}&STATUS={5}&BOR_U={6}&BOR_PW={7}","UI_DATA","","","","KNOWNUSER","HML_OK", unr, pass);
String cookieHeader;
HttpWebRequest wreq = (HttpWebRequest) WebRequest.Create(postUrl);
wreq.Referer = "https://universitylibrary.com/loan/DB=4/LNG=DU/USERINFO_LOGIN";
wreq.KeepAlive = true;
wreq.ContentLength = 119;
wreq.Host = "universitylibrary.com";
wreq.ContentType = "application/x-www-form-urlencoded";
wreq.Method = "POST";
wreq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";
wreq.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
wreq.SendChunked = true;
wreq.TransferEncoding = "gzip, deflate";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
wreq.ContentLength = bytes.Length;
using (Stream os = wreq.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
cookieHeader = "";
try
{
HttpWebResponse resp = (HttpWebResponse) wreq.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
}
catch (WebException ex)
{
Console.WriteLine(ex.Status);
Console.WriteLine(ex.Response);
}
The whole thing still doesn't work though, same issue as before.
Is it possible that HttpWebRequest can't handle https or that something else is missing for https to work? (HTTP & HTTPS seem to be syntactically identical and the port is correctly set to 443, the real difference seems to lay in the additional SSL/TLS layer, maybe I need to add this somewhere?)

When you are returned the login page it probably means that you have not been correctly authenticated. There could be several reasons for this, but in the end it is because you not mirroring the HTTP communication from the manual login on the website correctly.
What I usually do is to use a monitor such as Fiddler to capture the full request/response pattern from the manual login, which I can then subsequently mirror in my code.

You can just modify your login page after checking login data and make the only output is "SuccessfullySignIn", for example.
If this is the data your receive,
$if (getReq == "SuccessfullySignIn")
{
//Do something
}
And try to use Redirect features in your webpage

Scraping a website to get the element name and id through C# web browser

I am trying to scrape a website to get the Textarea information.
I'm using:
HtmlDocument doc = this.webBrowser1.Document;
When I look at the view source it shows <textarea name="message" class="profile">
But when I try to access this textarea with:
HtmlDocument doc = this.webBrowser1.Document;
doc.GetElementsByTagName("textarea")
.GetElementsByName("message")[0]
.SetAttribute("value", "Hello");
It shows the error:
Value of '0' is not valid for 'index'. 'index' should be between 0 and -1.
Parameter name: index
Any Help?

For your current need you can simply use this:
doc.GetElementsByTagName("textarea")[0].InnerText = "Hello";
For complex things you can use HtmlDocument class with MSHTML class.

I can entrust HtmlAgilityPack to you!
I'd like to think that you try to access a website that uses cookies to determine if a user is logged in (or not). If not, it will force you to register/log-in else you aren't allowed to see anything. Am I right?
Your browser stores that cookies, your C# does not! (broadly speaking)
You need to create a cookie container to solve that problem.
Your C#-App may log-in, request a cookie/session, may grab the Cookies from the responseheader and then you should be able to scrape the profiles or whatever you want.
Get the Post Data, which is send to server. You can use tools/addons like Fiddler, Tamper, ect..
E.g. PostdataString: user_name=TESTUSER&password=TESTPASSWORD&language=en&action%3Asubmit=Submit
Here is a snippet you can use.
//Create the PostData
string strPostData = "user_name=" + txtUser.Text + "&password=" + txtPass.Text + "&language=en&action%3Asubmit=Submit";
CookieContainer tempCookies = new CookieContainer();
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(strPostData);
//Create the Cookie
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.website.com/login.php");
request.Method = "POST";
request.KeepAlive = true;
request.AllowAutoRedirect = false;
request.Accept = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.ContentType = "application/x-www-form-urlencoded";
request.Referer = "http://www.website.com/login.php";
request.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1";
request.ContentLength = data.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(data, 0, data.Length);
HttpWebResponse response;
response = (HttpWebResponse)request.GetResponse();
string sRequestHeaderBuffer = Convert.ToString(response.Headers);
requestStream.Close();
//Stream(-output) of the new website
StreamReader postReqReader = new StreamReader(response.GetResponseStream());
//RichTextBox to see the new source.
richTextBox1.Text = postReqReader.ReadToEnd();
You will need to adjust the Cookie-parameters in between and add your current sessionid aswell to the code. This depends on the requested website you visit.
E.g.:
request.Headers.Add("Cookie", "language=en_US.UTF-8; StationID=" + sStationID + "; SessionID=" + sSessionID);

programmatically clicking links in c#

I'am trying to make a script to programmatically.
I managed to get the hole html page to a string, Now I want to somehow click the elements that I have there. I'm kind of lost so any info could help.
I'v tried to get the document as a HtmlDocument but for some reason when I use the getElementById method it doesnt find the element.
Please, Any info would help.
Thanks.
Currently this is the code i'v got,
It brings me up to the point where i have string that is it's value is the html document, now I need to some how extract the relavent tag and click it programmatically.
Thanks for your inputs,
Still waiting for one that can help me.
string email = "someemail*";`enter code here`
string pw = "somepass";
string PostData = String.Format("email={0}&pass={1}", email, pw);
CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://www.facebook.com/*******");
req.CookieContainer = cookieContainer;
req.Method = "POST";
req.ContentLength = PostData.Length;
req.ContentType = "application/x-www-form-urlencoded";
req.AllowAutoRedirect = true;
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] loginDataBytes = encoding.GetBytes(PostData);
req.ContentLength = loginDataBytes.Length;
Stream stream = req.GetRequestStream();
stream.Write(loginDataBytes, 0, loginDataBytes.Length);
HttpWebResponse webResp = (HttpWebResponse)req.GetResponse();
Stream datastream = webResp.GetResponseStream();
StreamReader reader = new StreamReader(datastream);
string sLine = "";
string json = "";
while (sLine != null)
{
sLine = reader.ReadLine();
json += sLine;
}
json.ToString();

perhaps you might want to look at WaitN, it allows you to do all this really really easily

A "click on a link" is the same thing as sending a HTTP request. If you can parse the URI from the document you have, you can create the HTTP request separately and send that.

Clicking on a link is done by issuing a HTTP-Get for the href of the link.
If there is JavaScript interactivity, then you need to take a webbrowser element, and inject a javascript, that on document.ready executes document.getElementById("whatever").click()
See
How do I programmatically click a link with javascript?
You can use HTML agility pack to parse a HTML document and extract the HREF argument.

C# request a webpage failed but successd using web browser (Well checked the header and cookies)

My friend is using C# to write a simple program for requesting a webpage.
However he encounter a problem when try to request a specified webpage.
He have already tried to set all the header and cookie inside the request, but it still got the timeout exception.
The example webpage is https://my.ooma.com
Here is the code:
string url = "https://my.ooma.com";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 30000;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5";
request.Method = "GET";
request.CookieContainer = new CookieContainer();
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3");
request.Headers.Add("Accept-Encoding:gzip,deflate,sdch");
request.Headers.Add("Accept-Language:en-US,en;q=0.8");
request.KeepAlive = true;
WebResponse myResponse = request.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream());
string result = sr.ReadToEnd();
sr.Close();
myResponse.Close();
All the headers is as same as when using Chrome to browse the webpage.
And he didn't see any cookies set by using the Chrome developer tool.
Do anyone can success request the page using C#?
Thanks a lot.

Sorry for being late.
The following code snippet should work just fine. I also tried with tour old URL that had "getodds.xgi" in it and it also worked fine.
The server uses a secure sockets layer (SSL) protocol for connections that use the Secure Hypertext Transfer Protocol (HTTPS) scheme only.
You don't need to set any UserAgent or Header if they were just intended to get response.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
WebRequest request = WebRequest.Create("http://my.ooma.com/");
string htmlResponse = string.Empty;
using (WebResponse response = request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
htmlResponse = reader.ReadToEnd();
reader.Close();
}
response.Close();
}

HttpWebRequest to https server works in fiddler but NOT from Visual Studio

Here is the scenario. I have written code use a digital certificate to GET cookies from a secured url, to in turn POST data back to another url using the retrieved cookies and same digital certificate. The GET works and cookies are retrieved, the POST comes back with an error 500. I installed fiddler to see what was going on...POST looks fine...cookies are present. I used the feature in fiddler that allows creating a request via drag and drop. POST the exact same POST recorded from C# code that was recorded in fiddler and it works!
What is Fiddler doing that Visual Studio is not? It must be doing something if fiddler can POST the data but Visual Studio returns an error 500. Here is the code below:
X509Certificate cert = new X509Certificate("mycert.pfx", "certpassword");
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
req.CookieContainer = new CookieContainer();
req.Method = "GET";
req.ClientCertificates.Add(cert);
HttpWebResponse Response = (HttpWebResponse)req.GetResponse();
CookieCollection ck = req.CookieContainer.GetCookies(req.RequestUri);
string strcookie = ck[0].Value;
string strcookie2 = ck[1].Value;
Response.Close();
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
req.CookieContainer = new CookieContainer();
Cookie auth = new Cookie("_wl_authcookie_", strcookie2);
Cookie jsess = new Cookie("JSESSIONID", strcookie);
auth.Domain = "server";
jsess.Domain = "server";
req.CookieContainer.Add(auth);
req.CookieContainer.Add(jsess);
req.ClientCertificates.Add(cert);
req.Method = "POST";
Byte[] data = ReadByteArrayFromFile("filewithdatatoPOST.txt");
req.ContentLength = data.Length;
Stream myStream = req.GetRequestStream();
myStream.Write(data, 0, data.Length);
myStream.Close();
HttpWebResponse Response2 = (HttpWebResponse)req.GetResponse();
Stream strm = Response2.GetResponseStream();
StreamReader sr2 = new StreamReader(strm);
Response2.Close();

Does your code work if you set
req.ServicePoint.Expect100Continue = false;
on all your WebRequests?

Solved!!! It wasn't related to the Expect100Continue.
So after weeks of troubleshooting....6 different programmers....I figured it out. I'm not sure if this is always true, but in this scenario the problem was that the url we were getting cookies from:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETcookies/fileUpload.html");
was not the same as the url that we were posting the data back to:
req = (HttpWebRequest)WebRequest.Create("https://servertoPOSTdatawithcookies/main");
Getting the cookies and posting back to the same url fixed the issue:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");
req = (HttpWebRequest)WebRequest.Create("https://servertoGETandPOSTcookies/main");