We Keep Coding

c# screen scraping and getting all the cookies for secured access to a website

I'm trying to access a website through c# program. There seems to be three cookies needed to access the website yet I only receive two in my cookie container so when I try to access other parts the website I can't. I first do a GET then a POST. The reason I programmed it this way because it seemed from the Chrome Dev tools I determined that it first used a GET for the first two and then a POST to login and get the third one. The POST shows a 302 Moved Temporarily and then right after that it's a redirect. Which I believe is the reason I can't obtain the last cookie can anyone shed any light?
cookieJar = new CookieContainer();
string formParams = string.Format("USERNAME={0}&PASSWORD={1}", username, password);
Console.Write(" \n 1st count before anything : " + cookieJar.Count + "\n"); // 0 cookies
//First go to the login page to obtain cookies
HttpWebRequest loginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
loginRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//.Connection = "keep-alive";
loginRequest.Method = "GET";
loginRequest.UseDefaultCredentials = true;
loginRequest.CookieContainer = cookieJar;
loginRequest.AllowAutoRedirect = false;
HttpWebResponse loginResponse = (HttpWebResponse)loginRequest.GetResponse();
Console.Write(" \n 2nd count after first response : " + cookieJar.Count + "\n"); // Only 2 are recorded.
//Create another request to actually log into website
HttpWebRequest doLogin = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
doLogin.Method = "POST";
doLogin.ContentType = "application/x-www-form-urlencoded";
doLogin.AllowAutoRedirect = false;
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
doLogin.ContentLength = bytes.Length;
using (Stream os = doLogin.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
oLogin.CookieContainer = cookieJar;
doLogin.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36";
doLogin.Referer = "https://server.com/login/login.jsp";
HttpWebResponse Response = (HttpWebResponse)doLogin.GetResponse();
Console.Write(" \n 3rd count after second repsonse : " + cookieJar.Count + "\n"); // still two

HttpWebRequest had a problem with cookies.
The problem was that a cookie that was assigned to "server.com" would be changed to ".server.com". However, "server.com" did not match ".server.com".
If you are using a framework older than (I think it is 3) you are probably experiencing this problem.
The work around is to use e.g. "www.server.com" in your request, this will match cookies assigned to ".server.com".

FreeAgent API (OAuth) From c#

I'm trying to connect to a API (In this case freeagent) to grab some data. I've used Googles OAuth Playground to generate me a token.
var req = (HttpWebRequest)WebRequest.Create("https://api.freeagent.com/v2/recurring_invoices");
req.Headers["Authorization"] = "Bearer " + Convert.ToBase64String(Encoding.ASCII.GetBytes(_accessToken));
req.ContentType = "application/xml";
req.Accept = "application/xml";
req.Method = "GET";
// and get the response
var resp = req.GetResponse();
var streamIn = new StreamReader(resp.GetResponseStream());
returnData = streamIn.ReadToEnd();
streamIn.Close();
return resp;
Now I'm loosly trying to conver the following : https://dev.freeagent.com/docs/using_curl
I constantly get a Bad Request HTTP 400 - Anyone have any suggestions on what could be causing this problem?

I have most of the api working using RestSharp:
https://github.com/nicwise/FreeAgent
I wrote that for my iPhone app, mobileAgent, but it's fairly basic
.NET, so it should work on anything.
My authentication injection code looks very much like yours:
protected void SetAuthentication(RestRequest request)
{
request.AddHeader("Authorization", "Bearer " +
Client.CurrentAccessToken.access_token);
}
Have you run fiddler or something similar to see exactly what is going
over the wire? that often helps.

The solution to this, is to pass the UserAgent as well.
Example...
req.UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17";

You shouldn't need to base64 encode the token, just set the Authorization header to "Bearer " + _accessToken. If this still doesn't work, try using http://httpbin.org/get as the URL to look at your request to see if there are any other possible problems.

C# http(s) GET request problems

Somehow I believe I am not doing my HTTP requests correctly.
This site:
https://www.bintube.com/login/
Has 2 html input fields with names:
ctl00$main$Login1$UserName and ctl00$main$Login1$Password
The HTML code for the submit button is:
<input type="submit" name="ctl00$main$Login1$LoginButton" value="Login" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("ctl00$main$Login1$LoginButton", "", true, "Login1", "", false, false))" id="ctl00_main_Login1_LoginButton" />
Strangely enough, it is not contained within form tags and unfortunately the onclick stuff is a black box for me.
So in order to get the right parameters for the GET request, I have to use Fiddler, since I do not know how the GET URL parameters (timestamp and hash) are generated. It looks something like this in fiddler:
http://support.bintube.com/access/remote/?name=myname&email=myemail%40gmail.com&external_id=myexternal_id&organization=BinTube.com&timestamp=1353061816&hash=2372492438924389243
However this URL is totally different from the former mentioned URL https://www.bintube.com/login/ (SSL). So I do not know if I missed anything until it jumped to http://support.bintube.com/ (that is what Fiddler shows me) and, also, I always have to use Fiddler to give me the URL for my C# code, in order to login with my code. Ideally, my code would figure everything out on it's own so I do not have to use Fiddler.
My code is:
var cc = new CookieContainer();
List<string> paras = new List<string>();
string url;
string html;
// data copied from Fiddler, (changed some of it of course):
paras.Add(String.Format("{0}={1}", "name", HttpUtility.UrlEncode("myname")));
paras.Add(String.Format("{0}={1}", "email", HttpUtility.UrlEncode("myemail")));
paras.Add(String.Format("{0}={1}", "external_id", HttpUtility.UrlEncode("myexternal_id")));
paras.Add(String.Format("{0}={1}", "organization", HttpUtility.UrlEncode("BinTube.com")));
paras.Add(String.Format("{0}={1}", "timestamp", HttpUtility.UrlEncode("1353061816")));
paras.Add(String.Format("{0}={1}", "hash", HttpUtility.UrlEncode("2372492438924389243")));
url = "http://support.bintube.com/access/remote/" + "?" + paras.Aggregate((x, y) => x + "&" + y);
var req = (HttpWebRequest)WebRequest.Create(url);
Console.WriteLine(url);
req.CookieContainer = cc;
req.Method = "GET";
req.AllowAutoRedirect = true;
// also copied from Fiddler
req.Host = "support.bintube.com";
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11";
req.KeepAlive = true;
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//req.Headers["Accept-Encoding"] = "gzip,deflate,sdch"; //<- this give gibberish response, why?
req.Headers["Accept-Language"] = "de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4";
req.Headers["Accept-Charset"] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
cc.Add(resp.Cookies);
//rewrite cookies for further requests (found that somewhere, but did not help)
string ckheader = cc.GetCookieHeader(req.RequestUri);
cc.SetCookies(new Uri("http://www.bintube.com/"), ckheader);
StreamReader r = new StreamReader(resp.GetResponseStream());
html = r.ReadToEnd();
The code works. Sometimes it stops working then I have to get new GET parameters from Fiddler, which is ugly design and I am not happy about it at all. The second, bigger, problem is I cannot use
https://www.bintube.com/search/ (requires login for results)
with my recieved cookies. The recieved cookies (from the first request) are _zendesk_session and _zendesk_shared_session cookies, nothing important (googled it). So somehow I believe I don't get the right cookie to advance further.
Maybe someone has some hints for me on why I am failing so badly?

Can't get HTML code through HttpWebRequest

I am trying to parse the HTML code of the page at http://odds.bestbetting.com/horse-racing/today in order to have a list of races, etc.
The problem is I am not being able to retrieve the HTML code of the page. Here is the C# code of the function:
public static string Http(string url) {
Uri myUri = new Uri(url);
// Create a 'HttpWebRequest' object for the specified url.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(myUri);
myHttpWebRequest.AllowAutoRedirect = true;
// Send the request and wait for response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
var stream = myHttpWebResponse.GetResponseStream();
var reader = new StreamReader(stream);
var html = reader.ReadToEnd();
// Release resources of response object.
myHttpWebResponse.Close();
return html;
}
When I execute the program calling the function it throws an exception on
HttpWebResponse myHttpWebResponse =
(HttpWebResponse)myHttpWebRequest.GetResponse();
which is:
Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones.
I have read this question but I don't seem to have the same problem.
I've also tried iguring something out sniffing the traffic with fiddler but can't see anything to where it redirects or something similar. I just have extracted these two possible redirections: odds.bestbetting.com/horse-racing/2011-06-10/byCourse
and odds.bestbetting.com/horse-racing/2011-06-10/byTime , but querying them produces the same result as above.
It's not the first time I do something like this, but I'm really lost on this one. Any help?
Thanks!

I finally found the solution... it effectively was a problem with the headers, specifically the User-Agent one.
I found after lots of searching a guy having the same problem as me with the same site. Although his code was different the important bit was that he set the UserAgent attribute of the request manually to that of a browser. I think I had done this before but I may had done it pretty bad... sorry.
The final code if it is of interest to any one is this:
public static string Http(string url) {
if (url.Length > 0)
{
Uri myUri = new Uri(url);
// Create a 'HttpWebRequest' object for the specified url.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(myUri);
// Set the user agent as if we were a web browser
myHttpWebRequest.UserAgent = #"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4";
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
var stream = myHttpWebResponse.GetResponseStream();
var reader = new StreamReader(stream);
var html = reader.ReadToEnd();
// Release resources of response object.
myHttpWebResponse.Close();
return html;
}
else { return "NO URL"; }
}
Thank you very much for helping.

There can be a dozen probable causes for your problem.
One of them is that the redirect from the server is pointing to an FTP site, or something like that.
It can also being that the server require some headers in the request that you're failing to provide.
Check what a browser would send to the site and try to replicate.

Screen Scrape a page of a web app - Internal Server Error

I am tring to screen scrape a page of a web app that just contains text and is hosted by a 3rd party. It's not a properly formed HTML page, however the text that is diplayed will tell us if the web app is up or down.
When I try to scrape the sreen it returns an error when it tries the WebRequest. The error is "The remote server returned an error: (500) Internal Server Error."
public void ScrapeScreen()
{
try
{
var url = textBox1.Text;
var request = WebRequest.Create(url);
var response = request.GetResponse();
var stream = response.GetResponseStream();
var reader = new StreamReader(stream);
var result = reader.ReadToEnd();
stream.Dispose();
reader.Dispose();
richTextBox1.Text = result;
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}
}
Any ideas how I can get the text from the page?

Some sites don't like the default UserAgent. Consider changing it to something real, like:
((HttpWebRequest)request).UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.125 Safari/533.4"

First, try this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
However, if you're just looking for text and not having to do any POST-ing of data to the server, you may want to look at the webClient class. It more closely resembles a real browser, and takes care of a lot of HTTP header stuff that you may end up having to twek if you stick with the HttpWebRequest class.