We Keep Coding

How to deal with JavaScript when fetching http Response in C# using HttpWebRequest?

I have a C# code (it's a Web Application, that's hosted on IIS) where I use HttpWebRequest to get HttpWebResponse. There I make request to any website & get the response as string, then I analysis the response string. But recently I get the response where JavaScript performs data fetching after page is loaded in browser.
I tried to debug this in firebug & saw that at bottom of response there's a JavaScript function that updates the dom elements after pageload. Is there any way that I could do the same in my C# code. I have searched on net about this found not solution till now.
Following is the code I am using:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
cookieList.Add(cookie);
}
string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
byte[] postBytes = Encoding.UTF8.GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
req.KeepAlive = true;
req.AutomaticDecompression = DecompressionMethods.GZip;
////set the cookie
req.CookieContainer = new CookieContainer();
foreach (Cookie cook in cookieList)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
req.CookieContainer.Add(cookie);
}
req.Headers.Add("Accept-Encoding", "gzip, deflate");
req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
req.Method = "POST";
req.Host = "login.example.com";
req.Referer = "https://login.example.com/Login/logout";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded;";
req.ContentLength = postBytes.Length;
//getting the request stream and posting data
StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
requestwriter.Write(postData);
requestwriter.Close();
HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
Stream responseStream = myHttpWebResponse.GetResponseStream();
StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
string responseString = myStreamReader.ReadToEnd();
myStreamReader.Close();
responseStream.Close();
myHttpWebResponse.Close();

I finally got the easy solution to my need. following is the link that I followed:
Link to tutorial
Following is the code that'll get the results:
First you'll need to import following:
using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;
Now the code:
var options = new PhantomJSOptions();
var driver = new PhantomJSDriver(options);
driver.Manage().Window.Size = new Size(1360, 728);
var size = driver.Manage().Window.Size;
driver.Navigate().GoToUrl("https://example.com/");
string url = driver.Url;
//the driver can now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement1 = driver.FindElementByName("username");
var pathElement2 = driver.FindElementByName("password");
var pathElement3 = driver.FindElementByXPath("//button[#class='SubmitButton']");
pathElement1.Clear();
pathElement1.SendKeys("username");
pathElement2.Clear();
pathElement2.SendKeys("password");
pathElement3.Click();
//Now get the response after login button click
source = driver.PageSource;

HttpWebrequests facebook login and request app token

I'm trying to login on facebook and retrive a token by using this link:
https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token
My code looks like this, but i get an invalid link when i'm requesting the link above.
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php?login_attempt=1");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "https://www.facebook.com/login.php?login_attempt=1";
string postData = String.Format("email={0}&pass={1}", email, pass);
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies); //recover cookies First request
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
cookies = getResponse.Cookies;
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
string sourceCode = sr.ReadToEnd();
}
//Get the token
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
webRequest.AllowAutoRedirect = false;
HttpWebResponse rresponse = (HttpWebResponse)webRequest.GetResponse();
if (rresponse.StatusCode == HttpStatusCode.Redirect)
{
Console.WriteLine("redirected to: " + rresponse.GetResponseHeader("Location"));
}
Please help me. Thanks in advance.

https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token
The URL does not allow the application configuration.: The application settings do not allow one or more of the URLs. Internet Site URL and Canvas URL or domain URLs must be sub-domains of application domains.

C# WebRequest Login Session

Okay I tried asking this question yesterday but i'm not sure if I gave enough info, i got an answer but it hasn't worked for me. Basically what i'm doing is the user opens this windows forms application and logs in. Afterwhich they enter some text into a textbox and click run. At this point the run function is making a webrequest to a server that requires a login (the login that is initially done after they open the program. For some reason its still not seeing that the user is logged in when performing the second request even though the cookies are added too a cookie container. I'm not sure what i'm doing wrong but I will post my code so you can further help me.
This is the function that is performed for the login when the user enters the application.
private void button1_Click(object sender, EventArgs e)
{
string paramaters = "authmethod=on&chkRememberMe=on&login-form-type=pwd&password=" + pw.Text + "&userid=" + uid.Text + "&username=" + uid.Text;
string strResponse;
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://www.url.com/login.form");
requestLogin.Method = "POST";
requestLogin.CookieContainer = cookieJar;
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.ContentLength = paramaters.Length;
StreamWriter stOut = new StreamWriter(requestLogin.GetRequestStream(), System.Text.Encoding.ASCII);
stOut.Write(paramaters);
stOut.Close();
HttpWebResponse responseLogin = (HttpWebResponse)requestLogin.GetResponse();
StreamReader stIn = new StreamReader(responseLogin.GetResponseStream());
strResponse = stIn.ReadToEnd();
stIn.Close();
//Add cookies to CookieJar (Cookie Container)
foreach (Cookie cookie in responseLogin.Cookies)
{
cookieJar.Add(new Cookie(cookie.Name.Trim(), cookie.Value.Trim(), cookie.Path, cookie.Domain));
richTextBox2.Text += cookie.Name.ToString() + Environment.NewLine + cookie.Value.ToString() + Environment.NewLine + cookie.Path.ToString() + Environment.NewLine + cookie.Domain.ToString();
}
if (strResponse.Contains("Log On Successful") || strResponse.Contains("already has a webseal session"))
{
foreach (Control cont in this.Controls)
{
cont.Visible = true;
}
loginPanel.SendToBack();
loginPanel.Visible = false;
}
else
{
MessageBox.Show("Login failed.");
}
}
This is the function that is ran when the user clicks the "run" button to initiate the tests on a consumer account.
private string runTestRequest(Uri url, string parameters)
{
string testResults = string.Empty;
HttpWebRequest runTest = (HttpWebRequest)WebRequest.Create(url);
runTest.CookieContainer = cookieJar;
runTest.Method = "POST";
runTest.ContentType = "application/x-www-form-urlencoded";
StreamWriter stOut = new StreamWriter(runTest.GetRequestStream(), System.Text.Encoding.ASCII);
stOut.Write(parameters);
stOut.Close();
StreamReader stIn = new StreamReader(runTest.GetResponse().GetResponseStream());
testResults = stIn.ReadToEnd();
stIn.Close();
return testResults;
}
And of course this is my cookie container object
public CookieContainer cookieJar = new CookieContainer();
P.S.: The domains of the webrequests are different. First being abc.com 2nd being 123.com The only problem is that the first domain (which is the login) is a global login for internal web applications like 123.com, so how would i use the login session from the 1st domain with the 2nd domain?
Can you please assist in helping me figure out what I am doing wrong.

I found out that what was happening was it was redircting to a subdomain on the same domain as the 2nd (123.com) to use the login. Evidently they had this global login system built on the multiple domains to pass the cookies. The code above DOES work and i do have it working now. Thanks!!

string url = "http://www.ABC/MemberShip/Login.aspx";// HttpContext.Current.Request.Url.AbsoluteUri.ToString().Replace("AutoLogin", "Login");
CookieContainer myCookieContainer = new CookieContainer();
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.CookieContainer = myCookieContainer;
request.Method = "GET";
request.KeepAlive = false;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
System.IO.Stream responseStream = response.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(responseStream, Encoding.UTF8);
string srcString = reader.ReadToEnd();
// get the page ViewState
string viewStateFlag = "id=\"__VIEWSTATE\" value=\"";
int i = srcString.IndexOf(viewStateFlag) + viewStateFlag.Length;
int j = srcString.IndexOf("\"", i);
string viewState = srcString.Substring(i, j - i);
// get page EventValidation
string eventValidationFlag = "id=\"__EVENTVALIDATION\" value=\"";
i = srcString.IndexOf(eventValidationFlag) + eventValidationFlag.Length;
j = srcString.IndexOf("\"", i);
string eventValidation = srcString.Substring(i, j - i);
string submitButton = "LoginButton";
// UserName and Password
string userName = "userid";
string password = "password";
// Convert the text into the url encoding string
viewState = System.Web.HttpUtility.UrlEncode(viewState);
eventValidation = System.Web.HttpUtility.UrlEncode(eventValidation);
submitButton = System.Web.HttpUtility.UrlEncode(submitButton);
// Concat the string data which will be submit
string formatString =
"txtUserName={0}&txtPassword={1}&btnSignIn={2}&__VIEWSTATE={3}&__EVENTVALIDATION={4}";
string postString =
string.Format(formatString, userName, password, submitButton, viewState, eventValidation);
// Convert the submit string data into the byte array
byte[] postData = Encoding.ASCII.GetBytes(postString);
// Set the request parameters
request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "POST";
request.Referer = url;
request.KeepAlive = false;
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; CIBA)";
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = myCookieContainer;
System.Net.Cookie ck = new System.Net.Cookie("TestCookie1", "Value of test cookie");
ck.Domain = request.RequestUri.Host;
request.CookieContainer.Add(ck);
request.CookieContainer.Add(response.Cookies);
request.ContentLength = postData.Length;
// Submit the request data
System.IO.Stream outputStream = request.GetRequestStream();
request.AllowAutoRedirect = true;
outputStream.Write(postData, 0, postData.Length);
outputStream.Close();
// Get the return data
response = request.GetResponse() as HttpWebResponse;
responseStream = response.GetResponseStream();
reader = new System.IO.StreamReader(responseStream, Encoding.UTF8);
srcString = reader.ReadToEnd();
Response.Write(srcString);
Response.End();

Difficulties with google authentication

I am trying to authenticate google with the following code but google sent me back to the login page again.
//STEP# 1
string loginURL = "https://www.google.com/accounts/ServiceLoginBox?service=analytics&nui=1&hl=en-US&continue=https%3A%2F%2Fwww.google.com%2Fanalytics%2Fsettings%2F%3Fet%3Dreset%26hl%3Den%26et%3Dreset%26hl%3Den-US";
request = (HttpWebRequest)WebRequest.Create(loginURL);
request.CookieContainer = cookieJar;
request.Method = "GET";
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
sr.Close();
}
galx = ExtractValue(serverResponse,"GALX","name=\"GALX\" value=\"");
Console.WriteLine(galx);
//Request# 2
string uriWithData = "https://www.google.com/accounts/ServiceLoginBoxAuth";
request = (HttpWebRequest)WebRequest.Create(uriWithData);
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
request.Method = "POST";
request.CookieContainer = cookieJar;
string param = string.Format("Email={0}&Passwd={1}&continue={2}&service=analytics&nui=1&dsh=8209101995200094904&GALX={3}&hl=en-US&PersistentCookie=yes","**my email address**",p,"",galx);
byte[] postArr = StrToByteArray(param);
request.ContentType = #"application/x-www-form-urlencoded";
request.ContentLength = param.Length;
Stream reqStream = request.GetRequestStream();
reqStream.Write(postArr,0,postArr.Length);
reqStream.Close();
response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
Console.WriteLine(serverResponse);
// Close and clean up the StreamReader
sr.Close();
}

I have no idea an I am pretty sure not many people are going to want to sift through that much code.
One thing that I see at a glance that may be causing problems is that you are playing with the cookies too much.
Create one CookieContainer and just pass it in with each request.
No need to 'transfer' or 'recreate' the container.

Try looking into google accounts api and GBaseService. I believe you will have to set HOSTED_OR_GOOGLE in accountType as Marty has done here in this post.

Why am I getting a 401 (Unauthorized) error when POSTing to Google Reader API?

I'm trying to interface with the Google Reader (undocumented/unofficial) API using information from this page. My first step is to get a SID and token, which works fine, but I can't seem to POST anything without getting a 401 error.
Here is the code I'm using to get my SID and token:
static string getSid()
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://www.google.com/accounts/ClientLogin?service=reader&Email=username&Passwd=password");
req.Method = "GET";
string sid;
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
using (var stream = response.GetResponseStream())
{
StreamReader r = new StreamReader(stream);
string resp = r.ReadToEnd();
int indexSid = resp.IndexOf("SID=") + 4;
int indexLsid = resp.IndexOf("LSID=");
sid = resp.Substring(indexSid, indexLsid - 5);
return sid;
}
}
And to generate a cookie and get the token:
static Cookie getCookie(string sid)
{
Cookie cookie = new Cookie("SID", sid, "/", ".google.com");
return cookie;
}
static string getToken(string sid, Cookie cookie)
{
string token;
string url = "http://www.google.com/reader/api/0/token";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(cookie);
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
using (var stream = response.GetResponseStream())
{
StreamReader r = new StreamReader(stream);
token = r.ReadToEnd();
return token;
}
}
Now if I try to do a POST (in this example, insert a new tag) using the following, I get the 401 error.
static void insertTag(string tag, Cookie cookie)
{
string result = "";
string data = Uri.EscapeDataString("a="+tag+"&T=" + Program.token);
byte[] buffer = Encoding.GetEncoding(1252).GetBytes(data);
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create
("http://www.google.com/reader/api/0/edit-tag?client=-");
WebReq.Method = "POST";
WebReq.Credentials = new NetworkCredential("username", "password");
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = buffer.Length;
Stream PostData = WebReq.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse WebResp = (HttpWebResponse)WebReq.GetResponse();
Stream Answer = WebResp.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
result = _Answer.ReadToEnd();
if (result.Length < 0)
result = "";
}
The error occurs on the line Stream Answer = WebResp.GetResponseStream();

You need to double check that you have a user agent set. I have run into this same problem before when i didnt have it set.
For example:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Or: MSDN Link
myHttpWebRequest=(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
myHttpWebRequest.UserAgent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";

Turns out I was using the wrong URL to access the Google Reader APIs thanks to some outdated documentation! The correct URL for adding labels in Google Reader (as of August 2009) is
http://www.google.com/reader/api/0/subscription/edit?client=scroll
with POST arguments
a=user/-/label/[your label]&s=feed/[feed url]&ac=edit&T=[token]