We Keep Coding

How to read a page that uses JS to block scraping?

I'm using Xamarin (C#). I tried this code to get data I need in my app:
String url = "http://mmehdirajabiigdl.gigfa.com/VideoImageDownloader.php?link=https://www.instagram.com/tv/CJ_YPTLJ8Zx/?igshid=1jcvto1p6ekxx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
string mm = sr.ReadToEnd();
The response is wrong because this page is using JS that blocks my codes from getting HTML codes.
Well how can I fix this? I know WebRequest has no "Enabling JS". Maybe I should use WebBrowser but Xamarin has no WebBrowser.

As i know, HttpWebRequest just does a HTTP request and you could scrape the static HTML. It does not support JavaScript.
You could execute Javascript with Xamarin Forms WebView.
For more details, please check the link below. https://www.xamarinhelp.com/xamarin-forms-webview-executing-javascript/

Webclient c# get cookie from url in loop

I'm trying to get cookie-response from a website.
For example each time I visit google.com I would like to retrieve new cookie values, is this possible? Below I'll have a code using webbrowser to retrieve cookies :
var cookies = FullWebBrowserCookie.GetCookieInternal(webBrowser3.Url,false);
I tried httpwebrequest also using codes I found on this site, but I can not get new cookie-values from the website.
All help would be appreciated
Regards,
dennis

Ok so I found a working method over here but i can not put this code in a loop.
var StringURL = "http://www.google.com";
HttpWebRequest request = null;
request = HttpWebRequest.Create(StringURL) as HttpWebRequest;
HttpWebResponse TheRespone = (HttpWebResponse)request.GetResponse();
String setCookieHeader = TheRespone.Headers[HttpResponseHeader.SetCookie];
textBox18.Text = TheRespone.Headers[HttpResponseHeader.SetCookie];
If i put this code into my application, which runs in a loop the application get stuck. When I put this code on a button, I can push button once and it prints out the cookie perfect, but when I hit the button again the app will get stuck also. So if there is anyone with a solution for this please be so kind to explain what I'm doing wrong.

Login to website using C#

Before everyone gets upset that this has been answered. I have scoured the web looking for how to do this and have tried a number of methods. Login to website, via C# and How to programmatically log in to a website to screenscape? Both of these were helpful but I cannot figure out why I cannot get past the login page. Here is my code:
string url = "https://www.advocare.com/login.aspx";
string url2 = "https://url.after.login";
HttpWebRequest wReq = WebRequest.Create(url) as HttpWebRequest;
wReq.KeepAlive = true;
wReq.Method = "POST";
wReq.AllowAutoRedirect = false;
wReq.ContentType = "application/x-www-form-urlencoded";
string postData = "ctl00$cphContent$txtUserName=Username&ctl00$cphContent$txtPassword=Password";
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
wReq.ContentLength = dataBytes.Length;
using (Stream postStream = wReq.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse wResp = wReq.GetResponse() as HttpWebResponse;
string pageSource;
wReq = WebRequest.Create(url2) as HttpWebRequest;
wReq.CookieContainer = new CookieContainer();
wReq.CookieContainer.Add(wResp.Cookies);
HttpWebResponse wResp2 = wReq.GetResponse() as HttpWebResponse;
using (StreamReader sr = new StreamReader(wResp2.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Everytime I look at pageSource it is the HTML for the login.aspx page. I must be missing something here. Maybe it's not taking the cookie, I don't know. One question I have aside from, why doesn't this work, is in the string postData = "". Are those suppose to be the name or id portion of the html tag? Any help on this is greatly appreciated as I am stumped and will have to find a different way. I would like to continue with the WebRequest and WebResponse instead of using WebBrowser. If I can't, oh well. Thanks again for any help!

What are you trying to do besides login? If its like QAing a site programically, i would suggest using selenium andcreate a c# app based off of that. If u want i can post a link to a base project for a selenium based project.

Don't necessarily view the page source, but look at the actual HTTP POST. Install a HTTP proxy such as Fiddler and then re-visit the page you are trying to emulate. Complete the HTTP POST request, and check out the results produced in the proxy. From there you'll be able to see the actual parameters, cookies, headers, etc. that are being passed and you can then attempt to replicate this in your code. It's often easy to miss something when simply viewing the HTML source but monitoring the network traffic is pretty straight forward.

get page source code after redirection

I tried some ways to get the page source code of the following website http://www.poppe-bedrijfswagens.nl. This website has a auto redirection set I think.
I tried following ways:
WebClient client = new WebClient();
string sourceCode = "";
sourceCode = client.DownloadString(address);
And
HttpWebRequest myWebRequest = (HttpWebRequest)HttpWebRequest.Create(address);
myWebRequest.AllowAutoRedirect = true;
myWebRequest.Method = "GET";
// make request for web page
HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse();
StreamReader myWebSource = new StreamReader(myWebResponse.GetResponseStream());
string myPageSource = myWebSource.ReadToEnd();
myWebResponse.Close();
I always get the source code of the first page but i need to get the source code of the page that the website redirected to.
The redirection for http://www.poppe-bedrijfswagens.nl is:
Type of redirect: “meta refresh” redirect after 0 second
Redirected to: http://www.poppe-bedrijfswagens.nl/daf-html/dealer_homepage.html
thanks in advance

The AllowAutoRedirect property is relevant when the redirection is done with an HTTP status code 302. A meta refresh isn't technically a redirection because you are loading the first page.
You can download the first page though and then search the DOM for the element you're interested in <meta http-equiv="refresh" content="0;url=HTTP://WWW.NEXT-URL.COM"> and then download the page you're interested in.

Issues retrieving facebook social plugin comments for page, C# HttpWebRequest class

I'm hoping I've done something knuckle-headed here and there is an easy answer. I'm simply trying to retrieve the list of comments for a page on my site. I use the social plug-in and then retrieve the comment id via the edge event. Server side I send the page id back and do a simple request using a HttpWebRequest. Worked well back in October, but now I get an 'internal error' response from FB. I can use the same url string put it into a browser and get the comments back in the browser in json.
StringBuilder url = new StringBuilder();
url.Append("https://graph.facebook.com/comments/?ids=" + comment.page);
string requestString = url.ToString();
HttpWebRequest request = WebRequest.Create(requestString) as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
Ideas? Thanks much in advance.

Since you're using the Facebook C# SDK (per your tag), try:
var url = "{your url}";
var api = new Facebook.FacebookClient(appId,appSec);
dynamic commentsObj = api.Get("/comments/?ids=" + url);
dynamic arrayOfComments = commentsObj[url].data