We Keep Coding

How to read a page that uses JS to block scraping?

I'm using Xamarin (C#). I tried this code to get data I need in my app:
String url = "http://mmehdirajabiigdl.gigfa.com/VideoImageDownloader.php?link=https://www.instagram.com/tv/CJ_YPTLJ8Zx/?igshid=1jcvto1p6ekxx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
string mm = sr.ReadToEnd();
The response is wrong because this page is using JS that blocks my codes from getting HTML codes.
Well how can I fix this? I know WebRequest has no "Enabling JS". Maybe I should use WebBrowser but Xamarin has no WebBrowser.

As i know, HttpWebRequest just does a HTTP request and you could scrape the static HTML. It does not support JavaScript.
You could execute Javascript with Xamarin Forms WebView.
For more details, please check the link below. https://www.xamarinhelp.com/xamarin-forms-webview-executing-javascript/

HttpWebResponse text does not appear to be JSON

I am making a rest call in which I get a HttpWebResponse that contains data. It seems the data is serialized, and I am trying to get the plain text of the request. I have been using the chrome extension Advanced Rest client, which when calling the same request it is able to display the text version of the json response.
From what I have read on here, you are required to deserialize into the expected object. However, it is pretty clear that chrome plugin has no idea about the object type and can still print out plain text.
Is it possible to do the same in c#?
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
// [code removed for setting json data via stream writer
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// This is where I am trying to figure out how to get plain readable text out of response.GetResponseStream()
}
Edit: If I simply use a StreamReader to get the text from the response stream, I get a bunch of binary data and not the plain json text.
Edit: realized the problem had to do with compression. This can be closed.

I'm not sure if got it right, but you can get the response as a string doing this:
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}

Turned out my problem was due to compression. I realized the header contained "Content-Encoding: gzip" so I searched on how to unzip with gzip compression and then the text was proper json. Thanks all

Is it possible to access a webpage without a webbrowser?

I want to visit a web page (it has to be accessed, nothing needs to be read, modified, etc. Just accessed). I don't want to use webbrowser.

Just do a cURL GET request.
curl http://example.com/
And if you want to use C#, then
using System.Net;
string url = "https://www.example.com/";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();

Also, you can use Fiddler to send request to the remote server (it is very helpful for service debuging).

try to use WebClient class:
for example:
WebClient client = new WebClient ();
string reply = client.DownloadString (address);

Login to website using C#

Before everyone gets upset that this has been answered. I have scoured the web looking for how to do this and have tried a number of methods. Login to website, via C# and How to programmatically log in to a website to screenscape? Both of these were helpful but I cannot figure out why I cannot get past the login page. Here is my code:
string url = "https://www.advocare.com/login.aspx";
string url2 = "https://url.after.login";
HttpWebRequest wReq = WebRequest.Create(url) as HttpWebRequest;
wReq.KeepAlive = true;
wReq.Method = "POST";
wReq.AllowAutoRedirect = false;
wReq.ContentType = "application/x-www-form-urlencoded";
string postData = "ctl00$cphContent$txtUserName=Username&ctl00$cphContent$txtPassword=Password";
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
wReq.ContentLength = dataBytes.Length;
using (Stream postStream = wReq.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse wResp = wReq.GetResponse() as HttpWebResponse;
string pageSource;
wReq = WebRequest.Create(url2) as HttpWebRequest;
wReq.CookieContainer = new CookieContainer();
wReq.CookieContainer.Add(wResp.Cookies);
HttpWebResponse wResp2 = wReq.GetResponse() as HttpWebResponse;
using (StreamReader sr = new StreamReader(wResp2.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Everytime I look at pageSource it is the HTML for the login.aspx page. I must be missing something here. Maybe it's not taking the cookie, I don't know. One question I have aside from, why doesn't this work, is in the string postData = "". Are those suppose to be the name or id portion of the html tag? Any help on this is greatly appreciated as I am stumped and will have to find a different way. I would like to continue with the WebRequest and WebResponse instead of using WebBrowser. If I can't, oh well. Thanks again for any help!

What are you trying to do besides login? If its like QAing a site programically, i would suggest using selenium andcreate a c# app based off of that. If u want i can post a link to a base project for a selenium based project.

Don't necessarily view the page source, but look at the actual HTTP POST. Install a HTTP proxy such as Fiddler and then re-visit the page you are trying to emulate. Complete the HTTP POST request, and check out the results produced in the proxy. From there you'll be able to see the actual parameters, cookies, headers, etc. that are being passed and you can then attempt to replicate this in your code. It's often easy to miss something when simply viewing the HTML source but monitoring the network traffic is pretty straight forward.

Extract contents from HTTP request and then get selected contents from it

just for learning purpose, i ma playing with page Request and response.I need to know how can i achieve this.What i want to do is to make a HTTP request from windows application and extract some content from it. For example
I am calling http://stackoverflow.com/questions
now from response i want to extract all question nodes which is in <div id="questions"> and format that and then display this in Table. Can some body guide me how to do that. I here that i can do that formating and extracting thingy from regular expression too but i m not sure how.
Thanks in advance
Lura

I suggest using the HTML Agility Pack - it will allow you to get the page directly and query it using XPath, similar to how XmlDocument works.

You can use HttpWebRequest to get Source content of page the as follows.
string url = #"http://stackoverflow.com/users";
System.Net.WebRequest request = System.Net.HttpWebRequest.Create(url);
System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
System.IO.StreamReader stream = new System.IO.StreamReader
(response.GetResponseStream(), System.Text.Encoding.GetEncoding("utf-8"));
XmlDocument rssDoc = new XmlDocument();
rssDoc.Load(stream);