I want to visit a web page (it has to be accessed, nothing needs to be read, modified, etc. Just accessed). I don't want to use webbrowser.
Just do a cURL GET request.
curl http://example.com/
And if you want to use C#, then
using System.Net;
string url = "https://www.example.com/";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
Also, you can use Fiddler to send request to the remote server (it is very helpful for service debuging).
try to use WebClient class:
for example:
WebClient client = new WebClient ();
string reply = client.DownloadString (address);
Related
I'm using Xamarin (C#). I tried this code to get data I need in my app:
String url = "http://mmehdirajabiigdl.gigfa.com/VideoImageDownloader.php?link=https://www.instagram.com/tv/CJ_YPTLJ8Zx/?igshid=1jcvto1p6ekxx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
string mm = sr.ReadToEnd();
The response is wrong because this page is using JS that blocks my codes from getting HTML codes.
Well how can I fix this? I know WebRequest has no "Enabling JS". Maybe I should use WebBrowser but Xamarin has no WebBrowser.
As i know, HttpWebRequest just does a HTTP request and you could scrape the static HTML. It does not support JavaScript.
You could execute Javascript with Xamarin Forms WebView.
For more details, please check the link below. https://www.xamarinhelp.com/xamarin-forms-webview-executing-javascript/
Before everyone gets upset that this has been answered. I have scoured the web looking for how to do this and have tried a number of methods. Login to website, via C# and How to programmatically log in to a website to screenscape? Both of these were helpful but I cannot figure out why I cannot get past the login page. Here is my code:
string url = "https://www.advocare.com/login.aspx";
string url2 = "https://url.after.login";
HttpWebRequest wReq = WebRequest.Create(url) as HttpWebRequest;
wReq.KeepAlive = true;
wReq.Method = "POST";
wReq.AllowAutoRedirect = false;
wReq.ContentType = "application/x-www-form-urlencoded";
string postData = "ctl00$cphContent$txtUserName=Username&ctl00$cphContent$txtPassword=Password";
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
wReq.ContentLength = dataBytes.Length;
using (Stream postStream = wReq.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse wResp = wReq.GetResponse() as HttpWebResponse;
string pageSource;
wReq = WebRequest.Create(url2) as HttpWebRequest;
wReq.CookieContainer = new CookieContainer();
wReq.CookieContainer.Add(wResp.Cookies);
HttpWebResponse wResp2 = wReq.GetResponse() as HttpWebResponse;
using (StreamReader sr = new StreamReader(wResp2.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Everytime I look at pageSource it is the HTML for the login.aspx page. I must be missing something here. Maybe it's not taking the cookie, I don't know. One question I have aside from, why doesn't this work, is in the string postData = "". Are those suppose to be the name or id portion of the html tag? Any help on this is greatly appreciated as I am stumped and will have to find a different way. I would like to continue with the WebRequest and WebResponse instead of using WebBrowser. If I can't, oh well. Thanks again for any help!
What are you trying to do besides login? If its like QAing a site programically, i would suggest using selenium andcreate a c# app based off of that. If u want i can post a link to a base project for a selenium based project.
Don't necessarily view the page source, but look at the actual HTTP POST. Install a HTTP proxy such as Fiddler and then re-visit the page you are trying to emulate. Complete the HTTP POST request, and check out the results produced in the proxy. From there you'll be able to see the actual parameters, cookies, headers, etc. that are being passed and you can then attempt to replicate this in your code. It's often easy to miss something when simply viewing the HTML source but monitoring the network traffic is pretty straight forward.
I have the following code which connects to my php server and retrieves data from it. The only thing is, I need to send the username and password securely from this webrequest to the PHP server. Looking at the docs for the webrequest class, there is a credentials property as well as a preauthenticate property. I'm assuming these are for the network credentials (all my users are in AD).
Is it possible to secure this post request with credentials or is this just a bad idea? I've also found SetBasicAuthHeader - I'll read up on this and see if it might help. All traffic will be on SSL from ASPX site to the PHP site
// variables to store parameter values
string url = "https://myphpserver.php";
// creates the post data for the POST request
string postData = "Username=" + username + "&Password=" + "&UID=" + UniqueRecID;
// create the POST request
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = postData.Length;
// POST the data
using (StreamWriter requestWriter2 = new StreamWriter(webRequest.GetRequestStream()))
{
requestWriter2.Write(postData);
}
// This actually does the request and gets the response back
HttpWebResponse resp = (HttpWebResponse)webRequest.GetResponse();
string responseData = string.Empty;
using (StreamReader responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream()))
{
// dumps the HTML from the response into a string variable
responseData = responseReader.ReadToEnd();
}
SetBasicAuthHeader is for HTTP Basic Access Authentication so won't help here as you're handling authentication at application level. Really, this is no more insecure than just going to the page in a browser. I see you're using SSL so your request will be encrypted anyway and you have nothing to worry about.
If you're concerned for some other reason (although I can't think why), it sounds like you have control over the PHP end so you could just encrypt the password and add an extra POST parameter so the server knows to decrypt it.
When using HTTPS your data is safe in the message and transport scope. It means no one can decode it or sniff the packets. I suggest you read this article HTTPS Wiki
Do I need to just slap some random garbage data in a WebRequest object to get by the HTTP status code 411 restriction on IIS?
I have an HttpPost action method in an MVC 3 app that consumes a POST request with all the relevant information passed in the querystring (no body needed).
[HttpPost] public ActionResult SignUp(string email) { ... }
It worked great from Visual Studio's built in web host, Cassini. Unfortunately, once the MVC code was live on IIS [7.5 on 2008 R2], the server is pitching back an HTTP error code when I hit it from my outside C# form app.
The remote server returned an error:
(411) Length Required.
Here is the calling code:
WebRequest request = WebRequest.Create("http://somewhere.com/signup/?email=a#b.com");
request.Method = "POST";
using (WebResponse response = request.GetResponse())
using (Stream responseStream = response.GetResponseStream())
using (StreamReader responseReader = new StreamReader(responseStream)) {
// Do something with responseReader.ReadToEnd();
}
Turns out you can get this to go through by simply slapping an empty content length on the request before you send it.
WebRequest request = WebRequest.Create("http://somewhere.com/signup/?email=a#b.com");
request.Method = "POST";
request.ContentLength = 0;
Not sure how explicitly giving an empty length vs. implying one makes a difference, but IIS was happy after I did. There are probably other ways around this, but this seems simple enough.
I believe you are required to set a Content-Length header anytime you post a request to a web server:
http://msdn.microsoft.com/en-us/library/system.web.httprequest.contentlength.aspx
You could try a GET request to test it.
When I try to fetch the content of this URL
http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html
using System.Net;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();
I don't get any response from the server. Please help me find out why this happens.
Thanks in advance!
It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.
Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.
Try setting request.Referer = "http://www.google.com";. Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.
I had the same problem and the cause was that I previously had set the method to HEAD and in later revisions had the need to parse the body.