I set up a proxy instance and used it with a webrequest object.
WebProxy a = new WebProxy("ip:port", true);
proxy.Credentials = new NetworkCredential("username", "password");
WebRequest b = WebRequest.Create("webpage url");
b.Proxy = proxy;
WebResponse c = req.GetResponse();
StreamReader d = new StreamReader(c.GetResponseStream());
d.ReadToEnd();//web page source
Works as it should, but I want to display the page in a web browser control without loss of information and design. If I set my control's document text to the source that was just downloaded. It has very bad formatting.
edit: Is there a way for me to apply the proxy object to the web browser control itself?
edit The WebBrowser control just uses IE's settings, so you don't have to set the proxy yourself. See http://social.msdn.microsoft.com/Forums/en-US/winforms/thread/f4dc3550-f213-41ff-a17d-95c917bed027/ how to set the IE proxy in code.
Well the problem here is that the HTML that you've received via the WebRequest contains relative paths to the CSS files that are not present in the current context. You can modify the HTML, by adding the following tag in the <head> section:
<base href="http://domainname.com/" />
After that the WebBrowser control resolves the relative CSS paths to the domain in this tag.
Related
I am trying to make a small app that can log in automatically on a website, get certain texts on the website and return to user.
To show what I have, I did below to make it log in,
System.Windows.Forms.HtmlDocument doc = logger.Document as System.Windows.Forms.HtmlDocument;
try
{
doc.GetElementById("loginUsername").SetAttribute("value", "myusername");
doc.GetElementById("loginPassword").SetAttribute("value", "mypassword");
doc.GetElementById("loginSubmit").InvokeMember("click");
And below to load html of the page
WebClient myClient = new WebClient();
Stream response = myClient.OpenRead(webbrowser.Url);
StreamReader reader = new StreamReader(response);
string src = reader.ReadToEnd(); // finally reading html and saving in variable
Now, it successfully loaded html but html of the page where it's not logged in. Is there a way to refer to current html somehow? Or another way to achieve my goals. Thank you for reading!
Use the Webclient class so you can use sessions and cookies.
check this Q&A: Using WebClient or WebRequest to login to a website and access data
Why don't you make REST API calls and send the data like username and password from your code itself?
Is there any Web API for the URL ? If yes , you can simply call the service and pass on the required parameters. The API shall return in JSON/XML which you can parse and extract information
Hello i making a simple httpwebrequest and then i read (StreamReader) the response and just want to get the html page of website,but i get only one laber(only one element of the page) in the browser all fine(i see all page) but when i try to set cookies to Deny\disable i also in the browser get this label(only one element of the page) and all is disappear.Sow i getting opinion if after i disabled cookies in browser i get the same page(like in code) that mean my HttpWebRequest is have settings cookies=deny/disable.
You can go to https://www.bbvanetcash.com/local_kyop/KYOPSolicitarCredenciales.html and disable cookies with F12 and you will see the difrance and also this page with one label.
Sow this my code any ideas what i need to change here?
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.bbvanetcash.com/local_kyop/KYOPSolicitarCredenciales.html");
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream streamResponseLogin = myHttpWebResponse.GetResponseStream();
StreamReader streamReadLogin = new StreamReader(streamResponseLogin);
LoginInfo = streamReadLogin.ReadToEnd();
Your code is receiving complete page content, but it cannot receive the dynamic contents. This is happening because the page you are trying to access relies on Cookies for maintaining session as well as JavaScript (it is using jQuery) for loading dynamic contents and providing rich user experience.
To successfully receive the whole page, your code must
support retrieving, storing and sending cookie objects across various HttpRequest and HttpResponse.
be able to execute JavaScript code to load the dynamic contents/markup of the page
To test 'if your code is receiving proper values or not' visit the site Web Sniffer and put your URL there.
As you can try on web-sniffer site, for www.google.com, the response you are getting is a redirect instruction.... that means, even to access the Google's home page, your code must understand HTTP status messages (302 there).
I'm trying to make an application in VS in C #.
Wanted navigate in webBrowser I have on my form through proxy.
But I can not.
With this code I can access a page by proxy.
But if I click within the page. I am already browsing by webBrowser directly without going through proxy.
proxyURI = new System.Uri("http://" + myProxy);
proxy = new System.Net.WebProxy(proxyURI);
request = (HttpWebRequest)WebRequest.Create("http://stackoverflow.com/");
request.Proxy = proxy;
response = (HttpWebResponse)request.GetResponse();
receiveStream = response.GetResponseStream();
webBrowser1.DocumentStream = receiveStream;
To make things easier is not possible to place the proxy in the WebBrowser?
Like:
webBrowser1.Document.GetElementById("login").byProxy("MyProxy").InvokeMember("click");
My apology but I'm noob in C #
Don't mix WebRequest and WebBrowser in this case, they have completely different Net sessions.
Use WebBrowser alone, and set the proxy with UrlMkSetSessionOption/INTERNET_OPTION_PROXY. Do this before you create WebBrowser (Otherwise, I'm not sure if existing WebBrowser instance picks up new proxy settings instantly).
I have a URL that I'd like to submit. This will cause a redirect on the target page. I'd then like to get the new URL.
This will occur inside of a library. So no winforms or WPF. Is there some type of web browser object available at this level (similar to the winforms web browser object)?
I don't need to do any page scraping. Just access to the new URL. But some type of eventing will need to be available so I'll know when it is ok to grab the new URL.
You can use HttpWebRequest:
HttpWebRequest httpWebRequest = (HttpWebRequest) WebRequest.Create(<submit URL>);
httpWebRequest.AllowAutoRedirect = true;
httpWebRequest.GetResponse();
// address after all redirections:
Uri address = httpWebRequest.Address;
I tried some ways to get the page source code of the following website http://www.poppe-bedrijfswagens.nl. This website has a auto redirection set I think.
I tried following ways:
WebClient client = new WebClient();
string sourceCode = "";
sourceCode = client.DownloadString(address);
And
HttpWebRequest myWebRequest = (HttpWebRequest)HttpWebRequest.Create(address);
myWebRequest.AllowAutoRedirect = true;
myWebRequest.Method = "GET";
// make request for web page
HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse();
StreamReader myWebSource = new StreamReader(myWebResponse.GetResponseStream());
string myPageSource = myWebSource.ReadToEnd();
myWebResponse.Close();
I always get the source code of the first page but i need to get the source code of the page that the website redirected to.
The redirection for http://www.poppe-bedrijfswagens.nl is:
Type of redirect: “meta refresh” redirect after 0 second
Redirected to: http://www.poppe-bedrijfswagens.nl/daf-html/dealer_homepage.html
thanks in advance
The AllowAutoRedirect property is relevant when the redirection is done with an HTTP status code 302. A meta refresh isn't technically a redirection because you are loading the first page.
You can download the first page though and then search the DOM for the element you're interested in <meta http-equiv="refresh" content="0;url=HTTP://WWW.NEXT-URL.COM"> and then download the page you're interested in.