I am currently trying to do a screen scrape using the following code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse theResponse = (HttpWebResponse) request.GetResponse();
using (StreamReader reader = new StreamReader(theResponse.GetResponseStream(), Encoding.UTF8))
{
string s = reader.ReadToEnd();
}
However, the data I am concerned with (an HTML table) is not part of the result. When I right click the page and ViewSource, I also do not see the HTML table I care about - however I do see it in the DOM when I use Firebug to inspect it.
It doesn't seem to be loaded via ajax either.
So - is there another way, using C#, to get the DOM as it exists in the Developer Tool view, rather than the ViewSource result?
Unfortunately, this page is not publicly available so I can't paste the URL.
It doesn't seem to be loaded via ajax either.
You don't need to use AJAX in order to dynamically add data to the DOM. You could perfectly fine use standard javascript.
To scrape such page you need a scraper that processes javascript. The WebBrowser control in WinForms does that. It allows you to load a web page and explore the DOM, just as you do in FireBug (except that the snapshot comes from IE because the WebBrowser is just a wrapper around IE).
But since the WebBrowser control is not designed to be used in a multithreaded environment (such as a web application) you will have to use a third party library to achieve that scraping task.
Have you used Fiddler or Ethereal to see what URL's are being connected to in the background? If you find the HTML table in the response from one of the URL's called in the background, you can scrape the data from that URL. Which URL/table are you trying to parse?
Related
I have some code that connects to an HTTP API, and is supposed to get an XML response. When the API link is placed in a browser, the browser downloads the XML as a file. However, when the code connects to the same API, HTML is returned. I've told the API owner but they don't think anything is wrong. Is there a way to capture the downloaded file instead of the HTML?
I've tried setting the headers to make my code look like a browser. Also tried using WebRequest instead of WebClient. But nothing works.
Here is the code, the URL works in the browser (file downloaded) but doesn't work for WebClient:
WebClient webClient = new WebClient();
string result = webClient.DownloadString(url);
The code should somehow get the XML file instead of the page HTML (actually the HTML doesn't appear in the browser, only the file).
The uri that you access may be a HTML page that have its own mechanism (like generating the actual download address which may be dynamically generated by the server and redirecting to it, in order to prevent external linking access) to access the real file.
It is supposed to use a browser background browser core like CefSharp to run the HTML and its javascript to let it navigate, and probably you may want to hook the download event to handle the downloading.
I think you need to add accept header to the WebClient object.
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.Accept] = "application/xml;q=1";
string result = webClient.DownloadString(url);
}
Thank you all for your input. In the end, it was caused by our vendor switching to TLS 1.2. I just had to force my code to use 1.2 and then it worked.
I used this below code from to download html content from google search url
HttpClient client = new HttpClient();
var response = await client.GetAsync("https://www.google.co.in/?gws_rd=ssl#q=chiranjeevi+movies");
var sdata = response.Content.ReadAsStringAsync().Result;
When i inspect on browse i find "klitem" class with div tag, but when read my response it is not showing, i don't no why it's not showing when i download the url content, any one please help me...
When I inspect on browser I find "klitem" class with div tag, but when read my response it is not showing, I don't no why it's not showing when I download the url content.
When you inspect on browser, you can find some HTML elements. Actually, they are elements in a DOM tree rendered by the web browser. They are not in the response content.
If you look at the Network tool of the develop tool, you will find when you call your url, the browser will send several requests and in the first request, it returns a html page like following
And this is the same content that read from your response. Unlike web browser, HttpClient can only get the response from one request. We can not get the final DOM tree from it. Google won't return the search result directly. As #Mehrzad said, the final result is possibly created dynamically via JavaScript. If you check the Network tool, you can find there are a lot of JavaScript requests in them instead of a static page.
I have 100 pages on my site, but I want download only part a page instead of all page content.
I want just one box of each page to download, the file size is 10 KB.
For this I Use WebClient and htmlagilitypack .
WebClient Client = new WebClient();
var result = Encoding.GetEncoding("UTF-8").GetString(Client.DownloadData(URL));
Unfortunately, that's not possible, because HTTP is not designed to deliver a specific part of a web page. It does support range requests, but for that you would need to know where exactly (in terms of bytes) the desired content is located.
You can
download the whole page and then
use a HTML parsing library to extract the part you need.
You cannot achieve this.
The only solution is changing the website structure itself. if you have control of the server -
Change the architecture of your website, making the data in the box accessible via an ajax call.
Now you can get the data via the WebClient.
If that data is already served via a API call, you can point your WebClient to that URI Instead.
Here is an example of structuring you website based on ajax -
AJAX with jQuery and ASP.NET
How can i get the web representation of a received http-request?
I recently asked a question on how i could circumvetnt a website censorship through a third anonymizer website.the result indicated i sniff the traffic and create the post url myself.
This stage is almost done.Now i need to get the page as a result.
If you're using Winforms, you could go with the WebBrowser class.
The WebBrowser control lets you host Web pages and other browser-enabled documents in your Windows Forms applications. You can use the WebBrowser control, for example, to provide integrated HTML-based user assistance or Web browsing capabilities in your application. Additionally, you can use the WebBrowser control to add your existing Web-based controls to your Windows Forms client applications.
(From MSDN)
You wish to save the response to disk. Ok, typically this is the sequence you'd take:
You make the HTTP request using the HttpWebRequest class.
Get the response using the GetResponse method. This will return an HttpWebResponse object.
Call GetResponseStream on that response object, and then
Write that stream to a file.
<sidenote> You should try and be a bit more clear in your question. I interpret the "web representation" of an HTTP request to be a "web page", as displayed in a browser. You never mentioned that you wanted to save the reply to disk.
I'm doing some automation work and can make my way around a site & post to HTML forms okay, but now I'm up against a new challenge, Ajax forms.
Since there's no source to read, I'm left wondering if it's possible to fill in an Ajax form progamatically, in C#. I'm currently using a non-visible axWebBrowser.
Thanks in advance for your help!
Yes, but I recommend using a different approach to requesting/responding to the server pages including the regular pages, and the AJAX handler pages.
In c#, try using the WebRequest/WebResponse or the more specialized HttpWebRequest/HttpWebResponse classes.
Ajax is no more than a "fancy" name for a technology that allows Javascript to make HTTP requests to a server which usually implements some handlers that produce specialized, light-weight content for the Javascript caller (comonly encoded as JSON).
Therefore in order to simulate AJAX calls, all you have to do is inspect your target application (the web page that you want to "post" to) and see what format is used for the AJAX communications - then replicate the page's Javascript behavior from C# using the WebREquest/WebResponse classes.
See Firebug - a great tool that allows you to inspect a web page to determine what calls it makes, to which pages and what those pages respond. It does a pretty good job at inspecting AJAX calls too.
Here's a very simple example of how to do a web request:
HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create("http://www.mysite.com");
using (HttpWebResponse resp = (HttpWebResponse)wReq.GetResponse())
{
// NOTE: A better approach would be to use the encoding returned by the server in
// the Response headers (I'm using UTF 8 for brevity)
using (StreamReader sr = new StreamReader(resp.GetResponseStream(), Encoding.UTF8))
{
string content = sr.ReadToEnd();
// Do something with the content
}
}
A POST is also a request, but with a different method. See this page for an example of how to do a very simple post.
EDIT - Details on Inspecting the page behavior with Firebug
What I mean by inspecting the page you're trying to replicate is to use a tool (I use Firebug - on Firefox) to determine the flow of information between the page and the server.
With Firebug, you can do this by using the "Net" and "Console" panels. The Net panel lists all requests executed by the browser while loading the page. While the "Console" will list communications between the page and the server that take place after the page has loaded. Those communications that take place after the page has loaded are essentially the AJAX calls that you'll want to replicate (Note: Network monitoring has to be enbled in Firebug for this to work)
Check out Michael Sync's tutorial to learn more about Firebug and experiment with the Console panel to learn more about the AJAX requests.
Regarding "replicate the page's behavior from C# using the WebRequest/WebResponse" - what you have to realize is that like I said earlier, the Javascript AJAX call is nothing more than an HTTP Request. It's an HTTP Request that the Javacript makes "behind the scenes", or out-of-band, to the web server. To replicate this, it is really no different than replicating a normal GET or a normal POST like I showed above. And this is where Firebug comes in to play. Using it you can view the requests, as the Javascript makes them - look at the Console panel, and see what the Request message looks like.
Then you can use the same technique as above, using the HttpWebRequest/HttpWebResponse to make the same type of request as the Javascript does, only do it from C# instead.
Gregg, I hope this clarifies my answer a little bit but beyond this I suggest playing with Firebug and maybe learning more about how the HTTP protocol works and how AJAX works as a technology.
Have you looked at using Selenium. AFAIK, you can write the test cases in C# and I know our testers have successfully used it before to UI Test a Ajax enabled ASP.NET site
http://seleniumhq.org/