I have a C#.net webform that does a simple response.write in content type JSON format. It works greats from every client I test it with - including a TinyWebDB API call from an Android phone.. but don't worry about that API for this question.
I added some serverside code to that web form to read and scrape a web page as follows.
System.Net.WebClient myWebClient = new System.Net.WebClient();
Stream myStream = myWebClient.OpenRead(what);
StreamReader sr = new StreamReader(myStream);
string s = sr.ReadToEnd();
I'm under the impression that code is all Server Side!?
I tested the page from IE, FF and Chrome.. all work great. However If I attempt to call the web form page from TinyWebDB the call works great and I get data back, but I get a 404 error on the server side read of the webpage??
It's almost as if System.Net.WebClient requires something from or is doing something on client itself. i thought the reading of the page was all happening serverside and behind the scenes on my serer. Why would my serverside code care about what browser or API initiated the call to the webform?
Should I be using another class?
Many Thanks.
That is serverside code.
Where does the variable "what" get set? Is the form TinyWebDB client munging your form input values?
Is it possible that the site you are requesting is rejecting your request because you're hitting it too fast and you've just been unlucky with it being that client every time?
Related
I have some code that connects to an HTTP API, and is supposed to get an XML response. When the API link is placed in a browser, the browser downloads the XML as a file. However, when the code connects to the same API, HTML is returned. I've told the API owner but they don't think anything is wrong. Is there a way to capture the downloaded file instead of the HTML?
I've tried setting the headers to make my code look like a browser. Also tried using WebRequest instead of WebClient. But nothing works.
Here is the code, the URL works in the browser (file downloaded) but doesn't work for WebClient:
WebClient webClient = new WebClient();
string result = webClient.DownloadString(url);
The code should somehow get the XML file instead of the page HTML (actually the HTML doesn't appear in the browser, only the file).
The uri that you access may be a HTML page that have its own mechanism (like generating the actual download address which may be dynamically generated by the server and redirecting to it, in order to prevent external linking access) to access the real file.
It is supposed to use a browser background browser core like CefSharp to run the HTML and its javascript to let it navigate, and probably you may want to hook the download event to handle the downloading.
I think you need to add accept header to the WebClient object.
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.Accept] = "application/xml;q=1";
string result = webClient.DownloadString(url);
}
Thank you all for your input. In the end, it was caused by our vendor switching to TLS 1.2. I just had to force my code to use 1.2 and then it worked.
I have 100 pages on my site, but I want download only part a page instead of all page content.
I want just one box of each page to download, the file size is 10 KB.
For this I Use WebClient and htmlagilitypack .
WebClient Client = new WebClient();
var result = Encoding.GetEncoding("UTF-8").GetString(Client.DownloadData(URL));
Unfortunately, that's not possible, because HTTP is not designed to deliver a specific part of a web page. It does support range requests, but for that you would need to know where exactly (in terms of bytes) the desired content is located.
You can
download the whole page and then
use a HTML parsing library to extract the part you need.
You cannot achieve this.
The only solution is changing the website structure itself. if you have control of the server -
Change the architecture of your website, making the data in the box accessible via an ajax call.
Now you can get the data via the WebClient.
If that data is already served via a API call, you can point your WebClient to that URI Instead.
Here is an example of structuring you website based on ajax -
AJAX with jQuery and ASP.NET
How can i get the web representation of a received http-request?
I recently asked a question on how i could circumvetnt a website censorship through a third anonymizer website.the result indicated i sniff the traffic and create the post url myself.
This stage is almost done.Now i need to get the page as a result.
If you're using Winforms, you could go with the WebBrowser class.
The WebBrowser control lets you host Web pages and other browser-enabled documents in your Windows Forms applications. You can use the WebBrowser control, for example, to provide integrated HTML-based user assistance or Web browsing capabilities in your application. Additionally, you can use the WebBrowser control to add your existing Web-based controls to your Windows Forms client applications.
(From MSDN)
You wish to save the response to disk. Ok, typically this is the sequence you'd take:
You make the HTTP request using the HttpWebRequest class.
Get the response using the GetResponse method. This will return an HttpWebResponse object.
Call GetResponseStream on that response object, and then
Write that stream to a file.
<sidenote> You should try and be a bit more clear in your question. I interpret the "web representation" of an HTTP request to be a "web page", as displayed in a browser. You never mentioned that you wanted to save the reply to disk.
I am currently trying to do a screen scrape using the following code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse theResponse = (HttpWebResponse) request.GetResponse();
using (StreamReader reader = new StreamReader(theResponse.GetResponseStream(), Encoding.UTF8))
{
string s = reader.ReadToEnd();
}
However, the data I am concerned with (an HTML table) is not part of the result. When I right click the page and ViewSource, I also do not see the HTML table I care about - however I do see it in the DOM when I use Firebug to inspect it.
It doesn't seem to be loaded via ajax either.
So - is there another way, using C#, to get the DOM as it exists in the Developer Tool view, rather than the ViewSource result?
Unfortunately, this page is not publicly available so I can't paste the URL.
It doesn't seem to be loaded via ajax either.
You don't need to use AJAX in order to dynamically add data to the DOM. You could perfectly fine use standard javascript.
To scrape such page you need a scraper that processes javascript. The WebBrowser control in WinForms does that. It allows you to load a web page and explore the DOM, just as you do in FireBug (except that the snapshot comes from IE because the WebBrowser is just a wrapper around IE).
But since the WebBrowser control is not designed to be used in a multithreaded environment (such as a web application) you will have to use a third party library to achieve that scraping task.
Have you used Fiddler or Ethereal to see what URL's are being connected to in the background? If you find the HTML table in the response from one of the URL's called in the background, you can scrape the data from that URL. Which URL/table are you trying to parse?
I'm doing some automation work and can make my way around a site & post to HTML forms okay, but now I'm up against a new challenge, Ajax forms.
Since there's no source to read, I'm left wondering if it's possible to fill in an Ajax form progamatically, in C#. I'm currently using a non-visible axWebBrowser.
Thanks in advance for your help!
Yes, but I recommend using a different approach to requesting/responding to the server pages including the regular pages, and the AJAX handler pages.
In c#, try using the WebRequest/WebResponse or the more specialized HttpWebRequest/HttpWebResponse classes.
Ajax is no more than a "fancy" name for a technology that allows Javascript to make HTTP requests to a server which usually implements some handlers that produce specialized, light-weight content for the Javascript caller (comonly encoded as JSON).
Therefore in order to simulate AJAX calls, all you have to do is inspect your target application (the web page that you want to "post" to) and see what format is used for the AJAX communications - then replicate the page's Javascript behavior from C# using the WebREquest/WebResponse classes.
See Firebug - a great tool that allows you to inspect a web page to determine what calls it makes, to which pages and what those pages respond. It does a pretty good job at inspecting AJAX calls too.
Here's a very simple example of how to do a web request:
HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create("http://www.mysite.com");
using (HttpWebResponse resp = (HttpWebResponse)wReq.GetResponse())
{
// NOTE: A better approach would be to use the encoding returned by the server in
// the Response headers (I'm using UTF 8 for brevity)
using (StreamReader sr = new StreamReader(resp.GetResponseStream(), Encoding.UTF8))
{
string content = sr.ReadToEnd();
// Do something with the content
}
}
A POST is also a request, but with a different method. See this page for an example of how to do a very simple post.
EDIT - Details on Inspecting the page behavior with Firebug
What I mean by inspecting the page you're trying to replicate is to use a tool (I use Firebug - on Firefox) to determine the flow of information between the page and the server.
With Firebug, you can do this by using the "Net" and "Console" panels. The Net panel lists all requests executed by the browser while loading the page. While the "Console" will list communications between the page and the server that take place after the page has loaded. Those communications that take place after the page has loaded are essentially the AJAX calls that you'll want to replicate (Note: Network monitoring has to be enbled in Firebug for this to work)
Check out Michael Sync's tutorial to learn more about Firebug and experiment with the Console panel to learn more about the AJAX requests.
Regarding "replicate the page's behavior from C# using the WebRequest/WebResponse" - what you have to realize is that like I said earlier, the Javascript AJAX call is nothing more than an HTTP Request. It's an HTTP Request that the Javacript makes "behind the scenes", or out-of-band, to the web server. To replicate this, it is really no different than replicating a normal GET or a normal POST like I showed above. And this is where Firebug comes in to play. Using it you can view the requests, as the Javascript makes them - look at the Console panel, and see what the Request message looks like.
Then you can use the same technique as above, using the HttpWebRequest/HttpWebResponse to make the same type of request as the Javascript does, only do it from C# instead.
Gregg, I hope this clarifies my answer a little bit but beyond this I suggest playing with Firebug and maybe learning more about how the HTTP protocol works and how AJAX works as a technology.
Have you looked at using Selenium. AFAIK, you can write the test cases in C# and I know our testers have successfully used it before to UI Test a Ajax enabled ASP.NET site
http://seleniumhq.org/