I'm doing some automation work and can make my way around a site & post to HTML forms okay, but now I'm up against a new challenge, Ajax forms.
Since there's no source to read, I'm left wondering if it's possible to fill in an Ajax form progamatically, in C#. I'm currently using a non-visible axWebBrowser.
Thanks in advance for your help!
Yes, but I recommend using a different approach to requesting/responding to the server pages including the regular pages, and the AJAX handler pages.
In c#, try using the WebRequest/WebResponse or the more specialized HttpWebRequest/HttpWebResponse classes.
Ajax is no more than a "fancy" name for a technology that allows Javascript to make HTTP requests to a server which usually implements some handlers that produce specialized, light-weight content for the Javascript caller (comonly encoded as JSON).
Therefore in order to simulate AJAX calls, all you have to do is inspect your target application (the web page that you want to "post" to) and see what format is used for the AJAX communications - then replicate the page's Javascript behavior from C# using the WebREquest/WebResponse classes.
See Firebug - a great tool that allows you to inspect a web page to determine what calls it makes, to which pages and what those pages respond. It does a pretty good job at inspecting AJAX calls too.
Here's a very simple example of how to do a web request:
HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create("http://www.mysite.com");
using (HttpWebResponse resp = (HttpWebResponse)wReq.GetResponse())
{
// NOTE: A better approach would be to use the encoding returned by the server in
// the Response headers (I'm using UTF 8 for brevity)
using (StreamReader sr = new StreamReader(resp.GetResponseStream(), Encoding.UTF8))
{
string content = sr.ReadToEnd();
// Do something with the content
}
}
A POST is also a request, but with a different method. See this page for an example of how to do a very simple post.
EDIT - Details on Inspecting the page behavior with Firebug
What I mean by inspecting the page you're trying to replicate is to use a tool (I use Firebug - on Firefox) to determine the flow of information between the page and the server.
With Firebug, you can do this by using the "Net" and "Console" panels. The Net panel lists all requests executed by the browser while loading the page. While the "Console" will list communications between the page and the server that take place after the page has loaded. Those communications that take place after the page has loaded are essentially the AJAX calls that you'll want to replicate (Note: Network monitoring has to be enbled in Firebug for this to work)
Check out Michael Sync's tutorial to learn more about Firebug and experiment with the Console panel to learn more about the AJAX requests.
Regarding "replicate the page's behavior from C# using the WebRequest/WebResponse" - what you have to realize is that like I said earlier, the Javascript AJAX call is nothing more than an HTTP Request. It's an HTTP Request that the Javacript makes "behind the scenes", or out-of-band, to the web server. To replicate this, it is really no different than replicating a normal GET or a normal POST like I showed above. And this is where Firebug comes in to play. Using it you can view the requests, as the Javascript makes them - look at the Console panel, and see what the Request message looks like.
Then you can use the same technique as above, using the HttpWebRequest/HttpWebResponse to make the same type of request as the Javascript does, only do it from C# instead.
Gregg, I hope this clarifies my answer a little bit but beyond this I suggest playing with Firebug and maybe learning more about how the HTTP protocol works and how AJAX works as a technology.
Have you looked at using Selenium. AFAIK, you can write the test cases in C# and I know our testers have successfully used it before to UI Test a Ajax enabled ASP.NET site
http://seleniumhq.org/
Related
I am building an angularJs app and need to have application_beginrequest where I need to get current url in the browser.
E.g. localhost:60607/#/login : need to get /login
localhost:60607/#/activity : need to get /activity
Also whenever user hits a refresh or when page loads again and goes into beginrequest it should have the same response.
I tried with context.Request.Path but it gives "/" only.
bookmarks are not sent by browser to server, bookmarks are used for navigating inside the page. You have to get book mark using javascript and send it using different parameter to get that bookmark value at server side.
Data after "#" symbol is supposed to be client side only (usually browsers strip away those pieces, application servers usually ignore them).
If your intent is to provide users the ability to refresh the page using Angular you can:
Provide isomorphic behavior (load the page server side) but if you are using Angular JS ver 1.x is not as simple as you imagine
Provide double routing behavior (*):
avoid using the "#" symbol for url and provide user simple url like /login or /activity
manage your route client side using angular as you are supposed to do already
manage your route server side and provide the right view when requested
you can try to centralize the view management providing them using a controller instead of static content
Using the second approach is probably simpler, but you will find that:
keeping the view state is possible only for simple views (the things get complex quite quickly we introducing multiple UI element state)
to manage the double routing you have to find a compromise between client/server
How can i get the web representation of a received http-request?
I recently asked a question on how i could circumvetnt a website censorship through a third anonymizer website.the result indicated i sniff the traffic and create the post url myself.
This stage is almost done.Now i need to get the page as a result.
If you're using Winforms, you could go with the WebBrowser class.
The WebBrowser control lets you host Web pages and other browser-enabled documents in your Windows Forms applications. You can use the WebBrowser control, for example, to provide integrated HTML-based user assistance or Web browsing capabilities in your application. Additionally, you can use the WebBrowser control to add your existing Web-based controls to your Windows Forms client applications.
(From MSDN)
You wish to save the response to disk. Ok, typically this is the sequence you'd take:
You make the HTTP request using the HttpWebRequest class.
Get the response using the GetResponse method. This will return an HttpWebResponse object.
Call GetResponseStream on that response object, and then
Write that stream to a file.
<sidenote> You should try and be a bit more clear in your question. I interpret the "web representation" of an HTTP request to be a "web page", as displayed in a browser. You never mentioned that you wanted to save the reply to disk.
I am currently developping a tool to get and parse some content of an external website (not mine). I will not paste the code as i don't think it does bring anything, but if in any manner you think it's useful, i will do it.
Here are the major steps of my tool:
Get the Web page using a regular webrequest/webresponse.
Parse the Web page to know how many pages should be parsed (the Web page parsed is a research result, so it can provide many pages of results)
As a page change in a regular browser is done by submitting a form, i did inspect all the POST parameters (hidden) of this form by parsing the web page.
Create the POST request with these parameters
Send the POST request to the server using WebClient and the UploadString() method.
Unfortunately, the last part doesn't work and throw a 500 error Invalid postback or callback argument. Event validation is enabled using ...
If it can help, in the hidden parameters of the form, a parameter named EventValidation is present, and i do provide it to the POST request.
Maybe someone could have an idea of what is going on as i am not so familiar with asp.
Please forgive my english mistakes
I am currently trying to do a screen scrape using the following code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse theResponse = (HttpWebResponse) request.GetResponse();
using (StreamReader reader = new StreamReader(theResponse.GetResponseStream(), Encoding.UTF8))
{
string s = reader.ReadToEnd();
}
However, the data I am concerned with (an HTML table) is not part of the result. When I right click the page and ViewSource, I also do not see the HTML table I care about - however I do see it in the DOM when I use Firebug to inspect it.
It doesn't seem to be loaded via ajax either.
So - is there another way, using C#, to get the DOM as it exists in the Developer Tool view, rather than the ViewSource result?
Unfortunately, this page is not publicly available so I can't paste the URL.
It doesn't seem to be loaded via ajax either.
You don't need to use AJAX in order to dynamically add data to the DOM. You could perfectly fine use standard javascript.
To scrape such page you need a scraper that processes javascript. The WebBrowser control in WinForms does that. It allows you to load a web page and explore the DOM, just as you do in FireBug (except that the snapshot comes from IE because the WebBrowser is just a wrapper around IE).
But since the WebBrowser control is not designed to be used in a multithreaded environment (such as a web application) you will have to use a third party library to achieve that scraping task.
Have you used Fiddler or Ethereal to see what URL's are being connected to in the background? If you find the HTML table in the response from one of the URL's called in the background, you can scrape the data from that URL. Which URL/table are you trying to parse?
I use Process.Start("firefox.exe", "http://localhost/page.aspx");
And how i can know page fails or no?
OR
How to know via HttpWebRequest, HttpWebResponse page fails or not?
When i use
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create("somepage.aspx");
HttpWebResponse loWebResponse = (HttpWebResponse)myReq.GetResponse();
Console.Write("{0},{1}",loWebResponse.StatusCode, loWebResponse.StatusDescription);
how can I return error details?
Not need additional plugins and frameworks. I want to choose this problem only by .net
Any Idea please
Use Watin to automate firefox instead of Process.Start. Its a browser automation framework that will let you monitor what is happening properly.
http://watin.sourceforge.net/
edit: see also Google Webdriver http://google-opensource.blogspot.com/2009/05/introducing-webdriver.html
If you are spawning a child-process, it is quite hard and you'd probably need to use each browser's specific API (it won't be the same between FF and IE, for example).
It doesn't help that in many cases the exe detects an existing instance and forwards the request there (so you can't trust the exit-code, since the page hasn't even been requested in the right exe yet).
Personally, I try to avoid assuming any particular browser for this scenario; just launch the url:
Process.Start("http://somesite.com");
This will use the user's default browser. You have to hope it appears though - you can't (reliably and robustly) check that externally without lots of work.
One other option is to read the data yourself (WebClient.Download*) - but this may have issues with complex cookies, login, user-agent awareness, etc.
Use HttpWebRequest class or WebClient class to check this. I don't think Process.Start will return something if the URL not exists.
Don't start the page in this form. Instead, create a local http://localhost:<port>/wrapper.html which loads http://localhost/page.aspx and then either http://localhost:<port>/pass.html or http://localhost:<port>/fail.html. localhost: is a trivial HTTP server interface implemented by your app.
The idea is that Javascript gives you an API inside the browser, which is far more standard than the APIs on the outside of browsers. Since the Javascript on wrapper.html comes from the same server and even port as the subsequent resources, this should satisfy the same-origin policies in current browsers.