I want to click on link after navigating to a website
webKitBrowser1.Navigate("http://www.somesite.com");
How to click on a link on this website assuming that the link's id is lnkId ?
Go to Google
In the default browser control that comes with Visual Studio, I can do that using the code below :
foreach (HtmlElement el in webBrowser1.Document.GetElementTagName("a")) {
if (el.GetAttribute("id") == "lnkId") {
el.InvokeMember("click");
}
}
What is the equivalent of the code above when I'm using WebkitDotNet control?
As the WebKit doesn't provide a Click() event (see here for details), you cannot do that in the above way. But a small trick may work as an equivalent of the original winforms way as below:
foreach (Node el in webKitBrowser1.Document.GetElementsByTagName("a"))
{
if (((Element) el).GetAttribute("id") == "lnkId")
{
string urlString = ((Element) el).Attributes["href"].NodeValue;
webKitBrowser1.Navigate(urlString);
}
}
Here what I am doing is casting the WebKit.DOM.Node object to its subclass WebKit.DOM.Element to get its Attributes. Then providing href to the NamedNodeMap, i.e. Attributes as the NodeName, you can easily extract the NodeValue, which is the target url in this case. You can then simply invoke the Navigate(urlString) method on the WebKitBrowser instance to replicate the click event.
I don't work with Windows and all my experience is on Webkit GTK. Following comments are based on that experience.
I am not sure which webkit .NET version you are using. Looks like there are multiple implementations. Assuming you are using the one mentioned by Wasif, you can evaluate javascript as mentioned in the example https://code.google.com/p/open-webkit-sharp/source/browse/JavaScriptExample/Form1.cs.
Actually if implementation is supporting javascript execution then you can do most, if not all the DOM operations. The API functions are usually same as javascript functions and most of the time call exact same functions internally despite of origination. Communication between your application and javascript can be little challenging, but if you can read alert messages, that also can be solved. It looks like this library does support alert handling mechanism. A tool I wrote at https://github.com/nhrdl/notesMD will show some examples of achieving this communication though it uses GTK version and is written in python.
Incidentally if you know the id of the element, then Document.GetElementById will save you the loop.
webKitBrowser1.StringByEvaluatingJavaScriptFromString("var inpt = document.createElement(\"input\"); inpt.setAttribute(\"type\", \"submit\"); inpt.setAttribute(\"id\", \"nut\"); inpt.setAttribute(\"type\", \"submit\"); inpt.setAttribute(\"name\", \"tmp\"); inpt.setAttribute(\"value\", \"tmp\"); var element = document.getElementById(\"lnk\"); element.appendChild(inpt);");
webKitBrowser1.StringByEvaluatingJavaScriptFromString("document.getElementById('nut').click();");
Related
I am coming back to work on a BOT that scraped data from a site once a day for my personal use.
However they have changed the code during COVID and now it seems they are loading in a lot of the content with Ajax/JavaScript.
I thought that if I did a WebRequest and obtained the response HTML from a URL, it should match the same content that I see in a browser (FF/Chrome) when I right click and "view source". I thought the actual DOM and generated source code would come later when those files were loaded as onload events fired, scripts lazily loaded and so on.
However the source HTML I obtain with my BOT is NOT the same as the HTML I see when viewing the source code. So my regular expressions that find certain links are not available to me.
Why am I seeing a difference between "view source" and a download of the HTML?
I can only think that when the page loads, SCRIPTS run that load other content into the page and that when I view source I am actually seeing a partial generated source rather than the original source code. Therefore is there a way I can call the page with my BOT, wait X seconds before obtaining the response to get this "onload" generated HTML?
Or even better a way for MY BOT (not using someone elses), to view generated source.
This BOT runs as a web service. I can find another site to scrape but it's just painful when I have all the regular expressions working on the source I see, except it's NOT the source my BOT obtains.
A bit confused at why my browser is showing me more content with a view source (not generated source), than my BOT gets when making a valid request.
Any help would be much appreciated this is almost an 8 year project that I have been doing on/off and this change has ruined one of the core parts of the system.
In response to OP's comment, here is the Java code for how to click at different parts on the screen to do this:
You could use Java's Robot class. I just learned about it a few days ago:
// Import
import java.awt.Robot;
// Code
void click(int x, int y, int btn) {
Robot robot = new Robot();
robot.mouseMove(x, y);
robot.mousePress(btn);
robot.mouseRelease(btn);
}
You would then run the click function with the x and y position to click, as well as the button (MouseEvent.BUTTON1, MouseEvent.BUTTON2, etc.)
After stringing together the right positions (this will vary depending on the screen) you could do just about anything.
To use shortcuts, just use the keyPress and keyRelease functions. Here is a good way to do this:
void key(int keyCode, boolean ctrl, boolean alt, boolean shift) {
if (ctrl)
robot.keyPress(KeyEvent.VK_CONTROL);
if (alt)
robot.keyPress(KeyEvent.VK_ALT);
if (shift)
robot.keyPress(KeyEvent.VK_SHIFT);
robot.keyPress(keyCode);
robot.keyRelease(keyCode);
if (ctrl)
robot.keyRelease(KeyEvent.VK_CONTROL);
if (alt)
robot.keyRelease(KeyEvent.VK_ALT);
if (shift)
robot.keyRelease(KeyEvent.VK_SHIFT);
}
Thus, something like Ctrl+Shift+I to open the inspect menu would look like this:
key(KeyEvent.VK_I, true, false, true);
Here are the steps to copy a website's code (from the inspector) with Google Chrome:
Ctrl + Shift + I
Right click the HTML tag
Select "Edit as HTML"
Ctrl + A
Ctrl + C
Then, you can use the technique from this StackOverflow to get the content from the clipboard:
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String text = (String) c.getData(DataFlavor.stringFlavor);
Using something like FileOutputStream to put the info into a file:
FileOutputStream output = new FileOutputStream(new File( PATH HERE ));
output.write(text.getBytes());
output.close();
I hope this helps!
I have seemed to have fixed it by just turning on the ability to store cookies in my custom HTTP (Bot/Scraper) class, that was being called from the class trying to obtain the data. Probably the site has a defense system to prevent visitors requesting pages and not the JS/CSS with a different session ID on each request.
However I would like to see some other examples because if it is just cookies then they could use JavaScript to test for JavaScript e.g an AJAX call to log if JS is actually on or some DOM manipulation to determine if you are really Human or not which would break it again.
Every site uses different methods to prevent scrapers, email harvesters, job rapists, link harvesters etc inc working out the standard time between requests for 100% verifiable humans and BOTS and then using those values to help determine spoofed user-agents etc. I wrote a whole system to stop BOTS at my last place of work and its a layered approach, just glad the cookies being enabled solved it on this site but it could easily be beefed up with other tricks to test for BOTS vs HUMANS.
I do know some Java, enough to work out what is going on anyway. My BOT is in C#.
I'm using the TestStack.White framework for a WinForms .Net 2.0 C# application's coded automated regression tests. I have no problem handling OK/Cancel message boxes (MessageBox.Show(...)) but I'm having trouble handling the Yes/No buttons when using MessageBoxButtons.YesNoCancel. Thing is, I do not know what the button names are.
These work in my test jigs:
public ButtonTestJig OkButton
{
get { return new ButtonTestJig(Window, "okButton"); }
}
public ButtonTestJig CancelButton
{
get { return new ButtonTestJig(Window, "cancelButton"); }
}
Does anybody know the equivalent "cancelButton" for the NO button and the equivalent "okButton" for the YES button? Or perhaps even better, how to find them. My googling failed me.
Thanks
You can use http://uiautomationverify.codeplex.com/releases/view/11366 or inspect.exe (which comes in the windows SDK to see what the automation Id's for those controls are.
I want to be able to launch multiple browser windows and get the html from them and interact with them like I can with the .net web browser control. The problem is the c# web browser control requires STA threading and that makes my program slower and error prone. Is their a way I can talk to the real IE browser or even Chrome, Firefox etc and do this?
I know there is paid third party solutions and stuff but I was wondering if anybody knew of any free libraries that are reliable and work. I can't use HttpWebRequest for what I am doing, so don't suggest that please.
I will be running on Windows and using C#.
I have used WebKit.NET. Its free and works great (supports flash, css, renders correctly, has a javascript engine). It may or may not be to extensive for what you are doing.
http://webkitdotnet.sourceforge.net/
There are plenty of guides, and the demo has a fully featured tab system.
EDIT: I was fooling around with it, and made an enhanced version of the demo
Awesomeium is a Webkit/Chromium (Chrome) based browser you can talk to via a .NET wrapper, much like the WebBrowser control. I'm not sure on the STA threading requirement though.
As for STA Threading making your application "slower and error prone" -- This sounds odd, perhaps you should investigate solving this.
For IE (only). Free:
Microsoft Internet Controls
using SHDocVw;
Microsoft HTML Object Library
using mshtml;
With these you can do things like
foreach (InternetExplorer brIns in _allWindows)
{
var htmlDoc = brIns.Document as HTMLDocument;
if (htmlDoc != null && htmlDoc.all.item(elementName) != null)
{
var elem = htmlDoc.all.item(elementName) as HTMLInputElement;
if (elem != null && attributeName == null)
{
_ieCurrentDoc = htmlDoc;
_currentHtml = htmlDoc.documentElement.outerHTML;
return true;
}
if (elem != null && elem.getAttribute(attributeName) != null)
{
_ieCurrentDoc = htmlDoc;
_currentHtml = htmlDoc.documentElement.outerHTML;
return true;
}
}
}
Or another webbrowser control: http://code.google.com/p/geckofx/
Embeding Firefox Brower In C# Using GeckoFX
You can't have your cake and eat it too, apparently.
I'm currently using the System.Windows.Forms.WebBrowser in my application. The program currently depends on using the GetElementsByTagName function. I use it to gather up all the elements of a certain type (either "input"s or "textarea"s), so I can sort through them and return the value of a specific one. This is the code for that function (my WebBrowser is named web1):
// returns the value from a element.
public String FetchValue(String strTagType, String strName)
{
HtmlElementCollection elems;
HtmlDocument page = web1.Document.Window.Frames[1].Document;
elems = page.GetElementsByTagName(strTagType);
foreach (HtmlElement elem in elems)
{
if (elem.GetAttribute("name") == strName ||
elem.GetAttribute("ref") == strName)
{
if (elem.GetAttribute("value") != null)
{
return elem.GetAttribute("value");
}
}
}
return null;
}
(points to note: the webpage I need to pull from is in a frame, and depending on circumstances, the element's identifying name will be either in the name or the ref attribute)
All of that works like a dream with the System.Windows.Forms.WebBrowser.
But what it is unable to do, is redirect the opening of a new window to remain in the application. Anything that opens in a new window shoots to the user's default browser, thus losing the session. This functionality can be easily fixed with the NewWindow2 event, which System.Windows.Forms.WebBrowser doesn't have.
Now forgive me for being stunned at its absence. I have but recently ditched VB6 and moved on to C# (yes VB6, apparently I am employed under a rock), and in VB6, the WebBrowser possessed both the GetElementsByTagName function and the NewWindow2 event.
The AxSHDocVw.WebBrowser has a NewWindow2 event. It would be more than happy to help me route my new windows to where I need them. The code to do this in THAT WebBrowser is (frmNewWindow being a simple form containing only another WebBrowser called web2 (Dock set to Fill)):
private void web1_NewWindow2(
object sender,
AxSHDocVw.DWebBrowserEvents2_NewWindow2Event e)
{
frmNewWindow frmNW = new frmNewWindow();
e.ppDisp = frmNW.web2.Application;
frmNW.web2.RegisterAsBrowser = true;
frmNW.Visible = true;
}
I am unable to produce on my own a way to replicate that function with the underwhelming regular NewWindow event.
I am also unable to figure out how to replicate the FetchValue function I detailed above using the AxSHDocVw.WebBrowser. It appears to go about things in a totally different way and all my knowledge of how to do things is useless.
I know I'm a sick, twisted man for this bizarre fantasy of using these two things in a single application. But can you find it in your heart to help this foolish idealist?
I could no longer rely on the workaround, and had to abandon System.Windows.Forms.WebBrowser. I needed NewWindow2.
I eventually figured out how to accomplish what I needed with the AxWebBrowser. My original post was asking for either a solution for NewWindow2 on the System.Windows.Forms.WebBrowser, or an AxWebBrowser replacement for .GetElementsByTagName. The replacement requires about 4x as much code, but gets the job done. I thought it would be prudent to post my solution, for later Googlers with the same quandary. (also in case there's a better way to have done this)
IHTMLDocument2 webpage = (IHTMLDocument2)webbrowser.Document;
IHTMLFramesCollection2 allframes = webpage.frames;
IHTMLWindow2 targetframe = (IHTMLWindow2)allframes.item("name of target frame");
webpage = (IHTMLDocument2)targetframe.document;
IHTMLElementCollection elements = webpage.all.tags("target tagtype");
foreach (IHTMLElement element in elements)
{
if (elem.getAttribute("name") == strTargetElementName)
{
return element.getAttribute("value");
}
}
The webbrowser.Document is cast into an IHTMLDocument2, then the IHTMLDocument2's frames are put into a IHTMLFramesCollection2, then I cast the specific desired frame into an IHTMLWindow2 (you can choose frame by index # or name), then I cast the frame's .Document member into an IHTMLDocument2 (the originally used one, for convenience sake). From there, the IHTMLDocument2's .all.tags() method is functionally identical to the old WebBrowser.Document.GetElementsByTagName() method, except it requires an IHTMLElementCollection versus an HTMLElementCollection. Then, you can foreach the collection, the individual elements needing to be IHTMLElement, and use .getAttribute to retrieve the attributes. Note that the g is lowercase.
The WebBrowser control can handle the NewWindow event so that new popup windows will be opened in the WebBrowser.
private void webBrowser1_NewWindow(object sender, CancelEventArgs e)
{
// navigate current window to the url
webBrowser1.Navigate(webBrowser1.StatusText);
// cancel the new window opening
e.Cancel = true;
}
http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/361b6655-3145-4371-b92c-051c223518f2/
The only solution to this I have seen was a good few years ago now, called csExWb2, now on Google code here.
It gives you an ExWebBrowser control, but with full-on access to all the interfaces and events offered by IE. I used it to get deep and dirty control of elements in a winforms-hosted html editor.
It may be a bit of a leap jumping straight into that, mind.
We are developing an application which needs to interact with the active document in IE.
Context: The app is a C#, .Net 3.5 desktop app. The goal is to highlight specific text elements in the web page on user request. For this we need to retrieve and interpret web page elements (the need for the return value) then act on them through another JS call. The operations that must be made in the web page are not all done at the same time so we must get some kind of "snapshot" of the interesting text elements (we do this on the Mac version of our app by returning a string containing an XML representation of those elements).
In .Net we used IHTMLDocument2's execScript method successfully to run some JavaScript inside the active IE document, but we can't seem to find a way to get a return value from the call. Based on the doc execScript returns an execution success/failure constant which is not what we need.
In essence what we need to do is to load some JavaScript from a text file into a string, then send it to IE for execution. Then we need to get a string back from the called script.
Any hints on what objects to use? How to proceed to get this functionality?
Thanks in advance!
My colleague found the solution, based on what Alun Harford said:
string jsToRun = "function myTest() { return document.title; } myTest();";
mshtml.IHTMLDocument2 myIHTMLDocument2 = GetSelectedIEWindow();
IE ie = IE.AttachToIE(Find.ByUrl(myIHTMLDocument2.url));
string jsReturn = ie.Eval(jsToRun);
jsReturn then contains the string value returned from myTest() in JavaScript. Note that there is no return before the myTest() function call in the script!
Have a look at the WatiN codebase. In there, IE.Eval does exactly what you're looking for.
If you are providing the html and script yourself you can do the following:
execute the javascript function
let the js function place the result in an html element
wait till the function is done running
retrieve the html element using document.getElementById
and retrieve the value
I'm not sure if there's a easier way to do this.
Well it is nasty but it can be done.
Try this:
[Guid("626FC520-A41E-11CF-A731-00A0C9082637"), InterfaceType(ComInterfaceType.InterfaceIsDual)]
interface IHTMLDocument
{
void Script([Out, MarshalAs(UnmanagedType.Interface)] out object ppScript);
}
public object RunScript(InternetExplorer ie, string scriptText)
{
IHTMLDocument doc = (IHTMLDocument)ie.Document;
object scriptObj;
doc.Script(out scriptObj);
Type t = scriptObj.GetType();
return t.InvokeMember("eval", System.Reflection.BindingFlags.InvokeMethod, null, scriptObj, new object[] { scriptText });
}
That will return your value in the object (just cast to what ever type you expected). Of course .NET 4 makes this even easier ;)