This link goes to an implementation of the imagination captcha imagination
The authors have themselves requested for people to make algorithms to try its efficiency against AI attacks.
Essentially the first page is asking for a mouse click anywhere on the image... My problem is that my algorithm comes up with the point (x,y) on the image but I want to emulate it real time on this link...
Can some one tell me how can i send the point values on this link and get back the message whether i was successful or not....
Essentially I am asking how can i emulate a mouse click on this link at the points that my algorithm gives using C#...
I am asking this only for studying the features of this captcha and its accuracy.
Thanks a lot
If you are able to execute JavaScript on that page directly, this code will do:
submitClick(document.getElementById("img").value, x, y, "tiled");
Otherwise, hit this url, substituting your own values for id, x, and y:
http://goldbach.cse.psu.edu/s/captcha/captcha_controller.php?id=87170&x=66&y=149&source=tiled
Parse the response - If your coordinates are correct, the response will contain "step 2". If not, the response will contain "step 1" and it will have a <div id="error">.
If you want to use their live site from code, I think you're talking about a screen scrape. I highly recommend looking into the HTML Agility Pack (available via nuget). This is going to allow you to read the DOM into your application and then interact with it however you please.
This could be a dumb answer but if you're trying to emulate a mouse click and find out if it's successful, why not use the Selenium Browser add-in to record your scripts / write' your own.
Then you can have a test suite to try against you're various different captchas.... hope this achieves what you're trying to do....
WebAii over at telerik has this feature. Here is some sample code i used at some point in the past customized for your situation. just put this in a class, left out the class container because it jacks up the formatting
protected static Manager _manager = null;
protected static Manager _manager = null;
protected Browser _main;
protected Find _find;
public WebAiiAutomater()
{
if (_manager != null)
{
foreach (var broswer in _manager.Browsers)
{
broswer.Close();
}
return;
}
var settings = new Settings(BrowserType.InternetExplorer, #"c:\log\") { ClientReadyTimeout = 60 * 1000 };
_manager = new Manager(settings);
_manager.Start();
_manager.LaunchNewBrowser();
_manager.ActiveBrowser.AutoWaitUntilReady = true;
_main = _manager.ActiveBrowser;
_find = _main.Find;
_main.NavigateTo(#"http://goldbach.cse.psu.edu/s/captcha/");
//start looping over your alogrithm trying different x,y coords against ClickImage(x,y
}
public bool ClickImage(int x, int y)
{
//var beginsearch = _find.ById("captcha_img"); //this should get you the image, but you don't need
_manager.Desktop.Mouse.Click(MouseClickType.LeftClick, x, y);
Thread.sleep(1000); //wait for postback - might be handled internally though
var errordiv = _find.ById("error");
return errordiv !=null;
}
Related
I am coming back to work on a BOT that scraped data from a site once a day for my personal use.
However they have changed the code during COVID and now it seems they are loading in a lot of the content with Ajax/JavaScript.
I thought that if I did a WebRequest and obtained the response HTML from a URL, it should match the same content that I see in a browser (FF/Chrome) when I right click and "view source". I thought the actual DOM and generated source code would come later when those files were loaded as onload events fired, scripts lazily loaded and so on.
However the source HTML I obtain with my BOT is NOT the same as the HTML I see when viewing the source code. So my regular expressions that find certain links are not available to me.
Why am I seeing a difference between "view source" and a download of the HTML?
I can only think that when the page loads, SCRIPTS run that load other content into the page and that when I view source I am actually seeing a partial generated source rather than the original source code. Therefore is there a way I can call the page with my BOT, wait X seconds before obtaining the response to get this "onload" generated HTML?
Or even better a way for MY BOT (not using someone elses), to view generated source.
This BOT runs as a web service. I can find another site to scrape but it's just painful when I have all the regular expressions working on the source I see, except it's NOT the source my BOT obtains.
A bit confused at why my browser is showing me more content with a view source (not generated source), than my BOT gets when making a valid request.
Any help would be much appreciated this is almost an 8 year project that I have been doing on/off and this change has ruined one of the core parts of the system.
In response to OP's comment, here is the Java code for how to click at different parts on the screen to do this:
You could use Java's Robot class. I just learned about it a few days ago:
// Import
import java.awt.Robot;
// Code
void click(int x, int y, int btn) {
Robot robot = new Robot();
robot.mouseMove(x, y);
robot.mousePress(btn);
robot.mouseRelease(btn);
}
You would then run the click function with the x and y position to click, as well as the button (MouseEvent.BUTTON1, MouseEvent.BUTTON2, etc.)
After stringing together the right positions (this will vary depending on the screen) you could do just about anything.
To use shortcuts, just use the keyPress and keyRelease functions. Here is a good way to do this:
void key(int keyCode, boolean ctrl, boolean alt, boolean shift) {
if (ctrl)
robot.keyPress(KeyEvent.VK_CONTROL);
if (alt)
robot.keyPress(KeyEvent.VK_ALT);
if (shift)
robot.keyPress(KeyEvent.VK_SHIFT);
robot.keyPress(keyCode);
robot.keyRelease(keyCode);
if (ctrl)
robot.keyRelease(KeyEvent.VK_CONTROL);
if (alt)
robot.keyRelease(KeyEvent.VK_ALT);
if (shift)
robot.keyRelease(KeyEvent.VK_SHIFT);
}
Thus, something like Ctrl+Shift+I to open the inspect menu would look like this:
key(KeyEvent.VK_I, true, false, true);
Here are the steps to copy a website's code (from the inspector) with Google Chrome:
Ctrl + Shift + I
Right click the HTML tag
Select "Edit as HTML"
Ctrl + A
Ctrl + C
Then, you can use the technique from this StackOverflow to get the content from the clipboard:
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String text = (String) c.getData(DataFlavor.stringFlavor);
Using something like FileOutputStream to put the info into a file:
FileOutputStream output = new FileOutputStream(new File( PATH HERE ));
output.write(text.getBytes());
output.close();
I hope this helps!
I have seemed to have fixed it by just turning on the ability to store cookies in my custom HTTP (Bot/Scraper) class, that was being called from the class trying to obtain the data. Probably the site has a defense system to prevent visitors requesting pages and not the JS/CSS with a different session ID on each request.
However I would like to see some other examples because if it is just cookies then they could use JavaScript to test for JavaScript e.g an AJAX call to log if JS is actually on or some DOM manipulation to determine if you are really Human or not which would break it again.
Every site uses different methods to prevent scrapers, email harvesters, job rapists, link harvesters etc inc working out the standard time between requests for 100% verifiable humans and BOTS and then using those values to help determine spoofed user-agents etc. I wrote a whole system to stop BOTS at my last place of work and its a layered approach, just glad the cookies being enabled solved it on this site but it could easily be beefed up with other tricks to test for BOTS vs HUMANS.
I do know some Java, enough to work out what is going on anyway. My BOT is in C#.
I'm trying to take a full screen screenshot of my page by using this code:
public void OpenEyesForVisualTesting(string testName) {
this.seleniumDriver.Driver = this.eyes.Open(this.seleniumDriver.Driver, "Zinc", testName);
}
public void CheckScreenForVisualTesting() {
this.eyes.Check("Zinc", Applitools.Selenium.Target.Window().Fully());
}
public void CloseEyes() {
this.eyes.close();
}
but instead I just get a half a page of the screenshot, I tried to contact Applitools but they just told me to replace eyes.checkwindow() to eyes.Check("tag", Target.Window().Fully()); which still didn't work.
If anyone can help me that would be great.
I work for Applitools and sorry for your troubles. Maybe you did not see our replies or they went to your spam folder. You need to set ForceFullPageScreenshot = true and StitchMode = StitchModes.CSS in order to capture a full page screenshot.
The below code example is everything you'd need to do in order to capture a full page image. Also, please make sure your .Net Eyes.Sdk version is >= 2.6.0 and Eyes.Selenium >= 2.5.0.
If you have any further questions or still encountering issues with this please feel free to email me directly. Thanks.
var eyes = new Eyes();
eyes.ApiKey = "your-api-key";
eyes.ForceFullPageScreenshot = true;
eyes.StitchMode = StitchModes.CSS;
eyes.Open(driver, "Zinc", testName, new Size(800, 600)); //last parameter is your viewport size IF testing on a browser. Do not set if testing on a mobile devices.
eyes.CheckWindow("Zinc");
//or new Fluet API method
eyes.Check("Zinc", Applitools.Selenium.Target.Window().Fully();
eyes.Close();
Use extent Reporting library, in that you can take screen shot as well as an interactive report on pass and fail cases.
here is the link how it works:
http://www.softwaretestingmaterial.com/generate-extent-reports/
If you want to take complete page Screenshot/ Particular element with Applitools. You can use lines of code:-
Full Screenshot
eyes.setHideScrollbars(false);
eyes.setStitchMode(StitchMode.CSS);
eyes.setForceFullPageScreenshot(true);
eyes.checkWindow();
Take screenshot for particular element
WebElement Button = driver.findElement(By.xpath("//button[#id='login-btn']"));
eyes.setHideScrollbars(false);
eyes.setStitchMode(StitchMode.CSS);
eyes.open(driver, projectName, testName);
eyes.setMatchLevel(MatchLevel.EXACT);
eyes.checkElement(Button );
(I am from Applitools.)
When setting forcefullpagescreenshoot to true, applitools will try to scroll the HTML element in the page, if the HTML is not the scrollable element then you will need to set it yourself:
eyes.Check(Target.Window().ScrollRootElement(selector))
In Firefox, you can see a scroll tag near elements that are scrollable.
You can't have your cake and eat it too, apparently.
I'm currently using the System.Windows.Forms.WebBrowser in my application. The program currently depends on using the GetElementsByTagName function. I use it to gather up all the elements of a certain type (either "input"s or "textarea"s), so I can sort through them and return the value of a specific one. This is the code for that function (my WebBrowser is named web1):
// returns the value from a element.
public String FetchValue(String strTagType, String strName)
{
HtmlElementCollection elems;
HtmlDocument page = web1.Document.Window.Frames[1].Document;
elems = page.GetElementsByTagName(strTagType);
foreach (HtmlElement elem in elems)
{
if (elem.GetAttribute("name") == strName ||
elem.GetAttribute("ref") == strName)
{
if (elem.GetAttribute("value") != null)
{
return elem.GetAttribute("value");
}
}
}
return null;
}
(points to note: the webpage I need to pull from is in a frame, and depending on circumstances, the element's identifying name will be either in the name or the ref attribute)
All of that works like a dream with the System.Windows.Forms.WebBrowser.
But what it is unable to do, is redirect the opening of a new window to remain in the application. Anything that opens in a new window shoots to the user's default browser, thus losing the session. This functionality can be easily fixed with the NewWindow2 event, which System.Windows.Forms.WebBrowser doesn't have.
Now forgive me for being stunned at its absence. I have but recently ditched VB6 and moved on to C# (yes VB6, apparently I am employed under a rock), and in VB6, the WebBrowser possessed both the GetElementsByTagName function and the NewWindow2 event.
The AxSHDocVw.WebBrowser has a NewWindow2 event. It would be more than happy to help me route my new windows to where I need them. The code to do this in THAT WebBrowser is (frmNewWindow being a simple form containing only another WebBrowser called web2 (Dock set to Fill)):
private void web1_NewWindow2(
object sender,
AxSHDocVw.DWebBrowserEvents2_NewWindow2Event e)
{
frmNewWindow frmNW = new frmNewWindow();
e.ppDisp = frmNW.web2.Application;
frmNW.web2.RegisterAsBrowser = true;
frmNW.Visible = true;
}
I am unable to produce on my own a way to replicate that function with the underwhelming regular NewWindow event.
I am also unable to figure out how to replicate the FetchValue function I detailed above using the AxSHDocVw.WebBrowser. It appears to go about things in a totally different way and all my knowledge of how to do things is useless.
I know I'm a sick, twisted man for this bizarre fantasy of using these two things in a single application. But can you find it in your heart to help this foolish idealist?
I could no longer rely on the workaround, and had to abandon System.Windows.Forms.WebBrowser. I needed NewWindow2.
I eventually figured out how to accomplish what I needed with the AxWebBrowser. My original post was asking for either a solution for NewWindow2 on the System.Windows.Forms.WebBrowser, or an AxWebBrowser replacement for .GetElementsByTagName. The replacement requires about 4x as much code, but gets the job done. I thought it would be prudent to post my solution, for later Googlers with the same quandary. (also in case there's a better way to have done this)
IHTMLDocument2 webpage = (IHTMLDocument2)webbrowser.Document;
IHTMLFramesCollection2 allframes = webpage.frames;
IHTMLWindow2 targetframe = (IHTMLWindow2)allframes.item("name of target frame");
webpage = (IHTMLDocument2)targetframe.document;
IHTMLElementCollection elements = webpage.all.tags("target tagtype");
foreach (IHTMLElement element in elements)
{
if (elem.getAttribute("name") == strTargetElementName)
{
return element.getAttribute("value");
}
}
The webbrowser.Document is cast into an IHTMLDocument2, then the IHTMLDocument2's frames are put into a IHTMLFramesCollection2, then I cast the specific desired frame into an IHTMLWindow2 (you can choose frame by index # or name), then I cast the frame's .Document member into an IHTMLDocument2 (the originally used one, for convenience sake). From there, the IHTMLDocument2's .all.tags() method is functionally identical to the old WebBrowser.Document.GetElementsByTagName() method, except it requires an IHTMLElementCollection versus an HTMLElementCollection. Then, you can foreach the collection, the individual elements needing to be IHTMLElement, and use .getAttribute to retrieve the attributes. Note that the g is lowercase.
The WebBrowser control can handle the NewWindow event so that new popup windows will be opened in the WebBrowser.
private void webBrowser1_NewWindow(object sender, CancelEventArgs e)
{
// navigate current window to the url
webBrowser1.Navigate(webBrowser1.StatusText);
// cancel the new window opening
e.Cancel = true;
}
http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/361b6655-3145-4371-b92c-051c223518f2/
The only solution to this I have seen was a good few years ago now, called csExWb2, now on Google code here.
It gives you an ExWebBrowser control, but with full-on access to all the interfaces and events offered by IE. I used it to get deep and dirty control of elements in a winforms-hosted html editor.
It may be a bit of a leap jumping straight into that, mind.
I am automating a task using webbrowser control , the site display pages using frames.
My issue is i get to a point , where i can see the webpage loaded properly on the webbrowser control ,but when it gets into the code and i see the html i see nothing.
I have seen other examples here too , but all of those do no return all the browser html.
What i get by using this:
HtmlWindow frame = webBrowser1.Document.Window.Frames[1];
string str = frame.Document.Body.OuterHtml;
Is just :
The main frame tag with attributes like SRC tag etc, is there any way how to handle this?Because as i can see the webpage completely loaded why do i not see the html?AS when i do that on the internet explorer i do see the pages source once loaded why not here?
ADDITIONAL INFO
There are two frames on the page :
i use this to as above:
HtmlWindow frame = webBrowser1.Document.Window.Frames[0];
string str = frame.Document.Body.OuterHtml;
And i get the correct HTMl for the first frame but for the second one i only see:
<FRAMESET frameSpacing=1 border=1 borderColor=#ffffff frameBorder=0 rows=29,*><FRAME title="Edit Search" marginHeight=0 src="http://web2.westlaw.com/result/dctopnavigation.aspx?rs=WLW12.01&ss=CXT&cnt=DOC&fcl=True&cfid=1&method=TNC&service=Search&fn=_top&sskey=CLID_SSSA49266105122&db=AK-CS&fmqv=s&srch=TRUE&origin=Search&vr=2.0&cxt=RL&rlt=CLID_QRYRLT803076105122&query=%22LAND+USE%22&mt=Westlaw&rlti=1&n=1&rp=%2fsearch%2fdefault.wl&rltdb=CLID_DB72585895122&eq=search&scxt=WL&sv=Split" frameBorder=0 name=TopNav marginWidth=0 scrolling=no><FRAME title="Main Document" marginHeight=0 src="http://web2.westlaw.com/result/dccontent.aspx?rs=WLW12.01&ss=CXT&cnt=DOC&fcl=True&cfid=1&method=TNC&service=Search&fn=_top&sskey=CLID_SSSA49266105122&db=AK-CS&fmqv=s&srch=TRUE&origin=Search&vr=2.0&cxt=RL&rlt=CLID_QRYRLT803076105122&query=%22LAND+USE%22&mt=Westlaw&rlti=1&n=1&rp=%2fsearch%2fdefault.wl&rltdb=CLID_DB72585895122&eq=search&scxt=WL&sv=Split" frameBorder=0 borderColor=#ffffff name=content marginWidth=0><NOFRAMES></NOFRAMES></FRAMESET>
UPDATE
The two url of the frames are as follows :
Frame1 whose html i see
http://web2.westlaw.com/nav/NavBar.aspx?RS=WLW12.01&VR=2.0&SV=Split&FN=_top&MT=Westlaw&MST=
Frame2 whose html i do not see:
http://web2.westlaw.com/result/result.aspx?RP=/Search/default.wl&action=Search&CFID=1&DB=AK%2DCS&EQ=search&fmqv=s&Method=TNC&origin=Search&Query=%22LAND+USE%22&RLT=CLID%5FQRYRLT302424536122&RLTDB=CLID%5FDB6558157526122&Service=Search&SRCH=TRUE&SSKey=CLID%5FSSSA648523536122&RS=WLW12.01&VR=2.0&SV=Split&FN=_top&MT=Westlaw&MST=
And the properties of the second frame whose html i do not get are in the picture below:
Thank you
I paid for the solution of the question above and it works 100 %.
What i did was use this function below and it returned me the count to the tag i was seeking which i could not find :S.. Use this to call the function listed below:
FillFrame(webBrowser1.Document.Window.Frames);
private void FillFrame(HtmlWindowCollection hwc)
{
if (hwc == null) return;
foreach (HtmlWindow hw in hwc)
{
HtmlElement getSpanid = hw.Document.GetElementById("mDisplayCiteList_ctl00_mResultCountLabel");
if (getSpanid != null)
{
doccount = getSpanid.InnerText.Replace("Documents", "").Replace("Document", "").Trim();
break;
}
if (hw.Frames.Count > 0) FillFrame(hw.Frames);
}
}
Hope it helps people .
Thank you
For taking html you have to do it that way:
WebClient client = new WebClient();
string html = client.DownloadString(#"http://stackoverflow.com");
That's an example of course, you can change the address.
By the way, you need using System.Net;
This works just fine...gets BODY element with all inner elements:
Somewhere in your Form code:
wb.Url = new Uri("http://stackoverflow.com");
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wbDocumentCompleted);
And here is wbDocumentCompleted:
void wb1DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var yourBodyHtml = wb.Document.Body.OuterHtml;
}
wb is System.Windows.Forms.WebBrowser
UPDATE:
The same as for the document, I think that your second frame is not loaded at the time you check for it's content...You can try solutions from this link. You will have to wait for your frames to be loaded in order to see its content.
The most likely reason is that frame index 0 has the same domain name as the main/parent page, while the frame index 1 has a different domain name. Am I correct?
This creates a cross-frame security issue, and the WB control just leaves you high and dry and doesn't tell you what on earth went wrong, and just leaves your objects, properties and data empty (will say "No Variables" in the watch window when you try to expand the object).
The only thing you can access in this situation is pretty much the URL and iFrame properties, but nothing inside the iFrame.
Of course, there are ways to overcome teh cross-frame security issues - but they are not built into the WebBrowser control, and they are external solutions, depending on which WB control you are using (as in, .NET version or pre .NET version).
Let me know if I have correctly identified your problem, and if so, if you would like me to tell you about the solution tailored to your setup & instance of the WB control.
UPDATE: I have noticed that you're doing a .getElementByTagName("HTML")(0).outerHTML to get the HTML, all you need to do is call this on the document object, or the .body object and that should do it. MyDoc.Body.innerHTML should get the the content you want. Also, notice that there are additional iFrames inside these documents, in case that is of relevance. Can you give us the main document URL that has these two URL's in it so we / I can replicate what you're doing here? Also, not sure why you are using DomElement but you should just cast it to the native object it wants to be cast to, either a IHTMLDocument2 or the object you see in the watch window, which I think is IHTMLFrameElement (if i recall correctly, but you will know what i mean once you see it). If you are trying to use an XML object, this could be the reason why you aren't able to get the HTML content, change the object declaration and casting if there is one, and give it a go & let us know :). Now I'm curious too :).
What am I doing:
My main intent is to enable user friendly text to speech for personal use on Win 7. Approach should work in Google Chrome, VS and Eclipse.
Code example:
Following code creates global keyboard hook for ctrl + alt + space, called hookEvent. If event fires, it starts/stops speaking clipboard contents ( that can be updated with ctrl + c ).
/// <summary>
/// KeyboardHook from: http://www.liensberger.it/web/blog/?p=207
/// </summary>
private readonly KeyboardHook hook = new KeyboardHook();
private readonly SpeechSynthesizer speaker = //
new SpeechSynthesizer { Rate = 3, Volume = 100 };
private void doSpeaking(string text)
{
// starts / stops speaking, while not blocking UI
if (speaker.State != SynthesizerState.Speaking)
speaker.SpeakAsync(text);
else
speaker.SpeakAsyncCancelAll();
}
private void hookEvent(object sender, KeyPressedEventArgs e)
{
this.doSpeaking(Convert.ToString(Clipboard.GetText()));
}
public Form1()
{
InitializeComponent();
hook.KeyPressed += new EventHandler<KeyPressedEventArgs>(hookEvent);
hook.RegisterHotKey(ModifierKeysx.Control|ModifierKeysx.Alt, Keys.Space);
}
Question:
I would prefer not using the clipboard. Or at least, restoring the value after, something like:
[MethodImpl(MethodImplOptions.Synchronized)]
private string getSelectedTextHACK()
{
object restorePoint = Clipboard.GetData(DataFormats.UnicodeText);
SendKeys.SendWait("^c");
string result = Convert.ToString(Clipboard.GetText());
Clipboard.SetData(DataFormats.UnicodeText, restorePoint);
return result;
}
What are my options?
Edit:
To my surprise, I found that my clipboard reader is the best way to go. I created a notification area app, that responds to left click (speaking clipboard) and right click (menu opens up). In menu the user can chance speed, speak or create a audio file.
MS provide accessibility tools that do cover what you're trying to do. If you take a look at documents about screen scraping. In short, every component is accessible in some manner, if you use some of the windows debugging tools you can get to see the component names/structures within. You can then use that, however, its complicated as most times you would need to be very specific for each application you intend to scrape from.
If you manage to scrape you dont need to use the clipboard, as you can access the text property of the apps direct. Its not something I've had to do, hence, Ive no code to offer off the top of my head, but the term "screen scraping" should point you in the right direction.
If to expand a little on what Bugfinder said, Microsoft provider a UI Automation Framework to solve problems like the one you mentioned:
In particular you can use the TextSelectionChangedEvent of TextPattern:
The problem with this solution is that it only works on supported operating systems and applications - and not all support this.
Your clipboard solution is acceptable for applications that do not provide a good automation interface.
But for many applications the UI Automation Framework will work well and will provide you with a far better solution.