I'm trying to log into a website and use Selenium to take a screenshot through code. So far, it's been working great except for one problem: every time I access the website, both manually and with Selenium, a popup asks me to confirm a security certificate.
The certificate is good, but I cannot for the life of me get Selenium to accept the window. I know it can deal with Alerts and accept them thusly:
driver.Navigate().GoToUrl("https://www.securewebsite.com");
driver.SwitchTo().Alert().Accept();
but the box doesn't disappear or react, and attempting to do anything with the driver besides accepting or declining the alert (with no effect) results in a Modal Dialog Present error.
I'm looking for either a way to turn off this popup or a way for Selenium to deal with it in code, preferably the latter. Let me know if I can provide any more information.
Alright, I couldn't figure it out, so I just disabled the popup. It's poor practice and binding the application to the specific instance of IE, but it was all I could come up with.
Related
I would like to build a bot - web crawler - to collect phone numbers.
I have a problem though: to see the phone number, a user must click something like "Show".
How can I solve this problem?
Check what the act of clicking on the button does. Does it call a Javascript function? Does that make an HTTP call to a backend? If so your bot should do that call instead of screen-scraping the first page. If not, does it just play with the DOM of the page to show an item on screen?
All the data you're looking for comes from some sort of back-end, so if you look in the developer tools of your browser when going through the page you can usually figure out what calls to script in order to get the data.
It is possible to make this harder (and that is what some sites to to protect themselves from scraping). Typically if you're in this situation, what you're doing is not entirely legal or nice. But technically it's very interesting, so here goes.
The best way to go forward is to run the site in a real browser (like PhantomJS, or Chrome) and use a framework like Webdriver to simulate browser interactions. This way you can pull most of the data out usually.
If you find that your ip gets blocked, you may use Tor and use multiple instances dynamically to hit the site... but make sure you ask the site owner nicely if you're allowed to do that of course.
I am using .NET 4 and trying to use the desktop authentication for the StackApps site via the web-browser control (WPF and/or WinForms) to develop a NNTP Bridge for accessing StackOverflow (https://stackapps.com/questions/4215/stackapp-nntp-bridge-for-accessing-stackexchange-forums-like-stackoverflow).
It seems that the login cannot be done, because the web browser hangs up, after the page from "StackExchange Login" is displayed.
I use the following URL:
https://stackexchange.com/oauth/dialog?client_id=1736&scope=no_expiry&redirect_uri=https://stackexchange.com/oauth/login_success
It works in the normal IE browser, but not in a WinForms or WPF window... Does anyone know what the problem is?
It is simple to repoduce:
Create a WinForms-Project
Add the "WebBrowser" control to the dialog
Double-Click on the Form1
Add the following code
private void Form1_Load(object sender, EventArgs e)
{
webBrowser1.Navigate("https://stackexchange.com/oauth/dialog?client_id=1736&scope=no_expiry&redirect_uri=https://stackexchange.com/oauth/login_success");
}
Start the application
Login by pressing the "login with Stack Exchange" account symbol
A new page gets loaded; it is displayed correctly, but you cannot enter your login name; the window hangs...
The same happens, if I use WPF-App and the WPF-WebBrowser-Control... it seems that it is stuck in an endless-loop in JavaScript...
Any hint on how to solve ths problem?
Or is it possible to debug the JavaScript in the WebBrowser-Control???
Fixing WebBrowser Control
I also had the issue of the WebBrowser control handing when trying to login.
Although requesting a token in IE (11) works, I found that IE itself also hangs when I put it in IE 7 emulation mode. This suggested to me that my previous attempt to make the WebBrowser control use a newer version had failed.
I found this article, Web Browser Control Specifying the IE Version, which suggests that for 32 bit applications in 64 bit mode, you need to set a different registry value.
So now, I've added two values in the registry:
HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION
and
HKEY_CURRENT_USER\Software\Wow6432Node\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_BROWSER_EMULATION
In both of them, I added a DWORD value named 'MyExecutable.exe' (where MyExecutable is the actual name of my executable). The value for each of them is 9000 which will work for IE9 and above. Watch out when using RegEdit to test this, it will default to hexadecimal instead of decimal. Also, make sure it is a DWORD value, not any other type.
This seems to do the trick. I can now run the application, go through the login process, and eventually I am redirected to the url specified by me, which I can then capture using the OnNavigate event of the webbrowser control.
Remaining issue
It doesn't really work perfectly. The first time I was redirected to some OpenID page as well, but at least the form didn't hang. With subsequent attempts, apparently the login (which succeeded before) is remembered and I get the message "Navigation to the webpage was cancelled" with a link to refresh the page. When I click that link, I am immediately redirected to the redirect_url I specified when requesting the login form. At least that part works, and I get an actual access_token and an expiry time, so for now I'm happy.
Update: After some testing, it turned out that the previous login was remembered. That causes the request uri to direct to the redirect_uri immediately. I used the OnBeforeNavigate event to detect this, but it isn't fired in this case. I now linked the NavigateComplete2 event, and that one is triggered in this scenario.
Fix for .NET?
I think for .NET the solution should be the same: add the executable (and MyExecutabl.vshost.exe as well, for debugging purposes in Visual Studio) to the first key. If it's a 32 bit executable running on 62 bit Windows, you might also need to add it to the second key, although I'm sure if that rule applies to .NET as well. I don't do C# on a daily basis, and I'm trying to get it to work in Delphi first, but if I find time to test this in C# I will post the update here.
In the end, it would be nice if the actual issue would be solved, and the JavaScript would work in IE7 mode as well, but at least this seems to be a proper work-around.
Unfortunately, I'm not a JavaScript developer, so I only could try to explain you how to debug a JavaScript that executes in the WebBrowser control.
This approach is for the Microsoft Visual Studio, I don't know if Delphi can provide similar functionality.
Enable Script Debugging (both Internet Explorer and Other) in Internet Explorer settings.
Disable Friendly HTTP messages in Internet Explorer settings.
Enable Display a notification about every script error in Internet Explorer settings.
From Visual Studio, start your WebBrowser hosting application without debugging (i. e., Ctrl + F5).
In Visual Studio, go to Debug → Attach to Process… and select your application in the list.
Hit the Select… button to the right of Attach to: field and choose Script code.
Hit the Attach button. Visual Studio starts the script debugger.
In your application, navigate to the deadlocking page by pressing the Login with Stack Exchange account symbol.
Go to Visual Studio and press the Pause button on the debugging toolbar.
Now you can look into your script code and investigate the code itself, the call stack, the variable values and so on. You can set breakpoints too. Perhaps you can then find the place where the script hangs. As I said before, I'm not a web developer and cannot help you with this…
Update:
I guess I can propose you a working solution.
My investigation shows that the WebBrowser hangs when it renders the content in the IE7 mode (what is the default mode even if you have IE10 in your system). You should force it to switch into IE9 mode. In IE9 mode, the page renders well and does not cause the script to stuck in an endless loop.
You can switch your WebBrowser to the IE9 mode using one of the following methods:
Define the global browser emulation mode for your application in Windows registry.
For 32 bit OS, go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_BROWSER_EMULATION.
For 64 bit OS, use HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_BROWSER_EMULATION.
In this node, create a new DWORD parameter called YourApplicationExeName.exe with a value 9000 (0x2710). You can create another entries for your *.vshost.exe executables, if you want this to work in Visual Studio debug mode.
Manipulate the source for the WebBrowser to switch it in the IE9 mode.
This will be more complicated. You need to alter the <head> tag of the html document adding a new <meta> tag preferably as a first element: <meta http-equiv="X-UA-Compatible" content="IE=9"/>.
This will cause the WebBrowser to switch its mode on document rendering. Since you can't change the compatibility mode after the document is rendered, you could use a proxy as a source for your WebBrowser, so this proxy will add the header.
I`ve successfully tested this with the 1st approach, and the second one should work too.
Hope that helps!
Project + Properties, Debug tab, tick the "Enable native code debugging" option. Ensure that you've got the Microsoft Symbol server enabled (Tools + Options, Debugging, Symbols), I know you do :)
You can now use Debug + Break All and see what's going on inside the browser. Do so repeatedly to get the lay of the land. You'll see that Javascript is executing (jscript8.dll) and constantly forcing the layout engine to recalculate layout, never getting the job done.
Browsers in general are vulnerable to Javascript that's stuck in an endless loop. A regular browser tends to put up a dialog after several dozen seconds to allow the user to abort the JS, but WebBrowser doesn't have that feature. Nor does it have the dev tools so debugging the JS isn't going to be a great joy either. This is going to be difficult to get fixed.
You might consider using the OAuth 2.0 api instead. Notes on usage are on this web page. Exactly how to integrate that with WebBrowser is a bit murky to me, I don't have a key to test this. Find help for this at the Stackapps site. You are probably not the first SE api user that ran into this problem.
I have a C# application which uses a System.Windows.Forms.WebBrowser.
The problem is: i'd like the user to navigate smoothly in my application, without prompts, without javascript windows popping up, without security prompts. Even if this requires some contents to be unavailable.
I just want to have one window (always one window, if a receive a new window event, i redirect it to the single window).
How can i do this?
I tried to use this.browser.ScriptErrorsSuppressed = true but i doesnt seem to work.
For example, if i test it on a browser page which performs text validation, i still receive a popup window saying that my text is invalid.
Thank you!
I've found a solution somewhere else, since it wasn't available here.
Here it is: http://www.codeproject.com/Articles/31163/Suppressing-Hosted-WebBrowser-Control-Dialogs
Basically, you have to hook the WM_INITDIALOG message.
It works wonders here.
Background:
I am creating a Windows Form App that automates order entry on a intranet Web Application. We have a large amount of order entry that needs to be done that will cost us a lot of money so I volenteered to automate the task.
Problem:
I am using the webbrowser class to navigate the web app. I have gotten very far but reached a road block. There is a part in the app that opens a web dialog page. How do I interact with the web dialog. My instance of the webbrowser class is still with the parent page. I am hoping someone can point me in the right direction.
You've got a number of options. To expand on the answers from others and add a new idea...
Do it using the webbrowser control: This is technically possible by either injecting javascript into the target page as demonstrated here or creating a JavaScript object and using it as a bridge via the webbrowser.objectforscripting property. This is very fragile - something as simple as the website changing an element's Id could break it. You also need to make sure your code doesn't interfere with the functioning of the form (clashing function names, etc...)
Do it using a postback: Monitor the communications between the web browser and the server (I personally prefer Firfox/Firebug but ie/Fiddler or Chrome/F12 are both good too). As long as you can replicate the actions of the browser exactly, the server can't know the difference. The problem here is that browsers are complex and the more secure a form is, the more demanding servers are. This means you may have to fake a login, get cookies, send them back on subsequnt requests, Handle Viewstate data and xss prevention variables. It's possible and it's far more robust than the first option but can be a pain to get working. If it's not a highly secure form,, this is your best bet. More information here
Do it by browser automation: Selenium is probably the best option here (as mentioned by others) but suffers from a similar flaw to the webbrowser control in that it's sensitive to changes on the form itself (But not as mcuh so as the webbrowser control).
Incidentally, if you have Visual Studio Ultimate/Test edition (and some others, not sure which), it includes a suite of testing tools including an excellent engine to automate load testing a website. This is also superb for tracking down what exactly a form does as you can see every step of the emulation.
Hope this helps
You have two choices depending of the level of complexity you need:
Use a HTTP Debugger like Fiddler to find out the POST data you
need to send to each page and mimic it via a HttpWebRequest.
Use a Browser Automation Tool like Selenium and do the job.
NOTE: Your action may be considered as spamming by the website so be ready for IP blocking, CAPTCHA...
You could give Selenium a go: http://seleniumhq.org/
UI automation is a far more intuitive approach to these types of tasks.
How to create a open a webpage as a popup using c# and need to run a function when the popup window is closed. my intention is to create a web login/logout and run a function after successful completion of the event
Well you've not given much away, but if I'm guessing that your architecture is ASP.NET, then you should have events on the server in your page's codebehind that can process that event. If you expand a bit on your requirements we can help you out a bit more.
Just for completeness, you should know that you can't just run C# code in a browser with html/javascript. You could run a Silverlight application but I don't think that's what you're after.
To summarise, make a web request and respond to it on the server. Popups are just webpages, so the architecture there is the same. When the request comes back, you can then run JS to close the popup and make the main browser window do something.
Personally I'd just have the main browser do the login, popups are cumbersome for users in web apps.
If you are using jQuery, I would strongly suggest using ThickBox http://jquery.com/demo/thickbox/ We use it in every single projects we do and it work very weel and it's easy to modify to have it do what you want.
You can use it to load another aspx page where your login code would reside and then pay particual attention to
function tb_remove() {
Which is called on close. This is where we added our code to return data to the page.
The short answer is, you can't. C# runs on the server and opening a popup window is a client side action. You will need to have JavaScript in your rendered markup to open the popup window when appropriate, or an anchor tag with target="_blank".
However, I agree with the other answers that popup windows are more of a pain than they are worth, they annoy users and lead to window management issues that are not always easy to solve especially when popup blockers are involved. A DOM based modal dialog is almost always a better solution.
I would go with Neil... Sorry to say this but you are exactly the type of person Jeff Atwood was talking about when he wrote this article...
http://www.codinghorror.com/blog/archives/001296.html
I would suggest you take the time to learn the difference between client and server functionality, languages and technology.
I would also suggest you listen to Neil, your usability skills also need serious work.
LOL - and if you think I'm being cruel, think how cruel you're being to your users... login in a pop up window... bah