We Keep Coding

Selenium ChromeDriver returns 403 errors on build server

We've created a Selenium test project that starts the (ASP.NET) web application and runs a couple of tests using the ChromeDriver. Locally this all runs fine (in headless and non-headless mode).
But on the build server (using an Azure DevOps agent) this fails without ever starting the tests. It looks like it fails when starting the ChromeDriver: the driver starts, but then it's immediately followed by 403 errors. It never gets to the part where it actually loads a webpage.
Any ideas where to look?

Answering my own question to document possible solutions.
After some rigorous investigation (which included using the source code to get to the bottom of things) we found out that the proxy server somehow got in the way. It turned out that the ChromeDriver tries to communicate over a local port (e.g. http://localhost:12345), which was redirected through the proxy server. This failed with a 403 error.
This gave us a lead on possible solutions. First we tried to use the .proxybypass file to exclude localhost addresses. This didn't work -- it turns out that this proxy bypass only works for https requests. And the ChromeDriver control commands are sent over http :-(
We then made sure that no proxy was used in our test code. We did this with the following lines:
var options = new ChromeOptions();
options.AddArgument("--no-sandbox");
options.AddArgument("headless");
options.AddArgument("ignore-certificate-errors");
options.Proxy = new Proxy()
{
Kind = ProxyKind.Direct
};
var driver = new ChromeDriver(options);
In addition to these settings (note that some arguments were added to solve other issues and might not apply to your own situation), we also disabled the proxy for other requests:
WebRequest.DefaultWebProxy = null;
HttpClient.DefaultProxy = new WebProxy()
{
BypassProxyOnLocal = true,
};
This allowed our tests to finally run on the build server without the 403 errors.
One last remark (which might be obvious) is to always run your tests in non-headless mode if you encounter any issues. This allowed us to see the "invalid certificate error" which would otherwise be hidden.

Recurring ERR_CONNECTION_RESET in C# implementation of Selenium ChromeDriver

I've been battling the following error over the past few weeks trying to run UI tests with Selenium (Chrome 92):
WebDriverException unknown error: net::ERR_CONNECTION_RESET. (Session info: chrome=92.0.4515.107)
This occurs most often trying to locate HTML elements using XPath.
Originally, I added a retry loop which checks if the WebDriverException is thrown three times, and if so, recycle the IWebDriver and try again (via IWebDriver.Quit() and IWebDriver.Dispose(). This does not fix the problem, nor does having my application restart after exceeding the retry limit. Thus, I'm starting to think this is a problem at the system-level (Windows Server 2016). Even a reboot does not always fix the issue - when it starts to fail, it just totally freezes and I end up having to completely rebuild the environment.
I'm using the Selenium WebDriver NuGet v92.0.4515.4300; I realize this isn't in total parity with the version listed above, but having the exact same versions does not fix the problem.
I know 'unknown error' is vague. Does anyone have suggestions for what could be causing the problem?
Thanks in advance.
EDIT: Posting browser arguments here, rather than cramming them into a comment. I forgot to mention, there is an instance of Fiddler which we run as a proxy on the local system and use in conjunction with some tests. It's started and ended with the testing application.
The following are injected as options when creating the Chrome instance:
--user-data-dir=C:\temp\{uniquedir}
options.AddUserProfilePreference("credentials_enable_service", false);
options.AddUserProfilePreference("profile.password_manager_enabled", false);
And these are added as command-line arguments through Selenium:
--lang=en-GB
--ignore-certificate-errors
--no-experiments
--disable-translate
--disable-plugins
--enable-logging
--no-sandbox
--dns-prefetch-disable
--disable-gpu
--disable-field-trial-config
(We are doing some security-related detections, hence the disabling of certificate errors.)

You mentioned that you use Fiddler, if the issue is reproducible only when it is started, you can try to regenerate the HTTPS certificates
The issue can be due to the application itself. If the error is shown when you are not using Selenium/ChromeDriver you can ask your developers/devops to check the IIS settings or follow one of the suggestions below.
Check your Internet options from computer Settings->Network&Internet->Proxy. On my computer "Automatically detect Settings" is enabled.
You can also try to delete the Chrome browser cache.
Flush the DNS (ipconfig /flushdns)
It could be some rule (firewall, internal DNS record) configured for the network where is your Windows Server.
Check if the issue is reproducible in incognito mode (without any browser extensions enabled). You can pass --incognito argument when starting the driver.
Issue could be caused by a certificate or SSL/TLS problem.

How to open the Default Chrome Profile through Selenium, ChromeDriver and GoogleChrome

I want to load a new Selenium ChromeDriver that is using Chrome as if I open Chrome from my dock (Essentially it'll have all my extensions, history, etc.)
When I use the following code:
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\User\\AppData\\Local\\Google\\Chrome\\User Data\\");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
It loads the Chrome browser with me signed into my Gmail and with all my extensions, just like I want, but the rest of my code:
chromeDriver.Navigate().GoToUrl("https://www.youtube.com/");
doesn't execute. But when I use the following
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\Andrea\\AppData\\Local\\Google\\Chrome\\User Data\\Default");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
The rest of my code executes perfectly (Notice the 'Default' added to the end of the first Argument). Any tips or suggestions on how I can get the first block of code (The one without 'Default' on the end) to execute the rest of my program would be great. Thanks!

I know this is an old question, but what worked for me is to do remove the "C:\" and replace all of the backslashes with forward slashes. So, with that from the original question, this should work to load the default profile:
options.AddArgument("user-data-dir=/Users/User/AppData/Local/Google/Chrome/User Data");

The Default Chrome Profile which you use for your regular tasks may contain either/all of the following items:
History
Bookmarks
Cookies
Extensions
Themes
Customized Fonts
All these configurations untill and unless are part of your Test Specification it would be a overkill to load them into the session initiated by Selenium WebDriver. Hence it will be a better approach if you create a dedicated New Chrome Profile for your tests and configure it with all the required configuration.
Here you will find a detailed discussion on How to create and open a Chrome Profile
Once you have created the dedicated New Chrome Profile for your tests you can easily invoke the Chrome Profile as follows:
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\User\\AppData\\Local\\Google\\Chrome\\User Data\\Profile 2");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
chromeDriver.Navigate().GoToUrl("https://www.youtube.com/");
Here you will find a detailed discussion on How to open URL through default Chrome profile using Python Selenium Webdriver

I have the same issue. I don't know how to fix it, I guess the root cause is white space in profile path.
I know a workaround for this. Just copy the C:\\Users\\Andrea\\AppData\\Local\\Google\\Chrome\\User Data to c:\myUserData (no space in the path).
Then add the argument.
options.AddArgument("user-data-dir=C:\\myUserData");

This is an Old question, but if you are facing this issue, all you have to do is close all tabs, Just shut down the chrome window..
Selenium can't use the data since it is already in use.
Hope you fond this helpful.

Using a proxy with Selenium

for the past two days I've been trying to use a proxy with Selenium, that's not exactly the issue though. The issue is that the proxy is private meaning it needs authentication to use it (Username and Password) but I can't figure out how to do it.
I'm using a Firefox driver, with a profile like so:
FirefoxProfile firefoxProfile = new FirefoxProfile();
firefoxProfile.SetPreference("network.proxy.type", 1);
firefoxProfile.SetPreference("network.proxy.http", "23.95.115.87");
firefoxProfile.SetPreference("network.proxy.http_port", 80);
var driver = new FirefoxDriver(firefoxProfile);
driver.Navigate().GoToUrl("http://ipchicken.com");
I figured that it would ask me for the username and password (in a dialog box) yet nothing happens, it just navigates to the webpage, and displays my own IP. I can't find anything really on this, any help guys? Thank you so much.

I am not an expert in Selenium but I can help you in making your proxy authentication free.
If you are on Windows, download something like CC-Proxy ( Its free for a single user) and add your proxy as a cascading proxy. This will create a local proxy server on your computer which won't require username/password. Then you can use the local proxy server in selenium.
If you are on Linux, you can use wine to run CC-Proxy or you can use tinyproxy or squid ( it is an overkill).
Comment if you face problem in setting up CC-Proxy or tinyproxy.

http authorization with webdriver and popup window

I'm trying to use Selenium WebDriver to automatically login in to a site with a user-name and password. I've done my research and I don't believe this feature is supported by WebDriver, so I need to find another way. The site I'm trying to automate logging into is located here.
When prompted to login a popup window comes up that doesn't seem to be part of the browser. I'm using Firefox and Chrome. It seems Windows API may be required? I already tried passing the credentials in the URL but that didn't work. Also tried sendkeys, but received a Windows exception that the application was not accepting Windows messages. I also tried switching the current handle using driver.windowhandles but the popup doesn't seem to be a new handle.
Does anybody have any ideas? I'm kinda stuck. The preliminary code to get to the popup window is:
IWebDriver driver = new FirefoxDriver();
driver.Navigate().GoToUrl("http://www.portal.adp.com");
string currentWindow = driver.CurrentWindowHandle;
IWebElement userLogin = driver.FindElement(By.Id("employee"));
userLogin.Click();

The popup you are seeing is prompted by web server and is a authentication prompt. Selenium doesn't support this operation.
One of the way to handle this limitation is to pass user and password in the url like like below:
http://user:password#example.com
More info available here : http://aleetesting.blogspot.in/2011/10/selenium-webdriver-tips.html

I wanted my answer out there because I think I've solved it. This answer does not require passing the credentials through the URL (for those of you that are unable to like me). It also does not require any custom Firefox Profiles or extensions to be installed or included with the solution or installed onto the browser eliminating cross-machine compatibility issues.
The issue with me was that the authentication could not be completed via passing the credentials through the URL because the login was behind a proxy.
So, I turned to windows automation toolkits and found AutoIT. Using AutoIT and Selenium, you can login automatically by sending the username and password to the windows dialog that appears. Here's how (note the steps below are for c#:
1 - Download AutoIT from http://www.autoitscript.com/site/autoit/downloads/
2 - Add the autoit .dll to your project references.
Right click on references, select Add Reference. Next click the browse button and browse to the dll location (most default installations it will be c:\Program Files (x86)\AutoIt3\AutoItX\AutoItX3.dll), and add to project.
3 - use AutoIT and Selenium like this (assuming your web driver is already initialized):
//Initialize AutoIT
var AutoIT = new AutoItX3();
//Set Selenium page load timeout to 2 seconds so it doesn't wait forever
Driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(2));
//Ingore the error
try
{
Driver.Url = url;
}
catch
{
return;
}
//Wait for the authentication window to appear, then send username and password
AutoIT.WinWait("Authentication Required");
AutoIT.WinActivate("Authentication Required");
AutoIT.Send("username");
AutoIT.Send("{TAB}");
AutoIt.Send("password");
AutoIT.Send("{ENTER}");
//Return Selenium page timeout to infinity again
Driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(-1));
Anyway, that's it, and it works like a charm :)
Also note that there are some special characters that need to be escaped in AutoIT using the sequence "{x}". For example, if your password is "!tRocks", you'd need to pass it into AutoIT as "{!}tRocks".
Happy automating.

FirefoxProfile profile = new FirefoxProfile();
profile.SetPreference("network.http.phishy-userpass-length", 255);
profile.SetPreference("network.automatic-ntlm-auth.trusted-uris", hostname);
Driver = new FirefoxDriver(profile);
hostname is your URL (example.com) then try to
Driver.Navigate().GoToUrl(http://user:password#example.com);

I just got done working on a prototype project that is supposed to handle exactly this kind of situation.
It utilizes BrowserMob, a popular open source proxy, to perform the authentication.
SeleniumBasicAuthWrapper Hope it helps! It is still a work in progress, but hopefully we'll get any kinks or defects ironed out in the near future.