We Keep Coding

NET5 - How to run numerous web acceptance tests on a single machine

Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.
By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.
Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.
A typical C# code to do that can be summarized as follows:
namespace WebAcceptanceTests
{
public static class Chrome
{
public static async Task Run(
Uri uri,
Func<ChromeDriver, Task> manipulate,
Action<ChromeDriver> validate)
{
var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
// To make Chrome window invisible.
options.AddArgument("--headless");
using var driver = new ChromeDriver(chromeDriverService, options);
try
{
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl(uri);
await manipulate(driver);
validate(driver);
}
finally
{
driver.Quit();
}
}
}
}
where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:
public static async Task Manipulate(ChromeDriver driver)
{
// Do some useful stuff here instead of just waiting.
await Task.Delay(60_000);
}
However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.
The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.
Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.
Thanks.

We achieve this using NUnit parallel run, parakkekuzabke by fixture.
Allocate driver during OneTimeSetup. Do whatever test need in single fixture.
On OneTimeTearDown, dispose driver.
We do this in base class that all web acceptance test fixtures are inherit
[Parallelizable(ParallelScope.Fixtures)]
public abstract class WebDriverTest
{
protected IDriver driver;
[OneTimeSetup]
public void PrepareDriver()
{
// ...
this.driver = new ChromeDriver(chromeDriverService, options);
// ...
}
[OneTimeTearDown]
public void CleanupDriver()
{
this.driver.Dispose();
}
[TearDown]
public void ScreenshotForFailedTest()
{
var testStatus = GetTestStatus();
if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
{
this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
// log more details if needed
}
}
}
[OneTimeTearDown] is executed even if there is failed tests
As bonus we take screen
Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.

Selenium WebDriver - Wait for long running queues

I have an application which is used to trigger long running data queues. By long running, I mean around 12-16 hours per queue and either of them cannot be executed in parallel. Each queue has individual steps which need to succeed before the next one runs.
I have already increased the timeouts while initializing ChromeDriver upto 1000 minutes
webDriver == new ChromeDriver(path,options,TimeSpan.FromMinutes(1000));
I am using WebDriverWait for checking after 1000 mins that all steps have been succeeded. In case of a failure, I still have to wait for 1000 minutes before I can tell the dev team about the failure.
Is there a better approach to solve this problem? It is also keeping my browser open for 1000 mins

Regarding your question -- is there a better way to solve this problem? With Selenium, not really. You'd have better luck taking a different approach, such as API, than through UI testing. However, it's still possible, just not ideal.
My best idea for this problem would be to set up some sort of controller that can manage your WebDriver instances and also keep track of the 12-16 hour queue time. Since I don't have any specific information about your project architecture or the queues you are testing, this will be a very generic implementation.
Here's a simple DriverManager class, that controls creating & terminating WebDriver sessions:
public class DriverManager
{
public IWebDriver CreateDriver
{
// code to initialize your WebDriver instance here
}
public void CloseWebDriverSession
{
Driver.Close();
Driver.Quit();
}
}
Next, here's a test case implementation that utilizes DriverManager to close & reopen WebDriver as needed.
public class TestBothQueues
{
// this driver instance will keep track of your session throughout the test case
public IWebDriver driver;
[Test]
public void ShouldRunBothQueues
{
// declare instance of DriverManager class
DriverManager manager = new DriverManager();
// start a webdriver instance
driver = manager.CreateDriver();
// run the first queue
RunFirstQueue();
// terminate the WebDriver so we don't have browser open for 12 hours
manager.CloseWebDriverSession();
// wait 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// start another WebDriver session to start the second queue
driver = manager.CreateDriver();
// run the second queue
RunSecondQueue();
// terminate when we are finished
manager.CloseWebDriverSession();
}
}
A few notes on this:
You can also convert this code into a while loop if you would like to start a WebDriver instance to check the queue on a time interval. For example, if the queue takes 12-16 hours to finish, you may want to wait 12 hours, then check the queue once per hour until you can verify it is completed. That would look something like this:
// first, wait initial 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// keep track of whether or not queue is finished
bool isQueueFinished = false;
while (!isQueueFinished);
{
// start webdriver instance to check the queue
IWebDriver driver = manager.CreateDriver();
// check if queue is finished
isQueueFinished = CheckIfQueueFinished(driver);
// if queue is finished, while loop will break
// if queue is not finished, close the WebDriver instance, and start again
if (!isQueueFinished)
{
// close the WebDriver since we won't need it
manager.CloseWebDriverSession();
// wait another hour
Thread.Sleep(TimeSpan.FromHours(1));
}
}
Hope this helps.

Waiting for render events with Selenium and React

I am getting started with using Selenium to test a React application that I have built.
In most simple JavaScript applications it is trivial to wire up a test where one uses the WebDriverWait class to wait for some event to signal that the test can proceed to the next step.
For example:
using (IWebDriver driver = new ChromeDriver())
{
driver.Navigate().GoToUrl($"{Properties.Settings.Default.ApplicationUrl}/Dashboard");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(d => d.Title.StartsWith("dashboard", StringComparison.OrdinalIgnoreCase));
//Can do next thing....
Assert.IsTrue(true);
}
However things get a bit more complicated if we want to test a React application. Given that each component in your React application will each individually invoke their own lifecycle events, in their own time, it is extremely difficult to know when a given control has completed its Render() lifecycle.
I have played with a simple solution like:
//Add something like this to each React component
componentDidMount() {
if (!window.document.hasMyComponentMounted)
window.document.hasMyComponentMounted = true;
}
and then I can do something like this in my tests:
using (IWebDriver driver = new ChromeDriver())
{
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
driver.Navigate().GoToUrl($"{Properties.Settings.Default.ApplicationUrl}/Dashboard");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(d => js.ExecuteScript("return window.document.hasHomeMounted"));
//Can do next thing....
Assert.IsTrue(true);
}
but this seems like a kinda sucky solution.
Has anyone encountered the same problem or know of potential workarounds (except for just using Thread.Sleep())?

I have written extensive automation for a React.js application, and as you mentioned, it was a very difficult problem waiting for components to render.
There is no concrete solution that can fix every issue with this. The solution you mentioned, to wait for components to mount, is a good one. You can make it cleaner by wrapping it in with WebDriver calls so that they automatically wait for the components to mount, and you do not have to specify that code in your test cases.
Here's an idea:
public static class WebDriverExtensions()
{
public static void WaitForComponents(this IWebDriver driver)
{
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(d => js.ExecuteScript("return window.document.hasHomeMounted"));
}
public static void GoToUrl(this IWebDriver driver, string url)
{
driver.Navigate().GoToUrl(url);
driver.WaitForComponents();
}
}
Then, your test case will look like:
using WebDriverExtensions;
using (IWebDriver driver = new ChromeDriver())
{
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
driver.GoToUrl($"{Properties.Settings.Default.ApplicationUrl}/Dashboard");
//Can do next thing....
Assert.IsTrue(true);
}
If the component you are checking is constantly changing, i.e. hasHomeMounted only works on the Home page, the solution is a little uglier. You could try something that takes a component name as a parameter to check if it is mounted:
public static void WaitForComponent(this IWebDriver driver, string componentName)
{
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(d => js.ExecuteScript($"return window.document.has{componentName}Mounted"));
}
As I mentioned, there is no catch-all way to accomplish this, and using Thread.Sleep() everywhere is not advisable. Your solution which checks the react component mounting is a good solution.
If you do not like the above solution, here's what I specifically used to handle most of my React UI testing:
public static void WaitAndClick(this IWebDriver driver, By by)
{
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(ExpectedConditions.ElementToBeClickable(by));
driver.FindElement(by).Click();
}
My issues with automating React UI involved elements taking too long to load. Most of the page would load, but the component I was trying to click would not exist sometimes. So I wrapped Click() in a WaitAndClick() method, to ensure the element is clickable before I try to perform the click. This worked for most of my cases.

Selenium WebDriver times out at 60 seconds no matter the interval wait time i set

So I'm running automated UI tests with Selenium Web Driver (using PhantomJS as a headless browser) on a web application. When I get to a certain point in my test -- a page where the web app takes over a minute to load the next page (up to 3 minutes some times) -- It will fail.
I get below error message:
Result Message:
Test method UI_Tests.UT_GI_Template.Delivery_UI threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote
WebDriver server for URL
http://localhost:45539/session/94ef38f0-a528-11e7-a7fd-69a0e29e333f/element/:wdc:1506697956275/value
timed out after 60 seconds. ---> System.Net.WebException: The request
was aborted: The operation has timed out.
I have set the wait time interval to 300 seconds yet it always times out after 60 seconds. Has anyone experienced this error? Here is my code:
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium.PhantomJS;
namespace UI_Tests
{
[TestClass]
public class UT_GI_Template {
private RemoteWebDriver wd;
[TestMethod]
public void Delivery_UI()
{
IWebDriver wd = new PhantomJSDriver();
try
{
WebDriverWait wait = new WebDriverWait(wd, new TimeSpan(0, 0, 300));
wd.Navigate().GoToUrl("example.com/QA");
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Click();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Clear();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).SendKeys("coffutt#icom.com");
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Click();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Clear();
....
Test will always time out on step below -- Enter on next button triggers a new page to load (which takes up to 3 minutes) -- I have checked and the rest of my UI tests in the suite run fine with below step commented out.
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_BodyContent_NextButton")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_BodyContent_NextButton")).SendKeys(Keys.Enter);
wait.Until(d => (d.FindElements(By.XPath("//*[#class='grid8 alpha omega']/h1")).Count != 0)); //
string madeIt5 = wd.FindElement(By.XPath("//*[#class='grid8 alpha omega']/h1")).Text;
Is there something I don't know about time intervals or that I am doing wrong? I have included no ImplicitWaits anywhere else in the code and have not included any System.Sleep(s). Why do my tests always timeout after 60 seconds when i have the interval time set to 300 seconds with WebDriverWait wait = new WebDriverWait(wd, new TimeSpan(0, 0, 300));? Any help would be greatly appreciated, thanks!!

After a lot of investigation (and about a million google searches) I have figured out and fixed the problem affecting my automated Selenium tests. The 60 second HTTP timeout error was coming from the Selenium RemoteWebDriver's default settings, this line in the source code >
https://github.com/SeleniumHQ/selenium/blob/9ca04253b7ed337852d50d02c72cb44a13169b71/dotnet/src/webdriver/Remote/RemoteWebDriver.cs#L69
protected static readonly TimeSpan DefaultCommandTimeout = TimeSpan.FromSeconds(60);
The only way to change this value of 60 seconds is to do so when you instantiate your webdriver, in my case I am using PhantomJS and the code looked like this >
var timeOutTime = TimeSpan.FromMinutes(4);
var options = new PhantomJSOptions();
options.AddAdditionalCapability("phantomjs.page.settings.resourceTimeout", timeOutTime);
RemoteWebDriver wd = new PhantomJSDriver("C:\\Users\\MyName\\Source\\Workspaces\\ICompany\\Application\\packages\\PhantomJS.2.0.0\\tools\\phantomjs\\", options, timeOutTime);
My old tests were timing out because of an ASP.net __doPostBack button that would take longer than 60 seconds to fetch data from my database when I selected a lot of arguments/items and therefore would trigger the default command timeout for HTTP requests seen in the WebDriver source code. I know this was the problem because the test would never time out on the postback with under 3 arguments/items selected.
Hope this solution can help someone else, cheers!

update 24-10-2022:
webBrowser = new ChromeDriver(chromeDriverService, chromeOptions, TimeSpan.FromMinutes(5));

Selenium and PhantomJS takes 30 seconds to open each link

I am trying to open a website and grab some data using Selenium with PhantomJS, however it takes a lot of time to open a website (about 30 second). And every time I open other link I have to wait 30+Seconds. What is wrong with my code ?
static void Main(string[] args)
{
IWebDriver browser = new PhantomJSDriver();
var URL = "http://www.cbssports.com/nba/playerrankings ";
browser.Navigate().GoToUrl(URL);
//Position
var title = browser.FindElements(By.CssSelector(".tableTitle"));
Console.WriteLine(title.First().Text);
Console.Read();
}
Things I have tried to do:
1.Set PhantomJS proxy type to none
2.Disable internet option: automatically detect settings
3.Disable IPv6 protocol
PhantomJS release notes claim, that there are some known issues with network performance on Microsoft Windows. According to release notes, solution is to set proxy type to none, however that doesn't work.

You have to wait 30 seconds because you haven't defined timeouts which are 30 seconds as default. You should use this predefined driver service.
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver browser = new PhantomJSDriver(phantomJSDriverService);
browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(0));

The default timeout for Selenium is 30 seconds. You are using browser.FindElements() (the plural version), which will wait the full 30 seconds before continuing!
You can reduce the timeout with browser.manage().timeouts().implicitlyWait(), or you can use explicit timeouts.