Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.
By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.
Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.
A typical C# code to do that can be summarized as follows:
namespace WebAcceptanceTests
{
public static class Chrome
{
public static async Task Run(
Uri uri,
Func<ChromeDriver, Task> manipulate,
Action<ChromeDriver> validate)
{
var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
// To make Chrome window invisible.
options.AddArgument("--headless");
using var driver = new ChromeDriver(chromeDriverService, options);
try
{
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl(uri);
await manipulate(driver);
validate(driver);
}
finally
{
driver.Quit();
}
}
}
}
where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:
public static async Task Manipulate(ChromeDriver driver)
{
// Do some useful stuff here instead of just waiting.
await Task.Delay(60_000);
}
However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.
The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.
Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.
Thanks.
We achieve this using NUnit parallel run, parakkekuzabke by fixture.
Allocate driver during OneTimeSetup. Do whatever test need in single fixture.
On OneTimeTearDown, dispose driver.
We do this in base class that all web acceptance test fixtures are inherit
[Parallelizable(ParallelScope.Fixtures)]
public abstract class WebDriverTest
{
protected IDriver driver;
[OneTimeSetup]
public void PrepareDriver()
{
// ...
this.driver = new ChromeDriver(chromeDriverService, options);
// ...
}
[OneTimeTearDown]
public void CleanupDriver()
{
this.driver.Dispose();
}
[TearDown]
public void ScreenshotForFailedTest()
{
var testStatus = GetTestStatus();
if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
{
this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
// log more details if needed
}
}
}
[OneTimeTearDown] is executed even if there is failed tests
As bonus we take screen
Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.
Related
I have an application which is used to trigger long running data queues. By long running, I mean around 12-16 hours per queue and either of them cannot be executed in parallel. Each queue has individual steps which need to succeed before the next one runs.
I have already increased the timeouts while initializing ChromeDriver upto 1000 minutes
webDriver == new ChromeDriver(path,options,TimeSpan.FromMinutes(1000));
I am using WebDriverWait for checking after 1000 mins that all steps have been succeeded. In case of a failure, I still have to wait for 1000 minutes before I can tell the dev team about the failure.
Is there a better approach to solve this problem? It is also keeping my browser open for 1000 mins
Regarding your question -- is there a better way to solve this problem? With Selenium, not really. You'd have better luck taking a different approach, such as API, than through UI testing. However, it's still possible, just not ideal.
My best idea for this problem would be to set up some sort of controller that can manage your WebDriver instances and also keep track of the 12-16 hour queue time. Since I don't have any specific information about your project architecture or the queues you are testing, this will be a very generic implementation.
Here's a simple DriverManager class, that controls creating & terminating WebDriver sessions:
public class DriverManager
{
public IWebDriver CreateDriver
{
// code to initialize your WebDriver instance here
}
public void CloseWebDriverSession
{
Driver.Close();
Driver.Quit();
}
}
Next, here's a test case implementation that utilizes DriverManager to close & reopen WebDriver as needed.
public class TestBothQueues
{
// this driver instance will keep track of your session throughout the test case
public IWebDriver driver;
[Test]
public void ShouldRunBothQueues
{
// declare instance of DriverManager class
DriverManager manager = new DriverManager();
// start a webdriver instance
driver = manager.CreateDriver();
// run the first queue
RunFirstQueue();
// terminate the WebDriver so we don't have browser open for 12 hours
manager.CloseWebDriverSession();
// wait 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// start another WebDriver session to start the second queue
driver = manager.CreateDriver();
// run the second queue
RunSecondQueue();
// terminate when we are finished
manager.CloseWebDriverSession();
}
}
A few notes on this:
You can also convert this code into a while loop if you would like to start a WebDriver instance to check the queue on a time interval. For example, if the queue takes 12-16 hours to finish, you may want to wait 12 hours, then check the queue once per hour until you can verify it is completed. That would look something like this:
// first, wait initial 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// keep track of whether or not queue is finished
bool isQueueFinished = false;
while (!isQueueFinished);
{
// start webdriver instance to check the queue
IWebDriver driver = manager.CreateDriver();
// check if queue is finished
isQueueFinished = CheckIfQueueFinished(driver);
// if queue is finished, while loop will break
// if queue is not finished, close the WebDriver instance, and start again
if (!isQueueFinished)
{
// close the WebDriver since we won't need it
manager.CloseWebDriverSession();
// wait another hour
Thread.Sleep(TimeSpan.FromHours(1));
}
}
Hope this helps.
Hoping to find my answer here I have spent the better half of a week trying to figure this out myself and I cannot fix my issue. A little background, I am new to C# and NUnit tests with webdriver, I am attempting to create a regression suit for the company I work for. We have a few products that are heavily integrated with ebay and so for part of the test I need to click on a button which takes me to the ebay login page and then I need to login. Seems simple enough (or so I thought). The issue I am running into is when hitting this ebay login page FF times out each time. I currently have my test setup to run in multiple browsers, chrome and IE are passing (they also hang a little) but FF has never passed.
This is odd to me because earlier in this test I go to ebay and sucessfully login. But when I have to link my company account to the ebay account this login page takes forever. I know we pass some unique token to ebay to link the accounts which is what I think causes the long load times.
So the failure with FF is always the same, timed out after 60 seconds. I have read other questions that seemed to be a similar issue (Selenium WebDriver throws Timeout exceptions sporadically) there are a few solutions the one I am interested in is setting the driver timeout to something greater than 60 seconds. I do not know how to do this with multiple browser setup I have going.
[TestFixture(typeof(FirefoxDriver))]
[TestFixture(typeof(ChromeDriver))]
[TestFixture(typeof(InternetExplorerDriver))]
public class UnitTest1<TWebDriver> where TWebDriver: IWebDriver, new()
{
PTGeneral General;
[TestFixtureSetUp]
public void SetUp()
{
General = new PTGeneral();
General.Driver = new TWebDriver();
}
I would really like to keep this setup as I like how I can test IE, FF, and Chrome, but I do not know how to implement a solution like this then.
new FirefoxDriver("FfBinaryPath", FfProfileInstance, TimeSpan.FromSeconds(180));
Any help would be greatly appreciated, if you would like more information ill be happy to provide, assuming I understand what you are asking ;)
Thank you all for even reading, this community is amazing.
Still struggling with this one. Here is the error message if that helps.
------ Run test started ------
NUnit VS Adapter 2.0.0.0 executing tests is started
Loading tests from C:\Users\jburns\documents\visual studio 2013\Projects\PTTest\PTTest\bin\Debug\PTTest.dll
Run started: C:\Users\jburns\documents\visual studio 2013\Projects\PTTest\PTTest\bin\Debug\PTTest.dll
TearDown failed for test fixture PTTest.UnitTest1<ChromeDriver>
The HTTP request to the remote WebDriver server for URL http://localhost:64706/session/0186901c828d3a1ad7523ecd41dedf9a/element timed out after 60 seconds.
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
at OpenQA.Selenium.Remote.DriverServiceCommandExecutor.Execute(Command commandToExecute)
at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementByXPath(String xpath)
at OpenQA.Selenium.By.<>c__DisplayClasse.<XPath>b__c(ISearchContext context)
at OpenQA.Selenium.By.FindElement(ISearchContext context)
at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
at PTTest.PTGeneral.IsElementPresent(By by) in c:\Users\jburns\Documents\Visual Studio 2013\Projects\PTTest\PTTest\PTGeneral.cs:line 42
at PTTest.PTGeneral.EmailCleanUP() in c:\Users\jburns\Documents\Visual Studio 2013\Projects\PTTest\PTTest\PTGeneral.cs:line 105
at PTTest.UnitTest1`1.TearDown() in c:\Users\jburns\Documents\Visual Studio 2013\Projects\PTTest\PTTest\UnitTest1.cs:line 29
TearDown failed for test fixture PTTest.UnitTest1<FirefoxDriver>
So I stripped out the multiple browser piece I had in the [TestFixture] and setup so that I could test only against FF and increased the driver timeout to 3m with
General.Driver = new FirefoxDriver(new FirefoxBinary(), new FirefoxProfile(), TimeSpan.FromSeconds(180));
This worked and made my tests pass. But I still need a solution that works when I run against multiple browsers, as I don't want to maintain 2 different projects when there should be a way to
Your problem is that you have to wait till a page is loading.
Then the solution will be to use waiting methods each time when a new page is being opening.
Put this method after navigating to each new page:
public void WaitForPageLoading(int secondsToWait = 600)
{
Stopwatch sw = new Stopwatch();
sw.Start();
try
{
while (sw.Elapsed.TotalSeconds < secondsToWait)
{
var pageIsReady = (bool)((IJavaScriptExecutor)Driver).ExecuteScript("return document.readyState == 'complete'");
if (pageIsReady)
break;
Thread.Sleep(100);
}
}
catch (Exception)
{
Driver.Dispose();
throw new TimeoutException("Page loading time out time has passed " + secondsToWait + " seconds");
}
finally
{
sw.Stop();
}
}
I am testing gmail login page in CodedUI testing and completed recording all actions.
Now i want to first launch login page of google page and i have implemented code as shown below.
BrowserWindow.CurrentBrowser = "IE";
this.UIMap.UIAdminloginMozillaFirWindow.LaunchUrl(new Uri("https://www.google.com"));
But Error is:
You can use the Selenium components for Coded UI Cross Browser Testing (https://visualstudiogallery.msdn.microsoft.com/11cfc881-f8c9-4f96-b303-a2780156628d), a set of extensions that translate Coded UI calls to WebDriver calls, and hence enabling support for Firefox and Chrome.
Download the installer and run it. If you build your tests using the test recorder, record as usual with Internet Explorer (recording does not work in Firefox or Chrome). In your code, before calling BrowserWindow.Launch("url"), set the browser type as follows:
BrowserWindow.CurrentBrowser = "Firefox"; // or "Chrome" or "IE"
Use almost all of the normal properties and methods of HtmlControl and its descendants. I know from experience that accessing HtmlControl.ControlDefinition will throw a NotSupportedException, and Mouse.StartDragging()/StopDragging() also does not work. Debugging can sometimes be interesting as well.
Add in Selenium for testing in Firefox.
[TestClass]
public class UnitTest1
{
FirefoxDriver firefox;
// This is the test to be carried out.
[TestMethod]
public void TestMethod1()
{
firefox = new FirefoxDriver();
firefox.Navigate().GoToUrl("http://www.google.com/");
IWebElement element = firefox.FindElement(By.Id("lst-ib"));
element.SendKeys("Google\n");
}
// This closes the driver down after the test has finished.
[TestCleanup]
public void TearDown()
{
firefox.Quit();
}
}
This issue began when I switched from testing on the www website to my localhost version of it. Working in VS 2012, I will begin debugging so the localhost is active, detach the process so I can test on it, then run any test I like. For a very basic example:
[Test]
public void CanGoToHomePage()
{
Pages.HomePage.Goto();
Assert.IsTrue(Pages.HomePage.IsAt());
}
And the functions it references are here:
public class HomePage
{
const string Url = "http://localhost:3738";
const string HomepageTitle = "FunnelFire - Home Page";
public void Goto()
{
Browser.Goto(Url);
}
public bool IsAt()
{
return Browser.Title == HomepageTitle;
}
}
And the actual selenium code here:
public static class Browser
{
static IWebDriver webDriver = new FirefoxDriver();
public static void Goto(string url)
{
webDriver.Manage().Window.Maximize();
webDriver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));
webDriver.Url = url;
}
}
Now the issue. The 10 second implicit wait that I added in Browser does successfully wait at most 10 seconds after loading a page to see if it can locate whatever information I want it to find, that is not the problem.
As I said earlier, after I switched to testing on localhost, suddenly I ran into a strange issue where a page would begin to load (i.e. screen still totally white, nothing finished) or even sometimes the next page would JUST barely finish loading and suddenly the test would just up and fail, pointing to the Assert of IsAt returning false even though the page it was loading was the correct one. I could run that test immediately once more and it would pass without a problem. Run it a third time and it could randomly fail again. I'm honestly not sure what is causing the issue and any help would be appreciated!
Implicit waits work only for finding elements. For waiting on the title of the page to be a certain value, you'll want to use an explicit wait. You can write your own version of this pattern, but in the .NET bindings, the WebDriver.Support.dll assembly has a WebDriverWait class to help with this. Its use would look something like this:
// WARNING! Untested code written from memory below. It has not
// been tested or even compiled in an IDE, so may be syntactically
// incorrect. The concept, however, should still be valid.
public void WaitForTitle(IWebDriver driver, string title, TimeSpan timeout)
{
WebDriverWait wait = new WebDriverWait(driver, timeout);
wait.Until((d) => { return d.Title == title; });
}
You could even modify your IsAt method to use this pattern, catching the WebDriverTimeoutException and returning false if the wait function times out.
I have written a number of tests for an MVC application using Selenium and the webdriver. These work on my dev machine without problem. We are using the PageFactory design. Currently I have an explicit wait of 3 seconds (although I have tested up to 10 seconds without change) on page load.
A relatively simple test is below:
The Test
[TestMethod]
public void Can_Log_In_With_Valid_Credential()
{
Pages.LoginPage.Goto();
var success = Pages.LoginPage.Login(Properties.Settings.Default.UserName,
Properties.Settings.Default.Password);
Assert.IsTrue(success);
Pages.HeaderPage.LogOut();
}
The Page and Related Logic
public static class Pages
{
public static LoginPage LoginPage
{
get
{
var loginPage = new LoginPage();
PageFactory.InitElements(Browser.Driver, loginPage);
return loginPage;
}
}
}
public class LoginPage : Page
{
public static string Url = Properties.Settings.Default.DomainAddress + "/Account/Logon";
public static string PageTitle = "Log On";
[FindsBy(How = How.Id, Using = "UserName")]
private IWebElement _userNameBox;
[FindsBy(How = How.Id, Using = "Password")]
private IWebElement _passwordBox;
[FindsBy(How = How.ClassName, Using = "validation-summary-errors")]
private IWebElement _validationErrors;
[FindsBy(How = How.CssSelector, Using = "div#LoginSubmit.signin input")]
private IWebElement _submitButton;
public void Goto()
{
Browser.Goto(Url);
}
public bool IsAt()
{
return Browser.Title == PageTitle;
}
public bool Login(string username, string password)
{
try
{
_userNameBox.SendKeys(username);
}
catch (Exception)
{
_userNameBox = ((IWebDriver) Browser.Driver).FindElement(By.Id("UserName"), 10);
_userNameBox.SendKeys(username);
}
_passwordBox.SendKeys(password);
_submitButton.Click();
return Browser.CurrentUrl != Properties.Settings.Default.DomainAddress + "/Account/Logon";
}
}
The problem arises on the build server. All of the tests fail with the following:
Class Initialization method
KepsPortalMvc.UserAccountManagementTests.Initialize threw exception.
OpenQA.Selenium.WebDriverTimeoutException:
OpenQA.Selenium.WebDriverTimeoutException: Timed out after 10 seconds
---> OpenQA.Selenium.NoSuchElementException: Unable to locate element: {"method":"id","selector":"UserName"}.
It appears that Firefox is not even loading the page. I have verified (via Task Manager) that both the webdriver and Firefox are being started by the BuildServiceAccount, however I have not yet found a way of monitoring the network activity (Fiddler is not showing me any HTTP traffic related to that Firefox instance at the very least).
Is there a way (short of putting something like Wireshark on the build server) to let me monitor the WebDriver-Firefox? It doesn't pop-up as it is run as a different account (I am assuming this is why).
Any help would be greatly appreciated.
Clarification
This is working on my Dev machine, but not on the Team Foundation Build server. We are running Team Foundation on a single server. The TFBuild Service account is what is currently running the Webdriver and Firefox. A process entry for Firefox appears in the Task Manager when it runs as does an entry for WebDriver. A WINDOW for Firefox does NOT. Fiddler does not show me any traffic for that Firefox instance (but I am unsure if that is because nothing is being loaded at all or it just doesn't show me traffic that is loaded in another users session).
A bit of an educated guess here, but it might be that your Test Environment is not set for UI tests. Look at this article on MSDN to see how it should be done.
Secondly, from my experience such a behavior might occur if the build agent runs a different version of msbuild. Make sure they are the same. For example building with VS 2010 and VS 2012 will use different assemblies, and thus provoking different results at runtime.
OpenQA.Selenium.NoSuchElementException
I think this error shows no Element is fpund. It can be caused, that FF lost focus in current window?
How many window have you opened ? or maybe after opening the URL adds either sleep or implicite wait until element isVisible ?
If I understand you right, you use Selenium to run UI tests on your local dev machine and want them to be run no TFS build machine.
This is not working by default, because the build process is a service without a UI and while it is running in the TFS Build service "session", you will not be able to see it. Therefore you need to setup TFS build to run as interactive process (http://msdn.microsoft.com/en-us/library/hh691189.aspx#agent_test), but this would mean that the server is always blocked for it. If it is a hardware machine somewhere, it is possible to turn on the monitor and use it with the build credentials, because it is not locked.
I did something similiar with MS Coded UI Tests and to see if it is working I used my account as build account, so I could open a RDP session to the server, which was the same like for the build. So I could see what was happening during the test, but closing the RDP ended up in failing the test. That's why I always waited for the tests to be finished and than restarted the machine.