We Keep Coding

NET5 - How to run numerous web acceptance tests on a single machine

Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.
By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.
Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.
A typical C# code to do that can be summarized as follows:
namespace WebAcceptanceTests
{
public static class Chrome
{
public static async Task Run(
Uri uri,
Func<ChromeDriver, Task> manipulate,
Action<ChromeDriver> validate)
{
var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
// To make Chrome window invisible.
options.AddArgument("--headless");
using var driver = new ChromeDriver(chromeDriverService, options);
try
{
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl(uri);
await manipulate(driver);
validate(driver);
}
finally
{
driver.Quit();
}
}
}
}
where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:
public static async Task Manipulate(ChromeDriver driver)
{
// Do some useful stuff here instead of just waiting.
await Task.Delay(60_000);
}
However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.
The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.
Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.
Thanks.

We achieve this using NUnit parallel run, parakkekuzabke by fixture.
Allocate driver during OneTimeSetup. Do whatever test need in single fixture.
On OneTimeTearDown, dispose driver.
We do this in base class that all web acceptance test fixtures are inherit
[Parallelizable(ParallelScope.Fixtures)]
public abstract class WebDriverTest
{
protected IDriver driver;
[OneTimeSetup]
public void PrepareDriver()
{
// ...
this.driver = new ChromeDriver(chromeDriverService, options);
// ...
}
[OneTimeTearDown]
public void CleanupDriver()
{
this.driver.Dispose();
}
[TearDown]
public void ScreenshotForFailedTest()
{
var testStatus = GetTestStatus();
if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
{
this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
// log more details if needed
}
}
}
[OneTimeTearDown] is executed even if there is failed tests
As bonus we take screen
Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.

Unable to connect to the remote server - due to Driver.Close() and Driver.Quit(), but how to fix?

I've been using Selenium and NUnit to do some automated testing, and up until now everything has been fine. The change I made recently was adding more than one test to a test class.
I'm pretty certain the issue lies with the code in my "Teardown" function in the test class. When I comment out
BrowserFactory.CloseAllDrivers();
Things run just fine.
This is the code for my "FrontEndAddItemToCartTest":
class FrontEndAddItemToCartTest : PageTest
{
[SetUp]
public void Initialize()
{
SetBrowser(BrowserFactory.BrowserType.Chrome); // Not headless
SetServer("testUrlNotGivenForSecurityPurposes");
StartTest(TestType.FrontEnd);
SetSize(MobileSize.XXLarge);
}
[Test]
public void StandardQuantityTest()
{
OrderItem standardQuantity = new OrderItem(new Product("500", ".25"), 500);
FrontEndActions.AddItemToCart(standardQuantity);
}
[Test]
public void CustomQuantityTest()
{
OrderItem customQuantity = new OrderItem(new Product("482", ".25"), 225);
FrontEndActions.AddItemToCart(customQuantity);
}
[TearDown]
public void EndTest()
{
BrowserFactory.CloseAllDrivers();
}
}
This is the error I get:
Message: OpenQA.Selenium.WebDriverException : Unexpected error. System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
with a bunch of other stuff that I don't believe is relevant.
That being said, I have code in "SetBrowser(...)" that initializes a
new ChromeDriver();
but that doesn't seem to be enough.
The methods at the top are there to avoid having to do too much Selenium-y looking stuff in each of the tests - to make things more maintainable by people other than just myself - but it's what you would expect from typical Driver setup. I'm not sure why the second test is what causes the issue, but since things work fine w/o the CloseAllDrivers() being run, I'm imagining it's that.

By the looks of the attributes, you're using MSTest? Is it executing tests in parallel?
I guess regardless, for good test isolation you would want to change the CloseAllDrivers method to only dispose the driver used in the test it's cleaning up. I'd recommend starting there and see if that has the same problem.
Also, is CloseAllDrivers calling driver.Quit() or driver.Dispose()? Either of those are the ones you want to use.

OK so the issue was a combination of things:
1. I had Drivers[driver].Close(); and Drivers[driver].Quit(); -- only having .Quit() resolved one issue. Not really sure why Close() was causing problems, to be quite honest. The other (window not properly closing in the end) was due to the following code in my BrowserFactory:
if (driver == null)
{
Driver = new ChromeDriver((ChromeOptions)options); // options created elsewhere
Drivers.Add("Chrome", Driver); // This adds the driver to the list of Drivers currently up.
}
else
{
Driver = new ChromeDriver((ChromeOptions)options); // same as before
Drivers["Chrome"] = Driver; // **this** wasn't here before. This was the issue. Essentially, I was calling ```Quit()``` on the first instance of the driver, not on the fresh one created by the second test.
}
Thanks for all the help, guys. A combination of me determined to figure this out and your responses got me to the solution :)

How to Launch URL in CodedUI testing in Mozilla Firefox 36.0.4

I am testing gmail login page in CodedUI testing and completed recording all actions.
Now i want to first launch login page of google page and i have implemented code as shown below.
BrowserWindow.CurrentBrowser = "IE";
this.UIMap.UIAdminloginMozillaFirWindow.LaunchUrl(new Uri("https://www.google.com"));
But Error is:

You can use the Selenium components for Coded UI Cross Browser Testing (https://visualstudiogallery.msdn.microsoft.com/11cfc881-f8c9-4f96-b303-a2780156628d), a set of extensions that translate Coded UI calls to WebDriver calls, and hence enabling support for Firefox and Chrome.
Download the installer and run it. If you build your tests using the test recorder, record as usual with Internet Explorer (recording does not work in Firefox or Chrome). In your code, before calling BrowserWindow.Launch("url"), set the browser type as follows:
BrowserWindow.CurrentBrowser = "Firefox"; // or "Chrome" or "IE"
Use almost all of the normal properties and methods of HtmlControl and its descendants. I know from experience that accessing HtmlControl.ControlDefinition will throw a NotSupportedException, and Mouse.StartDragging()/StopDragging() also does not work. Debugging can sometimes be interesting as well.

Add in Selenium for testing in Firefox.
[TestClass]
public class UnitTest1
{
FirefoxDriver firefox;
// This is the test to be carried out.
[TestMethod]
public void TestMethod1()
{
firefox = new FirefoxDriver();
firefox.Navigate().GoToUrl("http://www.google.com/");
IWebElement element = firefox.FindElement(By.Id("lst-ib"));
element.SendKeys("Google\n");
}
// This closes the driver down after the test has finished.
[TestCleanup]
public void TearDown()
{
firefox.Quit();
}
}

C# Webdriver - Page Title assert fails before page loads

This issue began when I switched from testing on the www website to my localhost version of it. Working in VS 2012, I will begin debugging so the localhost is active, detach the process so I can test on it, then run any test I like. For a very basic example:
[Test]
public void CanGoToHomePage()
{
Pages.HomePage.Goto();
Assert.IsTrue(Pages.HomePage.IsAt());
}
And the functions it references are here:
public class HomePage
{
const string Url = "http://localhost:3738";
const string HomepageTitle = "FunnelFire - Home Page";
public void Goto()
{
Browser.Goto(Url);
}
public bool IsAt()
{
return Browser.Title == HomepageTitle;
}
}
And the actual selenium code here:
public static class Browser
{
static IWebDriver webDriver = new FirefoxDriver();
public static void Goto(string url)
{
webDriver.Manage().Window.Maximize();
webDriver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));
webDriver.Url = url;
}
}
Now the issue. The 10 second implicit wait that I added in Browser does successfully wait at most 10 seconds after loading a page to see if it can locate whatever information I want it to find, that is not the problem.
As I said earlier, after I switched to testing on localhost, suddenly I ran into a strange issue where a page would begin to load (i.e. screen still totally white, nothing finished) or even sometimes the next page would JUST barely finish loading and suddenly the test would just up and fail, pointing to the Assert of IsAt returning false even though the page it was loading was the correct one. I could run that test immediately once more and it would pass without a problem. Run it a third time and it could randomly fail again. I'm honestly not sure what is causing the issue and any help would be appreciated!

Implicit waits work only for finding elements. For waiting on the title of the page to be a certain value, you'll want to use an explicit wait. You can write your own version of this pattern, but in the .NET bindings, the WebDriver.Support.dll assembly has a WebDriverWait class to help with this. Its use would look something like this:
// WARNING! Untested code written from memory below. It has not
// been tested or even compiled in an IDE, so may be syntactically
// incorrect. The concept, however, should still be valid.
public void WaitForTitle(IWebDriver driver, string title, TimeSpan timeout)
{
WebDriverWait wait = new WebDriverWait(driver, timeout);
wait.Until((d) => { return d.Title == title; });
}
You could even modify your IsAt method to use this pattern, catching the WebDriverTimeoutException and returning false if the wait function times out.

How to use ApprovalTests on Teamcity?

I am using Approval Tests. On my dev machine I am happy with DiffReporter that starts TortoiseDiff when my test results differ from approved:
[UseReporter(typeof (DiffReporter))]
public class MyApprovalTests
{ ... }
However when the same tests are running on Teamcity and results are different tests fail with the following error:
System.Exception : Unable to launch: tortoisemerge.exe with arguments ...
Error Message: The system cannot find the file specified
---- System.ComponentModel.Win32Exception : The system cannot find the file
specified
Obviously it cannot find tortoisemerge.exe and that is fine because it is not installed on build agent. But what if it gets installed? Then for each fail another instance of tortoisemerge.exe will start and nobody will close it. Eventually tons of tortoisemerge.exe instances will kill our servers :)
So the question is -- how tests should be decorated to run Tortoise Diff on local machine
and just report errors on build server? I am aware of #IF DEBUG [UseReporter(typeof (DiffReporter))] but would prefer another solution if possible.

There are a couple of solutions to the question of Reporters and CI. I will list them all, then point to a better solution, which is not quite enabled yet.
Use the AppConfigReporter. This allows you to set the reporter in your AppConfig, and you can use the QuietReporter for CI.
There is a video here, along with many other reporters. The AppConfigReporter appears at 6:00.
This has the advantage of separate configs, and you can decorate at the assembly level, but has the disadvantage of if you override at the class/method level, you still have the issue.
Create your own (2) reporters. It is worth noting that if you use a reporter, it will get called, regardless as to if it is working in the environment. IEnvironmentAwareReporter allows for composite reporters, but will not prevent a direct call to the reporter.
Most likely you will need 2 reporters, one which does nothing (like a quiet reporter) but only works on your CI server, or when called by TeamCity. Will call it the TeamCity Reporter. And One, which is a multiReporter which Calls teamCity if it is working, otherwise defers to .
Use a FrontLoadedReporter (not quite ready). This is how ApprovalTests currently uses NCrunch. It does the above method in front of whatever is loaded in your UseReporter attribute. I have been meaning to add an assembly level attribute for configuring this, but haven't yet (sorry) I will try to add this very soon.
Hope this helps.
Llewellyn

I recently came into this problem myself.
Borrowing from xunit and how they deal with TeamCity logging I came up with a TeamCity Reporter based on the NCrunch Reporter.
public class TeamCityReporter : IEnvironmentAwareReporter, IApprovalFailureReporter
{
public static readonly TeamCityReporter INSTANCE = new TeamCityReporter();
public void Report(string approved, string received) { }
public bool IsWorkingInThisEnvironment(string forFile)
{
return Environment.GetEnvironmentVariable("TEAMCITY_PROJECT_NAME") != null;
}
}
And so I could combine it with the NCrunch reporter:
public class TeamCityOrNCrunchReporter : FirstWorkingReporter
{
public static readonly TeamCityOrNCrunchReporter INSTANCE =
new TeamCityOrNCrunchReporter();
public TeamCityOrNCrunchReporter()
: base(NCrunchReporter.INSTANCE,
TeamCityReporter.INSTANCE) { }
}
[assembly: FrontLoadedReporter(typeof(TeamCityOrNCrunchReporter))]

I just came up with one small idea.
You can implement your own reporter, let's call it DebugReporter
public class DebugReporter<T> : IEnvironmentAwareReporter where T : IApprovalFailureReporter, new()
{
private readonly T _reporter;
public static readonly DebugReporter<T> INSTANCE = new DebugReporter<T>();
public DebugReporter()
{
_reporter = new T();
}
public void Report(string approved, string received)
{
if (IsWorkingInThisEnvironment())
{
_reporter.Report(approved, received);
}
}
public bool IsWorkingInThisEnvironment()
{
#if DEBUG
return true;
#else
return false;
#endif
}
}
Example of usage,
[UseReporter(typeof(DebugReporter<FileLauncherReporter>))]
public class SomeTests
{
[Test]
public void test()
{
Approvals.Verify("Hello");
}
}
If test is faling, it still would be red - but reporter would not came up.
The IEnvironmentAwareReporter is specially defined for that, but unfortunatelly whatever I return there, it still calls Report() method. So, I put the IsWorkingInThisEnvironment() call inside, which is a little hackish, but works :)
Hope that Llywelyn can explain why it acts like that. (bug?)

I'm using CC.NET and I do have TortoiseSVN installed on the server.
I reconfigured my build server to allow the CC.NET service to interact with the desktop. When I did that, TortiseMerge launched. So I think what's happening is that Approvals tries to launch the tool, but it cant because CC.NET is running as a service and the operating system prevents that behavior by default. If TeamCity runs as a service, you should be fine, but you might want to test.