We Keep Coding

NET5 - How to run numerous web acceptance tests on a single machine

Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.
By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.
Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.
A typical C# code to do that can be summarized as follows:
namespace WebAcceptanceTests
{
public static class Chrome
{
public static async Task Run(
Uri uri,
Func<ChromeDriver, Task> manipulate,
Action<ChromeDriver> validate)
{
var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
// To make Chrome window invisible.
options.AddArgument("--headless");
using var driver = new ChromeDriver(chromeDriverService, options);
try
{
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl(uri);
await manipulate(driver);
validate(driver);
}
finally
{
driver.Quit();
}
}
}
}
where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:
public static async Task Manipulate(ChromeDriver driver)
{
// Do some useful stuff here instead of just waiting.
await Task.Delay(60_000);
}
However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.
The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.
Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.
Thanks.

We achieve this using NUnit parallel run, parakkekuzabke by fixture.
Allocate driver during OneTimeSetup. Do whatever test need in single fixture.
On OneTimeTearDown, dispose driver.
We do this in base class that all web acceptance test fixtures are inherit
[Parallelizable(ParallelScope.Fixtures)]
public abstract class WebDriverTest
{
protected IDriver driver;
[OneTimeSetup]
public void PrepareDriver()
{
// ...
this.driver = new ChromeDriver(chromeDriverService, options);
// ...
}
[OneTimeTearDown]
public void CleanupDriver()
{
this.driver.Dispose();
}
[TearDown]
public void ScreenshotForFailedTest()
{
var testStatus = GetTestStatus();
if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
{
this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
// log more details if needed
}
}
}
[OneTimeTearDown] is executed even if there is failed tests
As bonus we take screen
Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.

Selenium WebDriver - Wait for long running queues

I have an application which is used to trigger long running data queues. By long running, I mean around 12-16 hours per queue and either of them cannot be executed in parallel. Each queue has individual steps which need to succeed before the next one runs.
I have already increased the timeouts while initializing ChromeDriver upto 1000 minutes
webDriver == new ChromeDriver(path,options,TimeSpan.FromMinutes(1000));
I am using WebDriverWait for checking after 1000 mins that all steps have been succeeded. In case of a failure, I still have to wait for 1000 minutes before I can tell the dev team about the failure.
Is there a better approach to solve this problem? It is also keeping my browser open for 1000 mins

Regarding your question -- is there a better way to solve this problem? With Selenium, not really. You'd have better luck taking a different approach, such as API, than through UI testing. However, it's still possible, just not ideal.
My best idea for this problem would be to set up some sort of controller that can manage your WebDriver instances and also keep track of the 12-16 hour queue time. Since I don't have any specific information about your project architecture or the queues you are testing, this will be a very generic implementation.
Here's a simple DriverManager class, that controls creating & terminating WebDriver sessions:
public class DriverManager
{
public IWebDriver CreateDriver
{
// code to initialize your WebDriver instance here
}
public void CloseWebDriverSession
{
Driver.Close();
Driver.Quit();
}
}
Next, here's a test case implementation that utilizes DriverManager to close & reopen WebDriver as needed.
public class TestBothQueues
{
// this driver instance will keep track of your session throughout the test case
public IWebDriver driver;
[Test]
public void ShouldRunBothQueues
{
// declare instance of DriverManager class
DriverManager manager = new DriverManager();
// start a webdriver instance
driver = manager.CreateDriver();
// run the first queue
RunFirstQueue();
// terminate the WebDriver so we don't have browser open for 12 hours
manager.CloseWebDriverSession();
// wait 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// start another WebDriver session to start the second queue
driver = manager.CreateDriver();
// run the second queue
RunSecondQueue();
// terminate when we are finished
manager.CloseWebDriverSession();
}
}
A few notes on this:
You can also convert this code into a while loop if you would like to start a WebDriver instance to check the queue on a time interval. For example, if the queue takes 12-16 hours to finish, you may want to wait 12 hours, then check the queue once per hour until you can verify it is completed. That would look something like this:
// first, wait initial 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// keep track of whether or not queue is finished
bool isQueueFinished = false;
while (!isQueueFinished);
{
// start webdriver instance to check the queue
IWebDriver driver = manager.CreateDriver();
// check if queue is finished
isQueueFinished = CheckIfQueueFinished(driver);
// if queue is finished, while loop will break
// if queue is not finished, close the WebDriver instance, and start again
if (!isQueueFinished)
{
// close the WebDriver since we won't need it
manager.CloseWebDriverSession();
// wait another hour
Thread.Sleep(TimeSpan.FromHours(1));
}
}
Hope this helps.

Task.Run occasionally fails silently when launched from MVC Controller

I am attempting to generate PDF copies of specific forms within my MVC application. As this is time consuming, and the client does not need to wait for this generation to happen, I'm trying to trigger this as a series of Fire and Forget Tasks.
One hang-up of note is that I need to have the HttpContext established, or some underlying pieces of the code that I can't alter won't work. I believe I have dealt with this problem, but I wanted to call it out in case it matters.
Here is the function I am calling...
private void AsyncPDFFormGeneration(string htmlOutput, string serverRelativePath, string serverURL, string signature, ScannedDocument document, HttpContext httpContext)
{
try
{
System.Web.HttpContext.Current = httpContext;
using (StreamWriter stw = new StreamWriter(Server.MapPath(serverRelativePath), false, System.Text.Encoding.Default))
{
stw.Write(htmlOutput);
}
Doc ABCDoc = new Doc();
ABCDoc.HtmlOptions.Engine = EngineType.Gecko;
int DocID = 0;
DocID = ABCDoc.AddImageUrl(serverURL + serverRelativePath + "?dumb=" + DateTime.Now.Hour.ToString() + DateTime.Now.Minute.ToString() + DateTime.Now.Second + DateTime.Now.Millisecond);
while (true)
{
ABCDoc.FrameRect();
if (!ABCDoc.Chainable(DocID))
break;
ABCDoc.TextStyle.LeftMargin = 100;
ABCDoc.Page = ABCDoc.AddPage();
DocID = ABCDoc.AddImageToChain(DocID);
}//End while (true...
for (int i = 1; i <= ABCDoc.PageCount; i++)
{
ABCDoc.PageNumber = i;
ABCDoc.Flatten();
}
ScannedDocuments.AddScannedDocument(document, ABCDoc.GetData());
System.IO.File.Delete(Server.MapPath(serverRelativePath));
}
catch (Exception e)
{
//Exception is logged to the database, and if that fails, to the Event Log
}
}
Within, I am writing the String output of the HTML contents of the MVC Form in question to an html file, handing the path to that file to the PDF writer, generating the PDF, and then deleting the html file.
I'm calling it inside of a Controller POST method, like so:
Task.Run(() => AsyncPDFFormGeneration(htmlOutput, serverRelativePath,
serverURL, signature, document, HttpContext.ApplicationInstance.Context));
This command is called as part of a foreach loop that constructs the forms, loads them into string format, and then passes them into a task. I've also tried this with
Task.Factory.StartNew
just in case something weird was going on with Task.Run, but that didn't produce a different result.
The problem I am having is that not all of the Tasks execute every time. If I run in Visual Studio and step my way through debugging, it works properly every time. However, when attempting to generate 11 forms sequentially, sometimes it generates all of them, sometimes it generated 3 or 4, sometimes it generates all but 1.
I have error logging set up to be as extensive as possible, but no exceptions are being thrown that I can find, and no generated html files are left lying around in my file structure on account of a thread aborted part-way.
There seems to be a slight correlation between how quickly the page comes back from the post, and how many of the forms are generated. A longer load time generally correlates to more of the forms being generated...but I was under the impression that shouldn't matter. I'm spinning these off to separate threads with their own copy of the HttpContext to take with them and carry around. Once launched, I did not think that the original thread should impact them.
Any ideas on why I'm only getting 3 successful Tasks on some attempts, all 11 on another attempt, and no exceptions?

Task.Run(() => AsyncPDFFormGeneration(htmlOutput, serverRelativePath,
serverURL, signature, document, HttpContext.ApplicationInstance.Context));
You have a subtle race condition on this line. The problem is with the HttpContext.ApplicationInstance.Context property. It will be evaluated when the task starts. If it happens before the end of the request, this is fine. But if for some reason the task takes a bit of time to start, then the request will complete first, and the HttpContext will be null. Therefore, you will have a null-reference exception, giving you the impression that the task didn't start (when, in fact, it did but crashed immediately outside of your try/catch).
To avoid that, just store the context in a local variable, and use it for Task.Run:
var context = HttpContext; // Or HttpContext.ApplicationInstance.Context, but I don't really see the point
Task.Run(() => AsyncPDFFormGeneration(htmlOutput, serverRelativePath, serverURL, signature, document, context));
That said, I don't know what API you are using that requres System.Web.HttpContext.Current to be set, but it seems a very bad choice for a fire-and-forget task. Even if you locally save the HttpContext, it'll still have been cleaned up, so I'm not sure it'll behave as expected.
Also, as was mentioned in the comments, launching fire-and-forget tasks on ASP.NET is dangerous. You should use HostingEnvironment.QueueBackgroundWorkItem instead.

I would try using await Task.WhenAll(task1, task2, task3, etc) as your application may be closing before all tasks have completed.

Why does instance of a IWebDriver class throws an exception in Iteration?

I am developing an application using selenium web driver. I am putting a set of task in the 'for' loop. The task is:
Open a URL using driver object of IWebDriver class.
Traverse to different different URLs
At the for loop, I am killing the instance of IWebDriver class with following:
driver.Close(); driver.Quit();
Now when the second iteration starts, the program does not perform the above task, rather it gives an exception.
When I research on the exception, I found the issue is with the driver.Close(); driver.Quit();. It's not even closing the browser, but its also removing the instance of a IWebDriver class. How can I handle this situation?
Here is the Sample code:
class callingfunction
{
public static IWebDriver driver = new ChromeDriver(Cpath);
public void function1()
{
for (int i = 0; i < 10; i++ )
{
driver.Navigate().GoToUrl("http://www.aaa.org");
driver.Navigate().GoToUrl("http://www.aaa.org/contactus");
driver.FindElement(By.Name("contact")).SendKeys(contact);
driver.Navigate().GoToUrl("http://www.aaa.org/aboutus");
//Code logic
driver.Close();
driver.Quit();
}
}
}

You're calling quit() on the WebDriver inside your loop, so on the second iteration you will indeed get errors.
The only place you need it is right at the very end of your test. The close() call is also unnecessary.
Furthermore, you're not waiting for anything to load, so without proper waits your driver.FindElement(By.Name("contact")).SendKeys(contact); will likely fail often.
Finally you're not doing anything with the 1st or 3rd page requests, so - given the lack of waits - those pages are likely never allowed to fully load.

I don't understand the scenario you are trying to accomplish but I'm hoping that in the process of simplifying the code, you've removed the bits that make this scenario make a little more sense. Having said that...
The problem is that you are closing and quitting the browser driver inside the for loop but never instantiate another browser instance. The browser instance is created when you do new ChromeDriver() so you will want that inside the for loop. Something like this should work.
class callingfunction
{
public static IWebDriver driver = null;
public void function1()
{
for (int i = 0; i < 10; i++)
{
driver = new ChromeDriver(Cpath); // launch browser
driver.Navigate().GoToUrl("http://www.aaa.org");
driver.Navigate().GoToUrl("http://www.aaa.org/contactus");
driver.FindElement(By.Name("contact")).SendKeys(contact);
driver.Navigate().GoToUrl("http://www.aaa.org/aboutus");
//Code logic
driver.Close(); // close browser
}
driver.Quit(); // close browser driver
}
}
Is there some reason you need to close the browser between runs? You can't just reuse the existing browser instance? Your test will run much faster... but sometimes there is a need to close it. Only you would know...

C# Webdriver - Page Title assert fails before page loads

This issue began when I switched from testing on the www website to my localhost version of it. Working in VS 2012, I will begin debugging so the localhost is active, detach the process so I can test on it, then run any test I like. For a very basic example:
[Test]
public void CanGoToHomePage()
{
Pages.HomePage.Goto();
Assert.IsTrue(Pages.HomePage.IsAt());
}
And the functions it references are here:
public class HomePage
{
const string Url = "http://localhost:3738";
const string HomepageTitle = "FunnelFire - Home Page";
public void Goto()
{
Browser.Goto(Url);
}
public bool IsAt()
{
return Browser.Title == HomepageTitle;
}
}
And the actual selenium code here:
public static class Browser
{
static IWebDriver webDriver = new FirefoxDriver();
public static void Goto(string url)
{
webDriver.Manage().Window.Maximize();
webDriver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));
webDriver.Url = url;
}
}
Now the issue. The 10 second implicit wait that I added in Browser does successfully wait at most 10 seconds after loading a page to see if it can locate whatever information I want it to find, that is not the problem.
As I said earlier, after I switched to testing on localhost, suddenly I ran into a strange issue where a page would begin to load (i.e. screen still totally white, nothing finished) or even sometimes the next page would JUST barely finish loading and suddenly the test would just up and fail, pointing to the Assert of IsAt returning false even though the page it was loading was the correct one. I could run that test immediately once more and it would pass without a problem. Run it a third time and it could randomly fail again. I'm honestly not sure what is causing the issue and any help would be appreciated!

Implicit waits work only for finding elements. For waiting on the title of the page to be a certain value, you'll want to use an explicit wait. You can write your own version of this pattern, but in the .NET bindings, the WebDriver.Support.dll assembly has a WebDriverWait class to help with this. Its use would look something like this:
// WARNING! Untested code written from memory below. It has not
// been tested or even compiled in an IDE, so may be syntactically
// incorrect. The concept, however, should still be valid.
public void WaitForTitle(IWebDriver driver, string title, TimeSpan timeout)
{
WebDriverWait wait = new WebDriverWait(driver, timeout);
wait.Until((d) => { return d.Title == title; });
}
You could even modify your IsAt method to use this pattern, catching the WebDriverTimeoutException and returning false if the wait function times out.