Looking to get some help around making my tests Parallelizable. I have a selenium c# setup that uses a combination of NUnit, C# and selenium to run tests in sequence locally on my machine or on the CI server.
I've looked into Parallelization of testing before but have been unable to make the jump, and running in a sequence was fine.
At the moment when I add the NUnit [Parallelizable] tag, I get an 'OpenQA.Selenium.WebDriverException : invalid session id' error, based on the reading I've done I need to make each new driver I call unique. However, I'm uncertain on how to do this? or even start for that matter... is this even possible within my current set up?
My tests are currently doing limited smoke tests and just removing the repetitive regression testing against multiple browsers, however, I foresee a need to vastly expand my coverage of testing capability.
I will probably be looking at getting Browserstack or Sauselab in the long term but obviously, that requires funding, and I need to get that signed off, so I will be looking to get it running locally for now.
here is a look at the basic set up of my code
test files:
1st .cs test file
{
[TestFixture]
[Parallelizable]
public class Featur2Tests1 : TestBase
{
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test1(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test2(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
}
}
2nd .cs test file
{
[TestFixture]
[Parallelizable]
public class FeatureTests2 : TestBase
{
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test1(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test2(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
}
}
TestBase.cs where my set up for each test
{
public class TestBase
{
public static IWebDriver driver;
public void Setup(string BrowserName, string Environment, string System)
{
Driver.Intialize(BrowserName);
//do additional setup before test run...
}
[TearDown]
public void CleanUp()
{
Driver.Close();
}
public static IEnumerable TestData
{
get
{
string[] browsers = Config.theBrowserList.Split(',');
string[] Environments = Config.theEnvironmentList.Split(',');
string[] Systems = Config.theSystemList.Split(',');
foreach (string browser in browsers)
{
foreach (string Environment in Environments)
{
foreach (string System in Systems)
{
yield return new TestCaseData(browser, Environment, System);
}
}
}
}
}
}
}
The IEnumerable TestData comes from a file called config.resx and contains the following data:
{Name}: {Value}
theBrowserList: Chrome,Edge,Firefox
theEnvironmentList: QA
theSystemList: WE
This is where I create my driver in Driver.cs
{
public class Driver
{
public static IWebDriver Instance { get; set; }
public static void Intialize(string browser)
{
string appDirectory = Directory.GetParent(AppDomain.CurrentDomain.BaseDirectory).Parent.Parent.Parent.FullName;
string driverFolder = $"{appDirectory}/Framework.Platform/bin/debug";
if (browser == "Chrome")
{
ChromeOptions chromeOpts = new ChromeOptions();
chromeOpts.AddUserProfilePreference("safebrowsing.enabled", true);
chromeOpts.AddArgument("start-maximized");
chromeOpts.AddArgument("log-level=3");
Instance = new ChromeDriver(driverFolder, chromeOpts);
}
else if (browser == "IE")
{
var options = new InternetExplorerOptions { EnsureCleanSession = true };
options.AddAdditionalCapability("IgnoreZoomLevel", true);
Instance = new InternetExplorerDriver(driverFolder, options);
Instance.Manage().Window.Maximize();
}
else if (browser == "Edge")
{
EdgeOptions edgeOpts = new EdgeOptions();
Instance = new EdgeDriver(driverFolder, edgeOpts);
Instance.Manage().Window.Maximize();
Instance.Manage().Cookies.DeleteAllCookies();
}
else if (browser == "Firefox")
{
FirefoxOptions firefoxOpts = new FirefoxOptions();
Instance = new FirefoxDriver(driverFolder, firefoxOpts);
Instance.Manage().Window.Maximize();
}
else { Assert.Fail($"Browser Driver; {browser}, is not currently supported by Initialise method"); }
}
public static void Close(string browser = "other")
{
if (browser == "IE")
{
Process[] ies = Process.GetProcessesByName("iexplore");
foreach (Process ie in ies)
{
ie.Kill();
}
}
else
{
Instance.Quit();
}
}
}
}
All your tests use the same driver, which is defined in TestBase as static. The two fixtures will run in parallel and will both effect the state of the driver. If you want two tests to run in parallel, they cannot both be using the same state, with the exception of constant or readonly values.
The first thing to do would be to make the driver an instance member, so that each of the derived fixtures is working with a different driver. If that doesn't solve the problem, it will at least take you to the next step toward a solution.
do not use static and that should help resolve your issue
public IWebDriver Instance { get; set; }
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
namespace Nunit_ParalelizeTest
{
public class Base
{
protected IWebDriver _driver;
[TearDown]
public void TearDown()
{
_driver.Close();
_driver.Quit();
}
[SetUp]
public void Setup()
{
_driver = new ChromeDriver();
_driver.Manage().Window.Maximize();
}
}
}
I see there is no [Setup] on top of setup method in the TestBase. Invalid session is caused because you are trying to close a window which is not there. Also try to replace driver.close() with driver.quit();
You should call the driver separately in each test, otherwise, nunit opens only one driver for all instances. Hope this makes sence to you.
Related
Question is probably really trivial but I cannot handle it in proper way. I'm using Selenium with NUnit, having two clases:
1) "DemoTest" which involves one simply test "DummyTest":
public class DemoTest : TestBase
{
public class RunTest
{
[Test, Category("Main-Tests"), Order(1)]
public void DummyTest()
{
}
}
}
2) "Test base" class where I want to place all of the NUnit/ driver attributes like: "SetUp" / "TearDown"
[TestFixture]
public class TestBase
{
public IWebDriver driver;
public IWebDriver Driver
{
get { return driver; }
set { driver = value; }
}
public string pageURL = "http://automationpractice.com/";
[SetUp]
public void SetUp()
{
driver = new ChromeDriver();
driver.Manage().Timeouts().PageLoad = TimeSpan.FromSeconds(15);
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(0);
driver.Navigate().GoToUrl(pageURL);
}
[TearDown]
public void TearDown()
{
driver.Close();
driver.Dispose();
}
}
}
As NUnit attributes are declared (SetUp section) my test from DemoTest class should at least move on the page under pageURL variable.
Result is that after running a test it's immediately jump on "passed" without opening the specified address.
"DemoTest" inherits from "Test base" class. Nuget packages are installed correctly. When I'm placing test inside the "Test base" class everything works correctly but I want to have it separated.
Try to fix DemoTest class as follows:
[TestFixture]
public class DemoTest : TestBase
{
[Test, Category("Main-Tests"), Order(1)]
public void DummyTest()
{
}
}
I am currently using the following code to shutdown my browsers after each test:
[TearDown]
public void StopBrowser()
{
if (Driver == null)
return;
Driver.Quit();
}
This works fine when running single tests however when I run multiple tests in Parallel using NUnit's [Parallelizable] tag, I get my tests starting to fail due to no such session errors, Unable to connect to the remote server errors and then an Object Reference error on top so it's definitely something to do with the parallel tests.
Please find below the code that I use in the [SetUp] method:
public IWebDriver Driver { get; set; }
public string TestUrl;
[SetUp]
public void Setup()
{
string Browser = "Chrome";
if (Browser == "Chrome")
{
ChromeOptions options = new ChromeOptions();
options.AddArgument("--start-maximized");
Driver = new ChromeDriver(options);
}
else if (Browser == "Firefox")
{
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService();
service.FirefoxBinaryPath = #"C:\Program Files (x86)\Mozilla Firefox\firefox.exe";
Driver = new FirefoxDriver(service);
}
else if (Browser == "Edge")
{
EdgeDriverService service = EdgeDriverService.CreateDefaultService();
Driver = new EdgeDriver(service);
}
BasePage Code:
public class BasePage<TObjectRepository> where TObjectRepository : BasePageObjectRepository
{
public BasePage(IWebDriver driver, TObjectRepository repository)
{
Driver = driver;
ObjectRepository = repository;
}
public IWebDriver Driver { get; }
internal TObjectRepository ObjectRepository { get; }
}
BasePageObjectRepository Code:
public class BasePageObjectRepository
{
protected readonly IWebDriver Driver;
public BasePageObjectRepository(IWebDriver driver)
{
Driver = driver;
}
}
My Tests simply call specific functions inside their relevant page and then the code inside the page simply has Selenium code that points to the elements in that Pages Object Repository i.e.
ObjectRepository.LoginButton.Click();
Im not sure if the way I have my tests set up is whats partially causing the issues but any help I can get in regards to this would be greatly appreciated
EDIT: Added more of my code in to help:
BaseTest Class:
[TestFixture]
public class BaseTest
{
public IWebDriver Driver { get; set; }
public string TestUrl;
[SetUp]
public void Setup()
{
string Browser = "Chrome";
if (Browser == "Chrome")
{
ChromeOptions options = new ChromeOptions();
options.AddArgument("--start-maximized");
Driver = new ChromeDriver(options);
}
else if (Browser == "Firefox")
{
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService();
service.FirefoxBinaryPath = #"C:\Program Files (x86)\Mozilla Firefox\firefox.exe";
Driver = new FirefoxDriver(service);
}
else if (Browser == "Edge")
{
EdgeDriverService service = EdgeDriverService.CreateDefaultService();
Driver = new EdgeDriver(service);
}
Driver.Manage().Window.Maximize();
string Environment;
Environment = "Test";
if (Environment == "Test")
{
TestUrl = "https://walberton-test.azurewebsites.net/";
}
else if (Environment == "Live")
{
TestUrl = "";
}
}
}
[OneTimeTearDown]
public void TeardownAllBrowsers()
{
Driver.Close();
Driver.Quit();
}
Example of One of the Tests:
[TestFixture]
public class TimeLogsTestChrome: BaseTest
{
[Test]
[Parallelizable]
[Category("Time Logs Test for Chrome")]
public void AssertTimeLogsHeading()
{
Driver = new ChromeDriver();
Driver.Manage().Window.Maximize();
var loginPage = new LoginPage(Driver);
var dashboard = new Dashboard(Driver);
var timeSheets = new Timesheets(Driver);
loginPage.GoTo("Test");
loginPage.EnterEmailAddress("Valid");
loginPage.EnterPassword("Valid");
loginPage.ClickLoginButtonForLoginToDashboard();
dashboard.ClickTimesheetsButton();
timeSheets.AssertHeading();
Driver.Close();
}
}
Example of Underlying Code to the Test:
internal class Timesheets : BasePage<TimesheetsPageObjectRepository>
{
public Timesheets(IWebDriver driver) : base(driver, new TimesheetsPageObjectRepository(driver))
{
}
internal void AssertHeading()
{
try
{
Assert.AreEqual("Timesheets", ObjectRepository.TimesheetHeading);
}
catch (Exception ex)
{
throw;
}
}
internal void ClickAddTimesheetButton()
{
ObjectRepository.AddTimesheetButton.Click();
}
internal void ClickSubmitTimeButton()
{
ObjectRepository.SubmitTimeButton.Click();
Thread.Sleep(5000);
}
}
Partial Example of the Page Object Repository that has most of the FindElement code:
internal class TimesheetsPageObjectRepository : BasePageObjectRepository
{
public TimesheetsPageObjectRepository(IWebDriver driver) : base(driver) { }
public string TimesheetHeading => Driver.FindElement(By.ClassName("pull-left")).Text;
}
NUnit tests in a fixture all share the same instance of the fixture. That's one of the fundamentals of NUnit's design that programmers have to keep in mind because it's easy for tests to step on one another's toes through use of shared state. You have to watch out for tests interfering with one another even in non-parallel situations but parallelism makes it worse.
In your case, the most likely culprit is the Driver property of the fixture. It is set by the setup of every test and whatever is held there is closed by every test. That's not what you want.
I also notice that the Driver is defined twice, once in the fixture and once in the base class. You have not posted enough code for me to understand why you are doing this, but it seems problematic.
You have to decide whether all the tests in a fixture are intended to use the same driver or not. I suspect that's not what you want, since the driver itself may have changeable state. However, you can set it up either way.
To use the same driver for all tests, make it a property of the fixture. Initialize it in a OneTimeSetUp method and release it in a OneTimeTearDown method.
To use a different driver for each test, do not make it a property. Instead, initialize a local variable in each test and release it in the test itself.
I want to run the test n times from beginning i.e. quit the driver and run the setup again. But retry attribute does not quit the driver, it just run the test case again.
[TestFixture(typeof(ChromeDriver))]
public class TestWithMultipleBrowsers<TWebDriver> where TWebDriver : IWebDriver, new()
{
#region Setup
private IWebDriver driver;
[TestFixtureSetUp]
public void CreateDriver()
{
if (typeof(TWebDriver).Name == "ChromeDriver")
{
driver = new ChromeDriver(#"C:\ChromeDriver");
}
else
{
driver = new TWebDriver();
}
}
[TestFixtureTearDown]
public void FixtureTearDown()
{
if (driver != null) driver.Quit();
}
[Test,Retry(2)]
[TestCase("jobsearch")]
[TestCase("employer")]
public void GoogleTest(string search)
{
driver.Navigate().GoToUrl("http://www.google.com/");
IWebElement query = driver.FindElement(By.Name("q"));
query.SendKeys(search + Keys.Enter);
Thread.Sleep(1000);
Assert.AreEqual(search + " - Google Search", driver.Title);
}
#endregion
}
I want to run the test n times from beginning i.e. quit the driver and run the setup again.
The reason the CreateDriver method isn't being called again is because you're using the [TestFixtureSetUp] attribute which only runs once for a [TestFixture]. If you want to run a setup method before each test, use the [Setup] attribute instead.
Same goes for the [TestFixtureTearDown] attribute. If that should occur after each test, you should use the [TearDown] attribute instead.
I am writing integration tests for ServiceStack with in-memory database and I ran into this exception: "System.IO.InvalidDataException ServiceStackHost.Instance has already been set" while trying to run multiple test classes together, each having its own AppHostHttpListenerBase. However, if I ran the test classes one at a time, it ran and passed without problems. One reason for having multiple classes is because I want to test the AppHost with different services/dependencies registered and also to group my tests logically. Below is a general snippet of my tests. I would like to be able run all the test at one go.
public class TestClassOne : IDisposable
{
string _endPoint = "http://localhost:54321/";
AppHostHttpListenerBase _appHost;
IDbConnectionFactory _dbConn = new OrmLiteConnectionFactory(":memory:", SqliteDialect.Provider);
public TestClassOne()
{
_appHost = new UnitTestAppHost(_dbConn, ...){};
_appHost.Init().Start(_endPoint);
}
[Fact]
public void Test()
{
...
using(var db = _dbConn.Open())
{
Assert.True(...);
}
}
public void Dispose()
{
_appHost.Dispose();
_appHost = null;
}
}
public class TestClassTwo : IDisposable
{
string _endPoint = "http://localhost:54321/";
AppHostHttpListenerBase _appHost;
IDbConnectionFactory _dbConn = new OrmLiteConnectionFactory(":memory:", SqliteDialect.Provider);
public TestClassTwo()
{
_appHost = new UnitTestAppHost(...){};
_appHost.Init().Start(_endPoint);
}
[Fact]
public void Test()
{
...
using(var db = _dbConn.Open())
{
Assert.True(...);
}
}
public void Dispose()
{
_appHost.Dispose();
_appHost = null;
}
}
I have tried running on another AppDomain, but it doesn't seems to be what I am looking for I think, because I need to do some Asserts on IDbConnection in the current running AppDomain (?), if that make any sense. Any suggestions on how I should be doing it? I'm using xUnit and Resharper's test runner btw.
I ended up fixing this by creating an AppHostSetupFixture class with a public static AppHost variable. Create a [SetUp] method that initializes your app host and a [TearDown] method that disposes it. Use AppHostSetupFixture.AppHost in your test classes.
[SetUpFixture]
public class AppHostSetupFixture
{
public static ServiceStackHost AppHost;
[SetUp]
public void Setup()
{
AppHost = new BasicAppHost(typeof(FeatureService).Assembly)
{
ConfigureContainer = container =>
{
var l = new List<string>();
l.Add(ConfigurationManager.ConnectionStrings["Redis"].ConnectionString);
container.Register<IRedisClientsManager>(c => new RedisManagerPool(l, new RedisPoolConfig() { MaxPoolSize = 40 }));
}
}
.Init();
}
[TearDown]
public void TearDown()
{
AppHost.Dispose();
}
}
This error is a result of trying to run multiple AppHosts per AppDomain. Each ServiceStack AppHost is a singleton and only allows a single AppHost per AppDomain.
I have a bootstrapper object that I'm trying to test (using xunit). The tests appear to pass, but I'm seeing some weird things in one of the test runners I use (ncrunch). I use both ncrunch and the resharper xunit runner. My idea was to take the assembly that the singleton is in, load it into a new appdomain, run my tests using reflection, then unload the app domain. As I said, the tests pass in both ncrunch and resharper, but ncrunch is not showing the execution paths that I expect. Here's the code:
public class Bootstrapper
{
private static Bootstrapper booted;
public Bootstrapper()
{
// performs boot tasks
}
public static void Boot()
{
if (booted == null)
{
var staticboot = new Bootstrapper();
Booted = staticboot;
}
}
public static Bootstrapper Booted
{
get
{
if (booted == null) throw new InvalidOperationException("Should call Boot() before accessing the booted object");
return booted;
}
set { booted = value; }
}
}
public class Tests
{
[Fact]
public void TryingToAccessBootedKernelBeforeBootThrowsException()
{
var setup = this.SetupTestingDomainWithAssembly("StackOverflowQuestion.Tests.dll");
var kernelType = setup.Item2.GetType("StackOverflowQuestion.Tests.Bootstrapper");
var bootedkernelProperty = kernelType.GetProperty("Booted");
try
{
bootedkernelProperty.GetValue(null);
}
catch (Exception e)
{
Assert.IsType(typeof(InvalidOperationException), e.InnerException);
}
AppDomain.Unload(setup.Item1);
}
[Fact]
public void CanAccessKernelAfterBooting()
{
var setup = this.SetupTestingDomainWithAssembly("StackOverflowQuestion.Tests.dll");
var kernelType = setup.Item2.GetType("StackOverflowQuestion.Tests.Bootstrapper");
var bootMethod = kernelType.GetMethod("Boot");
bootMethod.Invoke(null, new object[] { });
var bootedkernelProperty = kernelType.GetProperty("Booted");
Assert.DoesNotThrow(() => bootedkernelProperty.GetValue(null));
AppDomain.Unload(setup.Item1);
}
[Fact]
public void BootIsIdempotent()
{
var setup = this.SetupTestingDomainWithAssembly("StackOverflowQuestion.Tests.dll");
var kernelType = setup.Item2.GetType("StackOverflowQuestion.Tests.Bootstrapper");
var bootMethod = kernelType.GetMethod("Boot");
bootMethod.Invoke(null, new object[] {});
var bootedkernelProperty = kernelType.GetProperty("Booted");
var bootedKernel = (Bootstrapper)bootedkernelProperty.GetValue(null);
bootMethod.Invoke(null, new object[] {});
var secondCall = (Bootstrapper)bootedkernelProperty.GetValue(null);
Assert.Equal(bootedKernel, secondCall);
AppDomain.Unload(setup.Item1);
}
private Tuple<AppDomain, Assembly> SetupTestingDomainWithAssembly(string assemblyPath)
{
// we guarantee that each domain will have a unique name.
AppDomain testingDomain = AppDomain.CreateDomain(DateTime.Now.Ticks.ToString());
var pancakesAssemblyName = new AssemblyName();
pancakesAssemblyName.CodeBase = assemblyPath;
var assembly = testingDomain.Load(pancakesAssemblyName);
return new Tuple<AppDomain, Assembly>(testingDomain, assembly);
}
}
Now, I recognize that there is some cleanup that needs to happen code-wise, but I was happy to see them all green. If I fiddle with them to make them fail, that works as expected. The only thing that's kind of smelly is that ncrunch is reporting weird execution paths. Specifically, ncrunch is showing that the line that throws the invalid operation exception is never executed.
I suppose it's possible that ncrunch has a bug when dealing with other application domains, but it's more likely that I don't actually understand what's going on with the app domains, but I'm not sure where to continue from here.
Also, I do know that singletons are bad, but I believe bootstrappers are one place where they actually are useful. You want to guarantee that they are only booted once.
Unless I'm missing something here.. it doesn't look like you are actually invoking anything in your other app domain. Your reflection is occurring in the current app domain. Take a look at the DoCallback method: http://msdn.microsoft.com/en-us/library/system.appdomain.docallback.aspx
public class Tests
{
[Fact]
public void TryingToAccessBootedKernelBeforeBootThrowsException()
{
var appDomain = AppDomain.Create(Guid.NewGuid());
try
{
appDomain.DoCallBack(new CrossAppDomainDelegate(TryingToAccessBootedKernelBeforeBootThrowsException_AppDomainCallback));
}
catch (Exception e)
{
Assert.IsType(typeof(InvalidOperationException), e.InnerException);
}
AppDomain.Unload(appDomain);
}
public static void TryingToAccessBootedKernelBeforeBootThrowsException_AppDomainCallback()
{
var bootstrapper = BootStrapper.Booted;
}
}