We Keep Coding

Selenium tests running in parallel causing error: invalid session id

Looking to get some help around making my tests Parallelizable. I have a selenium c# setup that uses a combination of NUnit, C# and selenium to run tests in sequence locally on my machine or on the CI server.
I've looked into Parallelization of testing before but have been unable to make the jump, and running in a sequence was fine.
At the moment when I add the NUnit [Parallelizable] tag, I get an 'OpenQA.Selenium.WebDriverException : invalid session id' error, based on the reading I've done I need to make each new driver I call unique. However, I'm uncertain on how to do this? or even start for that matter... is this even possible within my current set up?
My tests are currently doing limited smoke tests and just removing the repetitive regression testing against multiple browsers, however, I foresee a need to vastly expand my coverage of testing capability.
I will probably be looking at getting Browserstack or Sauselab in the long term but obviously, that requires funding, and I need to get that signed off, so I will be looking to get it running locally for now.
here is a look at the basic set up of my code
test files:
1st .cs test file
{
[TestFixture]
[Parallelizable]
public class Featur2Tests1 : TestBase
{
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test1(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test2(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
}
}
2nd .cs test file
{
[TestFixture]
[Parallelizable]
public class FeatureTests2 : TestBase
{
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test1(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
[Test]
[TestCaseSource(typeof(TestBase), "TestData")]
public void test2(string BrowserName, string Environment, string System)
{
Setup(BrowserName, Environment, System);
//Run test steps....
}
}
}
TestBase.cs where my set up for each test
{
public class TestBase
{
public static IWebDriver driver;
public void Setup(string BrowserName, string Environment, string System)
{
Driver.Intialize(BrowserName);
//do additional setup before test run...
}
[TearDown]
public void CleanUp()
{
Driver.Close();
}
public static IEnumerable TestData
{
get
{
string[] browsers = Config.theBrowserList.Split(',');
string[] Environments = Config.theEnvironmentList.Split(',');
string[] Systems = Config.theSystemList.Split(',');
foreach (string browser in browsers)
{
foreach (string Environment in Environments)
{
foreach (string System in Systems)
{
yield return new TestCaseData(browser, Environment, System);
}
}
}
}
}
}
}
The IEnumerable TestData comes from a file called config.resx and contains the following data:
{Name}: {Value}
theBrowserList: Chrome,Edge,Firefox
theEnvironmentList: QA
theSystemList: WE
This is where I create my driver in Driver.cs
{
public class Driver
{
public static IWebDriver Instance { get; set; }
public static void Intialize(string browser)
{
string appDirectory = Directory.GetParent(AppDomain.CurrentDomain.BaseDirectory).Parent.Parent.Parent.FullName;
string driverFolder = $"{appDirectory}/Framework.Platform/bin/debug";
if (browser == "Chrome")
{
ChromeOptions chromeOpts = new ChromeOptions();
chromeOpts.AddUserProfilePreference("safebrowsing.enabled", true);
chromeOpts.AddArgument("start-maximized");
chromeOpts.AddArgument("log-level=3");
Instance = new ChromeDriver(driverFolder, chromeOpts);
}
else if (browser == "IE")
{
var options = new InternetExplorerOptions { EnsureCleanSession = true };
options.AddAdditionalCapability("IgnoreZoomLevel", true);
Instance = new InternetExplorerDriver(driverFolder, options);
Instance.Manage().Window.Maximize();
}
else if (browser == "Edge")
{
EdgeOptions edgeOpts = new EdgeOptions();
Instance = new EdgeDriver(driverFolder, edgeOpts);
Instance.Manage().Window.Maximize();
Instance.Manage().Cookies.DeleteAllCookies();
}
else if (browser == "Firefox")
{
FirefoxOptions firefoxOpts = new FirefoxOptions();
Instance = new FirefoxDriver(driverFolder, firefoxOpts);
Instance.Manage().Window.Maximize();
}
else { Assert.Fail($"Browser Driver; {browser}, is not currently supported by Initialise method"); }
}
public static void Close(string browser = "other")
{
if (browser == "IE")
{
Process[] ies = Process.GetProcessesByName("iexplore");
foreach (Process ie in ies)
{
ie.Kill();
}
}
else
{
Instance.Quit();
}
}
}
}

All your tests use the same driver, which is defined in TestBase as static. The two fixtures will run in parallel and will both effect the state of the driver. If you want two tests to run in parallel, they cannot both be using the same state, with the exception of constant or readonly values.
The first thing to do would be to make the driver an instance member, so that each of the derived fixtures is working with a different driver. If that doesn't solve the problem, it will at least take you to the next step toward a solution.

do not use static and that should help resolve your issue
public IWebDriver Instance { get; set; }
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
namespace Nunit_ParalelizeTest
{
public class Base
{
protected IWebDriver _driver;
[TearDown]
public void TearDown()
{
_driver.Close();
_driver.Quit();
}
[SetUp]
public void Setup()
{
_driver = new ChromeDriver();
_driver.Manage().Window.Maximize();
}
}
}

I see there is no [Setup] on top of setup method in the TestBase. Invalid session is caused because you are trying to close a window which is not there. Also try to replace driver.close() with driver.quit();

You should call the driver separately in each test, otherwise, nunit opens only one driver for all instances. Hope this makes sence to you.

Testing Thread Safety With Activator C#

I am working on a little project at home trying to improve my coding skills. I am currently working on building a factory. I was telling some co-workers about the way it was implemented and they had informed me that it could have issues with threading and deadlocks. I created a unit test, but I am unsure if this is valid or would even catch the issue. Here is my setup.
public class InventoryFactory : BaseFactory, IInventoryFactory
{
public IInventoryEngine Create(InventoryFactoryConfiguration configuration)
{
IInventoryRepository repository = BuildInventoryRepository(configuration.RepositoryConfiguration);
IInventoryService service = BuildInventoryService(configuration.ServiceConfiguration);
IInventoryEngine engine = BuildInventoryEngine(configuration.EngineConfiguration, repository, service);
return engine;
}
private IInventoryRepository BuildInventoryRepository(FactoryConfigurationModel configuration)
{
IInventoryRepository repository =
base.CreateInstance<IInventoryRepository>(configuration.AssemblyFile, configuration.FullyQualifiedClassName);
return repository;
}
private IInventoryService BuildInventoryService(FactoryConfigurationModel configuration)
{
IInventoryService service =
base.CreateInstance<IInventoryService>(configuration.AssemblyFile, configuration.FullyQualifiedClassName);
return service;
}
private IInventoryEngine BuildInventoryEngine(FactoryConfigurationModel configuration, IInventoryRepository repository, IInventoryService service)
{
IInventoryEngine engine =
base.CreateInstance<IInventoryEngine>(configuration.AssemblyFile, configuration.FullyQualifiedClassName, repository, service);
return engine;
}
}
public class BaseFactory
{
protected T CreateInstance<T>(string assemblyFile, string fullyQualifiedClassName, params object[] arguments)
{
Assembly assembly = Assembly.LoadFrom(assemblyFile);
Type classType = assembly.GetType(fullyQualifiedClassName);
T instance = (T)Activator.CreateInstance(classType, arguments);
return instance;
}
}
This is my current unit test to try seeing if any issues would occur.
[TestMethod]
public void InventoryFactoryThreadTest()
{
Task task1 = Task.Factory.StartNew(() =>
{
FactoryConfigurationModel repositoryConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel serviceConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel engineConfiguration = new FactoryConfigurationModel("foo", "bar");
InventoryFactoryConfiguration factoryConfiguration = new InventoryFactoryConfiguration(repositoryConfiguration, serviceConfiguration, engineConfiguration);
InventoryFactory factory = new InventoryFactory();
try
{
for (int i = 0; i < 150000; i++)
{
IInventoryEngine engine = factory.Create(factoryConfiguration);
}
}
catch (System.Exception ex)
{
Assert.Fail(ex.StackTrace);
}
});
Task task2 = Task.Factory.StartNew(() =>
{
FactoryConfigurationModel repositoryConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel serviceConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel engineConfiguration = new FactoryConfigurationModel("foo", "bar");
InventoryFactoryConfiguration factoryConfiguration = new InventoryFactoryConfiguration(repositoryConfiguration, serviceConfiguration, engineConfiguration);
InventoryFactory factory = new InventoryFactory();
try
{
for (int i = 0; i < 150000; i++)
{
IInventoryEngine engine = factory.Create(factoryConfiguration);
}
}
catch (System.Exception ex)
{
Assert.Fail(ex.StackTrace);
}
});
Task task3 = Task.Factory.StartNew(() =>
{
FactoryConfigurationModel repositoryConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel serviceConfiguration = new FactoryConfigurationModel("foo", "bar");
FactoryConfigurationModel engineConfiguration = new FactoryConfigurationModel("foo", "bar");
InventoryFactoryConfiguration factoryConfiguration = new InventoryFactoryConfiguration(repositoryConfiguration, serviceConfiguration, engineConfiguration);
InventoryFactory factory = new InventoryFactory();
try
{
for (int i = 0; i < 150000; i++)
{
IInventoryEngine engine = factory.Create(factoryConfiguration);
}
}
catch (System.Exception ex)
{
Assert.Fail(ex.StackTrace);
}
});
Task.WaitAll(task1, task2, task3);
}
From what I understand, the potential threading issue would be with the Activator.CreateInstance() method. My questions are: Is this a valid test to be able to see if threading is not safe. If this is not, could you tell me why?
Thank you ahead of time!

Calling Activator.CreateInstance is effectively the same as calling new Something(). Unless there's something in the class's constructor doing something really odd that affects some shared state, thread safety and deadlocks won't be a concern.
Inventing wheels is fun, but I recommend looking at some that have already been invented. Windsor, Autofac, and other Ioc/DI containers are really good at creating object instances, including scanning specified assemblies for interface implementations. You can specify whether you want it to create a new class instance each time or return the same instance over and over. You don't have to call constructors, which makes it easier to create classes with lots of complex nested dependencies. That in turn frees you up to write smaller, more testable classes without having to worry about how you'll construct them. It's awesome. If you're at the point where you see a need to create factories like this then it's probably time to look at IoC containers.
Also, there's a problem with testing to see if something is thread safe. Multithreading issues are unpredictable so it's hard to rule them out with tests. If something isn't thread safe it could pass all sorts of tests but blow up in another environment. That's not to say that the testing is bad. It's just never 100% conclusive.

I don't see the threading issue since you are creating and returning new individual instances of type T. May want to see if Assembly.LoadFrom is thread safe.
To be safe:
public class BaseFactory
{
private static object _monitor = new object();
protected T CreateInstance<T>(string assemblyFile, string fullyQualifiedClassName, params object[] arguments)
{
lock(_monitor)
{
Assembly assembly = Assembly.LoadFrom(assemblyFile);
Type classType = assembly.GetType(fullyQualifiedClassName);
T instance = (T)Activator.CreateInstance(classType, arguments);
return instance;
}
}
}

Unit test that legacy code reacts to ThreadAbortException in a certain way

I've got an existing bit of legacy code that I want to get under test. Here's a repro of the essentials:
public class LegacyUnit
{
private readonly ICollaborator collaborator;
public LegacyUnit(ICollaborator collaborator)
{
this.collaborator = collaborator;
}
public object GetStuff(HttpContextBase context, string input)
{
try
{
if (input == "")
{
context.Response.End();
}
collaborator.DoOtherStuff();
return "Done!";
}
catch (ThreadAbortException)
{ }
return null;
}
}
Now, this legacy unit has some issues, but for now I'm just trying to get it under test. Specifically, I want to test that collaborator.DoOtherStuff is not called if Response.End() raised a ThreadAbort.
The problem: how do you raise such an exception?
I've read through this question and its answers on ThreadAbortException, and understand that it's special. However, I don't see from those posts how to handle this in unit tests.
Here's my attempt:
[Test]
public void DoesNotCallCollaboratorOnThreadAbort()
{
var testResponseMock = new Mock<HttpResponseBase>();
var testContextMock = new Mock<HttpContextBase>();
var collaboratorMock = new Mock<ICollaborator>();
testContextMock.Setup(x => x.Response).Returns(testResponseMock.Object);
testResponseMock.Setup(x => x.End()).Throws<ThreadAbortException>(); // Compile error
var unit = new LegacyUnit(collaboratorMock.Object);
unit.GetStuff(testContextMock.Object, "");
collaboratorMock.Verify(c => c.DoOtherStuff(), Times.Never);
}
Obviously the compiler complains: ThreadAbortException has no available constructor. Also, it's sealed (probably for good reasons), so creating a "testable" sub-class won't work.
What is the proper way to get such code under test? Is it even feasible, or is the LegacyUnit just too test-unfriendly?
Full, minimal repro (empty .NET 4.5 class library with NUnit 2.6.4 and Moq 4.5.9):
public interface ICollaborator
{
void DoOtherStuff();
}
public class LegacyUnit
{
private readonly ICollaborator collaborator;
public LegacyUnit(ICollaborator collaborator)
{
this.collaborator = collaborator;
}
public object GetStuff(HttpContextBase context, string input)
{
try
{
if (input == "") context.Response.End();
collaborator.DoOtherStuff();
return "Done!";
}
catch (ThreadAbortException)
{ }
return null;
}
}
[TestFixture]
public class LegacyUnitTests
{
[Test]
public void DoesNotCallCollaboratorOnThreadAbort()
{
var testResponseMock = new Mock<HttpResponseBase>();
var testContextMock = new Mock<HttpContextBase>();
var collaboratorMock = new Mock<ICollaborator>();
testContextMock.Setup(x => x.Response).Returns(testResponseMock.Object);
testResponseMock.Setup(x => x.End()).Throws<ThreadAbortException>(); // Compile error here
var unit = new LegacyUnit(collaboratorMock.Object);
unit.GetStuff(testContextMock.Object, "");
collaboratorMock.Verify(c => c.DoOtherStuff(), Times.Never);
}
}

ThreadAbortException is raised in the target thread by calling Abort on it. You can create a thread to run the test and call Abort in your mock of testResponseMock.End e.g.
testContextMock.Setup(x => x.Response).Returns(testResponseMock.Object);
var unit = new LegacyUnit(collaboratorMock.Object);
var thread = new Thread(() => unit.GetStuff(testContextMock.Object, ""));
testResponseMock.Setup(x => x.End()).Callback(() => { Thread.CurrentThread.Abort(); });
thread.Start();
thread.Join();
collaboratorMock.Verify(c => c.DoOtherStuff(), Times.Never);

ServiceStack integration tests: ServiceStackHost.Instance has already been set

I am writing integration tests for ServiceStack with in-memory database and I ran into this exception: "System.IO.InvalidDataException ServiceStackHost.Instance has already been set" while trying to run multiple test classes together, each having its own AppHostHttpListenerBase. However, if I ran the test classes one at a time, it ran and passed without problems. One reason for having multiple classes is because I want to test the AppHost with different services/dependencies registered and also to group my tests logically. Below is a general snippet of my tests. I would like to be able run all the test at one go.
public class TestClassOne : IDisposable
{
string _endPoint = "http://localhost:54321/";
AppHostHttpListenerBase _appHost;
IDbConnectionFactory _dbConn = new OrmLiteConnectionFactory(":memory:", SqliteDialect.Provider);
public TestClassOne()
{
_appHost = new UnitTestAppHost(_dbConn, ...){};
_appHost.Init().Start(_endPoint);
}
[Fact]
public void Test()
{
...
using(var db = _dbConn.Open())
{
Assert.True(...);
}
}
public void Dispose()
{
_appHost.Dispose();
_appHost = null;
}
}
public class TestClassTwo : IDisposable
{
string _endPoint = "http://localhost:54321/";
AppHostHttpListenerBase _appHost;
IDbConnectionFactory _dbConn = new OrmLiteConnectionFactory(":memory:", SqliteDialect.Provider);
public TestClassTwo()
{
_appHost = new UnitTestAppHost(...){};
_appHost.Init().Start(_endPoint);
}
[Fact]
public void Test()
{
...
using(var db = _dbConn.Open())
{
Assert.True(...);
}
}
public void Dispose()
{
_appHost.Dispose();
_appHost = null;
}
}
I have tried running on another AppDomain, but it doesn't seems to be what I am looking for I think, because I need to do some Asserts on IDbConnection in the current running AppDomain (?), if that make any sense. Any suggestions on how I should be doing it? I'm using xUnit and Resharper's test runner btw.

I ended up fixing this by creating an AppHostSetupFixture class with a public static AppHost variable. Create a [SetUp] method that initializes your app host and a [TearDown] method that disposes it. Use AppHostSetupFixture.AppHost in your test classes.
[SetUpFixture]
public class AppHostSetupFixture
{
public static ServiceStackHost AppHost;
[SetUp]
public void Setup()
{
AppHost = new BasicAppHost(typeof(FeatureService).Assembly)
{
ConfigureContainer = container =>
{
var l = new List<string>();
l.Add(ConfigurationManager.ConnectionStrings["Redis"].ConnectionString);
container.Register<IRedisClientsManager>(c => new RedisManagerPool(l, new RedisPoolConfig() { MaxPoolSize = 40 }));
}
}
.Init();
}
[TearDown]
public void TearDown()
{
AppHost.Dispose();
}
}

This error is a result of trying to run multiple AppHosts per AppDomain. Each ServiceStack AppHost is a singleton and only allows a single AppHost per AppDomain.

Including exception details when using a Castle WCF Facility hosted service

I'm having some trouble unit testing a bit of code while utilising the Wcf Facility for Castle Windsor. It seems to refuse to include Exception Details when an Exception is thrown, I only get to see empty FaultExceptions. This is my test setup:
First, here's a stub of the service that I will be connecting to:
[ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public abstract class StubAilDataService : IAilDataService
{
public virtual Method1()
{
}
/* More methods */
}
Notice that I've specified IncludeExceptionDetailsInFaults and set it to true.
This is how I host the stubbed service:
private ServiceHost _host;
private StubAilDataService _rhinoService;
[TestFixtureSetUp]
public void FixtureSetup()
{
var sba = new ServiceDebugBehavior {IncludeExceptionDetailInFaults = true};
_rhinoService = MockRepository.GeneratePartialMock<StubAilDataService>();
_host = new ServiceHost(_rhinoService);
_host.AddServiceEndpoint(typeof(IAilDataService), new WSHttpBinding("wsSecure"), "http://localhost:8080/Service");
_host.Open();
_container.AddFacility<WcfFacility>().Register(
Component.For<IServiceBehavior>().Instance(sba),
Component.For<IAilDataService>()
.LifeStyle.PerWcfSession()
.ActAs(new DefaultClientModel
{
Endpoint =
WcfEndpoint.BoundTo(new WSHttpBinding("wsSecure"))
.At("http://localhost:8080/Service")
}) // More stuff
);
}
I've done a PartialMock in an attempt to keep the Include.. attribute on the mocked object.
And the test. Notice that I tell my mocked service to throw a very specific exception here.
[Test]
[ExpectedException(typeof(AggregateException))]
public void AnalyzeProductCreationJobs_Should_Throw_Aggregate_Exception_If_A_DataService_Call_Throws()
{
//Arrange
_rhinoService.Expect(
s => s.CategoryIsInAgility(Arg<string>.Matches(str => str.Equals("000103")), Arg<Settings>.Is.Anything))
.Throw(new FaultException<InvalidOperationException>(new InvalidOperationException("FAIL!")));
var product = new Product { CategoryCode = "000103" };
var analyzer = TypeResolver.Resolve<ProductAnalyzer>();
//Act
analyzer.AnalyzeProductCreationJobs(product);
}
And finally, the code I'm actually testing:
public class ProductAnalyzer
{
private readonly IDataServiceClient _dataClient;
public ProductAnalyzer(IDataServiceClient dataClient)
{
_dataClient = dataClient;
}
public IEnumerable<IAdsmlJob<CreateResponse>> AnalyzeProductCreationJobs(Product product)
{
IList<IAdsmlJob<CreateResponse>> creationJobs = new List<IAdsmlJob<CreateResponse>>();
var task = Task.Factory.StartNew(() =>
{
// This is where the exception set up in my .Expect gets thrown.
bool categoryIsInAgility = _dataClient.CategoryIsInAgility(product.CategoryCode);
// Logic
}); // Continued by more tasks
try
{ task.Wait(); }
catch (AggregateException ae)
{
ae.Flatten().Handle(ex => ex is TaskCanceledException);
}
}
I would expect the service to crash and throw the exception I've set it up to throw - but the Wcf Facility seems to strip away the exception that is thrown and replace it with an empty FaultException instead.
Am i missing something? There are quite a few components working together here - and I'm not 100% sure where things go wrong.

You have to explicitly declare the type of fault exception the method throws on the interface.
Example:
[ServiceContract(Namespace = "http://www.example.com")]
public interface IAilDataService
{
[OperationContract]
[FaultContract(typeof(OperationPermissionFault))]
[FaultContract(typeof(InvalidOperationException))]
void Method1();
}
See http://msdn.microsoft.com/en-us/library/system.servicemodel.faultcontractattribute.aspx