Can't run xunit tests in parallel
Created 2 classes with a nonsense test which only sleeps and returns. Then inherited from an empty base class. The tests ran in parallel.
Changed the 2 classes to inherit from a real base class I need to use, which contains methods and some constants, and the tests don't run in parallel anymore.
The base class I'm inheriting from does not contain any testing related code (all fact annotated functions are in the child classes).
I'm using xUnit 2.4.1 with C# and .Net 4.7.2
I suspect there is something in the base class which prohibits running in parallel, but don't understand what. I have several async methods there which utilize an auto-generated REST client.
From documentation:
How does xUnit.net decide which tests can run against each other in
parallel? It uses a concept called test collections to make that
decision.
By default, each test class is a unique test collection. Tests within
the same test class will not run in parallel against each other.
Different classes/types can be executed in parallel. Even if class derived of some base class, it is still considered as different class.
But tests executes by Visual Studio, and by default tests executes on single thread. Check, did you enabled Tests in Parallel feature in Visual Studio test explorer?
Just to be sure, I were able to execute tests of different classes, which derived of one base class in parallel, by running tests without Visual Studio.
Update
In case when test class derives class which include own test methods, then inherited methods will be run as part of current class and so will not be run in parallel with methods of current class.
But derived class tests methods will be executed as part of this specific class and will be run in parallel with other classes which derives from it.
In this case you should observe those methods to be executed twice, one as part of "base" class and part of derived class.
Related
I have an NUnit test project containing a bunch of Test classes/fixtures each of which inherits from an abstract base class hierarchy. These sealed test classes all have a TestFixtureSource attribute attached at the class level:
[TestFixtureSource(typeof(ExecutionBrowsers))]
public sealed class MyTestClass : TestBase
Where ExecutionBrowsers is defined:
internal sealed class ExecutionBrowsers : IEnumerable
{
public IEnumerator GetEnumerator()
{
yield return Browser.Chrome;
yield return Browser.Edge;
yield return Browser.Firefox;
}
}
So essentially each test class will be instantiated 3 times, once for each browser. I want to run these tests in parallel in such as way that a browser does not have more than one test using it at the same time (I have a hard limitation on this - lets not get into that). So what I did was to add a .cs file at the root of the project and stick the following attributes in it:
[assembly: NUnit.Framework.FixtureLifeCycle(NUnit.Framework.LifeCycle.InstancePerTestCase)]
[assembly: NUnit.Framework.Parallelizable(NUnit.Framework.ParallelScope.Fixtures)]
[assembly: NUnit.Framework.LevelOfParallelism(3)]
This doesn't quite work though, it does not restrict tests to one per browser at any given time. It will start off with the first test in the first test class (some classes have more than one test) running that on each of the three browsers. However if one browser takes longer than the others it will get out of sync and begin executing two tests on one browser.
How can I achieve the behaviour that I want?
Well see, we have a situation unit test frameworks do not run tests of the same collection asynchronously so you must rethink the structure of your unit test so that they are separate from each other, I didn't see enough of your structure to be able to assist in this restructuring
You're trying to achieve fine control by putting attributes at the assembly level. There are lots of ways that can go wrong and you have discovered one of them. I recommend avoiding use of assembly-level ParallelizableAttribute unless you are absolutely sure that the specified parallel behavior will work for all your test fixtures as well as any you or others may add in the future. ;-)
Instead, add [Parallelizable] to the class. It will apply to each of your instances and will allow them to run against one another. The individual test cases will be non-parallelizable by default with respect to one another.
For the other attributes, you should eliminate [FixtureLifeCycle] unless you have a specific reason why you need it, i.e. unless your tests are running in parallel and changing the state of the fixture. You should only use [LevelOfParallelism] if it is needed for performance and should not count on it to keep any particular set of tests from running with one another.
You have not said how you run the tests. The above will work if you are running straight nunit console plus framework from the command line. If you are using Visual Studio, there are some other considerations because Test Explorer can change what NUnit thinks you are doing based on how it runs the tests.
I've a scenario where I'm adding a common code which every Test Case needs to invoke before executing the [TestMethod] (Similar to a [TestInitialize]/[AssemblyInitialize]).
However there are 3125 Tests and I'll need to add this [TestInitialize] along with some [DeploymentItem] attribute on all these Test Classes.
I could think of these ways
Manually change every Test Class to include TestInitialize (which is not efficient)
Have an Abstract Base class which will have TestInitialize and let every Test Case extend from this Base Class. But This again needs change in all 3125 Test Classes to extend them from the Base Class.
I could use [AssemblyInitialize] and use it. These still need change in around 82 assemblies. Some assemblies already have [AssemblyInitialize] so it will need a deeper look while pasting the code.
I was wondering - if there was a way in c#, where we can enforce every TestCase to invoke a certain method before running the tests without having to touch all these 3125 files?
I am trying to write a unit test for the following method:
public override void Run()
{
base.Run();
this.wpfApplication.Run();
}
Note that "this.wpfApplication" is an instance of System.Windows.Application.
I am using Microsoft's Test Explorer to run my tests (in VS Express 2013).
If I run the unit test it will hang on the .Run() line.
How can I create a unit test to run this method?
I already have a unit test to test the base class's implementation of the Run method. The base class is part of a framework which supports console and WPF apps.
As with almost all other cases where I call a .NET class method, I ended up writing an interface (e.g. IWpfRun) and used a mock implementation in my unit tests. It does mean that I need to do more in my constructors (as I need to specify which implementation to use).
Now my unit tests are concerned with my methods rather than mixing in concerns of .NET class methods such as running a WPF app, logging to a file etc.
My IWpfRun interface has 2 implementations. One is my WpfRun class that wraps the System.Windows.Application.Run method in override of my interface's Run method. The second implementation is my WpfRunMock class, that overrides the same method with an update of a property. I can assert the property value has been updated after the Run method has been called.
I am using selenium web driver to a run a number of tests.
I have a base class which includes a lot of tests.
In my second class, called 'People', I have another set of tests. The People class inherits the Base class.
I initialise some of the tests in the base class, when run when I run the tests for the People class. My problem is, it also runs all the tests in the base class, whether or not I initialise them. This leaves me running 100 tests, which takes forever, and when I only really wanted to test about 50.
Is there any setting to stop selenium web driver from doing this?
As far as I understand your problem, it is normal. Your test framework will load your class 'People' and search for all tests. Since the tests defined in your base class belongs also to your class 'People' (for the framework it makes no difference), the framework will execute them as well.
You should'nt put any test in your base class nor initialise thing for your class 'People'. The base class should contains only utility/convenience methods and the startup/shutdown methods that are common to ALL your tests (for People and the rest).
In sub class 'People', you put all the tests and the startup/shutdown methods related to 'People'.
In sub class 'Toto', you put all the tests and the startup/shutdown methods related to 'Toto'.
etc.
Hope this help.
I have a number of xUnit tests across multiple classes. A lot of these classes rely on a setup method being called before each test and a clean up method being called after each test.
In each test class I have a constructor which calls the setup method and a dispose method like the following to clean up:
public override void Dispose()
{
Cleanup();
}
Each test class extends a base class, which extends a class which inherits IDisposable.
When I run the tests in an individual test class all the tests run fine and the dispose method is called correctly.
When I run all the tests in my solution using the test explorer in Visual Studio, a lot of the tests are failing because the clean up method isn't being called. When I debug the tests I'm not seeing dispose being called.
I installed the xunit.runner.visualstudio NuGet package to get the tests to run in Visual Studio 2019. I'm using xUnit 2.4.1.
Does anybody know why dispose is not being called when I run all the tests at once?
it does create some shared state which needs to be initialised before
each test and cleaned up afterwards
xUnit executes tests of different classes in parallel by default. Because you are using shared state between tests you need to execute all tests sequentially.
For executing tests sequentially, group all tests which using shared state into one "Test Collection".
From docs
When to use: when you want to create a single test context and share it among tests in several test classes, and have it cleaned up after
all the tests in the test classes have finished.
https://xunit.net/docs/shared-context