I thought I understood how MbUnit's parallel test execution worked, but the behaviour I'm seeing differs sufficiently much from my expectation that I suspect I'm missing something!
I have a set of UI tests that I wish to run concurrently. All of the tests are in the same assembly, split across three different namespaces. All of the tests are completely independent of one another, so I'd like all of them to be eligible for parallel execution.
To that end, I put the following in the AssemblyInfo.cs:
[assembly: DegreeOfParallelism(8)]
[assembly: Parallelizable(TestScope.All)]
My understanding was that this combination of assembly attributes should cause all of the tests to be considered [Parallelizable], and that the test runner should use 8 threads during execution. My individual tests are marked with the [Test] attribute, and nothing else. None of them are data-driven.
However, what I actually see is at most 5-6 threads being used, meaning that my test runs are taking longer than they should be.
Am I missing something? Do I need to do anything else to ensure that all of my 8 threads are being used by the runner?
N.B. The behaviour is the same irrespective of which runner I use. The GUI, command line and TD.Net runners all behave the same as described above, again leading me to think I've missed something.
EDIT: As pointed out in the comments, I'm running v3.1 of MbUnit (update 2 build 397). The documentation suggests that the assembly level [parallelizable] attribute is available, but it does also seem to reference v3.2 of the framework despite that not yet being available.
EDIT 2: To further clarify, the structure of my assembly is as follows:
assembly
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
EDIT 3: OK, I've now noticed that if I only ever run one fixture at a time, the maximum number of tests running concurrently is always 8. As soon as I select multiple fixtures, it drops to either 5 or 6. If I take the contents of two fixtures (currently they contain 12 tests each) and drop them into the same fixture (for a total of 24 tests in that one fixture) that fixture will also always run 8 tests concurrently.
This seems to show that it isn't an issue in the individual tests, but rather in how the assembly level attributes percolate down to the fixture, or how the test runner consumes those attributes.
Additionally, I also observed (when running two fixtures) that once one of the two fixtures had been executed in its entirety, the runner starts to execute more tests concurrently when its back down to running only one fixture. For me right now, the first fixture gets done executing when there are 7 tests left to run in the second fixture. As soon as that happens, the number of tests running concurrently jumps up from the previous 5 or 6 to the maximum available of 7.
According to the release note of Gallio v3.0.6:
MbUnit helps you get the most out of your multi-core CPU. Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.
Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.
Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them. (We might add a feature to set this at the fixture or assembly level later based on user feedback.)
Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular. For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism. If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].
The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another. By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.
If you don't like the default then you can override the degree of parallelism at the assembly-level like this:
I don't know if it helps. Maybe Jeff could give more details as he had implemented that feature.
Ran into same problem, my findings
[assembly: Parallelizable(...)] at assembly level overrides fixture Parallelizable attributes and will result in fixture tests being run one at a time but at a fixture parallel level. Seems to have a max of between 5-6 fixtures in parallel.
[Parallelizable(TestScope.Descendants)] at fixture level will result in fixtures being run one at a time but the tests being run in parallel. Seems to have no max on tests in parallel.
Ultimately due to the assembly level constraint on fixture parallel limits, the only way is to use fixture level attributes and have fixtures tests run in parallel.
I would suggest creating less fixture and more tests per fixture to get around this issue. You could always launch multiple runners per assembly fixture perhaps.
Shame this is the case.
Override does not work for more than 5 test to run at a time.
We have 25 systems on Sauce labs available to execute 25 scripts at a time, we over ride the DegreeOfParallelism to 20 only 5 executes at a time.
[assembly: DegreeOfParallelism(20)] - Do not Work for Mbunit
Related
I understand NUnit runs tests in parallel, which isn't what this question is about exactly. In a single test, I can have two local socket connections open using TcpClient on different ports, which can communicate with one another in the same unit test. The problem is the address space is of course shared by the thread executing the unit test, so it doesn't simulate communication through a network how I want.
If I have a test called Test1 with one TcpClient, and whilst it's running, I run Test2 with the other TcpClient, in these two tests I can have both the unit tests talking to one another and communicating - testing the protocol I've designed with separate address spaces, which does give me the simulation of a network I desire, and allows me to do the proper test assertions.
Now the question is, is there any way to do this automatically? I'd like to have a single test that can run Test1 & Test2, where both these tests communicate via their respective sockets, instead of starting them both up manually in parallel.
NUnit is not designed to run tests in different processes (nor is any other unit testing framework I'm aware of). It's designed to start as many tests as possible in the same process, to increase performance (creating processes is expensive).
Of course, that doesn't mean it's impossible to do this, but it's not a domain of NUnit itself. The question doesn't give much information about how the tests are structure, but I'm assuming there's a test assembly for the server and another test assembly for the client. Only when they run at the same time, the tests work.
I would go ahead and create a third test assembly that just starts the other two, really creating processes manually, e.g. launching
dotnet test ServerUnitTests.dll
dotnet test ClientUnitTests.dll
["Integration Tests" would be the better term here, but that's a detail]
I do not think that is possible.you can decide which test should run after another, so basically order. but usually, tests are not required to be run in parallel.
I'm having a task with writing specific test and then creating setup which allows me to run test in "X" different instances of the same browser in parallel.
Using Selenium with NUnit
Trying to google that but still having difficulties to cope with that:
1 - Mark test with [Parallelizable] annotation
2 - Try finding .runsettings file and set there number of threads (did not found it...)
3 - Marking a tests to run five times on five instances/ threads (most probably it's done by NUnit annotation but having troubles to find which one exactly)
Our current C#/Nunit 2.6.3 test framework has a regression suite that takes over 35 hours to run on a single pc,fixtures with some tests lasting as long as 20 minutes. Setting up batches of tests to run on multiple machines is time consuming and inefficient so I'm trying to migrate the tests to NUnit 3 to get the benefit of parallel execution on Selenium Grid.
It is my aim to have 12 nodes each running a single instances of IE. However it appears the NUnit3 Test adapter for VS is trying to run all tests simultaneously.
As I will always be executing tests from more fixtures than I will have nodes it is important that fixtures will sit in a queue until a node becomes available. In practice a test fixture may have to wait a couple of hours for a free node.
For my current configuration experiment I have the following set up:
A hub with the following config:java -jar selenium-server-standalone-2.48.2.jar -role hub -newSessionWaitTimeout:-1 -browserTimeout 120 -timeout 3600
A single node in default config.
Two test fixtures, each with 10 tests. The test fixtures have the following attribute: [Parallelizable(ParallelScope.Self)]
In this situation I would expect that as only a single node that is supporting a single instance of IE then only a single test would be executed. The hub would then send the next test in the queue to the node when it became free. However it appears that both test fixtures are being run simultaneously. One test is pushed to the node but tests on the other fixture are failing with the following message:
Result Message:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:4444/wd/hub/session timed out after 60 seconds.
----> System.Net.WebException : The operation has timed out
When I used grid on Eclipse in a Java/JUnit framework I had no problems. The hub would queue tests until a node became free without any timeout, using the default config.
Does anyone know the correct configuration or is this a problem with the NUnit 3 Test adapter? Browser choice is unfortunately fixed as IE.
I found that number of parallel threads can be controlled by setting the LevelOfParallelism Attribute in the AssemblyInfo.
//Determines the number parallel threads that run simultaneously
[assembly: LevelOfParallelism(7)]
If this attribute is set to the number of nodes available then the tests queue as I expected.
I decided to switch from current solution (using modified NUnit by our team some years ago NDistribUnit which run tests on VirtualMachines and then gather results on hub server) to Selenium Grid 2.
Option with ParallelizableAttribute was tried.
Unfortunately I noticed that IWebDriver was stored in Global variable (puhh). This caused tests to start several browser instances, but tests used single IWebDriver -> tests execution happened in single browser -> tests were run under single process, but with several 'worker' threads. It was tried using 2 VMs as 'nodes' and local PC as Hub.
I know best solution is to change invalid idea to store driver in global variable, but it'll take too much time: there are 3k+ heavy UI tests to be updated; many static methods expected to have driver as global var to be updated also.
As well NUnit 3.0 provides Option to run several assemblies in parallel. To run several test projects it's good, but currently we have 1 assembly per one application.
It would be nice to run tests for one application (one assembly) in parallel.
Is there other ways to use GRID + NUnit 3 here to make it work?
Finally, existed solution were refactored: now each test during execution has own driver. Due to this change a lot of code was re-written (it appears that too much methods expected to have IwebDriver as global variable)
Actually, there are 2 options to do that:
Refactoring - it's done for one test project.
Along with removing static variables (initial refactoring purpose) other code was changed as well.
A great minus - significant effort was required.
Using TeamCity agents for parallel run.
I forgot to mention finally tests are being executed on TeamCity, but by single agent.
For left 'old' tests (where Driver instance was stored in static variable) several TC agents were configured to run only several classes from tests solution.
This option is very 'fast' and doesn't require big code changes.
I hope somebody can help me.
I have a lot of UnitTest to my C# application in VS2010, and therefore I want to execute them in parallel so I can benefit of my four core machine.
This is "easily" done by adding parallelTestCount="0" to the execution in the Local.testsettings.
But some of my UnitTest (around 50) are not thread safe and instead of reworking them, I just want them to be run in not parallel mode.
Is that possible and if so how to do it?
You can't change the parallelism on a single test or set of tests. You could try to create a second test settings file and a second assembly containing your "unsafe" tests; you can define the folder that test assemblies are loaded from under the "Unit test" tab of the test settings dialog.
That said, your tests should be thread-safe. A unit test should be able to run in any order, in any environment, and always pass -- unless, of course, something changed in the code they're testing.