I'm having a task with writing specific test and then creating setup which allows me to run test in "X" different instances of the same browser in parallel.
Using Selenium with NUnit
Trying to google that but still having difficulties to cope with that:
1 - Mark test with [Parallelizable] annotation
2 - Try finding .runsettings file and set there number of threads (did not found it...)
3 - Marking a tests to run five times on five instances/ threads (most probably it's done by NUnit annotation but having troubles to find which one exactly)
Related
We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
I decided to switch from current solution (using modified NUnit by our team some years ago NDistribUnit which run tests on VirtualMachines and then gather results on hub server) to Selenium Grid 2.
Option with ParallelizableAttribute was tried.
Unfortunately I noticed that IWebDriver was stored in Global variable (puhh). This caused tests to start several browser instances, but tests used single IWebDriver -> tests execution happened in single browser -> tests were run under single process, but with several 'worker' threads. It was tried using 2 VMs as 'nodes' and local PC as Hub.
I know best solution is to change invalid idea to store driver in global variable, but it'll take too much time: there are 3k+ heavy UI tests to be updated; many static methods expected to have driver as global var to be updated also.
As well NUnit 3.0 provides Option to run several assemblies in parallel. To run several test projects it's good, but currently we have 1 assembly per one application.
It would be nice to run tests for one application (one assembly) in parallel.
Is there other ways to use GRID + NUnit 3 here to make it work?
Finally, existed solution were refactored: now each test during execution has own driver. Due to this change a lot of code was re-written (it appears that too much methods expected to have IwebDriver as global variable)
Actually, there are 2 options to do that:
Refactoring - it's done for one test project.
Along with removing static variables (initial refactoring purpose) other code was changed as well.
A great minus - significant effort was required.
Using TeamCity agents for parallel run.
I forgot to mention finally tests are being executed on TeamCity, but by single agent.
For left 'old' tests (where Driver instance was stored in static variable) several TC agents were configured to run only several classes from tests solution.
This option is very 'fast' and doesn't require big code changes.
I'm setting up automated acceptance tests for an Asp.Net MVC project using Gherkin scenarios, Specflow steps and Selenium remote webdriver. The aim is to run the tests from Jenkins in parallel on multiple nodes using the selenium server hub.
At the moment I'm trying to get them to run in parallel locally and having some difficulties. Is it even possible to run those tests in parallel with the Gherkin/Specflow setup or does the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently ?
If it's not impossible how exactly can it be done ? And if it is impossible what else could be done to speed up the testing process ?
It has nothing to do with
the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently
It's totally possible to run your MSTest cases in parallel. As far as I'm aware the max number of hung tests shouldn't exceed 5. However I've used custom tool for this purpose, spawning every test in a separate thread, it's easy achievable with TPL.
You'll need to use the selenium grid to do this, and I would recommend putting the grid on another computer or virtual machine if you can.
You can run the same test on multiple browsers using a combination of 1) the task parallel library and 2) the dynamic object class. I've written about the specifics here http://blog.dmbcllc.com/running-selenium-in-parallel-with-any-net-unit-testing-tool/
I thought I understood how MbUnit's parallel test execution worked, but the behaviour I'm seeing differs sufficiently much from my expectation that I suspect I'm missing something!
I have a set of UI tests that I wish to run concurrently. All of the tests are in the same assembly, split across three different namespaces. All of the tests are completely independent of one another, so I'd like all of them to be eligible for parallel execution.
To that end, I put the following in the AssemblyInfo.cs:
[assembly: DegreeOfParallelism(8)]
[assembly: Parallelizable(TestScope.All)]
My understanding was that this combination of assembly attributes should cause all of the tests to be considered [Parallelizable], and that the test runner should use 8 threads during execution. My individual tests are marked with the [Test] attribute, and nothing else. None of them are data-driven.
However, what I actually see is at most 5-6 threads being used, meaning that my test runs are taking longer than they should be.
Am I missing something? Do I need to do anything else to ensure that all of my 8 threads are being used by the runner?
N.B. The behaviour is the same irrespective of which runner I use. The GUI, command line and TD.Net runners all behave the same as described above, again leading me to think I've missed something.
EDIT: As pointed out in the comments, I'm running v3.1 of MbUnit (update 2 build 397). The documentation suggests that the assembly level [parallelizable] attribute is available, but it does also seem to reference v3.2 of the framework despite that not yet being available.
EDIT 2: To further clarify, the structure of my assembly is as follows:
assembly
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
- namespace
- fixture
- tests (each carrying only the [Test] attribute)
- fixture
- tests (each carrying only the [Test] attribute)
EDIT 3: OK, I've now noticed that if I only ever run one fixture at a time, the maximum number of tests running concurrently is always 8. As soon as I select multiple fixtures, it drops to either 5 or 6. If I take the contents of two fixtures (currently they contain 12 tests each) and drop them into the same fixture (for a total of 24 tests in that one fixture) that fixture will also always run 8 tests concurrently.
This seems to show that it isn't an issue in the individual tests, but rather in how the assembly level attributes percolate down to the fixture, or how the test runner consumes those attributes.
Additionally, I also observed (when running two fixtures) that once one of the two fixtures had been executed in its entirety, the runner starts to execute more tests concurrently when its back down to running only one fixture. For me right now, the first fixture gets done executing when there are 7 tests left to run in the second fixture. As soon as that happens, the number of tests running concurrently jumps up from the previous 5 or 6 to the maximum available of 7.
According to the release note of Gallio v3.0.6:
MbUnit helps you get the most out of your multi-core CPU. Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.
Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.
Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them. (We might add a feature to set this at the fixture or assembly level later based on user feedback.)
Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular. For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism. If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].
The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another. By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.
If you don't like the default then you can override the degree of parallelism at the assembly-level like this:
I don't know if it helps. Maybe Jeff could give more details as he had implemented that feature.
Ran into same problem, my findings
[assembly: Parallelizable(...)] at assembly level overrides fixture Parallelizable attributes and will result in fixture tests being run one at a time but at a fixture parallel level. Seems to have a max of between 5-6 fixtures in parallel.
[Parallelizable(TestScope.Descendants)] at fixture level will result in fixtures being run one at a time but the tests being run in parallel. Seems to have no max on tests in parallel.
Ultimately due to the assembly level constraint on fixture parallel limits, the only way is to use fixture level attributes and have fixtures tests run in parallel.
I would suggest creating less fixture and more tests per fixture to get around this issue. You could always launch multiple runners per assembly fixture perhaps.
Shame this is the case.
Override does not work for more than 5 test to run at a time.
We have 25 systems on Sauce labs available to execute 25 scripts at a time, we over ride the DegreeOfParallelism to 20 only 5 executes at a time.
[assembly: DegreeOfParallelism(20)] - Do not Work for Mbunit
Has anyone found a way to run Selenium RC / Selenium Grid tests, written in C# in parallel?
I've currently got a sizable test suite written using Selenium RC's C# driver. Running the entire test suite takes a little over an hour to complete. I normally don't have to run the entire suite so it hasn't been a concern up to now, but it's something that I'd like to be able to do more regularly (ie, as part of an automated build)
I've been spending some time recently poking around with the Selenium Grid project whose purpose essentially is to allow those tests to run in parallel. Unfortunately, it seems that the TestDriven.net plugin that I'm using runs the tests serially (ie, one after another). I'm assuming that NUnit would execute the tests in a similar fashion, although I haven't actually tested this out.
I've noticed that the NUnit 2.5 betas are starting to talk about running tests in parallel with pNUnit, but I haven't really familiarized myself enough with the project to know for sure whether this would work.
Another option I'm considering is separating my test suite into different libraries which would let me run a test from each library concurrently, but I'd like to avoid that if possible since I'm not convinced this is a valid reason for splitting up the test suite.
I am working on this very thing and have found Gallio latest can drive mbUnit tests in parallel. You can drive them against a single Selenium Grid hub, which can have several remote control servers listening.
I'm using the latest nightly from Gallio to get the ParallelizableAttribute and DegreeOfParallelismAttribute.
Something things I've noticed is I cannot rely on TestSet and TestTeardown be isolated the parallel tests. You'll need the test to look something like this:
[Test] public void Foo(){
var s = new DefaultSelenium("http://grid", 4444, "*firefox",
"http://server-under-test");
s.Start();
s.Open("mypage.aspx");
// Continue
s.Stop();
}
Using the [SetUp] attribute to start the Selenium session was causing the tests to not get the remote session from s.Start().
I wrote PNUnit as an extension for NUnit almost three years ago and I'm happy to see it was finally integrated into NUnit.
We use it on a daily basis to test our software under different distros and combinations. Just to give an example: we've a test suite of heavy tests (long ones) with about 210 tests. Each of them sets up a server and runs a client in command line running several operations (up to 210 scenarios).
Well, we use the same suite to run the tests on different Linux combinations and windows variations, and also combined ones like a windows server with a linux client, windows xp, vista, then domain controller, out of domain, and so on. We use the same binaries and then just have "agents" launched at several boxes.
We use the same platform for: balancing load test load -> I mean, running in chunks faster. Running several combinations at the same time, and what I think is more interesting: defining multi client scenarios: two clients wait for the server to start up, then launch operations, synch with each other and so on. We also use PNUnit for load testing (hundreds of boxes against a single server).
So, if you have any questions about how to set it up (which is not simple yet, I'm afraid), don't hesitate to ask.
Also I wrote an article long ago about it at DDJ: http://www.ddj.com/architect/193104810
Hope it helps
I don't know if no answer counts as an answer but I'd say you have researched everything and you really came up with the 2 possible solutions...
Test Suite runs tests in parallel
Split the test suite up
I am at a loss for any thing else.