We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
Related
I decided to switch from current solution (using modified NUnit by our team some years ago NDistribUnit which run tests on VirtualMachines and then gather results on hub server) to Selenium Grid 2.
Option with ParallelizableAttribute was tried.
Unfortunately I noticed that IWebDriver was stored in Global variable (puhh). This caused tests to start several browser instances, but tests used single IWebDriver -> tests execution happened in single browser -> tests were run under single process, but with several 'worker' threads. It was tried using 2 VMs as 'nodes' and local PC as Hub.
I know best solution is to change invalid idea to store driver in global variable, but it'll take too much time: there are 3k+ heavy UI tests to be updated; many static methods expected to have driver as global var to be updated also.
As well NUnit 3.0 provides Option to run several assemblies in parallel. To run several test projects it's good, but currently we have 1 assembly per one application.
It would be nice to run tests for one application (one assembly) in parallel.
Is there other ways to use GRID + NUnit 3 here to make it work?
Finally, existed solution were refactored: now each test during execution has own driver. Due to this change a lot of code was re-written (it appears that too much methods expected to have IwebDriver as global variable)
Actually, there are 2 options to do that:
Refactoring - it's done for one test project.
Along with removing static variables (initial refactoring purpose) other code was changed as well.
A great minus - significant effort was required.
Using TeamCity agents for parallel run.
I forgot to mention finally tests are being executed on TeamCity, but by single agent.
For left 'old' tests (where Driver instance was stored in static variable) several TC agents were configured to run only several classes from tests solution.
This option is very 'fast' and doesn't require big code changes.
I'm setting up automated acceptance tests for an Asp.Net MVC project using Gherkin scenarios, Specflow steps and Selenium remote webdriver. The aim is to run the tests from Jenkins in parallel on multiple nodes using the selenium server hub.
At the moment I'm trying to get them to run in parallel locally and having some difficulties. Is it even possible to run those tests in parallel with the Gherkin/Specflow setup or does the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently ?
If it's not impossible how exactly can it be done ? And if it is impossible what else could be done to speed up the testing process ?
It has nothing to do with
the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently
It's totally possible to run your MSTest cases in parallel. As far as I'm aware the max number of hung tests shouldn't exceed 5. However I've used custom tool for this purpose, spawning every test in a separate thread, it's easy achievable with TPL.
You'll need to use the selenium grid to do this, and I would recommend putting the grid on another computer or virtual machine if you can.
You can run the same test on multiple browsers using a combination of 1) the task parallel library and 2) the dynamic object class. I've written about the specifics here http://blog.dmbcllc.com/running-selenium-in-parallel-with-any-net-unit-testing-tool/
The company I work for uses Visual Studio to develop its website and all of its features, and there is also a separate site that's been developed for testing the site. This 'testing' site can run individual test cases against the website, and must be run for each possible case.
Everything is written in VB.NET and each time the program is run a single thread is created to run the test. However, at the 'end' of the test the thread seems to still lingers. The stop button in Visual Studio must be manually clicked in order to terminate the application. Also, a process icon lingers in the task bar long after the application has closed.
It appears to me that the program is not correctly terminating all threads run during the tests, but I'm not sure if this is an issue worth brining up in the office, so I ask the following question...
What is the purpose of properly closing an application and all threads running on it, and what are the consequences, if any, of not doing so?
Well it's probably a small problem now, but it's not a good practice, IMHO. Imagine what would happen if the same code was now being executed by a continuous integration server, for instance, TeamCity (or Jenkins, or...), and the unit tests are being run continuously and automatically, by said build server.
What would happen to the build status when those threads fail to close down cleanly? We often face this problem due to bad design decisions in threading, or due to simple (and possibly, idiotic) mistakes in our unit testing code. The net effect though, is a hung build process.
I've seen CI servers hang for almost half a day before someone (mercifully) killed the build process. Essentially, this indicates a problem in our code that may or may not become a huge issue. If this was server-side code, there is potential for this code to lead to a pretty bad situation. My advice would be to dig out your introspection toolkits (memory profiling, perf profiling, etc) and see what exactly is going on, and resolve it.
We had a similar problem with an application that is being called to index SPA pages on our application server. It was throwing an exception in some cases and threads were not closing. The biggest downside is that it will consume the servers memory which is bad
Another downside as it runs as a web application that it will consume available ports and stop running when it run out of available ports.
The code should be modified to peacefully kill the thread after finishing or on exceptions and of course report any.
We have a unit tests project which is still using the "old-style" private accessors in many tests.
Since they're a maintenance nightmare, we're trying to get rid of them, and move to new new Microsoft Fakes framework, using Shims where needed.
Recently we wrote some new unit tests which use Shims, and noticed that for some reason this caused a few OTHER, old, tests, which were not modified, to run considerably slower. By slower I mean run times of about ~10sec instead of ~900millisec for the affected tests.
Running the affected tests on their own didn't seems to have this effect though - it only occurs when running them after tests with Shims.
Initially we thought this might be simply due to initialization problems, causing tests to influence one another.
However, after some experimentation, we found that the slowdown occurs even without actually adding any new test code. Simply adding the following snippet before one of the slowed-down tests caused the same effect of the test running slower:
using (ShimsContext.Create()) {}
Debugging seemed to show that the the code being tested was indeed running much slower (not the unit test code itself) , but we couldn't identify which part of it. We're not able to identify why these tests are affected while others are not.
At this point we tried profiling these tests (using the new "profile test" option in VisualStudio). However, it turns out that profiling tests with Shims is not possible for some reason. The following exception was thrown:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: UnitTestIsolation instrumentation failed to initialize. Please restart Visual Studio and rerun this test
As a last resort we also tried moving all tests using Shims to a separate test project in the same solution. This did seem to help, and all test run times returned to normal. We used test playlists to run each project's tests before the other's, and in both cases run times were OK. It's not really a solution though, and feels more like circumventing the actual issue.
So, we're not sure how to proceed. Any thoughts and ideas would be helpful.
Thanks.
The Microsoft documentation, Better Unit Testing with Microsoft Fakes (RTM).pdf, states that you will see a performance decrease when using Shims.
This article also goes over the performance impact of Shims:
http://technet.microsoft.com/en-us/windows/jj863250.aspx
"Other" tests should be explicitly executed in shimless context (ShimsContext.ExecuteWithoutShims), because it looks like even disposed ShimsContext in other tests may have detours to logic that doesn't use shims.
I am using temporary database for my unit tests that are created and deleted with each test but have been very careful to ensure that it shares the same setup as my 'real' database (enlist=false;multipleactiveresultsets=true;etc) but have been constantly running into MSDTC escalation issues wherever TransactionScope is being used.
After switching TransactionScope for Transaction I have solved the MSDTC escalation issue and run straight into a new problem - the transaction works perfectly as I use the application in the web browser, but it fails in unit testing with the following error:
SqlConnection does not support parallel transactions.
Without posting tons of code, has anyone else run into similar problems where the test environment and dev environment appear to work differently? Is there any general reason why a parallel transaction would work in dev and not in unit test mode?
If I cannot simulate exactly what is happening with my tests then what is the point in testing at all?
One big difference that I failed to account for was where I may be using RenderAction in a view, and therefore calling more controller actions than the test would, which may explain MSDTC/Transaction issue.