I was running tests in VS2013 with Resharper 8, which integrated with an external process and redirected its output. I'm not seeking guidance on what qualifies as a unit test, because I find the test runner to be a nicer driver for any test than a console app.
This process was a database and so my SetUp and TearDown methods unsurprisingly did the opposites of each other and would absolutely step on each other if allowed to run concurrently.
They did not get to step on each other, because whenever two tests were started by the running simultaneously, they would lock up.
Latest Version 7 of Resharper did not have this issue.
Tinkering with the unit test settings in Resharper > Options > Unit Testing did not help.
How to run NUnit test fixtures serially? -- is close to working.
On my test fixture base (and I always run base.SetUp() at the beginning of derived SetUp), I added the following:
[SetUp]
protected virtual void SetUp()
{
FixtureHandle.Wait();
}
[TearDown]
protected virtual void TearDown()
{
FixtureHandle.Set();
}
FixtureHandle is a ManualResetEvent. This prevents NUnit from being too damn good and fast.
The above answer, from #nik.shornikov, will work, but in Resharper 2016.2 and later (and probably also in earlier versions) you can do this via an option, without writing any code. Please see my answer to Is there a way to *prevent* ReSharper from running an assembly's unit tests in parallel? for full details.
Related
I have created a number of long running integration tests. They take around 5-10 minutes to complete. I don't want them to run Everytime the developers run tests from VS as it will take a long time. They are purely ad hoc tests
Is there a way to mark a test as ignored but still allow developers to run it locally if they explicitly select and run the test without removing ignore tag each time? Also i want these tests to run during overnight build via TFS and MS test runner
Suggest you look into using Test Categories.
By categorising your long running tests separately, you can choose what to include on test runs, e.g.
[TestMethod]
[TestCategory("Integration")]
public void MyLongRunningTest()
{
// Test Code
}
And to run:
mstest /testcontainer:TestProject.dll /category:"Integration"
I have configured some tests to run in parallel using Selenium and Nunit but sometimes one of the tests kills off early and miss behaves. It's not always the same test and they are basic examples so am not sure what's happening.
I also followed the example in the following link (https://www.youtube.com/watch?v=18zrtO1l7EU) to the letter but it also does not work. Sometimes one of the tests halts and doesn't continue and then fails. I've tried with both chrome and IE drivers but the same thing happens. Could this be an Nunit version issue?
I think this was because I was initially setting the Base weddriver as Static. Since taking the static off it works fine.
We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
We have a unit tests project which is still using the "old-style" private accessors in many tests.
Since they're a maintenance nightmare, we're trying to get rid of them, and move to new new Microsoft Fakes framework, using Shims where needed.
Recently we wrote some new unit tests which use Shims, and noticed that for some reason this caused a few OTHER, old, tests, which were not modified, to run considerably slower. By slower I mean run times of about ~10sec instead of ~900millisec for the affected tests.
Running the affected tests on their own didn't seems to have this effect though - it only occurs when running them after tests with Shims.
Initially we thought this might be simply due to initialization problems, causing tests to influence one another.
However, after some experimentation, we found that the slowdown occurs even without actually adding any new test code. Simply adding the following snippet before one of the slowed-down tests caused the same effect of the test running slower:
using (ShimsContext.Create()) {}
Debugging seemed to show that the the code being tested was indeed running much slower (not the unit test code itself) , but we couldn't identify which part of it. We're not able to identify why these tests are affected while others are not.
At this point we tried profiling these tests (using the new "profile test" option in VisualStudio). However, it turns out that profiling tests with Shims is not possible for some reason. The following exception was thrown:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: UnitTestIsolation instrumentation failed to initialize. Please restart Visual Studio and rerun this test
As a last resort we also tried moving all tests using Shims to a separate test project in the same solution. This did seem to help, and all test run times returned to normal. We used test playlists to run each project's tests before the other's, and in both cases run times were OK. It's not really a solution though, and feels more like circumventing the actual issue.
So, we're not sure how to proceed. Any thoughts and ideas would be helpful.
Thanks.
The Microsoft documentation, Better Unit Testing with Microsoft Fakes (RTM).pdf, states that you will see a performance decrease when using Shims.
This article also goes over the performance impact of Shims:
http://technet.microsoft.com/en-us/windows/jj863250.aspx
"Other" tests should be explicitly executed in shimless context (ShimsContext.ExecuteWithoutShims), because it looks like even disposed ShimsContext in other tests may have detours to logic that doesn't use shims.
Has anyone found a way to run Selenium RC / Selenium Grid tests, written in C# in parallel?
I've currently got a sizable test suite written using Selenium RC's C# driver. Running the entire test suite takes a little over an hour to complete. I normally don't have to run the entire suite so it hasn't been a concern up to now, but it's something that I'd like to be able to do more regularly (ie, as part of an automated build)
I've been spending some time recently poking around with the Selenium Grid project whose purpose essentially is to allow those tests to run in parallel. Unfortunately, it seems that the TestDriven.net plugin that I'm using runs the tests serially (ie, one after another). I'm assuming that NUnit would execute the tests in a similar fashion, although I haven't actually tested this out.
I've noticed that the NUnit 2.5 betas are starting to talk about running tests in parallel with pNUnit, but I haven't really familiarized myself enough with the project to know for sure whether this would work.
Another option I'm considering is separating my test suite into different libraries which would let me run a test from each library concurrently, but I'd like to avoid that if possible since I'm not convinced this is a valid reason for splitting up the test suite.
I am working on this very thing and have found Gallio latest can drive mbUnit tests in parallel. You can drive them against a single Selenium Grid hub, which can have several remote control servers listening.
I'm using the latest nightly from Gallio to get the ParallelizableAttribute and DegreeOfParallelismAttribute.
Something things I've noticed is I cannot rely on TestSet and TestTeardown be isolated the parallel tests. You'll need the test to look something like this:
[Test] public void Foo(){
var s = new DefaultSelenium("http://grid", 4444, "*firefox",
"http://server-under-test");
s.Start();
s.Open("mypage.aspx");
// Continue
s.Stop();
}
Using the [SetUp] attribute to start the Selenium session was causing the tests to not get the remote session from s.Start().
I wrote PNUnit as an extension for NUnit almost three years ago and I'm happy to see it was finally integrated into NUnit.
We use it on a daily basis to test our software under different distros and combinations. Just to give an example: we've a test suite of heavy tests (long ones) with about 210 tests. Each of them sets up a server and runs a client in command line running several operations (up to 210 scenarios).
Well, we use the same suite to run the tests on different Linux combinations and windows variations, and also combined ones like a windows server with a linux client, windows xp, vista, then domain controller, out of domain, and so on. We use the same binaries and then just have "agents" launched at several boxes.
We use the same platform for: balancing load test load -> I mean, running in chunks faster. Running several combinations at the same time, and what I think is more interesting: defining multi client scenarios: two clients wait for the server to start up, then launch operations, synch with each other and so on. We also use PNUnit for load testing (hundreds of boxes against a single server).
So, if you have any questions about how to set it up (which is not simple yet, I'm afraid), don't hesitate to ask.
Also I wrote an article long ago about it at DDJ: http://www.ddj.com/architect/193104810
Hope it helps
I don't know if no answer counts as an answer but I'd say you have researched everything and you really came up with the 2 possible solutions...
Test Suite runs tests in parallel
Split the test suite up
I am at a loss for any thing else.