I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.
My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?
Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?
Important note
MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.
Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.
When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.
Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.
Visual Studio 2015 and earlier.
MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:
Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.
In Visual Studio 2015 update 1 and later
Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.
It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.
Related
I have some Specflow(3.9.40)-based UI test using Selenium Webdriver and NUnit(3.13.2) Framework.
I would like these tests to not run when I hit 'Run All' in the Test Explorer.
In my StepDefinition class I have tried adding the Explicit Attribute:
[TestFixture, Explicit("Only run when called explicitly")]
[Binding]
public class LoginStepDefinitions
{ ...
, but for some reason it does not seem to work - The UI Tests still run.
I imagine there may be something I can add to the [BeforeScenario] in my HookInitialization class, but I am stumped:
[BeforeScenario]
public void FirstBeforeScenario()
{
var seleniumDriver = new SeleniumDriver(_scenarioContext);
_scenarioContext.Set(seleniumDriver, "SeleniumDriver");
}
Is there a way for me to tell Visual Studio Test Explorer to ignore the UI Tests or its folder ?
The basic problem is that NUnit's Explicit setting is unique among the frameworks supported by Test Explorer, which knows nothing of this concept of "explicitness."
If Test Explorer calls the NUnit adapter telling it to run all, then NUnit will honor the Explicit setting. However, in some situations, it simply passes a list of all the tests to NUnit. In that case, all tests appear (to NUnit) to have been run explicitly.
Which way Test Explorer calls NUnit is dependent on the version you are running and it's only possible to tell how it calls by debugging or by observing the outcome. To add to the confusion, some versions of the NUnit3 adapter use heuristics to try to guess whether an explicit test was called explicitly. This doesn't solve your problem but should make it easier to understand what is going on.
The best workaround, if you are using TestExplorer, is to avoid Explicit and to set up categories, which control the running of the tests. Fortunately, TestExplorer and NUnit have pretty much the same understanding of categories.
You can filter tests first in Test Explorer, then the "Run All" button is scoped to only those tests in the Test Explorer panel. If you desire to run all tests except those in a certain project, enter this into the test filter search box in the Test Explorer panel:
-project:NameOfTheTestProject
If your test project has space characters in the name:
-project:"Name of the test project"
Note the - character before the word project. This is crucial. This character tells Visual Studio Test Explorer to invert your filter criteria — it is used to exclude tests in a certain project.
If you have these UI tests intermingled with other kinds of tests, obviously filtering out an entire project is not desirable. Consider placing the UI tests in a specific folder and filter on the FullName instead. For example, if your SpecFlow tests exist in a folder called "Specs", then omit those tests using:
-FullName:Specs
Again, include the - character (with no spaces) before the FUllName criteria to exclude tests in that namespace. This works with SpecFlow because the namespaces for C# classes generated from .feature files follow the folder structure of your test project, just like adding a .cs file does.
This technique should work with any unit test provider, not just NUnit.
Question:
Can I use separate unit test runner for subgroup of unit tests in my build? All of those that would need separate runner process are contained within single .dll
Or at least is it possible to specify order of unit tests?
Background:
I have some unit tests that are testing integration with native components that makes the process memory dirty and so in production code I recycle my process after using them. (it's python integration for .net and some packages are not designed for python engine unload and reload).
However the unit tests are only isolated by app domains - so they still remain in same process and can colide.
You can use [TestCategory] NUnit attribute to create different test group. After grouping, you could run only specific group from TeamCity server. You could also divide it into different steps.
But also as a variant, you could use [OneTimeSetUp] and [OneTimeTearDown] attributes.
Useful links:
https://msdn.microsoft.com/en-us/library/dd286683.aspx - description for TestCategory attribute.
http://nunit.org/docs/2.5/consoleCommandLine.html - how you can run your test categories from nunit-console.
https://confluence.jetbrains.com/display/TCD9/Getting+Started+with+NUnit#GettingStartedwithNUnit-Case1.CommandLine - how you could use nunit-console inside team city.
Second approach:
https://github.com/nunit/docs/wiki/OneTimeSetUp-Attribute
https://github.com/nunit/docs/wiki/OneTimeTearDown-Attribute
Turns out TeamCity supports separation of test assemblies by individual test runner processes - option called 'Run process per assembly' in NUnit build step configuration:
More details here: https://confluence.jetbrains.com/display/TCD10/NUnit (search 'Run process per assembly')
I am trying to run tests a second time programmatically, that exist in the same project.
Because if a SetUp of tests was failed then tests and SetUp will run again.
I use Selenium Webdriver to run tests and a lot of then has SetUp pre-actions. For example: a pre-action is to buy a ticket and then a set of tests are run to check if all is OK.
Often my SetUp pre-action fails (error occurred) and then no test are run. I would like to give them a second change, that is, to run them again.
A great solution that might work for me is this SO question, but it requires NUnit 3.
There is no NUnit.Core package in NUnit 3 so the solution shows a lot of errors. I'm also not sure that I have to use it if my tests methods are placed in the same project (so maybe I don't need to load an assembly).
Any ideas of how to run a test method from code if this method is placed in the same project?
The missing methods and assemblies are part of the internal implementation of NUnit 2.6.4 and were never intended for public use. The solution you point to is only "great" while it works, but because it uses internal classes and methods, there are no guarantees.
For NUnit 3.0, you are in luck because there is now a public API for use in running tests. By using a public / published API, you gain a guarantee that we will keep it working in the future.
The documentation for the API is now located at Test-Engine-API. Here is a simple example of its use...
ITestEngine engine = TestEngineActivator.CreateInstance();
TestPackage package = new TestPackage("path/to/assembly");
ITestRunner runner = engine.GetRunner(package);
XmlNode result = runner.Run(this, TestFilter.Empty);
You should reference the nunit.engine.api assembly (not nunit.engine) in order to use this. Your class making these calls is assumed to implement ITestEventListener.
I have a few test projects, one of which has many unit tests, all very related to doing certain calculations. Running all unit tests takes a long time. So, I'm looking for a way to easily split tests into nightly unit tests and all other unit tests. How can I conveniently specify which test fixtures and test methods should be nightly? In other words, is there a way to split tests from within the same project into separate assemblies?
What I've thought of and tried so far:
Just separate the nightly tests into their own project manually and only run that new project on a nightly basis. This would mean moving files around in the project anytime I wanted to make a test fixture run nightly or regularly, i.e. not desirable.
Come up with a build script that separates these tests at compile time into separate projects. Then I'd end up with separate assemblies. This seems too complicated, but maybe it's the only option.
I would love to be able to use a class attribute to specify which tests will run nightly, e.g.:
[TestClass, Nightly]
public class MyTestClass { }
Any ideas?
You can have as many unit test projects for the same solution as you need.
If you want to go with separate project (i.e. your build system only setup to run test from separate project) I'd have new project sharing tests from old one - create new project and add files from old one as "link" (there is small option on "Open" button of add file dialog to do so).
Alternatively you can add attributes to tests (i.e. TestCategoryAttribute) and make your test runner respect those.
I try to use NUnit with SnippetCompiler http://www.sliver.com/dotnet/SnippetCompiler/
I have added references to nunit.framework.dll in snippetcompiler (Menu Tools, References) and compiled nunit sample http://www.nunit.org/index.php?p=quickStart&r=2.5.2 to bank.dll
but when I open bank.dll in NUnit GUI it fails saying it cannot load NUnit assembly or one of its dependencies.
Is it possible to fix this ?
I couldn't even get v2.0.8.3 of SnippetCompiler to include the reference. It let me do it, but it wouldn't compile.
In any case writing unit tests isn't the purpose of SnippetCompiler. It's designed to do quick spikes - try something and see if it works. In other words, it's throwaway code.
In addition, the version for .NET 3.5 (the one I'm using) is an alpha release; the developer does not seem to be maintaining this. (Not to put down the author - this was an awesome tool that saved me lots of time!)
For writing spikes against modern versions of .NET, I've switched to LINQPad. Change Edit/Preferences/Query to C# Program and it's very similar to SnippetCompiler. The basic version is free; for a small fee, the registered version provides IntelliSense.
Even if you're writing learning tests with NUnit, you'll want to preserve those tests. Use Visual Studio (or another IDE) and create a separate Class Library project for your tests.
Maybe this is the solution
http://weblogs.asp.net/rosherove/archive/2008/02/21/ad-hoc-unit-tests-with-snippet-compiler.aspx
Ad hoc Unit Tests with Snippet
Compiler
If you're a fan of snippet compiler
(if you're not you should seriously
check it out)Travis Illig published a
little template for writing Typemock
Isolator test inside this handy little
tool.
the reasons you'd need a specialized
template in the first place to write
these kinds of tests in snippet
compiler:
1) Typemock Isolator uses the .NET
profiling APIs to work its magic, so
the .net process running your tests
need to have a couple of environment
variables enabled to work
2) His code template actually creates
and runs a new process that triggers
nunit-console.exe with the path of the
current code you just wrote in snippet
compiler allowing you to effecively
write and run unit tests in snippet
compiler!
3) the nunit-console process will
already have the env. vars as
mentioned in the first item set to it.
Travis' template will work for
anything nunit can run, not just
typemock isolator tests, which is
pretty cool.