Question:
Can I use separate unit test runner for subgroup of unit tests in my build? All of those that would need separate runner process are contained within single .dll
Or at least is it possible to specify order of unit tests?
Background:
I have some unit tests that are testing integration with native components that makes the process memory dirty and so in production code I recycle my process after using them. (it's python integration for .net and some packages are not designed for python engine unload and reload).
However the unit tests are only isolated by app domains - so they still remain in same process and can colide.
You can use [TestCategory] NUnit attribute to create different test group. After grouping, you could run only specific group from TeamCity server. You could also divide it into different steps.
But also as a variant, you could use [OneTimeSetUp] and [OneTimeTearDown] attributes.
Useful links:
https://msdn.microsoft.com/en-us/library/dd286683.aspx - description for TestCategory attribute.
http://nunit.org/docs/2.5/consoleCommandLine.html - how you can run your test categories from nunit-console.
https://confluence.jetbrains.com/display/TCD9/Getting+Started+with+NUnit#GettingStartedwithNUnit-Case1.CommandLine - how you could use nunit-console inside team city.
Second approach:
https://github.com/nunit/docs/wiki/OneTimeSetUp-Attribute
https://github.com/nunit/docs/wiki/OneTimeTearDown-Attribute
Turns out TeamCity supports separation of test assemblies by individual test runner processes - option called 'Run process per assembly' in NUnit build step configuration:
More details here: https://confluence.jetbrains.com/display/TCD10/NUnit (search 'Run process per assembly')
Related
I have some Specflow(3.9.40)-based UI test using Selenium Webdriver and NUnit(3.13.2) Framework.
I would like these tests to not run when I hit 'Run All' in the Test Explorer.
In my StepDefinition class I have tried adding the Explicit Attribute:
[TestFixture, Explicit("Only run when called explicitly")]
[Binding]
public class LoginStepDefinitions
{ ...
, but for some reason it does not seem to work - The UI Tests still run.
I imagine there may be something I can add to the [BeforeScenario] in my HookInitialization class, but I am stumped:
[BeforeScenario]
public void FirstBeforeScenario()
{
var seleniumDriver = new SeleniumDriver(_scenarioContext);
_scenarioContext.Set(seleniumDriver, "SeleniumDriver");
}
Is there a way for me to tell Visual Studio Test Explorer to ignore the UI Tests or its folder ?
The basic problem is that NUnit's Explicit setting is unique among the frameworks supported by Test Explorer, which knows nothing of this concept of "explicitness."
If Test Explorer calls the NUnit adapter telling it to run all, then NUnit will honor the Explicit setting. However, in some situations, it simply passes a list of all the tests to NUnit. In that case, all tests appear (to NUnit) to have been run explicitly.
Which way Test Explorer calls NUnit is dependent on the version you are running and it's only possible to tell how it calls by debugging or by observing the outcome. To add to the confusion, some versions of the NUnit3 adapter use heuristics to try to guess whether an explicit test was called explicitly. This doesn't solve your problem but should make it easier to understand what is going on.
The best workaround, if you are using TestExplorer, is to avoid Explicit and to set up categories, which control the running of the tests. Fortunately, TestExplorer and NUnit have pretty much the same understanding of categories.
You can filter tests first in Test Explorer, then the "Run All" button is scoped to only those tests in the Test Explorer panel. If you desire to run all tests except those in a certain project, enter this into the test filter search box in the Test Explorer panel:
-project:NameOfTheTestProject
If your test project has space characters in the name:
-project:"Name of the test project"
Note the - character before the word project. This is crucial. This character tells Visual Studio Test Explorer to invert your filter criteria — it is used to exclude tests in a certain project.
If you have these UI tests intermingled with other kinds of tests, obviously filtering out an entire project is not desirable. Consider placing the UI tests in a specific folder and filter on the FullName instead. For example, if your SpecFlow tests exist in a folder called "Specs", then omit those tests using:
-FullName:Specs
Again, include the - character (with no spaces) before the FUllName criteria to exclude tests in that namespace. This works with SpecFlow because the namespaces for C# classes generated from .feature files follow the folder structure of your test project, just like adding a .cs file does.
This technique should work with any unit test provider, not just NUnit.
I have an NUnit test project called Tests. Within this project I have a number of files with their own class that run separate tests, including UnitTests.cs, IntegrationTests.cs, etc.
When I run dotnet test project I want only the unit tests within UnitTests.cs to be run, not the IntegrationTests.cs.
I want to avoid using command line arguments to ensure this. Is there anything I can write in the source code of IntegrationTests.cs to exclude it from dotnest test, whilst still being able to run them manually from the IDE?
I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.
My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?
Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?
Important note
MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.
Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.
When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.
Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.
Visual Studio 2015 and earlier.
MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:
Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.
In Visual Studio 2015 update 1 and later
Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.
It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.
I have a few test projects, one of which has many unit tests, all very related to doing certain calculations. Running all unit tests takes a long time. So, I'm looking for a way to easily split tests into nightly unit tests and all other unit tests. How can I conveniently specify which test fixtures and test methods should be nightly? In other words, is there a way to split tests from within the same project into separate assemblies?
What I've thought of and tried so far:
Just separate the nightly tests into their own project manually and only run that new project on a nightly basis. This would mean moving files around in the project anytime I wanted to make a test fixture run nightly or regularly, i.e. not desirable.
Come up with a build script that separates these tests at compile time into separate projects. Then I'd end up with separate assemblies. This seems too complicated, but maybe it's the only option.
I would love to be able to use a class attribute to specify which tests will run nightly, e.g.:
[TestClass, Nightly]
public class MyTestClass { }
Any ideas?
You can have as many unit test projects for the same solution as you need.
If you want to go with separate project (i.e. your build system only setup to run test from separate project) I'd have new project sharing tests from old one - create new project and add files from old one as "link" (there is small option on "Open" button of add file dialog to do so).
Alternatively you can add attributes to tests (i.e. TestCategoryAttribute) and make your test runner respect those.
I am using ncover code central put on a server and collector on my desktop . now I m able to find the coverage of exe (my appliation) when manual tests or tests on gui by launching the application are executed but i want to profile the dll of my application not exe to find the code coverage using my reference unit Test dlls and posting the coverage on code central.
Please suggest how to do that
When covering unit tests, you need to profile the test runner process, not your application exe. If you are running NUnit, for example:
Create a new project on Code Central.
Go to the "Processes" tab and click "Edit Match Rules"
Add a match rule of Match type="Regex", Process matching="nunit-agent." NUnit ultimately triggers nunit-agent.exe to run the tests, so this is the process you want to cover.
Verify the new project is synced to Collector.
Run your unit tests.