I have a few test projects, one of which has many unit tests, all very related to doing certain calculations. Running all unit tests takes a long time. So, I'm looking for a way to easily split tests into nightly unit tests and all other unit tests. How can I conveniently specify which test fixtures and test methods should be nightly? In other words, is there a way to split tests from within the same project into separate assemblies?
What I've thought of and tried so far:
Just separate the nightly tests into their own project manually and only run that new project on a nightly basis. This would mean moving files around in the project anytime I wanted to make a test fixture run nightly or regularly, i.e. not desirable.
Come up with a build script that separates these tests at compile time into separate projects. Then I'd end up with separate assemblies. This seems too complicated, but maybe it's the only option.
I would love to be able to use a class attribute to specify which tests will run nightly, e.g.:
[TestClass, Nightly]
public class MyTestClass { }
Any ideas?
You can have as many unit test projects for the same solution as you need.
If you want to go with separate project (i.e. your build system only setup to run test from separate project) I'd have new project sharing tests from old one - create new project and add files from old one as "link" (there is small option on "Open" button of add file dialog to do so).
Alternatively you can add attributes to tests (i.e. TestCategoryAttribute) and make your test runner respect those.
Related
Question:
Can I use separate unit test runner for subgroup of unit tests in my build? All of those that would need separate runner process are contained within single .dll
Or at least is it possible to specify order of unit tests?
Background:
I have some unit tests that are testing integration with native components that makes the process memory dirty and so in production code I recycle my process after using them. (it's python integration for .net and some packages are not designed for python engine unload and reload).
However the unit tests are only isolated by app domains - so they still remain in same process and can colide.
You can use [TestCategory] NUnit attribute to create different test group. After grouping, you could run only specific group from TeamCity server. You could also divide it into different steps.
But also as a variant, you could use [OneTimeSetUp] and [OneTimeTearDown] attributes.
Useful links:
https://msdn.microsoft.com/en-us/library/dd286683.aspx - description for TestCategory attribute.
http://nunit.org/docs/2.5/consoleCommandLine.html - how you can run your test categories from nunit-console.
https://confluence.jetbrains.com/display/TCD9/Getting+Started+with+NUnit#GettingStartedwithNUnit-Case1.CommandLine - how you could use nunit-console inside team city.
Second approach:
https://github.com/nunit/docs/wiki/OneTimeSetUp-Attribute
https://github.com/nunit/docs/wiki/OneTimeTearDown-Attribute
Turns out TeamCity supports separation of test assemblies by individual test runner processes - option called 'Run process per assembly' in NUnit build step configuration:
More details here: https://confluence.jetbrains.com/display/TCD10/NUnit (search 'Run process per assembly')
I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.
My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?
Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?
Important note
MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.
Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.
When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.
Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.
Visual Studio 2015 and earlier.
MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:
Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.
In Visual Studio 2015 update 1 and later
Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.
It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.
I'm working with Unit Test using VS2013 Professional. In particular I'm using the NUnit framework (NUnit TestAdapter for VS2013). My problem is that when I run my tests then VS starts building all the projects inside the solution. Currently the Unit Test project does not reference any solution project.
If I simply code a single test method like:
[Test]
public void SimpleTestMethod(){
Assert.That("a", Is.EqualTo("a"));
}
and the Unit Test project is in a Solution with N project, when I run my test then VS will build all N-1 project... In my case this behavior is boring because it takes too much time (the solution contains many projects) and some projects contain errors.
Is there a way to run my SimpleTestMethod() without complete solution building?
Break your test project to multiple projects that only reference a subset of the solution's projects.
This is also good test housekeeping - have a separate unit test project for each solution project instead of one huge project with dependencies to anything else. There are several advantages to this:
Tests run faster
It's a lot easier to isolate test cases, especially configuration settings
You can version projects and their test cases together
A good naming practice is to name your test projects the same as their target projects with a .Tests suffix. You can also create a solution folder (not a real folder) called "Tests" and move the test projects in it.
As for the why: Test runners use the Unit Test assembly and its dependencies to run their tests. If any of the assembly's dependencies change, the assembly and the dependencies have to be rebuilt. Visual Studio doesn't know what the external tool will call so it has to build all changed assemblies and their dependents.
If the build fails, there are no valid assemblies for the test runner to use so VS has to rebuild the entire solution before the runner can work. In this case, the obvious solution is to fix the error.
There are some stopgap options you can use until you can fix the error:
Temporarily remove the broken project from the build configuration
Split the solution so that you have a solution that can be built and tested
I struggled with this for a very long time as well. I actually hated the automatic build process, even when everything was successful.
I started to run tests through the command line. No build process is necessary. You can write your own .bat files and keep logs of test results. There are plenty of command line parameters that can be added to customize for what you are looking for.
https://msdn.microsoft.com/en-us/library/jj155796.aspx
I'm doing some refactoring on a big project that have two types of testing projects: Unit and Integration. The Unit testing projects are supposed to have tests that inherits UnitTest class, and the Integration testing projects, tests that inherit IntegrationTest. The problem now, is that in "Unit" there are some files that are "Integration" tests, and vice versa. Do you know a way to "easily" move files and keep the configuration in a way that the tests still runs?
I created a batch, and it moved the files correctly. The problem is that the .csproj doesn't update automatically and also that the're some classes that are not tests, but that are used by the tests. (Mock up classes for example).
What i want is a project with several files in it. Each file has a set of tests. Then I want to be able to run each of those tests.
So right now I have a project but I only have one file: Project.cs. When i run the test through the console/GUI All them tests from Project.cs are run (or a subset if i use the /fixture command).
What I would like (or would like to know if its possible) would be the following.
Have a Project called Tests. In that I have several files FileTests.cs, ControlTests.cs, MiscTests.cs, etc each of these have various tests. Then there is also a Program.cs. I would be able to run the Tests project and choose which of the FileTests, ControlTests, etc I run.
Is that possible?
Cheers
Sounds like you are looking for NUnit test categories. You can attribute you test with the correct category and then run just tests belonging to that category. This gives you much more flexibility than having to define them physically as files or assemblies.