I've gone through tutorials on writing unit tests with a "Test" project for my .net 4.5 app but I don't understand how they are running.
When I run unit tests that test specific methods of classes in my projects are the entire projects that the methods are recompiled for each test?
I'm confused about how the tests interact with the methods they are testing. Are the tests themselves compiled? I guess they would have to be because it's c#. Is there a separate binary for the tests?
You actually can't run unit tests independently. Typically they will be executed by a "test runner," e.g. the built-in MSTest test running or an NUnit test runner adapter. The process works like this:
You write the test, marking test classes with a specific attribute, e.g. [TestClass] or [TestFixture].
The test classes are compiled into a DLL.
You start your test runner using menu items in Visual Studio (or it is started as part of the automated build process).
The test runner loads the unit test DLL.
The test runner loads all of the types in the DLL and uses reflection to locate the types (classes) that have the magic attibute that indicates that it contains tests.
The test runner instantiates the test class/fixture.
The test runner iterates over the methods in the DLL that have a [Test] or [TestMethod] attribute.
The test runner invokes each method using Invoke().
The test runner displays the results of its test in the test runner UI, or outputs it to a report.
Related
I have an NUnit test project called Tests. Within this project I have a number of files with their own class that run separate tests, including UnitTests.cs, IntegrationTests.cs, etc.
When I run dotnet test project I want only the unit tests within UnitTests.cs to be run, not the IntegrationTests.cs.
I want to avoid using command line arguments to ensure this. Is there anything I can write in the source code of IntegrationTests.cs to exclude it from dotnest test, whilst still being able to run them manually from the IDE?
Question:
Can I use separate unit test runner for subgroup of unit tests in my build? All of those that would need separate runner process are contained within single .dll
Or at least is it possible to specify order of unit tests?
Background:
I have some unit tests that are testing integration with native components that makes the process memory dirty and so in production code I recycle my process after using them. (it's python integration for .net and some packages are not designed for python engine unload and reload).
However the unit tests are only isolated by app domains - so they still remain in same process and can colide.
You can use [TestCategory] NUnit attribute to create different test group. After grouping, you could run only specific group from TeamCity server. You could also divide it into different steps.
But also as a variant, you could use [OneTimeSetUp] and [OneTimeTearDown] attributes.
Useful links:
https://msdn.microsoft.com/en-us/library/dd286683.aspx - description for TestCategory attribute.
http://nunit.org/docs/2.5/consoleCommandLine.html - how you can run your test categories from nunit-console.
https://confluence.jetbrains.com/display/TCD9/Getting+Started+with+NUnit#GettingStartedwithNUnit-Case1.CommandLine - how you could use nunit-console inside team city.
Second approach:
https://github.com/nunit/docs/wiki/OneTimeSetUp-Attribute
https://github.com/nunit/docs/wiki/OneTimeTearDown-Attribute
Turns out TeamCity supports separation of test assemblies by individual test runner processes - option called 'Run process per assembly' in NUnit build step configuration:
More details here: https://confluence.jetbrains.com/display/TCD10/NUnit (search 'Run process per assembly')
I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.
My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?
Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?
Important note
MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.
Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.
When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.
Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.
Visual Studio 2015 and earlier.
MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:
Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.
In Visual Studio 2015 update 1 and later
Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.
It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.
I am wondering if I can run my NUnit tests in parallel, I'm using the R# test runner and there is the option of running assemblies in parallel.
I have a Test Project with all of the tests and they are all 'grouped' and ordered by namespace. Is there any ability to be able to run tests in namespaces in parallel?
Example of what I mean by 'grouping'
I would go with the NUnit console and that gives you some commandline options to run the tests by specifying TestFixtures
nunit /fixture:NUnit.Tests.AssertionTests nunit.tests.dll
to load only the NUnit.Tests.AssertionTests in the nunit.tests.dll assembly. The name specified after the /fixture option may be that of a TestFixture class, or a namespace
I have never personally tested parallel execution with commandline but theoretically it can be done
See this
We are having almost 20000 unit test written for project and if I make any change which can fail existing unit test. Now there are almost 100 unit test are failing and I have list of unit test case failing.
Question is there any way I can add all those 100 failing unit test cases either resharper unit test session or to Nunit.
Is there any nunit adapter or resharper extension that will allow me to give list of unit test case names either from file like csv file add it to nunit or resharper unit session?
Those failing unit test cases can be from different category. I need to manually run or add to unit test session.
You could use NUnit console,
nunit-console /runlist:testlist.txt nunit.tests.dll
Refer : http://nunit.org/index.php?p=consoleCommandLine&r=2.6.3