I have a question on running UnitTests sequentially. Unfortunately in scenario it is not an option to run them parallel or mock the database. The project is written in .NET core 3.1 and the UnitTests need to execute database operations before and after a Unittest has run.
After reading https://www.meziantou.net/mstest-v2-execute-tests-in-parallel.htm and a lot of other articles about sequential UnitTesting I came up with this (simplified):
BaseClass:
namespace XY.Test
{
[TestClass]
public class BaseTest: TimerModel
{
private static readonly DbCreator Creator = new DbCreator();
public static readonly DbConnectionManager ConnectionManager = new DbConnectionManager();
[TestInitialize]
public void BaseTestInitialize()
{
CreateTestData();
}
[TestCleanup]
public void BaseTestCleanup()
{
RemoveTestData();
}
public void CreateTestData()
{
RemoveTestData();
Creator.ExecuteSqlFromScript(ConnectionManager, #"Resources\CreateTestData.sql");
}
public void RemoveTestData()
{
Creator.ExecuteSqlFromScript(ConnectionManager, #"Resources\EmptyTestDataTables.sql");
}
}
}
TestClass:
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] //<-- Also tried out Workers = 1 and Scope = ExecutionScope.MethodLevel
namespace XY.Test.Models
{
[TestClass]
public class TerminalConfigModelTest: BaseTest
{
[TestMethod]
[DoNotParallelize]
public void TestMethod1()
{
...
}
[TestMethod]
[DoNotParallelize]
public void TestMethod2()
{
...
}
}
}
For some reason, no matter what I do, the UnitTests are being executed parallel. What do I have to change in order to have them executed sequentially?
When I execute all tests in the test class, the TestInitialize of the base class is called twice before the TestCleanup is run. This causes the CreateTestData method to fail as indexes prevent a double insert of the test data.
What I would expect:
TestInitialize1 is called
TestMethod1 is executed
TestCleanup1 is called
TestInitialize2 is called
TestMethod2 is executed
TestCleanup2 is called
...
What happens:
TestInitialize1 is called
TestMethod1 is executed
TestInitialize2 is called before TestCleanup1 is called
TestMethod2 execution fails
Am I missunderstanding the [DoNotParallelize] option?
Paralelism isn't the problem here, my tests are definitely sequential and [ClassCleanup] also screwed me over. It's just unintuitive and weird, more info here.
I wanted to use ordered tests but it seems it's a legacy functionality only in MSTest-v1 and new versions of Visual Studio don't even support it.
Best thing i can tell you now is just don't use [ClassCleanup].
Use [TestCleanup] or [AssemblyCleanup].
I'm working on a project where I have to use something like soft assertion in NUnit in C#.
I need to implement a feature which says that if a test failed, go to the next test without showing exception but note that this test was failed.
I know that it's inadvisable to use multiple asserts but it's necessary because I got a form where f.e field with surname can fail but next tests are independent and should still run.
public class SoftAssertionTest
{
public static void AreEqual(object expected, object actual)
{
try
{
Assert.AreEqual(expected, actual);
}
catch (Exception e)
{
//Catch exception but remember that test failed and got to the
//next
}
}
}
Expected result is that all tests run without exceptions but finally result show Fail status.
[TestFixture]
public class TestClass
{
[Test]
public static void Test()
{
SoftAssertionTest.AreEqual(1, 2);
SoftAssertionTest.AreEqual(3, 4);
}
}
Any ideas?
NUnit 3 has Assert.Warn if you just want to warn on failures and Assert.Multiple if you want to run multiple asserts and fail the test if any of the individual asserts fail, but ensure that all of the asserts are run.
Try SoftAssertion
SoftAssert softAssert = new SoftAssert();
softAssert.True(false);
softAssert.False(true);
softAssert.NotNull(null);
softAssert.VerifyAll();
Good morning,
As context, I am working in an automated test suite using nUnit 3.0.1 (with Selenium). The tests are being runned directly from Visual Studio 2013' Test Explorer.
My problem is this: There is some Setting up functionality that I am trying to make automatic when starting to run the test suite. These are things that need to be done only ONCE for all the suite of tests, so using the [SetUp] Attribute for each test class is not what I intend.
When I run the Tests however, my setting up is not even touched :(. Debugging confirms that the setting up is not being done.
So far I have tried using: [SetUpFixture] along with [OneTimeSetUp] or [SetUp] (in many combinations), also tried removing the namespace. I am still not getting what I need.
I am fairly new with nUnit, so I would like some guidance about what could be wrong, or some alternatives to making sure the scripted setup is run.
I hope you followed this pattern:
In the below MySetupClass which defines the Test Suite setup and teardown methods which are executed only once at the start and end of the test runs.
Refer SetupFixture documentation.
MySetupClass.cs
namespace NUnit.Tests
{
using System;
using NUnit.Framework;
[SetUpFixture]
public class MySetUpClass
{
[OneTimeSetUp]
public void RunBeforeAnyTests()
{
Console.WriteLine("SetupFixture - OneTimeSetup");
}
[OneTimeTearDown]
public void RunAfterAllTests()
{
Console.WriteLine("Suite TearDown - OneTimeTearDown");
}
}
}
In the following SuccessTests.cs, Setup and TearDown are executed before and after each test, since these methods are defined in a class marked with atttribute TestFixture. OneTimeSetup & OneTimeTearDown attributes defines methods in the test case to be executed once before and after all the test methods in the class.
Refer to the documentation of Setup attribute [here][2].
SuccessTests.cs
namespace NUnit.Tests
{
using System;
using NUnit.Framework;
[TestFixture]
public class SuccessTests
{
[SetUp]
public void SetUp()
{ Console.WriteLine("Test Setup"); }
[TearDown]
public void TearDown()
{ Console.WriteLine("Test Teardown"); }
[OneTimeSetup]
public void OneTimeSetup()
{ Console.WriteLine("Test Fixture - OneTimeSetup"); }
[OneTimeTearDown]
public void OneTimeTearDown()
{ Console.WriteLine("Test Fixture - OneTimeTearDown"); }
[Test]
public void Test1()
{ Console.WriteLine("Actual Test1"); }
[Test]
public void Test2()
{ Console.WriteLine("Actual Test2"); }
}
}
Running the above tests will give this result.
Suite Setup - OneTimeSetup
Test Fixture - OneTimeSetup
Test Setup
Actual Test1
Test Teardown
Test Setup
Actual Test2
Test Teardown
Test Fixture TearDown - OneTimeTearDown
Suite TearDown - OneTimeTearDown
I have a simple NUnit project in C#. I have a file TestFixture.cs that successfully runs its four tests. If I want to add additional tests in a new CS file, how do I do this? If I simply copy the code from TestFixture.cs into a new TestFixture2.cs and rename the TestFixture classes to TestFixture3 and 4 respectively, they do not execute. I would think they would automatically run. Do I need to tell the runner specifically if it is a file other than TestFixture.cs? Here is my code:
namespace NUnitTest1
{
[TestFixture]
public class TestFixture3
{
[Test]
public void TestTrue()
{
Assert.IsTrue(true);
}
// This test fail for example, replace result or delete this test to see all tests pass
[Test]
public void TestFault()
{
Assert.IsTrue(true);
}
}
[TestFixture]
public class TestFixture4
{
[Test]
public void TestTrue()
{
Assert.IsTrue(true);
}
// This test fail for example, replace result or delete this test to see all tests pass
[Test]
public void TestFault()
{
Assert.IsTrue(true);
}
}
}
My unit test runs by executing a command line program using the following code:
namespace NUnitTest1
{
class Program
{
[STAThread]
static void Main(string[] args)
{
string[] my_args = { Assembly.GetExecutingAssembly().Location };
int returnCode = NUnit.ConsoleRunner.Runner.Main(my_args);
if (returnCode != 0)
Console.Beep();
}
}
}
NUnit will automatically get all types marked with TestFixture attribute from test assembly when you load it (it does not matter whether fixtures in one .cs file or in separate). Then NUnit will search for methods marked with Test or TestCase attributes and load them. If NUnit don't see some test, then make sure you loaded latest version of your test assembly.
NOTE: If you are using NUnit test runner, then there is nice setting Reload when test assembly changes. With this option turned on NUnit will automatically reload test assembly when you rebuild it in Visual Studio.
By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?
I just want to point out that while most of the responders assumed these were unit tests, the question did not specify that they were.
nUnit is a great tool that can be used for a variety of testing situations. I can see appropriate reasons for wanting to control test order.
In those situations I have had to resort to incorporating a run order into the test name. It would be great to be able to specify run order using an attribute.
NUnit 3.2.0 added an OrderAttribute, see:
https://github.com/nunit/docs/wiki/Order-Attribute
Example:
public class MyFixture
{
[Test, Order(1)]
public void TestA() { ... }
[Test, Order(2)]
public void TestB() { ... }
[Test]
public void TestC() { ... }
}
Your unit tests should each be able to run independently and stand alone. If they satisfy this criterion then the order does not matter.
There are occasions however where you will want to run certain tests first. A typical example is in a Continuous Integration situation where some tests are longer running than others. We use the category attribute so that we can run the tests which use mocking ahead of the tests which use the database.
i.e. put this at the start of your quick tests
[Category("QuickTests")]
Where you have tests which are dependant on certain environmental conditions, consider the TestFixtureSetUp and TestFixtureTearDown attributes, which allow you to mark methods to be executed before and after your tests.
Wanting the tests to run in a specific order does not mean that the tests are dependent on each other - I'm working on a TDD project at the moment, and being a good TDDer I've mocked/stubbed everything, but it would make it more readable if I could specify the order which the tests results are displayed - thematically instead of alphabetically. So far the only thing I can think of is to prepend a_ b_ c_ to classes to classes, namespaces and methods. (Not nice) I think a [TestOrderAttribute] attribute would be nice - not stricly followed by the framework, but a hint so we can achieve this
Regardless of whether or not Tests are order dependent... some of us just want to control everything, in an orderly fashion.
Unit tests are usually created in order of complexity. So, why shouldn't they also be run in order of complexity, or the order in which they were created?
Personally, I like to see the tests run in the order of which I created them. In TDD, each successive test is naturally going to be more complex, and take more time to run. I would rather see the simpler test fail first as it will be a better indicator as to the cause of the failure.
But, I can also see the benefit of running them in random order, especially if you want to test that your tests don't have any dependencies on other tests. How about adding an option to test runners to "Run Tests Randomly Until Stopped"?
I am testing with Selenium on a fairly complex web site and the whole suite of tests can run for more than a half hour, and I'm not near to covering the entire application yet. If I have to make sure that all previous forms are filled in correctly for each test, this adds a great deal of time, not just a small amount of time, to the overall test. If there's too much overhead to running the tests, people won't run them as often as they should.
So, I put them in order and depend on previous tests to have text boxes and such completed. I use Assert.Ignore() when the pre-conditions are not valid, but I need to have them running in order.
I really like the previous answer.
I changed it a little to be able to use an attribute to set the order range:
namespace SmiMobile.Web.Selenium.Tests
{
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using NUnit.Framework;
public class OrderedTestAttribute : Attribute
{
public int Order { get; set; }
public OrderedTestAttribute(int order)
{
Order = order;
}
}
public class TestStructure
{
public Action Test;
}
class Int
{
public int I;
}
[TestFixture]
public class ControllingTestOrder
{
private static readonly Int MyInt = new Int();
[TestFixtureSetUp]
public void SetUp()
{
MyInt.I = 0;
}
[OrderedTest(0)]
public void Test0()
{
Console.WriteLine("This is test zero");
Assert.That(MyInt.I, Is.EqualTo(0));
}
[OrderedTest(2)]
public void ATest0()
{
Console.WriteLine("This is test two");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
}
[OrderedTest(1)]
public void BTest0()
{
Console.WriteLine("This is test one");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
}
[OrderedTest(3)]
public void AAA()
{
Console.WriteLine("This is test three");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
}
[TestCaseSource(sourceName: "TestSource")]
public void MyTest(TestStructure test)
{
test.Test();
}
public IEnumerable<TestCaseData> TestSource
{
get
{
var assembly =Assembly.GetExecutingAssembly();
Dictionary<int, List<MethodInfo>> methods = assembly
.GetTypes()
.SelectMany(x => x.GetMethods())
.Where(y => y.GetCustomAttributes().OfType<OrderedTestAttribute>().Any())
.GroupBy(z => z.GetCustomAttribute<OrderedTestAttribute>().Order)
.ToDictionary(gdc => gdc.Key, gdc => gdc.ToList());
foreach (var order in methods.Keys.OrderBy(x => x))
{
foreach (var methodInfo in methods[order])
{
MethodInfo info = methodInfo;
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
object classInstance = Activator.CreateInstance(info.DeclaringType, null);
info.Invoke(classInstance, null);
}
}).SetName(methodInfo.Name);
}
}
}
}
}
}
I know this is a relatively old post, but here is another way to keep your test in order WITHOUT making the test names awkward. By using the TestCaseSource attribute and having the object you pass in have a delegate (Action), you can totally not only control the order but also name the test what it is.
This works because, according to the documentation, the items in the collection returned from the test source will always execute in the order they are listed.
Here is a demo from a presentation I'm giving tomorrow:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NUnit.Framework;
namespace NUnitTest
{
public class TestStructure
{
public Action Test;
}
class Int
{
public int I;
}
[TestFixture]
public class ControllingTestOrder
{
private static readonly Int MyInt= new Int();
[TestFixtureSetUp]
public void SetUp()
{
MyInt.I = 0;
}
[TestCaseSource(sourceName: "TestSource")]
public void MyTest(TestStructure test)
{
test.Test();
}
public IEnumerable<TestCaseData> TestSource
{
get
{
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test one");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
}
}).SetName(#"Test One");
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test two");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
}
}).SetName(#"Test Two");
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test three");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
}
}).SetName(#"Test Three");
}
}
}
}
I am working with Selenium WebDriver end-to-end UI test cases written in C#, which are run using NUnit framework. (Not unit cases as such)
These UI tests certainly depend on order of execution, as other test needs to add some data as a precondition. (It is not feasible to do the steps in every test)
Now, after adding the 10th test case, i see NUnit wants to run in this order:
Test_1
Test_10
Test_2
Test_3
..
So i guess i have to too alphabetisize the test case names for now, but it would be good to have this small feature of controlling execution order added to NUnit.
There are very good reasons to utilise a Test Ordering mechanism. Most of my own tests use good practice such as setup/teardown. Others require huge amounts of data setup, which can then be used to test a range of features. Up till now, I have used large tests to handle these (Selenium Webdriver) integration tests. However, I think the above suggested post on https://github.com/nunit/docs/wiki/Order-Attribute has a lot of merit. Here is an example of why ordering would be extremely valuable:
Using Selenium Webdriver to run a test to download a report
The state of the report (whether it's downloadable or not) is cached for 10 minutes
That means, before every test I need to reset the report state and then wait up to 10 minutes before the state is confirmed to have changed, and then verify the report downloads correctly.
The reports cannot be generated in a practical / timely fashion through mocking or any other mechanism within the test framework due to their complexity.
This 10 minute wait time slows down the test suite. When you multiply similar caching delays across a multitude of tests, it consumes a lot of time. Ordering tests could allow data setup to be done as a "Test" right at the beginning of the test suite, with tests relying on the cache to bust being executed toward the end of the test run.
Usually Unit Test should be independent, but if you must, then you can name your methods in alphabetical order ex:
[Test]
public void Add_Users(){}
[Test]
public void Add_UsersB(){}
[Test]
public void Process_Users(){}
or you can do..
private void Add_Users(){}
private void Add_UsersB(){}
[Test]
public void Process_Users()
{
Add_Users();
Add_UsersB();
// more code
}
This question is really old now, but for people who may reach this from searching, I took the excellent answers from user3275462 and PvtVandals / Rico and added them to a GitHub repository along with some of my own updates. I also created an associated blog post with some additional info you can look at for more info.
Hope this is helpful for you all. Also, I often like to use the Category attribute to differentiate my integration tests or other end-to-end tests from my actual unit tests. Others have pointed out that unit tests should not have an order dependency, but other test types often do, so this provides a nice way to only run the category of tests you want and also order those end-to-end tests.
If you are using [TestCase], the argument TestName provides a name for the test.
If not specified, a name is generated based on the method name and the arguments provided.
You can control the order of test execution as given below:
[Test]
[TestCase("value1", TestName = "ExpressionTest_1")]
[TestCase("value2", TestName = "ExpressionTest_2")]
[TestCase("value3", TestName = "ExpressionTest_3")]
public void ExpressionTest(string v)
{
//do your stuff
}
Here I used the method name "ExpressionTest" suffix with a number.
You can use any names ordered alphabetical
see TestCase Attribute
I'm suprised the NUnit community hasn't come up with anything, so I went to create something like this myself.
I'm currently developing an open-source library that allows you to order your tests with NUnit. You can order test fixtures and ordering "ordered test specifications" alike.
The library offers the following features:
Build complex test ordering hierarchies
Skip subsequent tests if an test in order fails
Order your test methods by dependency instead of integer order
Supports usage side-by-side with unordered tests. Unordered tests are executed first.
The library is actually inspired in how MSTest does test ordering with .orderedtest files. Please look at an example below.
[OrderedTestFixture]
public sealed class MyOrderedTestFixture : TestOrderingSpecification {
protected override void DefineTestOrdering() {
TestFixture<Fixture1>();
OrderedTestSpecification<MyOtherOrderedTestFixture>();
TestFixture<Fixture2>();
TestFixture<Fixture3>();
}
protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
}
You should not depend on the order in which the test framework picks tests for execution. Tests should be isolated and independent. In that they should not depend on some other test setting the stage for them or cleaning up after them. They should also produce the same result irrespective of the order of the execution of tests (for a given snapshot of the SUT)
I did a bit of googling. As usual, some people have resorted to sneaky tricks (instead of solving the underlying testability/design issue
naming the tests in an alphabetically ordered manner such that tests appear in the order they 'need' to be executed. However NUnit may choose to change this behavior with a later release and then your tests would be hosed. Better check in the current NUnit binaries to Source Control.
VS (IMHO encouraging the wrong behavior with their 'agile tools') has something called as "Ordered tests" in their MS testing framework. I didn't waste any time reading up but it seems to be targeted towards the same audience
See Also: characteristics of a good test
In case of using TestCaseSource the key is to override string ToString method, How that works:
Assume you have TestCase class
public class TestCase
{
public string Name { get; set; }
public int Input { get; set; }
public int Expected { get; set; }
}
And a list of TestCases:
private static IEnumerable<TestCase> TestSource()
{
return new List<TestCase>
{
new TestCase()
{
Name = "Test 1",
Input = 2,
Expected = 4
},
new TestCase()
{
Name = "Test 2",
Input = 4,
Expected = 16
},
new TestCase()
{
Name = "Test 3",
Input = 10,
Expected = 100
}
};
}
Now lets use it with a Test method and see what happen:
[TestCaseSource(nameof(TestSource))]
public void MethodXTest(TestCase testCase)
{
var x = Power(testCase.Input);
x.ShouldBe(testCase.Expected);
}
This will not test in order and the output will be like this:
So if we added override string ToString to our class like:
public class TestCase
{
public string Name { get; set; }
public int Input { get; set; }
public int Expected { get; set; }
public override string ToString()
{
return Name;
}
}
The result will change and we get the order and name of test like:
Note:
This is just example to illustrate how to get name and order in test, the order is taken numerically/alphabetically so if you have more than ten test I suggest making Test 01, Test 02.... Test 10, Test 11 etc because if you make Test 1 and at some point Test 10 than the order will be Test 1, Test 10, Test 2 .. etc.
The input and Expected can be any type, string, object or custom class.
Beside order, the good thing here is that you see the test name which is more important.