what I have here is my script in selenium c# using NUnit framework with multiple test as you can see below and it's connected to my TestRail and I am wondering if there is any way to check if my test is passed before the actual test is done.
[Test, Order(0)]
public void Login()
{
Navigate.ToLoginPage();
Log.In("user");
TestRailServer.Result("1"); //1 is testcase id
}
[Test, Order(1)]
public void RegisterUser()
{
Add.User();
TestRailServer.Result("2"); //2 is testcase id
}
The TestRailServer.Result("1") you see in the test is my connection in testrail it will decide if the test case in testrail is passed or failed based in the TestMethod(Login and RegisterUser)
Here's my code in TestRail connection
private static TestRailClient trClient;
public static void Result(string testCaseID)
{
ulong testRunID = "testRailID";
string testrailComment;
trClient = new TestRailClient("https://client.testrail.net/",
"username", "password");
var result = TestContext.CurrentContext.Result.Outcome.Status;
var testrailStatus = result switch
{
TestStatus.Failed => ResultStatus.Failed,
TestStatus.Passed => ResultStatus.Passed,
_ => ResultStatus.Retest
};
testrailComment = (testrailStatus == ResultStatus.Failed) ? TestContext.CurrentContext.Result.Message : "Passed";
trClient.AddResultForCase(testRunID, ulong.Parse(testCaseID), testrailStatus, testrailComment);
}
But the problem is the TestRailServer.Result is called before the test is done so I need a way to tell if my test passed before it executes the TestRailServer
Thank for the help if there is any and apology if you find it difficult to undestand
I've not used testrail, but I've used Nunit a fair amount.
My understand is that NUnit only knows if it has failed by doing an assert. If you don't have an assert in your code (and I don't see any at the level you've posted) your tests will always pass on completion. The challenge with your approach is that a failing assert will stop the test and you potentially won't get any result. That's how NUnit is designed - it's for unit tests that should take miliseconds afterall.
Instead of you ending every test line interacting with the reporter, move it to a [Teardown] method. That way the test completes (in a pass or fail state) and reporting is done after the event.
Something like:
public string testId;
[Test, Order(0)]
public void test1()
{
testId= "1";
Navigate.ToLoginPage();
Log.In("user");
//assert here
}
[Test, Order(1)]
public void test2()
{
testId= "2";
Add.User();
//assert here
}
[TearDown]
public void TearDown()
{
TestRailServer.Result(testId);
}
(and add asserts to the tests if you don't already) :-)
I have a Mock object whose method I'm trying to setup by throwing an Exception when executed, for a particular unit test case using the Moq framework.
var mockMySvc = new Mock<IMySvc>();
mockMySvc
.Setup(x=>x.SomeMethod())
.Throws<Exception>();
//Execution of some code
//At the assertions
mockMySvc.VerifyAll();
At runtime, the code complains about all the expections of mockMySvc not having been met despite the exception being thrown. Am I missing something or does the .VerifyAll() method not work with the .Throws() functionality.
I don't know about your way of setting it up, but I always do it this way:
Assert.Throws<Exception>(() => myclass.SomeMethod());
This way you don't need to verify anything.
Based on your comment this is how you make sure the exception is thrown inside the method, so that you can check your code inside catch blocks.
[Test]
public void Test1()
{
_filmService.Setup(f => f.FindById(It.IsAny<int>())).Throws<Exception>();
_filmController.Test();
_filmService.Verify(f => f.Exists(It.IsAny<Film>()), Times.Once);
}
Actual code:
public ActionResult Test()
{
try
{
_filmService.FindById(-1);
}
catch (System.Exception)
{
_filmService.Exists(null);
}
return View();
}
This is just an example I tested in my code and it works correctly.
I write this helloworld unit test and build it. It always have this build error: "The command://...../Nunit.ConsoleRunner3.6/nunit3-console.exe exited with code 1".I search all the stackoverflow but there is no a single thread about Nunit test exist code 1, do you know where can I find the information for this code?
using NUnit.Framework;
namespace SomeName
{
[TestFixture]
class IdentifierTest
{
[SetUp]
public void SetUp()
{
}
[TearDown]
public void TearDown()
{
}
[Test]
public void DetectIDTest()
{
Assert.AreEqual("hi","ho");
}
}
}
A positive exit code indicates the number of tests that has failed (in your case 1) as per this line in the NUnit console runner source code. Additional (negative) failure codes also have meaning per these six lines. This is substantiated by a discussion about exit codes here, I have attached the response as an image for easy reference
I wrote a unit test in such a way that it should throw AnException or AnotherException, both deriving from AnExceptionBaseException. I then proceeded to add an ExpectedExceptionAttribute for the base exception, only to find that my test will still be marked as failed.
Test Name: Call_Should_Throw_If_Device_Is_Not_Ready Test
...
Result Message: Test method
DiskManagementTests.DiskFreeSpaceTests.Call_Should_Throw_If_Device_Is_Not_Ready
threw exception System.IO.FileNotFoundException, but exception
System.IO.IOException was expected. Exception message:
System.IO.FileNotFoundException: The device is not ready. (Exception
from HRESULT: 0x80070015)
This seems like a reasonable design decision because, in this particular case, the exception is generated from an HRESULT return code. That makes it nearly impossible to determine which exception will be thrown. At least not without copying the code logic from the unit that my test is supposed to ...test.
My code (I believe this can throw either FileNotFound or DirectoryNotFound):
[TestMethod]
[ExpectedException(typeof(IOException))]
public void Call_Should_Throw_If_Device_Is_Not_Ready()
{
foreach (DriveInfo drive in DriveInfo.GetDrives().Where(drive => !drive.IsReady))
{
DiskFreeSpace diskFreeSpace = DiskManagement.GetDiskFreeSpace(drive.RootDirectory.FullName);
Assert.Fail("API call did not fail even though the drive reports that it is not ready.");
}
Assert.Inconclusive("All drives were ready. Try testing with an empty disc drive.");
}
Do I need to reconsider the way I write unit tests?
EDIT
This scenario is supported after all. All it really took was setting AllowDerivedTypes to true.
[TestMethod]
[ExpectedException(typeof(IOException), AllowDerivedTypes = true)]
public void Call_Should_Throw_If_Device_Is_Not_Ready()
{
// ...
}
You can create your own ExpectedException attribute that will check if the thrown exception inherites the Exception defined in the attribute.
public sealed class MyExpectedException : ExpectedExceptionBaseAttribute
{
private Type _expectedExceptionBaseType;
public MyExpectedException(Type expectedExceptionType)
{
_expectedExceptionBaseType = expectedExceptionType;
}
protected override void Verify(Exception exception)
{
Assert.IsNotNull(exception);
Assert.IsTrue(exception.GetType().IsInstanceOfType(typeof(_expectedExceptionBaseType)) ||
exception.GetType().IsSubclassOf(typeof(_expectedExceptionBaseType)));
}
}
and change the attribute to your test:
[MyExpectedException(typeof(IOException))]
By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?
I just want to point out that while most of the responders assumed these were unit tests, the question did not specify that they were.
nUnit is a great tool that can be used for a variety of testing situations. I can see appropriate reasons for wanting to control test order.
In those situations I have had to resort to incorporating a run order into the test name. It would be great to be able to specify run order using an attribute.
NUnit 3.2.0 added an OrderAttribute, see:
https://github.com/nunit/docs/wiki/Order-Attribute
Example:
public class MyFixture
{
[Test, Order(1)]
public void TestA() { ... }
[Test, Order(2)]
public void TestB() { ... }
[Test]
public void TestC() { ... }
}
Your unit tests should each be able to run independently and stand alone. If they satisfy this criterion then the order does not matter.
There are occasions however where you will want to run certain tests first. A typical example is in a Continuous Integration situation where some tests are longer running than others. We use the category attribute so that we can run the tests which use mocking ahead of the tests which use the database.
i.e. put this at the start of your quick tests
[Category("QuickTests")]
Where you have tests which are dependant on certain environmental conditions, consider the TestFixtureSetUp and TestFixtureTearDown attributes, which allow you to mark methods to be executed before and after your tests.
Wanting the tests to run in a specific order does not mean that the tests are dependent on each other - I'm working on a TDD project at the moment, and being a good TDDer I've mocked/stubbed everything, but it would make it more readable if I could specify the order which the tests results are displayed - thematically instead of alphabetically. So far the only thing I can think of is to prepend a_ b_ c_ to classes to classes, namespaces and methods. (Not nice) I think a [TestOrderAttribute] attribute would be nice - not stricly followed by the framework, but a hint so we can achieve this
Regardless of whether or not Tests are order dependent... some of us just want to control everything, in an orderly fashion.
Unit tests are usually created in order of complexity. So, why shouldn't they also be run in order of complexity, or the order in which they were created?
Personally, I like to see the tests run in the order of which I created them. In TDD, each successive test is naturally going to be more complex, and take more time to run. I would rather see the simpler test fail first as it will be a better indicator as to the cause of the failure.
But, I can also see the benefit of running them in random order, especially if you want to test that your tests don't have any dependencies on other tests. How about adding an option to test runners to "Run Tests Randomly Until Stopped"?
I am testing with Selenium on a fairly complex web site and the whole suite of tests can run for more than a half hour, and I'm not near to covering the entire application yet. If I have to make sure that all previous forms are filled in correctly for each test, this adds a great deal of time, not just a small amount of time, to the overall test. If there's too much overhead to running the tests, people won't run them as often as they should.
So, I put them in order and depend on previous tests to have text boxes and such completed. I use Assert.Ignore() when the pre-conditions are not valid, but I need to have them running in order.
I really like the previous answer.
I changed it a little to be able to use an attribute to set the order range:
namespace SmiMobile.Web.Selenium.Tests
{
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using NUnit.Framework;
public class OrderedTestAttribute : Attribute
{
public int Order { get; set; }
public OrderedTestAttribute(int order)
{
Order = order;
}
}
public class TestStructure
{
public Action Test;
}
class Int
{
public int I;
}
[TestFixture]
public class ControllingTestOrder
{
private static readonly Int MyInt = new Int();
[TestFixtureSetUp]
public void SetUp()
{
MyInt.I = 0;
}
[OrderedTest(0)]
public void Test0()
{
Console.WriteLine("This is test zero");
Assert.That(MyInt.I, Is.EqualTo(0));
}
[OrderedTest(2)]
public void ATest0()
{
Console.WriteLine("This is test two");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
}
[OrderedTest(1)]
public void BTest0()
{
Console.WriteLine("This is test one");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
}
[OrderedTest(3)]
public void AAA()
{
Console.WriteLine("This is test three");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
}
[TestCaseSource(sourceName: "TestSource")]
public void MyTest(TestStructure test)
{
test.Test();
}
public IEnumerable<TestCaseData> TestSource
{
get
{
var assembly =Assembly.GetExecutingAssembly();
Dictionary<int, List<MethodInfo>> methods = assembly
.GetTypes()
.SelectMany(x => x.GetMethods())
.Where(y => y.GetCustomAttributes().OfType<OrderedTestAttribute>().Any())
.GroupBy(z => z.GetCustomAttribute<OrderedTestAttribute>().Order)
.ToDictionary(gdc => gdc.Key, gdc => gdc.ToList());
foreach (var order in methods.Keys.OrderBy(x => x))
{
foreach (var methodInfo in methods[order])
{
MethodInfo info = methodInfo;
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
object classInstance = Activator.CreateInstance(info.DeclaringType, null);
info.Invoke(classInstance, null);
}
}).SetName(methodInfo.Name);
}
}
}
}
}
}
I know this is a relatively old post, but here is another way to keep your test in order WITHOUT making the test names awkward. By using the TestCaseSource attribute and having the object you pass in have a delegate (Action), you can totally not only control the order but also name the test what it is.
This works because, according to the documentation, the items in the collection returned from the test source will always execute in the order they are listed.
Here is a demo from a presentation I'm giving tomorrow:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NUnit.Framework;
namespace NUnitTest
{
public class TestStructure
{
public Action Test;
}
class Int
{
public int I;
}
[TestFixture]
public class ControllingTestOrder
{
private static readonly Int MyInt= new Int();
[TestFixtureSetUp]
public void SetUp()
{
MyInt.I = 0;
}
[TestCaseSource(sourceName: "TestSource")]
public void MyTest(TestStructure test)
{
test.Test();
}
public IEnumerable<TestCaseData> TestSource
{
get
{
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test one");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
}
}).SetName(#"Test One");
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test two");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
}
}).SetName(#"Test Two");
yield return new TestCaseData(
new TestStructure
{
Test = () =>
{
Console.WriteLine("This is test three");
MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
}
}).SetName(#"Test Three");
}
}
}
}
I am working with Selenium WebDriver end-to-end UI test cases written in C#, which are run using NUnit framework. (Not unit cases as such)
These UI tests certainly depend on order of execution, as other test needs to add some data as a precondition. (It is not feasible to do the steps in every test)
Now, after adding the 10th test case, i see NUnit wants to run in this order:
Test_1
Test_10
Test_2
Test_3
..
So i guess i have to too alphabetisize the test case names for now, but it would be good to have this small feature of controlling execution order added to NUnit.
There are very good reasons to utilise a Test Ordering mechanism. Most of my own tests use good practice such as setup/teardown. Others require huge amounts of data setup, which can then be used to test a range of features. Up till now, I have used large tests to handle these (Selenium Webdriver) integration tests. However, I think the above suggested post on https://github.com/nunit/docs/wiki/Order-Attribute has a lot of merit. Here is an example of why ordering would be extremely valuable:
Using Selenium Webdriver to run a test to download a report
The state of the report (whether it's downloadable or not) is cached for 10 minutes
That means, before every test I need to reset the report state and then wait up to 10 minutes before the state is confirmed to have changed, and then verify the report downloads correctly.
The reports cannot be generated in a practical / timely fashion through mocking or any other mechanism within the test framework due to their complexity.
This 10 minute wait time slows down the test suite. When you multiply similar caching delays across a multitude of tests, it consumes a lot of time. Ordering tests could allow data setup to be done as a "Test" right at the beginning of the test suite, with tests relying on the cache to bust being executed toward the end of the test run.
Usually Unit Test should be independent, but if you must, then you can name your methods in alphabetical order ex:
[Test]
public void Add_Users(){}
[Test]
public void Add_UsersB(){}
[Test]
public void Process_Users(){}
or you can do..
private void Add_Users(){}
private void Add_UsersB(){}
[Test]
public void Process_Users()
{
Add_Users();
Add_UsersB();
// more code
}
This question is really old now, but for people who may reach this from searching, I took the excellent answers from user3275462 and PvtVandals / Rico and added them to a GitHub repository along with some of my own updates. I also created an associated blog post with some additional info you can look at for more info.
Hope this is helpful for you all. Also, I often like to use the Category attribute to differentiate my integration tests or other end-to-end tests from my actual unit tests. Others have pointed out that unit tests should not have an order dependency, but other test types often do, so this provides a nice way to only run the category of tests you want and also order those end-to-end tests.
If you are using [TestCase], the argument TestName provides a name for the test.
If not specified, a name is generated based on the method name and the arguments provided.
You can control the order of test execution as given below:
[Test]
[TestCase("value1", TestName = "ExpressionTest_1")]
[TestCase("value2", TestName = "ExpressionTest_2")]
[TestCase("value3", TestName = "ExpressionTest_3")]
public void ExpressionTest(string v)
{
//do your stuff
}
Here I used the method name "ExpressionTest" suffix with a number.
You can use any names ordered alphabetical
see TestCase Attribute
I'm suprised the NUnit community hasn't come up with anything, so I went to create something like this myself.
I'm currently developing an open-source library that allows you to order your tests with NUnit. You can order test fixtures and ordering "ordered test specifications" alike.
The library offers the following features:
Build complex test ordering hierarchies
Skip subsequent tests if an test in order fails
Order your test methods by dependency instead of integer order
Supports usage side-by-side with unordered tests. Unordered tests are executed first.
The library is actually inspired in how MSTest does test ordering with .orderedtest files. Please look at an example below.
[OrderedTestFixture]
public sealed class MyOrderedTestFixture : TestOrderingSpecification {
protected override void DefineTestOrdering() {
TestFixture<Fixture1>();
OrderedTestSpecification<MyOtherOrderedTestFixture>();
TestFixture<Fixture2>();
TestFixture<Fixture3>();
}
protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
}
You should not depend on the order in which the test framework picks tests for execution. Tests should be isolated and independent. In that they should not depend on some other test setting the stage for them or cleaning up after them. They should also produce the same result irrespective of the order of the execution of tests (for a given snapshot of the SUT)
I did a bit of googling. As usual, some people have resorted to sneaky tricks (instead of solving the underlying testability/design issue
naming the tests in an alphabetically ordered manner such that tests appear in the order they 'need' to be executed. However NUnit may choose to change this behavior with a later release and then your tests would be hosed. Better check in the current NUnit binaries to Source Control.
VS (IMHO encouraging the wrong behavior with their 'agile tools') has something called as "Ordered tests" in their MS testing framework. I didn't waste any time reading up but it seems to be targeted towards the same audience
See Also: characteristics of a good test
In case of using TestCaseSource the key is to override string ToString method, How that works:
Assume you have TestCase class
public class TestCase
{
public string Name { get; set; }
public int Input { get; set; }
public int Expected { get; set; }
}
And a list of TestCases:
private static IEnumerable<TestCase> TestSource()
{
return new List<TestCase>
{
new TestCase()
{
Name = "Test 1",
Input = 2,
Expected = 4
},
new TestCase()
{
Name = "Test 2",
Input = 4,
Expected = 16
},
new TestCase()
{
Name = "Test 3",
Input = 10,
Expected = 100
}
};
}
Now lets use it with a Test method and see what happen:
[TestCaseSource(nameof(TestSource))]
public void MethodXTest(TestCase testCase)
{
var x = Power(testCase.Input);
x.ShouldBe(testCase.Expected);
}
This will not test in order and the output will be like this:
So if we added override string ToString to our class like:
public class TestCase
{
public string Name { get; set; }
public int Input { get; set; }
public int Expected { get; set; }
public override string ToString()
{
return Name;
}
}
The result will change and we get the order and name of test like:
Note:
This is just example to illustrate how to get name and order in test, the order is taken numerically/alphabetically so if you have more than ten test I suggest making Test 01, Test 02.... Test 10, Test 11 etc because if you make Test 1 and at some point Test 10 than the order will be Test 1, Test 10, Test 2 .. etc.
The input and Expected can be any type, string, object or custom class.
Beside order, the good thing here is that you see the test name which is more important.