I have written the xUnit test cases in C#. That test class contains so many methods. I need to run the whole test cases in a sequence. How can I set the test case sequence in xUnit?
In xUnit 2.* this can be achieved using the TestCaseOrderer attribute to designate an ordering strategy, which can be used to reference an attribute that is annotated on each test to denote an order.
For example:
Ordering Strategy
[assembly: CollectionBehavior(DisableTestParallelization = true)]
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = 0;
foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
priority = attr.GetNamedArgument<int>("Priority");
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
{
list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
foreach (TTestCase testCase in list)
yield return testCase;
}
}
static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key) where TValue : new()
{
TValue result;
if (dictionary.TryGetValue(key, out result)) return result;
result = new TValue();
dictionary[key] = result;
return result;
}
}
Attribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public TestPriorityAttribute(int priority)
{
Priority = priority;
}
public int Priority { get; private set; }
}
Test Cases
[TestCaseOrderer("FullNameOfOrderStrategyHere", "OrderStrategyAssemblyName")]
public class PriorityOrderExamples
{
[Fact, TestPriority(5)]
public void Test3()
{
// called third
}
[Fact, TestPriority(0)]
public void Test2()
{
// called second
}
[Fact, TestPriority(-5)]
public void Test1()
{
// called first
}
}
xUnit 2.* ordering samples here
Testpriority: at the bottom of this page.
[PrioritizedFixture]
public class MyTests
{
[Fact, TestPriority(1)]
public void FirstTest()
{
// Test code here is always run first
}
[Fact, TestPriority(2)]
public void SeccondTest()
{
// Test code here is run second
}
}
BTW, I have the same problem right now. And yes, it is not the clean art.. but QA wanted a manual test.. so an automated test with a specific order already is a big leap for them.. (cough) and yes, it is not really unit testing..
If you really have the need to prioritize your tests (probably not your unit tests) you can use Xunit.Priority.
I have used it for some integration testing and works really well and simple without the overhead of having to write your prioritization classes, for simple case scenarios
For some reason, XUnit.Priority didn't work for me. In my test cases, it wasn't running the tests in the priority order specified.
So I tried XUnitPriorityOrderer, which is similar to use but was working (To quickly test it, save the following code in a text editor as OrderedXUnitTests.linq, then open it with LinqPad 6 and execute it. Alternatively, you can also copy the TestClass to Visual Studio and add XUnit, XUnit.Runner.VisualStudio and XUnitPriorityOrderer):
<Query Kind="Program">
<NuGetReference>XUnitPriorityOrderer</NuGetReference>
<Namespace>Xunit</Namespace>
<Namespace>XUnitPriorityOrderer</Namespace>
</Query>
#load "xunit"
// using XUnitPriorityOrderer
// see: https://github.com/frederic-prusse/XUnitPriorityOrderer
void Main()
{
RunTests(); // Call RunTests() or press Alt+Shift+T to initiate testing.
}
#region private::Tests
[TestCaseOrderer(CasePriorityOrderer.TypeName, CasePriorityOrderer.AssembyName)]
public class TestClass
{
static List<string> Order { get; set; }
public TestClass()
{
Order = Order ?? new List<string>();
}
[Fact, Order(2)]
void Test_Xunit_AnotherTest()
{
Order.Add("Test_Xunit_AnotherTest");
Assert.True(3 + 1 == 4);
}
[Fact, Order(1)]
void Test_Xunit()
{
Order.Add("Test_XUnit");
Assert.True(1 + 1 == 2);
}
[Fact, Order(99)]
void Print_Order()
{
Order.Add("Print_Order");
var strOrder = string.Join(", ", Order.ToArray());
strOrder.Dump("Execution Order");
Assert.True(true);
}
}
#endregion
This will run the tests in given order (Order(1), Order(2) and then Order(99)) and will dump the executed tests finally (test method Print_Order()).
You can't, by design. It's deliberately random in order to prevent anyone getting one of those either by desire or by accident.
The randomness is only for a given Test class, so you may be able to achieve your goals by wrapping items you want to control the order of inside a nested class - but in that case, you'll still end up with random order whenever you have more than two Test Methods in a class.
If you're trying to manage the building up of fixtures or context, the built-in IUseFixture<T> mechanism may be appropriate. See the xUnit Cheat Sheet for examples.
But you really need to tell us more about what you're trying to do or we'll just have to get speculative.
Related
Could someone tell me i this is a correct unit test for FillUpContainer()?
namespace CoffeeMaker
{
[TestClass()]
public class WaterContainer
{
//private WaterContainer WaterC;
WaterContainer WaterC = new WaterContainer(0);
int level = WaterC.Level;
[TestMethod()]
public void FillUpWaterTest()
{
if (WaterC.Level == 0)
{
bool result;
result = WaterC.FillUpWater();
}
level.Equals(5);
}
}
}
No, it's not. level.Equals(5) is not an assertion, it's just an expression, the result of which you throw away by not assigning to anything.
A better test might look something like this:
[TestClass()]
public class WaterContainer
{
[TestMethod()]
public void WhenWaterContainerIsEmpty_FillingItUp_CausesCorrectWaterLevel()
{
// arrange
WaterContainer water = new WaterContainer(0);
// act
water.FillUpWater();
// assert
Assert.AreEqual(5, water.Level);
}
}
Notice the method name describes what's being tested, and the test is self-contained (all local variables).
The Assert statement will cause the test to actually fail if the water level is not correct.
Instead of this:
level.Equals(5);
I believe you want this:
Assert.AreEqual(5, WaterC.Level);
This will cause the unit test to fail if WaterC.Level != 5.
I have the following simple class Data annotation to control the area of phone number:
public class PhoneAreaAttribute : ValidationAttribute, IClientValidatable
{
public const string ValidInitNumber = "0";
public const int MinLen = 2;
public const int MaxLen = 4;
public override bool IsValid(object value)
{
var area = (string)value;
if (string.IsNullOrWhiteSpace(area))
{
return true;
}
if (area.StartsWith(PhoneAreaAttribute.ValidInitNumber))
{
return false;
}
if (!Regex.IsMatch(area, #"^[\d]+$"))
{
return false;
}
if (!area.LengthBetween(PhoneAreaAttribute.MinLen, PhoneAreaAttribute.MaxLen))
{
return false;
}
return true;
}
public IEnumerable<ModelClientValidationRule> GetClientValidationRules(ModelMetadata metadata, ControllerContext context)
{
var rule = new ModelClientValidationRule
{
ErrorMessage = FormatErrorMessage(metadata.GetDisplayName()),
ValidationType = "phoneArea",
};
yield return rule;
}
}
I do not know how it would be a correct unit test for this class.
thanks.
Okay, basically testing an attribute is the same as testing any regular class. I took your class and reduced it a little bit so I could run it (you created some extension methods, which I didn't want to recreate). Below you find this class definition.
public class PhoneAreaAttribute : ValidationAttribute
{
public const string ValidInitNumber = "0";
public override bool IsValid(object value)
{
var area = (string)value;
if (string.IsNullOrEmpty(area))
{
return true;
}
if (area.StartsWith(PhoneAreaAttribute.ValidInitNumber))
{
return false;
}
return true;
}
}
Note beforehand: some of my naming conventions for unit tests might differ from the ones you use (there are a few out there).
Now we will create a Unit Test. I understood that you already have a Test Project, if you don't have one, just create one. In this test project, you will create a new unit test (Basic Unit Test), let's name it PhoneAreaAttributeTest.
As good practice, I create a test initialiser to create all shared "resources", in this case a new instance of the PhoneAreaAttribute class. Yes, you can just create an instance, like you are used to with "regular" classes (as a matter of fact, there isn't that much difference between "regular" classes and your attribute class).
Now we are ready to start writing the tests for the methods. Basically you will want to handle all possible scenarios. I will show you here two scenarios that are possible in my (reduced) IsValid method. First I will see whether the given object parameter can be cased to a string (this is a first scenario / TestMethod). Second I will see whether the path of "IsNullOrEmpty" is properly handled (this is a second scenario / TestMethod).
As you can see, it is just a regular unit test. These are just the very basics. If you still have questions, I would like to also suggest you to read some tutorials.
Here is the PhoneAreaAttributeTest test class:
[TestClass]
public class PhoneAreaAttributeTest
{
public PhoneAreaAttribute PhoneAreaAttribute { get; set; }
[TestInitialize]
public void PhoneAreaAttributeTest_TestInitialise()
{
PhoneAreaAttribute = new PhoneAreaAttribute();
}
[TestMethod]
[ExpectedException(typeof(InvalidCastException))]
public void PhoneAreaAttributeTest_IsValid_ThrowsInvalidCastException()
{
object objectToTest = new object();
PhoneAreaAttribute.IsValid(objectToTest);
}
[TestMethod]
public void PhoneAreaAttributeTest_IsValid_NullOrEmpty_True()
{
string nullToTest = null;
string emptoToTest = string.Empty;
var nullTestResult = PhoneAreaAttribute.IsValid(nullToTest);
var emptyTestResult = PhoneAreaAttribute.IsValid(emptoToTest);
Assert.IsTrue(nullTestResult, "Null Test should return true.");
Assert.IsTrue(emptyTestResult, "Empty Test should return true.");
}
}
When considering how to "correctly" test this class, consider the following:
The cyclomatic complexity (CC) of IsValid is 5.
The method relies on two other methods IsNullOrWhiteSpace and LengthBetween. I believe both of these have an additional CC of 2.
There is the chance of throwing an InvalidCastException. This represents another potential test case.
In total, you have potentially 8 cases in which you need to test. Using xUnit.net and Fluent Assertions* (you could do something similar in NUnit as well), you could write the following unit tests to "correctly" test this method:
public class PhoneAreaAttributeTests
{
[Theory]
[InlineData("123", true)]
[InlineData(" ", true)]
[InlineData(null, true)]
public void IsValid_WithCorrectInput_ReturnsTrue(
object value, bool expected)
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
// Exercise
var actual = phoneAreaAttribute.IsValid(value);
// Verify
actual.Should().Be(expected, "{0} should be valid input", value);
// Teardown
}
[Theory]
[InlineData("012", false)]
[InlineData("A12", false)]
[InlineData("1", false)]
[InlineData("12345", false)]
public void IsValid_WithIncorrectInput_ReturnsFalse(
object value, bool expected)
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
// Exercise
var actual = phoneAreaAttribute.IsValid(value);
// Verify
actual.Should().Be(expected, "{0} should be invalid input", value);
// Teardown
}
[Fact]
public void IsValid_UsingNonStringInput_ThrowsExcpetion()
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
const int input = 123;
// Exercise
// Verify
Assert.Throws<InvalidCastException>(
() => phoneAreaAttribute.IsValid(input));
// Teardown
}
// Simple factory method so that if we change the
// constructor, we don't have to change all our
// tests reliant on this object.
public PhoneAreaAttribute CreatePhoneAreaAttribute()
{
return new PhoneAreaAttribute();
}
}
*I like using Fluent Assertions, and in this case it helps because we can specify a message to let us know when an assert fails, which one is the failing assertion. These data-driven tests are nice in that they can reduce the number of similar test methods we would need to write by grouping various permutations together. When we do this it is a good idea to avoid Assertion Roulette by the custom message as explained. By the way, Fluent Assertions can work with many testing frameworks.
When looking at the post here, it looks like I should be able to create several objects using CreateMany(), iterate over them using foreach, and then return them as an array.
What I'm seeing is that each iteration seems to create new objects each time. Is this expected behavior?
Entity to create:
public class TestEntity
{
public int Id { get; private set; }
public string SomeString { get; set; }
public void SetId(int value)
{
this.Id = value;
}
}
Sample Program.cs:
private static int id;
static void Main(string[] args)
{
var fixture = new Fixture();
IEnumerable<TestEntity> testEntities =
fixture.Build<TestEntity>().CreateMany(5);
Output(testEntities);
foreach (var testEntity in testEntities)
{
testEntity.SetId(id++);
Console.WriteLine(
string.Format("CHANGED IN FOREACH:: hash: {0}, id: {1}, string: {2}",
testEntity.GetHashCode(), testEntity.Id, testEntity.SomeString));
}
Output(testEntities);
}
private static void Output(IEnumerable<TestEntity> testEntities)
{
foreach (var testEntity in testEntities)
{
Console.WriteLine(
string.Format("hash: {0}, id: {1}, string: {2}",
testEntity.GetHashCode(), testEntity.Id, testEntity.SomeString));
}
}
I created an issue here (which can probably be deleted if this is expected behavior).
Edit 2011-06-02
To get the behavior I was expecting, and if I don't want to modify the AutoFixture behavior, I can use an extension method:
var fixture = new Fixture();
TestEntity[] testEntities = fixture.Build<TestEntity>().CreateMany(5).ToArray();
This is indeed the expected default behavior. There are many reasons for that, but basically it boils down to that when you ask for a IEnumerable<T> AutoFixture actually goes to great lengths to ensure that you get only what you ask for.
This is surprising behavior to many. The good news is that you can change it.
fixture.Customizations.Add(new StableFiniteSequenceRelay());
This will change the behavior such that subsequently all sequences are stable. You can package that method call into a Customization for better reusability. That might look something like this (totally optional):
public class StableFiniteSequenceCustomization : ICustomization
{
public void Customize(IFixture fixture)
{
fixture.Customizations.Add(new StableFiniteSequenceRelay());
}
}
I have a test class as follows:
[TestFixture("someurl1", new[] { "param1", "param2" }, 15)]
[TestFixture("someurl2", new[] { "param3" }, 15)]
public class my_test
{
public my_test(string url, string[] fields, int someVal)
{
// test setup
}
}
When running this test in ReSharper 6.1 and NUnit 2.5.10, the test is not run twice, as expected, it only runs once. In the test results I see listed
my_test("someurl1", System.String[], 15)
This makes me think that the two fixtures are being treated as the same, and that NUnit isn't differentiating between the string arrays in the two tests.
As a workaround I have added a dummy parameter in the constructor. If I set this to a different value for each fixture, then all the tests run.
Is it not possible to have TestFixtures with arrays containing different values? I've just upgraded from ReSharper 5 so I'm wondering if that is related. I have read about some issues with parameterised tests in 6.x.
[TestFixture("someurl1", "param1|param2", 15)]
[TestFixture("someurl2", "param3", 15)]
public class my_test
{
private string[] _fields;
public my_test(string url, string fieldList, int someVal)
{
_fields = fieldList.Split('|');
// test setup
}
[Test]
public void listFields()
{
foreach (var field in _fields)
{
Console.WriteLine(field);
}
}
}
I think this is related to newing-up an array in the TestFixture constructor, I read somewhere that you can't do that.
Have you tried creating individual tests under it as a generic TestFixture??? Something like
[TestFixture]
public class my_test
{
private bool my_test(string url, string[] fields, int someVal)
{
// test setup
return DidTestCompleteOk;
}
[Test]
public void TestURL1()
{
Assert.IsTrue( my_test("someurl1", new[] { "param1", "param2" }, 15));
}
[Test]
public void TestURL2()
{
Assert.IsTrue( my_test("someurl2", new[] { "param3" }, 15) );
}
}
It turns out this is the absolute error message if ANYTHING goes wrong in a parameterized test fixture's constructor. You don't get the actual exception information returned like when stuff fails in other code.
So you probably should move your setup code to a [SetUp] or [TestFixtureSetUp] or reallly be sure your constructor is executing without error. But really you should be doing the first suggestion, and only save the test fixture parameters in the constructor and do something with them in another method.
I'm using Moq and I have the following interface:
public interface IGameBoard : IEnumerable<PieceType>
{
...
}
public class GameBoardNodeFactory
{
public virtual GameBoardNode Create (int row, int column, IGameBoard gameBoard)
{
...
}
}
Then I have a test like this:
var clonedGameBoardMock = new Mock<IGameBoard> (MockBehavior.Loose);
var gameBoardNodeFactoryMock = new Mock<GameBoardNodeFactory> ();
gameBoardNodeFactoryMock.Setup (x =>
x.Create (
position.Row,
position.Column,
clonedGameBoardMock.Object)).Returns (new GameBoardNode { Row = position.Row, Column = position.Column });
But then gameBoardNodeFactoryMock.Object.Create (position.Row, position.Column, clonedGameBoardMock.Object) throws a NullReferenceException. I tried to create a mock for the IGameBoard such that it doesn't extend IEnumerable<PieceType> interface and then it works.
Any help is appreciated.
You would need to create a Setup for GetEnumerator() if it's being called. Something like:
var mockPieces = new List<PieceType>;
clonedGameBoardMock.Setup(g => g.GetEnumerator()).Returns(mockPieces.GetEnumerator());
Note sure if that's the issue in this case, but worth noting if you ever need to mock IEnumerable<T>.
The answer by #DanBryant was also the key to our solution. However, the enumerator in that case might be accidentally reused. Instead, I suggest using:
clonedGameBoardMock.Setup(g => g.GetEnumerator()).Returns(() => mockPieces.GetEnumerator());
Here's a full repro (new class library using NUnit 2.6.4 and Moq 4.2):
public interface IMyThing<T> : IEnumerable<T>
{
string Name { get; set; }
IMyThing<T> GetSub<U>(U key);
}
public interface IGenericThing
{
string Value { get; set; }
}
public class Pet
{
public string AnimalName { get; set; }
}
public class Unit
{
public IEnumerable<Pet> ConvertInput(IMyThing<IGenericThing> input)
{
return input.GetSub("api-key-123").Select(x => new Pet { AnimalName = x.Value });
}
}
[TestFixture]
public class Class1
{
[Test]
public void Test1()
{
var unit = new Unit();
Mock<IMyThing<IGenericThing>> mock = new Mock<IMyThing<IGenericThing>>();
Mock<IMyThing<IGenericThing>> submock = new Mock<IMyThing<IGenericThing>>();
var things = new List<IGenericThing>(new[] { new Mock<IGenericThing>().Object });
submock.Setup(g => g.GetEnumerator()).Returns(() => things.GetEnumerator());
mock.Setup(x => x.GetSub(It.IsAny<string>())).Returns(submock.Object);
var result = unit.ConvertInput(mock.Object);
Assert.That(result, Is.Not.Null.And.Not.Empty);
Assert.That(result, Is.Not.Null.And.Not.Empty); // This would crash if the enumerator wasn't returned through a Func<>...
}
}
For what it's worth / to make this question pop up to that one lone Googler with the same problem I had: the above is an abstracted version of the Couchbase .NET client's IView<T> interface, which also implements IEnumerable<T>.
A null reference in this situation usually means your setup was never met. Meaning it was never called with the exact values you set it up for. To debug this I would make your match less constraining by using It.IsAny() and so on to make sure the test will match on any call to the mocked function. In most cases this is good enough. Any reason why your are trying to match on specific values?
Okay if anyone is interested, I updated Moq to version 4 and now everything works as expected.