We Keep Coding

Theory test iterations with none constant type parameters don't show as separate iterations in the Test Explorer

I have been writing unit tests in an xUnit project for a .NET project. Because I had a lot of combinations of parameters for the same test (3000+) I decided to use the [ClassData(typeof(TestDataEnumerable))] in order to have the different options be generated.
The first test has only two parameters, both enums (constants which is important later). The second test contains a third params parameter which are of type object.
When I run the tests in VS Test Explorer I get this output for the tests (not actual code).
Namespace.FirstTest.GivenPrecondition_WhenCondition_ThenMethodReturns(firstParam:Standard, secondParam: Denied)
Namespace.SecondTest.GivenPrecondition_WhenCondition_ThenMethodReturns(firstParam: Standard, secondParam: Denied, thirdParams: [ObjectToStringResult, ObjectToStringResult])
As well as the Test Explorer showing a dropdown under the test method with each of the Theory iterations for the First Test. But it doesn't show all the iterations for the second test (with the non-constant parameter - the fact that it is a params doesn't affect this, test with just allowed a single object type).
TestClass >
> First Test
- IterationOne
- IterationTwo
Second Test
As well as this the XML output from XunitXML.TestLogger cuts the names off before the parameters are shown when it contains a none constant type.
The code for one such test is below, Permission is just a class containing a Name. It is a class rather than just a string value to allow inheritance I believe so can't be changed to a constant type.
[Theory]
[ClassData(typeof(PermissionPermutations))]
public void GivenUser_WhenUserHasCorrectPermissions_LoggedInMenuReturnsExpectedMenuItems(EUserType userType, ERovUserType rovUserType, params Permission[] permissions)
{
var user = UserBuilder.Create(userType, rovUserType)
.WithPermissions(permissions)
.Build();
var results = GetActualResults(user);
var expected = GetExpectedResults(user);
Assert.Equal(result.Count(), expected.Count());
Assert.All(results, (name) => Assert.Contains(name, expected));
Assert.All(expected, (name) => Assert.Contains(name, result));
}
public class PermissionPermutations : IEnumerable<object[]>
{
public IEnumerator<object[]> GetEnumerator()
{
foreach (var (userType, rovUserType) in UserTypeConstants.GetAllCombinations())
{
foreach (var permission in PermissionConstants.AllPermissions)
{
yield return new object[] { userType, rovUserType, permission };
}
}
}
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
The PermissionConstants class looks like this so a total of 88 permissions exist (I tested it with a few thousand iterations and it's not the number of iterations that causes the Explorer to hide the iterations):
public static class PermissionConstants
{
public static Permission[] AllPermissions => new[]
{
new Permission { Name = "CanViewUsers" },
new Permission { Name = "CanEditUsers" },
... there are 88 of these total
};
}
And the UserTypeConstants class looks like this, I've changed the names of the enums for privacy, but theEUserType and ERovUserType are just enums:
public static class UserTypeConstants
{
public static EUserType[] AllUserTypes => new[]
{
EUserType.Standard,
EUserType.Admin,
EUserType.Restricted,
EUserType.SuperUser,
EUserType.SystemUser
};
public static ERovUserType[] AllRovUserTypes => new[]
{
ERovUserType.Level0,
ERovUserType.Level1,
ERovUserType.Level2,
ERovUserType.Level3,
ERovUserType.Level4,
ERovUserType.Level5
};
public static IEnumerable<(EUserType, ERovUserType)> GetAllCombinations() {
foreach (var userType in AllUserTypes)
{
foreach(var rovUserType in AllRovUserTypes)
{
yield return (userType, rovUserType);
}
}
}
}

Could some one check for me if my unit test for the FillUpWater is correct

Could someone tell me i this is a correct unit test for FillUpContainer()?
namespace CoffeeMaker
{
[TestClass()]
public class WaterContainer
{
//private WaterContainer WaterC;
WaterContainer WaterC = new WaterContainer(0);
int level = WaterC.Level;
[TestMethod()]
public void FillUpWaterTest()
{
if (WaterC.Level == 0)
{
bool result;
result = WaterC.FillUpWater();
}
level.Equals(5);
}
}
}

No, it's not. level.Equals(5) is not an assertion, it's just an expression, the result of which you throw away by not assigning to anything.
A better test might look something like this:
[TestClass()]
public class WaterContainer
{
[TestMethod()]
public void WhenWaterContainerIsEmpty_FillingItUp_CausesCorrectWaterLevel()
{
// arrange
WaterContainer water = new WaterContainer(0);
// act
water.FillUpWater();
// assert
Assert.AreEqual(5, water.Level);
}
}
Notice the method name describes what's being tested, and the test is self-contained (all local variables).
The Assert statement will cause the test to actually fail if the water level is not correct.

Instead of this:
level.Equals(5);
I believe you want this:
Assert.AreEqual(5, WaterC.Level);
This will cause the unit test to fail if WaterC.Level != 5.

How can I inject primitive values using RhinoAutoMocker?

I have been using RhinoAutoMocker for unit testing, and it works very well in almost all cases. I'm currently having trouble figuring out is how to use it when the Class Under Test has primitive constructor arguments.
Let's say I have two classes defined as such:
public class AddAnswerValidator : BaseValidator
{
public AddAnswerValidator(Answer answerToAdd,
IAnswerRepository answerRepository)
{
...some code
}
public override bool Validates()
{
...some code
}
}
public class RemoveAnswerValidator : BaseValidator
{
public RemoveAnswerValidator(int answerIDToRemove,
IAnswerRepository answerRepository)
{
...some code
}
public override bool Validates()
{
...some code
}
}
An example test for each are as follows:
[Test]
public void AddAnswerValidatorTests_Validates_ValidInput_ReturnsTrue()
{
var autoMocker = new RhinoAutoMocker<AddAnswerValidator>();
var fakeAnswer = new Answer();
autoMocker.Inject<Answer>(fakeAnswer);
var result = autoMocker.ClassUnderTest.Validates();
Assert.IsTrue(result);
}
[Test]
public void RemoveAnswerValidatorTests_Validates_ValidInput_ReturnsTrue()
{
var autoMocker = new RhinoAutoMocker<RemoveAnswerValidator>();
var fakeAnswerID = 1;
autoMocker.Inject<int>(fakeAnswerID);
var result = autoMocker.ClassUnderTest.Validates();
Assert.IsTrue(result);
}
The first test (for AddAnswerValidator) works fine. The second test (for RemoveAnswerValidator) fails with a StructureMap 202 Error "No default instance defined for plugin family RemoveAnswerValidator" error. I'm working under the assumption that the second test is failing because StructureMap isn't resolving the integer constructor argument.
I've read through this post regarding RhinoAutoMocker Injection for collections and I've been tracing through the source code on GitHub, but I don't understand why the primitive value isn't being injected.
I've even tried substituting some of the overloaded Inject methods available on the container object such as:
autoMocker.Inject<int>(fakeAnswerID);
with
autoMocker.Container.Inject<int>("answerIDToRemove", fakeAnswerID);
but using the name of the constructor argument doesn't produce any different results.
--
In the long run, this isn't a huge problem since I can always just create an instance of the Class Under Test and create my own mocks, it would just be nice to be able to use a consistent methodology across all of my tests.

I know, it's a little bit too late, but i had the same problem and managed to solve it with integer parameter:
var autoMocker = new RhinoAutoMocker<RemoveAnswerValidator>();
automocker.Container.Configure(c=>
c.For<RemoveAnswerValidator>()
.Use<RemoveAnswerValidator>()
.Ctor<int>()
.Is(1));

While I never did find a way to inject a primitive using AutoMocker, I ended up working around the problem by creating a parameter object and injecting that instead.
The parameter object then includes the necessary primitives as a properties, in addition to the other (former) parameters. The example above would be changed to:
public class RemoveAnswerValidator : BaseValidator
{
public RemoveAnswerValidator(RemoveAnswerValidatorParameters parms)
{
...some code
}
public override bool Validates()
{
...some code
}
}
public class RemoveAnswerValidatorParameters
{
public int AnswerID { get; set; }
public IAnswerRepository AnswerRepository { get; set; }
}
(Then in the test class)
[Test]
public void RemoveAnswerValidatorTests_Validates_ValidInput_ReturnsTrue()
{
var autoMocker = new RhinoAutoMocker<RemoveAnswerValidator>();
var fakeAnswerParameters = new FakeAnswerParameters()
{
AnswerID = 1,
AnswerRepository = autoMocker.Get<IAnswerRepository>()
};
autoMocker.Inject<RemoveAnswerValidatorParameters>(fakeAnswer);
var result = autoMocker.ClassUnderTest.Validates();
Assert.IsTrue(result);
}

How to do a Unit Test for a custom Data Annotation

I have the following simple class Data annotation to control the area of phone number:
public class PhoneAreaAttribute : ValidationAttribute, IClientValidatable
{
public const string ValidInitNumber = "0";
public const int MinLen = 2;
public const int MaxLen = 4;
public override bool IsValid(object value)
{
var area = (string)value;
if (string.IsNullOrWhiteSpace(area))
{
return true;
}
if (area.StartsWith(PhoneAreaAttribute.ValidInitNumber))
{
return false;
}
if (!Regex.IsMatch(area, #"^[\d]+$"))
{
return false;
}
if (!area.LengthBetween(PhoneAreaAttribute.MinLen, PhoneAreaAttribute.MaxLen))
{
return false;
}
return true;
}
public IEnumerable<ModelClientValidationRule> GetClientValidationRules(ModelMetadata metadata, ControllerContext context)
{
var rule = new ModelClientValidationRule
{
ErrorMessage = FormatErrorMessage(metadata.GetDisplayName()),
ValidationType = "phoneArea",
};
yield return rule;
}
}
I do not know how it would be a correct unit test for this class.
thanks.

Okay, basically testing an attribute is the same as testing any regular class. I took your class and reduced it a little bit so I could run it (you created some extension methods, which I didn't want to recreate). Below you find this class definition.
public class PhoneAreaAttribute : ValidationAttribute
{
public const string ValidInitNumber = "0";
public override bool IsValid(object value)
{
var area = (string)value;
if (string.IsNullOrEmpty(area))
{
return true;
}
if (area.StartsWith(PhoneAreaAttribute.ValidInitNumber))
{
return false;
}
return true;
}
}
Note beforehand: some of my naming conventions for unit tests might differ from the ones you use (there are a few out there).
Now we will create a Unit Test. I understood that you already have a Test Project, if you don't have one, just create one. In this test project, you will create a new unit test (Basic Unit Test), let's name it PhoneAreaAttributeTest.
As good practice, I create a test initialiser to create all shared "resources", in this case a new instance of the PhoneAreaAttribute class. Yes, you can just create an instance, like you are used to with "regular" classes (as a matter of fact, there isn't that much difference between "regular" classes and your attribute class).
Now we are ready to start writing the tests for the methods. Basically you will want to handle all possible scenarios. I will show you here two scenarios that are possible in my (reduced) IsValid method. First I will see whether the given object parameter can be cased to a string (this is a first scenario / TestMethod). Second I will see whether the path of "IsNullOrEmpty" is properly handled (this is a second scenario / TestMethod).
As you can see, it is just a regular unit test. These are just the very basics. If you still have questions, I would like to also suggest you to read some tutorials.
Here is the PhoneAreaAttributeTest test class:
[TestClass]
public class PhoneAreaAttributeTest
{
public PhoneAreaAttribute PhoneAreaAttribute { get; set; }
[TestInitialize]
public void PhoneAreaAttributeTest_TestInitialise()
{
PhoneAreaAttribute = new PhoneAreaAttribute();
}
[TestMethod]
[ExpectedException(typeof(InvalidCastException))]
public void PhoneAreaAttributeTest_IsValid_ThrowsInvalidCastException()
{
object objectToTest = new object();
PhoneAreaAttribute.IsValid(objectToTest);
}
[TestMethod]
public void PhoneAreaAttributeTest_IsValid_NullOrEmpty_True()
{
string nullToTest = null;
string emptoToTest = string.Empty;
var nullTestResult = PhoneAreaAttribute.IsValid(nullToTest);
var emptyTestResult = PhoneAreaAttribute.IsValid(emptoToTest);
Assert.IsTrue(nullTestResult, "Null Test should return true.");
Assert.IsTrue(emptyTestResult, "Empty Test should return true.");
}
}

When considering how to "correctly" test this class, consider the following:
The cyclomatic complexity (CC) of IsValid is 5.
The method relies on two other methods IsNullOrWhiteSpace and LengthBetween. I believe both of these have an additional CC of 2.
There is the chance of throwing an InvalidCastException. This represents another potential test case.
In total, you have potentially 8 cases in which you need to test. Using xUnit.net and Fluent Assertions* (you could do something similar in NUnit as well), you could write the following unit tests to "correctly" test this method:
public class PhoneAreaAttributeTests
{
[Theory]
[InlineData("123", true)]
[InlineData(" ", true)]
[InlineData(null, true)]
public void IsValid_WithCorrectInput_ReturnsTrue(
object value, bool expected)
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
// Exercise
var actual = phoneAreaAttribute.IsValid(value);
// Verify
actual.Should().Be(expected, "{0} should be valid input", value);
// Teardown
}
[Theory]
[InlineData("012", false)]
[InlineData("A12", false)]
[InlineData("1", false)]
[InlineData("12345", false)]
public void IsValid_WithIncorrectInput_ReturnsFalse(
object value, bool expected)
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
// Exercise
var actual = phoneAreaAttribute.IsValid(value);
// Verify
actual.Should().Be(expected, "{0} should be invalid input", value);
// Teardown
}
[Fact]
public void IsValid_UsingNonStringInput_ThrowsExcpetion()
{
// Setup
var phoneAreaAttribute = CreatePhoneAreaAttribute();
const int input = 123;
// Exercise
// Verify
Assert.Throws<InvalidCastException>(
() => phoneAreaAttribute.IsValid(input));
// Teardown
}
// Simple factory method so that if we change the
// constructor, we don't have to change all our
// tests reliant on this object.
public PhoneAreaAttribute CreatePhoneAreaAttribute()
{
return new PhoneAreaAttribute();
}
}
*I like using Fluent Assertions, and in this case it helps because we can specify a message to let us know when an assert fails, which one is the failing assertion. These data-driven tests are nice in that they can reduce the number of similar test methods we would need to write by grouping various permutations together. When we do this it is a good idea to avoid Assertion Roulette by the custom message as explained. By the way, Fluent Assertions can work with many testing frameworks.

How to set the test case sequence in xUnit

I have written the xUnit test cases in C#. That test class contains so many methods. I need to run the whole test cases in a sequence. How can I set the test case sequence in xUnit?

In xUnit 2.* this can be achieved using the TestCaseOrderer attribute to designate an ordering strategy, which can be used to reference an attribute that is annotated on each test to denote an order.
For example:
Ordering Strategy
[assembly: CollectionBehavior(DisableTestParallelization = true)]
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = 0;
foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
priority = attr.GetNamedArgument<int>("Priority");
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
{
list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
foreach (TTestCase testCase in list)
yield return testCase;
}
}
static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key) where TValue : new()
{
TValue result;
if (dictionary.TryGetValue(key, out result)) return result;
result = new TValue();
dictionary[key] = result;
return result;
}
}
Attribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public TestPriorityAttribute(int priority)
{
Priority = priority;
}
public int Priority { get; private set; }
}
Test Cases
[TestCaseOrderer("FullNameOfOrderStrategyHere", "OrderStrategyAssemblyName")]
public class PriorityOrderExamples
{
[Fact, TestPriority(5)]
public void Test3()
{
// called third
}
[Fact, TestPriority(0)]
public void Test2()
{
// called second
}
[Fact, TestPriority(-5)]
public void Test1()
{
// called first
}
}
xUnit 2.* ordering samples here

Testpriority: at the bottom of this page.
[PrioritizedFixture]
public class MyTests
{
[Fact, TestPriority(1)]
public void FirstTest()
{
// Test code here is always run first
}
[Fact, TestPriority(2)]
public void SeccondTest()
{
// Test code here is run second
}
}
BTW, I have the same problem right now. And yes, it is not the clean art.. but QA wanted a manual test.. so an automated test with a specific order already is a big leap for them.. (cough) and yes, it is not really unit testing..

If you really have the need to prioritize your tests (probably not your unit tests) you can use Xunit.Priority.
I have used it for some integration testing and works really well and simple without the overhead of having to write your prioritization classes, for simple case scenarios

For some reason, XUnit.Priority didn't work for me. In my test cases, it wasn't running the tests in the priority order specified.
So I tried XUnitPriorityOrderer, which is similar to use but was working (To quickly test it, save the following code in a text editor as OrderedXUnitTests.linq, then open it with LinqPad 6 and execute it. Alternatively, you can also copy the TestClass to Visual Studio and add XUnit, XUnit.Runner.VisualStudio and XUnitPriorityOrderer):
<Query Kind="Program">
<NuGetReference>XUnitPriorityOrderer</NuGetReference>
<Namespace>Xunit</Namespace>
<Namespace>XUnitPriorityOrderer</Namespace>
</Query>
#load "xunit"
// using XUnitPriorityOrderer
// see: https://github.com/frederic-prusse/XUnitPriorityOrderer
void Main()
{
RunTests(); // Call RunTests() or press Alt+Shift+T to initiate testing.
}
#region private::Tests
[TestCaseOrderer(CasePriorityOrderer.TypeName, CasePriorityOrderer.AssembyName)]
public class TestClass
{
static List<string> Order { get; set; }
public TestClass()
{
Order = Order ?? new List<string>();
}
[Fact, Order(2)]
void Test_Xunit_AnotherTest()
{
Order.Add("Test_Xunit_AnotherTest");
Assert.True(3 + 1 == 4);
}
[Fact, Order(1)]
void Test_Xunit()
{
Order.Add("Test_XUnit");
Assert.True(1 + 1 == 2);
}
[Fact, Order(99)]
void Print_Order()
{
Order.Add("Print_Order");
var strOrder = string.Join(", ", Order.ToArray());
strOrder.Dump("Execution Order");
Assert.True(true);
}
}
#endregion
This will run the tests in given order (Order(1), Order(2) and then Order(99)) and will dump the executed tests finally (test method Print_Order()).

You can't, by design. It's deliberately random in order to prevent anyone getting one of those either by desire or by accident.
The randomness is only for a given Test class, so you may be able to achieve your goals by wrapping items you want to control the order of inside a nested class - but in that case, you'll still end up with random order whenever you have more than two Test Methods in a class.
If you're trying to manage the building up of fixtures or context, the built-in IUseFixture<T> mechanism may be appropriate. See the xUnit Cheat Sheet for examples.
But you really need to tell us more about what you're trying to do or we'll just have to get speculative.