How to programmatically tell NUnit to repeat a test?
Background:
I'm running NUnit from within my C# code, using a SimpleNameFilter and a RemoteTestRunner. My application reads a csv file, TestList.csv, that specifies what tests to run. Up to that point everything works ok.
Problem:
The problem is when I put the same test name two times in my TestList file. In that case, my application correctly reads and loads the SimpleNameFilter with two instances of the test name. This filter is then passed to the RemoteTestRunner. Then, Nunit executes the test only once. It seems that when Nunit sees the second instance of a test it already ran, it ignores it.
How can I override such behavior? I'd like to have NUnit run the same test name two times or more as specified in my TestList.csv file.
Thank you,
Joe
http://www.nunit.org/index.php?p=testCase&r=2.5
TestCaseAttribute serves the dual purpose of marking a method with
parameters as a test method and providing inline data to be used when
invoking that method. Here is an example of a test being run three
times, with three different sets of data:
[TestCase(12,3, Result=4)]
[TestCase(12,2, Result=6)]
[TestCase(12,4, Result=3)]
public int DivideTest(int n, int d)
{
return( n / d );
}
Running an identical test twice should have the same result. An individual test can either pass or it can fail. If you have tests that work sometimes and fail another then it feels like the wrong thing is happening. Which is why NUnit doesn't support this out of the box. I imagine it would also cause problems in the reporting of the results of the test run, does it say that test X worked, or failed if both happened?
The closest thing you're going to get in Nunit is something like the TestCaseSource attribute (which you already seem to know about). You can use TestCaseSource to specify a method, which can in turn, read from a file. So, you could for example have a file "cases.txt" which looks like this:
Test1,1,2,3
Test2,wibble,wobble,wet
Test1,2,3,4
And then use this from your tests like so:
[Test]
[TestCaseSource("Test1Source")]
public void Test1(string a, string b, string c) {
}
[Test]
[TestCaseSource("Test2Source")]
public void Test2(string a, string b, string c) {
}
public IEnumerable Test1Source() {
return GetCases("Test1");
}
public IEnumerable Test2Source() {
return GetCases("Test2");
}
public IEnumerable GetCases(string testName) {
var cases = new List<IEnumerable>();
var lines = File.ReadAllLines(#"cases.txt").Where(x => x.StartsWith(testName));
foreach (var line in lines) {
var args = line.Split(',');
var currentcase = new List<object>();
for (var i = 1; i < args.Count(); i++) {
currentcase.Add(args[i]);
}
cases.Add(currentcase.ToArray());
}
return cases;
}
This is obviously a very basic example, that results in Test1 being called twice and Test2 being called once, with the arguments from the text file. However, this is again only going to work if the arguments passed to the test are different, since nunit uses the arguments to create a unique test name, although you could work around this by having the test source generate a unique number for each method call and passing it to the test as an extra argument that the test simply ignores.
An alternative would be for you to run the nunit from a script that calls nunit over and over again for each line of the file, although I imagine this may cause you other issues when you're consolidating the reporting from the multiple runs.
Related
I use Theory with MemberData like this:
[Theory]
[MemberData(nameof(TestParams))]
public void FijewoShortcutTest(MapMode mapMode)
{
...
and when it works, it is all fine, but when it fails XUnit iterates over all data I pass as parameters. In my case it is fruitless attempt, I would like to stop short -- i.e. when the first set of parameters make the test to fail, stop the rest (because they will fail as well -- again, it is my case, not general rule).
So how to tell XUnit to stop Theory on the first fail?
The point of a Theory is to have multiple independent tests running the same code of different data. If you only actually want one test, just use a Fact and iterate over the data you want to test within the method:
[Fact]
public void FijewoShortcutTest()
{
foreach (MapMode mapMode in TestParams)
{
// Test code here
}
}
That will mean you can't easily run just the test for just one MapMode though. Unless it takes a really long time to execute the tests for some reason, I'd just live with "if something is badly messed up, I get a lot of broken tests".
This question already has answers here:
How to run a test many times with data read from .csv file (data driving)
(3 answers)
Closed 6 years ago.
I Have a desktop Application in which I have 200 Test cases with different input parameters
Now the issue is Every time I am recording the Each and every test case with different input parameters
Is there is any way so that I can copy the code and change the parameters so that my code remains same for all the test cases only input parameters change
There are a few things to address here. Firstly, you can run the test using a data driven approach as described in the link above.
More importantly, in my opinion anyway, is how you are writing your test so that they can be data driven and what exactly are you testing that you need so many combinations?
When writing tests, it is important to have reusable code to test. I would recommend looking at something like Code First Scaffolding or Coded UI Page Modeling (I wrote the page modeling stuff). With these approaches, your test code is far more maintainable and flexible (easier to change by hand). This would allow for extremely simple data driven tests.
public void WhenPerformingCalculation_ThenResultIsCorrect() {
// imagine calculator with two numbers and a sign
var testResult =
modelUnderTest.LeftSideNumber.SetValue(3) // set first number
.Operator.SetValue("*") // set sign
.RightSideNumber.SetValue(10) // set right number
.Evaluate.Click() // press evaluate button
.Result; // get the result
Assert.AreEqual(testResult, 30);
}
becomes
public class CalculationParameters
{
public double LeftNumber {get;set;}
public string Operator {get;set;}
public double RightNumber {get;set;}
public double Result {get;set;}
public override string ToString(){ return $"{LeftNumber} {Operator} {RightNumber} = {Result}"; }
}
public void WhenPerformingCalculation_ThenResultIsCorrect() {
ICollection<CalculationParameters> parameters = getParameters();
List<Exception> exceptions = new List<Exception>();
foreach(CalculationParameters parameter in parameters)
{
try
{
var testResult =
modelUnderTest.LeftSideNumber.SetValue(parameter.LeftNumber) // set first number
.Operator.SetValue(parameter.Operator) // set sign
.RightSideNumber.SetValue(parameter.RightNumber) // set right number
.Evaluate.Click() // press evaluate button
.Result; // get the result
Assert.AreEqual(testResult, parameter.Result);
}
catch (Exception e)
{
exceptions.Add(new Exception($"Failed for parameters: {parameter}", e));
}
}
if(exceptions.Any()){
throw new AggregateException(exceptions);
}
}
Secondly, why do you need to test so many combinations of input / output in a given test? If you are testing things like, "Given the login page, when supplying invalid credentials, then a warning is supplied to the user." How many invalid inputs do you really need to test? There would be a second test for valid credentials and no data driving is necessary.
I would caution you to be careful that you are not testing stuff that should be a unit test in your UI. It sounds like you are testing different combinations of inputs to see if the UI generates the correct output, which should probably be a unit test of your underlying system. When testing the UI, typically it is sufficient to test that the bindings to your view models are correct and not test that calculations or other server logic is performed accurately.
My provided example shows what I would NOT test client side unless that calculator only exists client side (no server side validation or logic regarding the calculation). Even in that case, I would probably get a javascript test runner to test the view model powering my calculator instead of using coded ui to do this test.
Would you be able to provide some example of the combinations of input/output you are testing?
You can use the event arguments to the application through command line with a batch script or you can create a function what will pass the requested parameters.
In the main header you can use
main(string eventargs[]);
where the string variable will be the event arguments from the command line
Im begginer in webdriver and c#. I want to use variable, from first test in another tests, how do I do that? I got to this point with some examples, but it does not work. I see that first test gets the login right, but when I start the second test and try to sendkeys, I get that loginName is null. (code is in short version, only to give you an idea of what Im trying to do)
[TestFixture]
public class TestClass
{
private IWebDriver driver;
private StringBuilder verificationErrors;
private string baseURL;
private bool acceptNextAlert = true;
static public String loginName;
static public String loginPassword;
[SetUp]
public void SetupTest()...
[TearDown]
public void TeardownTest()...
[Test]
public void GetLoginAndPassword()
{
loginName = driver.FindElement(By.XPath("...")).Text;
loginPassword = driver.FindElement(By.XPath("...")).Text;
Console.WriteLine(loginName);
}
[Test]
public void Test1()
{
driver.FindElement(By.Id("UserNameOrEmail")).SendKeys(loginName);
driver.FindElement(By.Id("Password")).SendKeys(loginPassword);
}
You cannot (and should not) send variables between tests. Tests methods are independent from another... and should actually Assert() something.
Your first method GetLoginAndPassword() isn't a test method per se but a utility method. If you use a Selenium PageObject pattern, this is probably a method of your PageObject class that you can run at the begining of your actual Test1() method.
The problem is that the methods marked with TestAttribute do not run sequentially in the same order you implemented them. Thus it might be possible that Test1 runs long before GetLoginAndPassword). You have to either call that method once from within the constructor or during test-initialization, or before every test-run.
[Test]
public void Test1()
{
GetLoginAndPassword();
driver.FindElement(By.Id("UserNameOrEmail")).SendKeys(loginName);
driver.FindElement(By.Id("Password")).SendKeys(loginPassword);
}
Probably your GetLoginAndPassword isn´t even a method to test but a method used from your tests (unless you actually have a method within your system under test called GetLoginAndPassword. However since there are no asserts at all your tests are somewhat weird.
The purpose of unit testing is to test if a specific unit (meaning a group of closely related classes) works as specified. It is not meant to test if your complete elaborate program works as specified.
Test driven design has the advantage that you are more aware what each function should do. Each function is supposes to transform a pre-condition into a specified post-condition, regardless of what you did before or after calling the function.
If your tests assume that other tests are run before your test is run, then you won't test the use case that the other functions are not called, or only some of these other required functions are called.
This leads to the conclusion that each test method should be able to be run independently. Each test should set up the precondition, call the function and check if the postcondition is met.
But what if my function A only works correctly if other function B is called?
In that case the specification of A ought to describe what happens if B was called before A was called, as well as what would happen if A was called without calling B first.
If your unit test would first test B and then A with the prerequisite that B was called, you would not test whether A would react according to specification without calling B.
Example.
Suppose we have a divider class, that will divide any number by a given denominator that can be set using a property.
public class Divider
{
public double Denominator {get; set;}
public double Divide(double numerator)
{
return numerator / this.Denominator;
}
}
It is obvious that in normal usage one ought to set property Denominator before calling Divide:
Divider divider = new divider(){Denominator = 3.14};
Console.WriteLine(divider.Divide(10);
Your specification ought to describe what happens if Divide is called without setting Denominator to a non-zero value. This description would be like:
If method Divide is called with a parameter value X and the value of Denominator is a non-zero Y, then the return value is X/Y. If the value of Denominator is zero, then the System.DivideByZeroException is thrown.
You should create at least two tests. One for the use case that Denominator was set at a proper non-zero value, and one for the use case that the Denominator is not set at all. And if you are very thorough: a test for the use case that the Denominator is first set to a non-zero value and then to a zero value.
Im running some tests on my code at the moment. My main test method is used to verify some data, but within that check there is a lot of potential for it to fail at any one point.
Right now, I've set up multiple Assert.Fail statements within my method and when the test is failed, the message I type is displayed as expected. However, if my method fails multiple times, it only shows the first error. When I fix that, it is only then I discover the second error.
None of my tests are dependant on any others that I'm running. Ideally what I'd like is the ability to have my failure message to display every failed message in one pass. Is such a thing possible?
As per the comments, here are how I'm setting up a couple of my tests in the method:
private bool ValidateTestOne(EntityModel.MultiIndexEntities context)
{
if (context.SearchDisplayViews.Count() != expectedSdvCount)
{
Assert.Fail(" Search Display View count was different from what was expected");
}
if (sdv.VirtualID != expectedSdVirtualId)
{
Assert.Fail(" Search Display View virtual id was different from what was expected");
}
if (sdv.EntityType != expectedSdvEntityType)
{
Assert.Fail(" Search Display View entity type was different from what was expected");
}
return true;
}
Why not have a string/stringbuilder that holds all the fail messages, check for its length at the end of your code, and pass it into Assert.Fail? Just a suggestion :)
The NUnit test runner (assuming thats what you are using) is designed to break out of the test method as soon as anything fails.
So if you want every failure to show up, you need to break up your test into smaller, single assert ones. In general, you only want to be testing one thing per test anyways.
On a side note, using Assert.Fail like that isn't very semantically correct. Consider using the other built-in methods (like Assert.Equals) and only using Assert.Fail when the other methods are not sufficient.
None of my tests are dependent on any others that I'm running. Ideally
what I'd like is the ability to have my failure message to display
every failed message in one pass. Is such a thing possible?
It is possible only if you split your test into several smaller ones.
If you are afraid code duplication which is usually exists when tests are complex, you can use setup methods. They are usually marked by attributes:
NUnit - SetUp,
MsTest - TestInitialize,
XUnit - constructor.
The following code shows how your test can be rewritten:
public class HowToUseAsserts
{
int expectedSdvCount = 0;
int expectedSdVirtualId = 0;
string expectedSdvEntityType = "";
EntityModelMultiIndexEntities context;
public HowToUseAsserts()
{
context = new EntityModelMultiIndexEntities();
}
[Fact]
public void Search_display_view_count_should_be_the_same_as_expected()
{
context.SearchDisplayViews.Should().HaveCount(expectedSdvCount);
}
[Fact]
public void Search_display_view_virtual_id_should_be_the_same_as_expected()
{
context.VirtualID.Should().Be(expectedSdVirtualId);
}
[Fact]
public void Search_display_view_entity_type_should_be_the_same_as_expected()
{
context.EntityType.Should().Be(expectedSdvEntityType);
}
}
So your test names could provide the same information as you would write as messages:
Right now, I've set up multiple Assert.Fail statements within my
method and when the test is failed, the message I type is displayed as
expected. However, if my method fails multiple times, it only shows
the first error. When I fix that, it is only then I discover the
second error.
This behavior is correct and many testing frameworks follow it.
I'd like to recommend stop using Assert.Fail() because it forces you to write specific messages for every failure. Common asserts provide good enough messages so you can replace you code with the following lines:
// Act
var context = new EntityModelMultiIndexEntities();
// Assert
Assert.Equal(expectedSdvCount, context.SearchDisplayViews.Count());
Assert.Equal(expectedSdVirtualId, context.VirtualID);
Assert.Equal(expectedSdvEntityType, context.EntityType);
But I'd recommend start using should-frameworks like Fluent Assertions which make your code mere readable and provide better output.
// Act
var context = new EntityModelMultiIndexEntities();
// Assert
context.SearchDisplayViews.Should().HaveCount(expectedSdvCount);
context.VirtualID.Should().Be(expectedSdVirtualId);
context.EntityType.Should().Be(expectedSdvEntityType);
I am writing unit tests using Nunit for a C# project.
I am trying to run a single test multiple times with different data using the TestCaseSource attribute.
I am doing this elsewhere without any problems, but now I am finding that the first time I run my tests, the code passes. The next time, it doesn't. Using some Console.WriteLine statements, I can see that the test data is different each time.
The method used to generate the data is internal to the test class, is not static and generates all required dependencies from scratch for each test.
--
I have a fake class which holds a queue of values to return when a given function is called. A new class is created for each test.
However, if the first time the test is run, it exhausts the queue, the next time it is run, no data is found. Surely this should be regenerated every time?
--
It is as if Nunit is not calling the method specified by the TestCaseSource attribute every time the test is run - only when the project is first loaded.
Is this expected? Is there a workaround?
EDIT:
Ok, here is a very basic example, below:
[TestFixture]
public class Tests
{
public interface IEntry
{
object Read();
}
[TestCaseSource("TestData")]
public void Test(Mock<IEntry> entry)
{
object o = entry.Object.Read();
object o2 = entry.Object.Read();
}
public System.Collections.IEnumerable TestData()
{
var entry = new Mock<IEntry>();
int call = 0;
entry.Setup(x => x.Read()).Returns(() =>
{
Console.WriteLine(call);
return null;
}).Callback(() =>
{
call++;
});
yield return new TestCaseData(entry);
}
}
If you watch the test output in Nunit, it should always display 0, followed by 1. In this case, each time the test is run it will be incremented. I.e. second run: 2 and 3, third run: 4 and 5, etc.
If you move the TestData code into Test, then the correct values are returned every time.
I assume you're using the NUnit GUI runner?
What you're seeing is (I believe, I can't find confirmation in the NUnit source easily) is an optimization of the test runner. Rather than re-create the values provided by the TestCaseSource attribute on every test run, it will only do so when the assembly under test changes.
If you change your code to remove the Moq dependency, it's a little clearer:
[TestFixture]
public class SampleTests
{
[TestCaseSource("TestData")]
public void Test(CallTracker callTracker)
{
callTracker.Call++;
callTracker.Call++;
}
public IEnumerable TestData()
{
yield return new TestCaseData(new CallTracker());
}
public class CallTracker
{
int call;
public int Call
{
get
{
return call;
}
set
{
call = value;
Console.WriteLine(call);
}
}
}
}
This yields the same behavior as your code. CallTracker is created anew whenever the TestCaseSource is evaluated, but since the call count keeps going up, the test runner must be resuing the same instance (for what I'm assuming is performance reasons).
Resharper's test runner in Visual Studio does not exhibit this behavior; it always shows 1 and 2 for repeated runs without recompiling. This is probably why it's slower to start running tests than the NUnit GUI runner is. Similarly, the NUnit console does not exhibit this behavior since it always starts cold.