Im running some tests on my code at the moment. My main test method is used to verify some data, but within that check there is a lot of potential for it to fail at any one point.
Right now, I've set up multiple Assert.Fail statements within my method and when the test is failed, the message I type is displayed as expected. However, if my method fails multiple times, it only shows the first error. When I fix that, it is only then I discover the second error.
None of my tests are dependant on any others that I'm running. Ideally what I'd like is the ability to have my failure message to display every failed message in one pass. Is such a thing possible?
As per the comments, here are how I'm setting up a couple of my tests in the method:
private bool ValidateTestOne(EntityModel.MultiIndexEntities context)
{
if (context.SearchDisplayViews.Count() != expectedSdvCount)
{
Assert.Fail(" Search Display View count was different from what was expected");
}
if (sdv.VirtualID != expectedSdVirtualId)
{
Assert.Fail(" Search Display View virtual id was different from what was expected");
}
if (sdv.EntityType != expectedSdvEntityType)
{
Assert.Fail(" Search Display View entity type was different from what was expected");
}
return true;
}
Why not have a string/stringbuilder that holds all the fail messages, check for its length at the end of your code, and pass it into Assert.Fail? Just a suggestion :)
The NUnit test runner (assuming thats what you are using) is designed to break out of the test method as soon as anything fails.
So if you want every failure to show up, you need to break up your test into smaller, single assert ones. In general, you only want to be testing one thing per test anyways.
On a side note, using Assert.Fail like that isn't very semantically correct. Consider using the other built-in methods (like Assert.Equals) and only using Assert.Fail when the other methods are not sufficient.
None of my tests are dependent on any others that I'm running. Ideally
what I'd like is the ability to have my failure message to display
every failed message in one pass. Is such a thing possible?
It is possible only if you split your test into several smaller ones.
If you are afraid code duplication which is usually exists when tests are complex, you can use setup methods. They are usually marked by attributes:
NUnit - SetUp,
MsTest - TestInitialize,
XUnit - constructor.
The following code shows how your test can be rewritten:
public class HowToUseAsserts
{
int expectedSdvCount = 0;
int expectedSdVirtualId = 0;
string expectedSdvEntityType = "";
EntityModelMultiIndexEntities context;
public HowToUseAsserts()
{
context = new EntityModelMultiIndexEntities();
}
[Fact]
public void Search_display_view_count_should_be_the_same_as_expected()
{
context.SearchDisplayViews.Should().HaveCount(expectedSdvCount);
}
[Fact]
public void Search_display_view_virtual_id_should_be_the_same_as_expected()
{
context.VirtualID.Should().Be(expectedSdVirtualId);
}
[Fact]
public void Search_display_view_entity_type_should_be_the_same_as_expected()
{
context.EntityType.Should().Be(expectedSdvEntityType);
}
}
So your test names could provide the same information as you would write as messages:
Right now, I've set up multiple Assert.Fail statements within my
method and when the test is failed, the message I type is displayed as
expected. However, if my method fails multiple times, it only shows
the first error. When I fix that, it is only then I discover the
second error.
This behavior is correct and many testing frameworks follow it.
I'd like to recommend stop using Assert.Fail() because it forces you to write specific messages for every failure. Common asserts provide good enough messages so you can replace you code with the following lines:
// Act
var context = new EntityModelMultiIndexEntities();
// Assert
Assert.Equal(expectedSdvCount, context.SearchDisplayViews.Count());
Assert.Equal(expectedSdVirtualId, context.VirtualID);
Assert.Equal(expectedSdvEntityType, context.EntityType);
But I'd recommend start using should-frameworks like Fluent Assertions which make your code mere readable and provide better output.
// Act
var context = new EntityModelMultiIndexEntities();
// Assert
context.SearchDisplayViews.Should().HaveCount(expectedSdvCount);
context.VirtualID.Should().Be(expectedSdVirtualId);
context.EntityType.Should().Be(expectedSdvEntityType);
Related
I use Theory with MemberData like this:
[Theory]
[MemberData(nameof(TestParams))]
public void FijewoShortcutTest(MapMode mapMode)
{
...
and when it works, it is all fine, but when it fails XUnit iterates over all data I pass as parameters. In my case it is fruitless attempt, I would like to stop short -- i.e. when the first set of parameters make the test to fail, stop the rest (because they will fail as well -- again, it is my case, not general rule).
So how to tell XUnit to stop Theory on the first fail?
The point of a Theory is to have multiple independent tests running the same code of different data. If you only actually want one test, just use a Fact and iterate over the data you want to test within the method:
[Fact]
public void FijewoShortcutTest()
{
foreach (MapMode mapMode in TestParams)
{
// Test code here
}
}
That will mean you can't easily run just the test for just one MapMode though. Unless it takes a really long time to execute the tests for some reason, I'd just live with "if something is badly messed up, I get a lot of broken tests".
Let me start by saying that I know about the DebuggerStepThroughAttribute and I'm using it in a number of places with much success.
However, I'm searching for a complementary solution that would work for my specific scenario, which I'll now illustrate...
Say I have a homegrown data-access framework. This framework comes with lots of unit tests which ensure that all my high-level data-access APIs are working as expected. In these tests, there is often a requirement to first seed some test-specific data to a one-off database, then execute the actual test on that data.
The thing is, I might rely on unit tests not just to give me a passive green/red indication about my code, but also to help me zero in on the source of occasional regression. Given the way I've written the tests, it's easy to imagine that a small subset of them could sometimes give me grief, because the code that performs the test data seeding and actual test code both use the same framework APIs at lower levels.
So for example, if my debugging of a failed test happened to require that I place a breakpoint inside one such common method, debugger would stop there a number of times (maybe an annoyingly large number of times!) before I'd get to the point I'm interested in (the actual test, not the seeding).
Leaving aside the fact that I could theoretically refactor everything and improve decoupling, I'm asking this:
Is there a general way to quickly and easily disable debugger breaking for a specific code block, including any sub-calls that might be made from that block, when any of the executed lines could have a breakpoint associated?
The only solution that I'm aware of is to use conditional breakpoints. I would need to set a certain globally accessible flag when entering the method that I wanted to exclude and clear it when exiting. Any conditional breakpoints would then have to require that flag must not be set.
But this seems tedious, because breakpoints are often added, removed, then added again, etc. Given the rudimentary breakpoint management support in Visual Studio this quickly becomes really annoying.
Is there another way? Preferably by manipulating the debugger directly or indirectly, similarly to how the DebuggerStepThroughAttribute does it for single method scope?
EDIT:
Here's a contrived example of what I might have:
public class MyFramework
{
public bool TryDoCommonWork(string s)
{
// Picture an actual breakpoint here instead of this line.
// As it is, debugger would stop here 3 times during the seeding
// phase and then one final time during the test phase.
Debugger.Break();
if (s != null)
{
// Do the work.
return true;
}
return false;
}
}
[TestClass]
public class MyTests
{
[TestMethod]
public void Test()
{
var fw = new MyFramework();
// Seeding stage of test.
fw.TryDoCommonWork("1");
fw.TryDoCommonWork("2");
fw.TryDoCommonWork("3");
// Test.
Assert.IsTrue(fw.TryDoCommonWork("X"));
}
}
What I'm really looking for is something roughly similar to this:
[TestClass]
public class MyTests
{
[TestMethod]
public void Test()
{
var fw = new MyFramework();
// Seeding stage of test with no debugger breaking.
using (Debugger.NoBreakingWhatsoever())
{
fw.TryDoCommonWork("1");
fw.TryDoCommonWork("2");
fw.TryDoCommonWork("3");
}
// Test with normal debugger breaking.
Assert.IsTrue(fw.TryDoCommonWork("X"));
}
}
Using Selenium in Visual Studio. Using NUnit to sort my testcases.
I'm writing a testcase that compares two serialnumbers with a if statement like this:
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
else
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
//method for stopping test missing!
I want to be able to abort the rest of the testsequence if the serialnumbers don't match.
Is there a "end/stop test" method or something similar I could put in else section?
I think you have a couple of options.
1) simply throw an exception (and fail the test)
Throwing an exception will fail a unit test. There are loads of different types of exceptions but the base is simply Exception. You can check the different types of exceptions available here. Where possible try to pick the exception that most closely represents the error (so bad arguments for example use ArgumentException or some derivative there of).
Your test would then look something like this:
[Test]
public void CompareVariables()
{
if (!string.Equals(serialNumberInfo, serialNumberReport))
throw new Exception($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
2) Use an assertion (and fail the test)
Unit tests are usually supposed to assert that something desirable happened. If that thing didn't happen then an exception should be thrown (which is often handled for you by some assertion framework).
So you could flip the test to do this:
[Test]
public void CompareVariables()
{
serialNumberInfo.ShouldBe(serialNumberReport);
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
This is done with Shouldly but there are countless assertion frameworks so pick your favourite. (mstest has one built in but I find it less readable but that is a personal preference).
Note, only use an assertion when you want to explicitly make sure that it should have happened. I.e. This needs to be true for my test to pass, rather than if this happened then abort. That's hard to explain so I hope that makes sense?
Exceptions for when something went wrong, Assertions for when something should have gone right.
3) Leave the test (and pass the test)
If the test exits without an exception being thrown (either manually or via an assertion framework) then the test is considered to be a passing test. Therefor if you wanted to treat this as a pass you could simply return from the test.
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
}
else
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
return;
}
// The rest of your tests
}
This will mark the test as passing, but mean the rest of the operations in the test are not run. I would try not to do this however - unless you really understand why you want this because you could start passing tests without knowing why they passed (i.e. without asserting anything)
I hope that helps
If you want to end the test early without failing it, simply use return.
[Test]
public void MyTest() {
//Do some stuff
if(!shouldContinue) {
return;
}
}
I do this reasonably often given certain conditions may warrant additional assertions, and other conditions may not. Throwing an exception will fail the test. This will not fail it.
Edit: I just noticed that the other responder mentioned this at the end of their answer. So ignore me :)
This question already has answers here:
How to run a test many times with data read from .csv file (data driving)
(3 answers)
Closed 6 years ago.
I Have a desktop Application in which I have 200 Test cases with different input parameters
Now the issue is Every time I am recording the Each and every test case with different input parameters
Is there is any way so that I can copy the code and change the parameters so that my code remains same for all the test cases only input parameters change
There are a few things to address here. Firstly, you can run the test using a data driven approach as described in the link above.
More importantly, in my opinion anyway, is how you are writing your test so that they can be data driven and what exactly are you testing that you need so many combinations?
When writing tests, it is important to have reusable code to test. I would recommend looking at something like Code First Scaffolding or Coded UI Page Modeling (I wrote the page modeling stuff). With these approaches, your test code is far more maintainable and flexible (easier to change by hand). This would allow for extremely simple data driven tests.
public void WhenPerformingCalculation_ThenResultIsCorrect() {
// imagine calculator with two numbers and a sign
var testResult =
modelUnderTest.LeftSideNumber.SetValue(3) // set first number
.Operator.SetValue("*") // set sign
.RightSideNumber.SetValue(10) // set right number
.Evaluate.Click() // press evaluate button
.Result; // get the result
Assert.AreEqual(testResult, 30);
}
becomes
public class CalculationParameters
{
public double LeftNumber {get;set;}
public string Operator {get;set;}
public double RightNumber {get;set;}
public double Result {get;set;}
public override string ToString(){ return $"{LeftNumber} {Operator} {RightNumber} = {Result}"; }
}
public void WhenPerformingCalculation_ThenResultIsCorrect() {
ICollection<CalculationParameters> parameters = getParameters();
List<Exception> exceptions = new List<Exception>();
foreach(CalculationParameters parameter in parameters)
{
try
{
var testResult =
modelUnderTest.LeftSideNumber.SetValue(parameter.LeftNumber) // set first number
.Operator.SetValue(parameter.Operator) // set sign
.RightSideNumber.SetValue(parameter.RightNumber) // set right number
.Evaluate.Click() // press evaluate button
.Result; // get the result
Assert.AreEqual(testResult, parameter.Result);
}
catch (Exception e)
{
exceptions.Add(new Exception($"Failed for parameters: {parameter}", e));
}
}
if(exceptions.Any()){
throw new AggregateException(exceptions);
}
}
Secondly, why do you need to test so many combinations of input / output in a given test? If you are testing things like, "Given the login page, when supplying invalid credentials, then a warning is supplied to the user." How many invalid inputs do you really need to test? There would be a second test for valid credentials and no data driving is necessary.
I would caution you to be careful that you are not testing stuff that should be a unit test in your UI. It sounds like you are testing different combinations of inputs to see if the UI generates the correct output, which should probably be a unit test of your underlying system. When testing the UI, typically it is sufficient to test that the bindings to your view models are correct and not test that calculations or other server logic is performed accurately.
My provided example shows what I would NOT test client side unless that calculator only exists client side (no server side validation or logic regarding the calculation). Even in that case, I would probably get a javascript test runner to test the view model powering my calculator instead of using coded ui to do this test.
Would you be able to provide some example of the combinations of input/output you are testing?
You can use the event arguments to the application through command line with a batch script or you can create a function what will pass the requested parameters.
In the main header you can use
main(string eventargs[]);
where the string variable will be the event arguments from the command line
How to programmatically tell NUnit to repeat a test?
Background:
I'm running NUnit from within my C# code, using a SimpleNameFilter and a RemoteTestRunner. My application reads a csv file, TestList.csv, that specifies what tests to run. Up to that point everything works ok.
Problem:
The problem is when I put the same test name two times in my TestList file. In that case, my application correctly reads and loads the SimpleNameFilter with two instances of the test name. This filter is then passed to the RemoteTestRunner. Then, Nunit executes the test only once. It seems that when Nunit sees the second instance of a test it already ran, it ignores it.
How can I override such behavior? I'd like to have NUnit run the same test name two times or more as specified in my TestList.csv file.
Thank you,
Joe
http://www.nunit.org/index.php?p=testCase&r=2.5
TestCaseAttribute serves the dual purpose of marking a method with
parameters as a test method and providing inline data to be used when
invoking that method. Here is an example of a test being run three
times, with three different sets of data:
[TestCase(12,3, Result=4)]
[TestCase(12,2, Result=6)]
[TestCase(12,4, Result=3)]
public int DivideTest(int n, int d)
{
return( n / d );
}
Running an identical test twice should have the same result. An individual test can either pass or it can fail. If you have tests that work sometimes and fail another then it feels like the wrong thing is happening. Which is why NUnit doesn't support this out of the box. I imagine it would also cause problems in the reporting of the results of the test run, does it say that test X worked, or failed if both happened?
The closest thing you're going to get in Nunit is something like the TestCaseSource attribute (which you already seem to know about). You can use TestCaseSource to specify a method, which can in turn, read from a file. So, you could for example have a file "cases.txt" which looks like this:
Test1,1,2,3
Test2,wibble,wobble,wet
Test1,2,3,4
And then use this from your tests like so:
[Test]
[TestCaseSource("Test1Source")]
public void Test1(string a, string b, string c) {
}
[Test]
[TestCaseSource("Test2Source")]
public void Test2(string a, string b, string c) {
}
public IEnumerable Test1Source() {
return GetCases("Test1");
}
public IEnumerable Test2Source() {
return GetCases("Test2");
}
public IEnumerable GetCases(string testName) {
var cases = new List<IEnumerable>();
var lines = File.ReadAllLines(#"cases.txt").Where(x => x.StartsWith(testName));
foreach (var line in lines) {
var args = line.Split(',');
var currentcase = new List<object>();
for (var i = 1; i < args.Count(); i++) {
currentcase.Add(args[i]);
}
cases.Add(currentcase.ToArray());
}
return cases;
}
This is obviously a very basic example, that results in Test1 being called twice and Test2 being called once, with the arguments from the text file. However, this is again only going to work if the arguments passed to the test are different, since nunit uses the arguments to create a unique test name, although you could work around this by having the test source generate a unique number for each method call and passing it to the test as an extra argument that the test simply ignores.
An alternative would be for you to run the nunit from a script that calls nunit over and over again for each line of the file, although I imagine this may cause you other issues when you're consolidating the reporting from the multiple runs.