I have a class that raises an event with an error message.
In some of my tests I am subscribing to the event and asserting that the error message is not empty.
[Test]
public MyMethod_DoBad_ErrorMessageNotEmpty()
{
var logic = new MyClass();
string ErrorMessage = String.Empty;
logic.DisplayError += delegate(string s)
{
ErrorMessage = s;
};
logic.DoItBadly();
Assert.IsFalse(String.IsNullOrWhiteSpace(ErrorMessage));
}
//MyClass
public void DoItBadly()
{
//do something naughty but not final
DisplayError("Naughty");
//some other problem arises
if (1==1)
DisplayError("Something else naughty");
}
However I am starting to find in edge case testing that my new tests that should fail, pass because it has raised an error event previously in the code before it has got to where I want it to.
Therefore should I be asserting that the error message contains a specified string?
However I am starting to find in edge case testing that my new tests that should fail, pass because it has raised an error event previously in the code before it has got to where I want it to.
That suggests you're either reusing an existing object between tests, or that your test is doing too much. If you can't help but do work before the real operation you want to test, you can write your test as:
// Construct objects
// Do setup work
// Check that there's no error message yet
// Do work you expect to fail
// Check that there *is* an error message
Of course you can check for the exact error message, but that's likely to end up being time-consuming. If you're using reasonably ad-hoc error reporting (not worrying about i18n etc) then I'd personally just check whether an error message is present or not.
I think you should test both this cases in different tests:
[Test]
public ShouldRaiseNaughtyErrorWhenDoBadly()
{
var logic = new MyClass();
string errorMessage = String.Empty;
logic.DisplayError += delegate(string s) {errorMessage = s; };
logic.DoItBadly();
Assert.That(errorMessage, Is.EqualTo("Naughty"));
}
[Test]
public ShouldRaiseElseNaughtyErrorWhenDoBadlyWithOtherProblem()
{
var logic = new MyClass();
string errorMessage = String.Empty;
logic.DisplayError += delegate(string s) {errorMessage = s; };
// do something for other problem condition
logic.DoItBadly();
Assert.That(errorMessage, Is.EqualTo("Something else naughty"));
}
Or, if you need to check both errors where raised:
[Test]
public ShouldRaiseBothErrors()
{
var logic = new MyClass();
List<string> errorMessages = new List<string>();
logic.DisplayError += delegate(string s) {errorMessages.Add(s); };
// do something for other problem condition
logic.DoItBadly();
Assert.That(errorMessages.Count, Is.EqualTo(2));
Assert.That(errorMessages[0], Is.EqualTo("Naughty"));
Assert.That(errorMessages[1], Is.EqualTo("Something else naughty"));
}
UPDATE:
Considering event-based nature of your notifications, you can catch them all and then search for some concrete error:
[Test]
public ShouldRaiseNaughtyErrorWhenDoBadly()
{
var logic = new MyClass();
List<string> errorMessages = new List<string>();
logic.DisplayError += delegate(string s) { errorMessages.Add(s); };
logic.DoItBadly();
Assert.That(errorMessages.Contains("Naughty"));
}
Ideally, you would want to isolate and abstract the areas of DoItBadly() that are polluting your error text with the error message, so that you can test the rest of the method without problem.
However, given the understanding that it can often be easier said than done, the next best thing would be to only populate ErrorMessage with s if a certain condition is met (or have a whitelist of error messages that would not populate ErrorMessage). So if you only set ErrorMessage if it is not the error you have deemed "acceptable," then your test should pass and your own requirement should be met.
Though even better would be to assert a positive outcome, instead of making your success case the absence of negative outcomes.
Related
I am trying to check that an exception that I throw gives the correct error message.
I have a method in a class that withdraws (substracts) from a value. If the value is less than 0, I throw an exception.
if (amount < 0)
{
throw new System.ArgumentOutOfRangeException("amount", AmountLessThanZeroMessage);
}
This is my error message:
public const string AmountLessThanZeroMessage = "Amount is less than zero";
However, when I try to write a unit test to see if I get the correct message, it fails because of the parameter. This is the test:
[Fact]
public void CannotWithdrawLessThanZero()
{
// Arrange
var account = new Account("User", 23);
// Act
account.DepositCash(100);
var thrownException = Assert.Throws<ArgumentOutOfRangeException>(() => account.WithdrawCash(-10));
// Assert
Assert.Equal(Account.AmountLessThanZeroMessage, thrownException.Message);
}
The result includes the parameter in the end, resulting in a failed test:
It seems the actual message includes which parameter it refers to. How do I correct this message? Should I just add the line (Parameter 'amount') to the expected string, or are there any better options?
You can create the exact same exception and use its message property. Like the code below
[Fact]
public void CannotWithdrawLessThanZero()
{
// Arrange
var account = new Account("User", 23);
var expectedException = new System.ArgumentOutOfRangeException("amount", AmountLessThanZeroMessage);
// Act
account.DepositCash(100);
var thrownException = Assert.Throws<ArgumentOutOfRangeException>(() => account.WithdrawCash(-10));
// Assert
Assert.Equal(expectedException.Message, thrownException.Message);
}
Messages changes often, message can be localized, so testing message equality will make tests more fragile.
Instead you can test that correct exception type was thrown, or even better and create domain specific exception to be sure that exception have been thrown for correct reasons.
public class WithdrawOfNegativeAmountNotAllowedException : Exception
{
public WithdrawOfNegativeAmountNotAllowedException(int amount)
: base($"Amount is less than zero ({amount})")
{
}
}
[Fact]
public void Cannot_withdraw_less_than_zero()
{
var account = new Account("User", 23);
account.DepositCash(100);
Action withdraw = () => account.WithdrawCash(-10);
withdraw.Should().Throw<WithdrawOfNegativeAmountNotAllowedException>();
}
In case you still want to test for correct message. Assertions of strings can be done in multiple ways, I would suggest to make assertion as loose as possible.
Test for part of the message which unlikely will change. For example test that message starts with some text or contain critical value.
[Fact]
public void Cannot_withdraw_less_than_zero()
{
var account = new Account("User", 23);
account.DepositCash(100);
Action withdraw = () => account.WithdrawCash(-10);
withdraw.Should()
.Throw<WithdrawOfNegativeAmountNotAllowedException>()
.And
.Message.Should().ContainEquivalentOf("Amount is less than zero");
}
I'm trying to learn automation with Selenium Webdriver using c#. I have my custom method Assert. What I did to continue the test after catching an AssertFailedException is using try-catch below is my code
public static void assert(string value, IWebElement element)
{
try
{
Assert.AreEqual(value, element.Text);
}
catch (AssertFailedException e)
{
Console.WriteLine(e.Message.ToString());
}
}
My problem is it catches all AssertFailedException(which is my goal) but the result of the test is PASSED in visual studio. My question is, how do I implement Continue on Failure and Fail the test if the Console contains Exceptions. Thankyou in advance guys!
As far as I understand, you want to do several checks inside your test, and at the very end of it to determine whether any of of them failed. You may need to write some custom code to achieve this. For example, you may introduce a class Assertion:
internal class Assertion
{
private readonly string title;
private readonly object expected;
private readonly object actual;
public Assertion(string title, object expected, object actual)
{
this.title = title;
this.expected = expected;
this.actual = actual;
}
public bool IsMatch()
{
return this.actual == this.expected;
}
public override string ToString()
{
return $"Title: {title}. Expected: {expected}. Actual: {actual}";
}
}
When your test is running, you will create new instances of Assertion class and store them in a list. At the end of the test, you can use the following method:
private static void VerifyAssertions(Assertion[] assertions)
{
var failedAssertions = assertions.Where(a => !a.IsMatch()).ToArray();
if (failedAssertions.Any())
{
throw new AssertFailedException(string.Join<Assertion>("; ", failedAssertions));
}
}
You could try using verify instead of assert for minor checks. Assert by default indicates a major checkpoint and script execution will be terminated on fail and if you catch that exception the reporting will be ignored - this is expected behaviour. However, a verify indicates that the script can continue even on fail - in this case the failed step will be reported and the script will continue.
To put it simply, use assert when you don't want the script to proceed on failure and use verify when you want the script to report the failure and proceed.
In NSubstitute, is it possible to specify a message that should be thrown if a Received fails? Something like the following:
[Test]
public void Should_execute_command()
{
var command = Substitute.For<ICommand>();
var something = new SomethingThatNeedsACommand(command);
something.DoSomething();
command.Received()
.Execute()
.Because("We should have executed the command that was passed in");
}
For comparison, in Moq, you can do this:
command.Verify(c => c.Execute, "We should have executed the command that was passed in");
And then you get that message as part of the test failure message in your test runner. This can help to make test failures easier to read/diagnose. Is there anything similar in NSubstitute?
There is no way to change the message that you receive in the ReceivedCallException; it is built directly from hard-coded strings in the NSubstitute.Core.ReceivedCallsExceptionThrower Throw method:
public void Throw(ICallSpecification callSpecification, IEnumerable<ICall> matchingCalls, IEnumerable<ICall> nonMatchingCalls, Quantity requiredQuantity)
{
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.AppendLine(string.Format("Expected to receive {0} matching:\n\t{1}", requiredQuantity.Describe("call", "calls"), callSpecification));
this.AppendMatchingCalls(callSpecification, matchingCalls, stringBuilder);
if (requiredQuantity.RequiresMoreThan<ICall>(matchingCalls))
{
this.AppendNonMatchingCalls(callSpecification, nonMatchingCalls, stringBuilder);
}
throw new ReceivedCallsException(stringBuilder.ToString());
}
Unless you're ready to dive in NSubstitute code, for now the best bet would to catch the ReceivedCallsException and throw your own message.
In my test program in Nunit, I want to verify that it's getting the write Argument Exception by verifying the message.
[Test]
public void ArgumentsWorkbookNameException()
{
const string workbookName = "Tester.xls";
var args = new[] { workbookName, "Sheet1", "Source3.csv", "Sheet2", "Source4.csv" };
Assert.Throws(typeof(ArgumentException), delegate { var appargs = new ApplicationArguments(args); }, "Invalid ending parameter of the workbook. Please use .xlsx");
}
After testing this out, this doesn't work when I modified the message in the main program.
int wbLength = args[0].Length;
// Telling the user to type in the correct workbook name file.
if (args[0].Substring(wbLength-5,5)!=".xlsx")
{
throw new ArgumentException(
"Invalid ending parameter of the workbook. Please use .xlsx random random");
}
The unit test still passed, regardless if I changed the message.
How do I do it? Or is there no such things in C#. My colleague said there are options like that in Ruby and RSPEC, but he's not 100% sure on C#.
Use the fluent interface to create assertions:
Assert.That(() => new ApplicationArguments(args),
Throws.TypeOf<ArgumentException>()
.With.Message.EqualTo("Invalid ending parameter of the workbook. Please use .xlsx random random"));
I agree with Jon that "such tests are unnecessarily brittle". However, there are at least two ways to check for exception message:
1: Assert.Throws returns an exception, so you can make an assertion for its message:
var exception = Assert.Throws<ArgumentException>(() => new ApplicationArguments(args));
Assert.AreEqual("Invalid ending parameter of the workbook. Please use .xlsx random random", exception.Message);
2: [HISTORICAL] Before NUnit 3, you could also use ExpectedException attribute. But, take a note that attribute waits for an exception in the whole tested code, not only in code which throws an exception you except. Thus, using this attribute is not recommended.
[Test]
[ExpectedException(typeof(ArgumentException), ExpectedMessage = "Invalid ending parameter of the workbook. Please use .xlsx random random")]
public void ArgumentsWorkbookNameException()
{
const string workbookName = "Tester.xls";
var args = new[] { workbookName, "Sheet1", "Source3.csv", "Sheet2", "Source4.csv" };
new ApplicationArguments(args);
}
You may also use FluentAssertions to do so, e.g.
subject.Invoking(y => y.Foo("Hello"))
.Should().Throw<InvalidOperationException>()
.WithMessage("Hello is not allowed at this moment");
The message parameter in Assert.Throws isn't the expected exception message; it's the error message to include with the assertion failure if the test fails.
I don't believe that NUnit supports testing the exception message out of the box, and I'd argue that such tests are unnecessarily brittle anyway. If you really want to write your own such helper method you can do so, but I personally wouldn't encourage it. (I very rarely specify a test failure message either, unless it's to include some diagnostic information. If a test fails I'm going to look at the test anyway, so the message doesn't add much.)
I would encourage you to use the generic overload instead though, and a lambda expression, for simplicity:
Assert.Throws<ArgumentException>(() => new ApplicationArguments(args));
(If that's your actual code by the way, there are other problems - try passing in new[] { "xyz" } as an argument...)
In .NET Core 3.1 MSTest project, this is how I did it.
[TestMethod]
public async Task SaveItemAsync_NameIsNull_ThrowsException()
{
var item = new Item
{
Name = null
};
var result = await Assert.ThrowsExceptionAsync<ArgumentException>(() => _service.SaveItemAsync(item));
Assert.AreEqual("The item's name must be set.", result.Message);
}
I'm writing a unit test for this one method which returns "void". I would like to have one case that the test passes when there is no exception thrown. How do I write that in C#?
Assert.IsTrue(????)
(My guess is this is how I should check, but what goes into "???")
I hope my question is clear enough.
Your unit test will fail anyway if an exception is thrown - you don't need to put in a special assert.
This is one of the few scenarios where you will see unit tests with no assertions at all - the test will implicitly fail if an exception is raised.
However, if you really did want to write an assertion for this - perhaps to be able to catch the exception and report "expected no exception but got this...", you can do this:
[Test]
public void TestNoExceptionIsThrownByMethodUnderTest()
{
var myObject = new MyObject();
try
{
myObject.MethodUnderTest();
}
catch (Exception ex)
{
Assert.Fail("Expected no exception, but got: " + ex.Message);
}
}
(the above is an example for NUnit, but the same holds true for MSTest)
In NUnit, you can use:
Assert.DoesNotThrow(<expression>);
to assert that your code does not throw an exception. Although the test would fail if an exception is thrown even if there was no Assert around it, the value of this approach is that you can then distinguish between unmet expectations and bugs in your tests, and you have the option of adding a custom message that will be displayed in your test output. A well-worded test output can help you locate errors in your code that have caused a test to fail.
I think it's valid to add tests to ensure that your code is not throwing exceptions; for example, imagine you are validating input and need to convert an incoming string to a long. There may be occasions when the string is null, and this is acceptable, so you want to ensure that the string conversion does not throw an exception. There will therefore be code to handle this occasion, and if you haven't written a test for it you will be missing coverage around an important piece of logic.
This helper class scratched my itch with MSTest. Maybe it can scratch yours also.
[TestMethod]
public void ScheduleItsIneligibilityJob_HasValid_CronSchedule()
{
// Arrange
var factory = new StdSchedulerFactory();
IScheduler scheduler = factory.GetScheduler();
// Assert
AssertEx.NoExceptionThrown<FormatException>(() =>
// Act
_service.ScheduleJob(scheduler)
);
}
public sealed class AssertEx
{
public static void NoExceptionThrown<T>(Action a) where T:Exception
{
try
{
a();
}
catch (T)
{
Assert.Fail("Expected no {0} to be thrown", typeof(T).Name);
}
}
}
Don't test that something doesn't happen. It's like assuring that code doesn't break. That's sort of implied, we all strive for non-breaking, bug-less code. You want to write tests for that? Why just one method? Don't you want all your methods being tested that they don't throw some exception? Following that road, you'll end up with one extra, dummy, assert-less test for every method in your code base. It brings no value.
Of course, if your requirement is to verify method does catch exceptions, you do test that (or reversing it a bit; test that it does not throw what it is supposed to catch).
However, the general approach/practices remain intact - you don't write tests for some artificial/vague requirements that are out of scope of tested code (and testing that "it works" or "doesn't throw" is usually an example of such - especially in scenario when method's responsibilities are well known).
To put it simple - focus on what your code has to do and test for that.
I like to see an Assert.Whatever at the end of each test, just for consistency... without one, can I really be sure there's not supposed to be one there?
For me, this is as simple as putting Assert.IsTrue(true);
I know I didn't accidentally put that code in there, and thus I should be confident enough at quick a skim through that this was as intended.
[TestMethod]
public void ProjectRejectsGappedVersioningByDefault() {
var files = new List<ScriptFile>();
files.Add(ScriptProjectTestMocks.GetVersion1to2());
files.Add(ScriptProjectTestMocks.GetVersion3to4());
Assert.Throws<ScriptProject.InvalidProjectFormatException>(() => {
var sut = new ScriptProject(files);
});
}
[TestMethod]
public void ProjectAcceptsGappedVersionsExplicitly() {
var files = new List<ScriptFile>();
files.Add(ScriptProjectTestMocks.GetVersion1to2());
files.Add(ScriptProjectTestMocks.GetVersion3to4());
var sut = new ScriptProject(files, true);
Assert.IsTrue(true); // Assert.Pass() would be nicer... build it in if you like
}
My friend Tim told me about ExpectedException. I really like this b/c it is more succinct, less code, and very explicit that you are testing for an exception.
[TestMethod()]
[ExpectedException(typeof(System.Exception))]
public void DivideTest()
{
int numerator = 4;
int denominator = 0;
int actual = numerator / denominator;
}
You can read way more about it here: ExpectedException Attribute Usage.
With Xunit you can use this:
var exception = Record.Exception(() =>
MethodUnderTest());
Assert.Null(exception);
or for async operations
var exception = await Record.ExceptionAsync(async () =>
await MethodUnderTestAsync());
Assert.Null(exception);
Another way which worked for me is to store it in a variable and check output.
var result = service.Run()
Assert.IsFalse(result.Errors.Any())
using Moq;
using Xunit;
[Fact]
public void UnitTest_DoesNotThrow_Exception()
{
var builder = new Mock<ISomething>().Object;
//Act
var exception = Record.Exception(() => builder.SomeMethod());
//Assert
Assert.Null(exception);
}