MSTest now allows you to test that a certain piece of code throws a certain exception type, through:
Assert.ThrowsException<MyException>(() => foo.Bar());
However, I need a more flexible exception test; I need to not only test the type of the exception, but check that its message starts with a certain string (rather than matching the exact exception string). Is there a way I can do this with MSTest? If not, what's the best way for me to do it? Is there another testing framework that handles this better?
Ideally, the Assert would take in a second func that passed in the Exception thrown, and this func could test the Exception however it wanted and return true or false to indicate that the assert had succeeded or failed.
var ex = Assert.ThrowsException<MyException>(() => foo.Bar());
Assert.IsTrue(ex.Message.StartsWith("prefix"));
Related
I have an application where I throw exceptions in constructor of one class.
I have unit and integration tests written in nUnit and I check if this exception is thrown when I want, like this:
Assert.That(() => _fileOperationsModel.OpenFile("abc"), Throws.Exception.TypeOf<MissingFieldException>());
And it worked great until I have started to use Autofac. I need to create this class which throws an exception using autofac, so when constructor is called and my exception is thrown, there is no MissingFieldException anymore, but Autofac.Core.DependencyResolutionException.
And in that excepetion's details, there are 4 inner exceptions. 3 of them are also DependencyResolutionException and the last of them is my MissingFieldException.
I was trying to test it like this:
var ex = Assert.Throws<Autofac.Core.DependencyResolutionException>(() => _fileOperationsModel.OpenFile("abc"));
Assert.That(ex.InnerException, Is.TypeOf<MissingFieldException>());
but it doesn't work coz InnerException is only one and it is also DependencyResolutionException.
Do you have any idea how I can handle this exceptions and how I can test it? I am new in Autofac and I was trying to find something about that but with no result.
EDIT:
I know I can do that like this
var ex = Assert.Throws<Autofac.Core.DependencyResolutionException>(() => _fileOperationsModel.OpenFile("abc"));
Assert.That(ex.InnerException, Is.TypeOf<DependencyResolutionException>());
Assert.That(ex.InnerException.InnerException, Is.TypeOf<DependencyResolutionException>());
Assert.That(ex.InnerException.InnerException.InnerException, Is.TypeOf<DependencyResolutionException>());
Assert.That(ex.InnerException.InnerException.InnerException.InnerException, Is.TypeOf<MissingFieldException>());
but maybe there is some better and not so awful solution?
Maybe this earlier can give you some inspiration.
Otherwise, you can try writing your own constraint matcher based on the default exception matcher,
however in line 50 iterating over the inner exceptions.
If both are not an option, you could resort to your own proposed solution. I do not think deep nested inner exception verification is implemented in NUnit at this moment. Moreover, I think you can do without verifying the intermediate DependencyResolutionExceptions.
I have a method that takes a JSON object and puts it through several stages of processing, updating values in the database at each stage. We wanted this method to be fault tolerant, and decided that the desired behaviour would be, if any processing stage failed, log an error to the database and carry on with the next stage of processing, rather than aborting.
I've just made several changes to the behaviour of one of the processing steps. I then ran our unit test suite, expecting several of the tests to fail due to the new behaviour and point me at potential problem areas. Instead, the tests all passed.
After investigating, I realised that the mock data the tests run against didn't include certain key values important for the new behaviour. The tests were in fact throwing exceptions when they ran, but the exceptions were being caught and handled - and, because the tests don't run with a logger enabled, they were completely suppressed. So the new code didn't change the data in a way that would cause the tests to fail, because it was silently erroring instead.
This seems like the sort of problem that unit tests are there to catch, and the fact that they showed no trace means they're not serving their purpose. Is there any way that I can use NUnit to assert that no exception was ever thrown, even if it was handled? Or alternatively, is there a sensible way to refactor that would expose this issue better?
(Working in C#, but the question seems fairly language-agnostic)
First and foremost, in the scenario you describe, the presence or absence of exceptions is secondary. If you wrote the code to produce a desired result while catching and handling exceptions, then that result - whether it's a return value or some other effect - is the most important thing to test.
If the exceptions that you didn't see caused that result to be incorrect, then testing for the correct result will always reveal the problems. If you don't know what the expected result will be and are only interested in whether or not exceptions are getting handled, something is wrong. We can never determine whether or not anything works correctly according to whether or not it throws exceptions.
That aside, here's how to test whether or not your code is catching and logging exceptions that you wouldn't otherwise be able to observe:
If you're injecting a logger that looks something like this:
public interface ILogger
{
void LogError(Exception ex);
void LogMessage(string message);
}
...then a simple approach is to create a test double which stores the exceptions so that you can inspect it and see what was logged.
public class ListLoggerDouble : ILogger
{
public List<Exception> Exceptions = new List<Exception>();
public List<string> Messages = new List<string>();
public void LogError(Exception ex)
{
Exceptions.Add(ex);
}
public void LogMessage(string message)
{
Messages.Add(message);
}
}
After you've executed the method you're testing you can assert that a collection contains the exception(s) or message(s) you expect. If you wish you can also verify that there are none, although it seems like that might be redundant if the result you're testing for is correct.
I wouldn't create a logger that throws an exception and then write a test that checks for a thrown exception. That makes it look like the expected behavior of your code is to throw an exception, which is exactly the opposite of what it does. Tests help us to document expected behaviors. Also, what will you do if you want to verify that you caught and logged two exceptions?
Using Selenium in Visual Studio. Using NUnit to sort my testcases.
I'm writing a testcase that compares two serialnumbers with a if statement like this:
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
else
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
//method for stopping test missing!
I want to be able to abort the rest of the testsequence if the serialnumbers don't match.
Is there a "end/stop test" method or something similar I could put in else section?
I think you have a couple of options.
1) simply throw an exception (and fail the test)
Throwing an exception will fail a unit test. There are loads of different types of exceptions but the base is simply Exception. You can check the different types of exceptions available here. Where possible try to pick the exception that most closely represents the error (so bad arguments for example use ArgumentException or some derivative there of).
Your test would then look something like this:
[Test]
public void CompareVariables()
{
if (!string.Equals(serialNumberInfo, serialNumberReport))
throw new Exception($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
2) Use an assertion (and fail the test)
Unit tests are usually supposed to assert that something desirable happened. If that thing didn't happen then an exception should be thrown (which is often handled for you by some assertion framework).
So you could flip the test to do this:
[Test]
public void CompareVariables()
{
serialNumberInfo.ShouldBe(serialNumberReport);
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
This is done with Shouldly but there are countless assertion frameworks so pick your favourite. (mstest has one built in but I find it less readable but that is a personal preference).
Note, only use an assertion when you want to explicitly make sure that it should have happened. I.e. This needs to be true for my test to pass, rather than if this happened then abort. That's hard to explain so I hope that makes sense?
Exceptions for when something went wrong, Assertions for when something should have gone right.
3) Leave the test (and pass the test)
If the test exits without an exception being thrown (either manually or via an assertion framework) then the test is considered to be a passing test. Therefor if you wanted to treat this as a pass you could simply return from the test.
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
}
else
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
return;
}
// The rest of your tests
}
This will mark the test as passing, but mean the rest of the operations in the test are not run. I would try not to do this however - unless you really understand why you want this because you could start passing tests without knowing why they passed (i.e. without asserting anything)
I hope that helps
If you want to end the test early without failing it, simply use return.
[Test]
public void MyTest() {
//Do some stuff
if(!shouldContinue) {
return;
}
}
I do this reasonably often given certain conditions may warrant additional assertions, and other conditions may not. Throwing an exception will fail the test. This will not fail it.
Edit: I just noticed that the other responder mentioned this at the end of their answer. So ignore me :)
I've read this question about simulating throwing exceptions. Answers suggest I create a mock object that pretends being a real object.
That's not what I want to have. If I replace a real object I lose part of real code. I want to invoke real code with minimal changes and have an exception thrown from inside it at random point of that code.
Is it possible to have an exception thrown at random point of code being called from a Unit test?
don't put randomness into your unit tests. it will only bring you troubles. a unit tests should always have some consistence. especially if you want him to tell you what went wrong when he went red. if you implement a random exception throwing into your unit tests it can happen that he SOMETIMES crashes and returns a red bar. maybe the next run it the error is gone again. this is really not what unit tests are here for and you will have big troubles finding the issue for that once-occurred failed test.
a much better approach is to systematically test critical parts of your code. replace each crucial method with a mocked object, which throws the type of exception you want to test for and cover each test case like this. this will give you much more information about what went wrong when you are looking for that error.
hope this helps.
I can think about this work around.
1- Make a function called RandomException(); that throw exception if a random value is divisible by 3 otherwise it won't throw the exception. make sure this function is encapsulcate in this block of code.
#if DEBUG
void RandomException()
{
// gen random value
// if random value % 3 == 0 throw the exception now
}
#else
void RandomException()
{
}
#endif
that way when you release the code these function calls won't affect the program.
I hope that idea helps you.
It seems like you are trying to test the exception handling, i dont think this is a good approach to throw an exception randomly and check if it gets cought, if you need to test handling in certain conditions you need to simulate the condition in a real object, like an illegal situation which should cause an exception.
You still can produce this behaviour through a local varibale which you can set from outside (in your Unittest) which will cause the code to throw an exception depending on this variable, but as saied i dont think its a good approach.
My code operates on data which "should" be correct. However during development there are occasions when I obtain invalid data.
When that happens I would like to raise the debug assert and, if user choose to continue, the code would filter out the invalid records and continue to operate on "safe" data.
// assert incorrect data
Debug.Assert(person.Items.All(item => item.IsValid), "Inconsistent data!");
// operate on filtered data
this.ItemViewModels = new ObservableCollection<ItemViewModel>(
person.Items
.Where(i =>item.IsValid) // Use only correct data
.Select(i => new ItemViewModel(lang, i)));
I would like to unit test the code path when I choose to operate on the filtered data.
Question: Is there a way how to get past the assert call in the unit test?
Some equivalent to clicking OK=Continue in the "Assertion Failed" dialogue?
TIA
In addition to SLaks's answer I would add that what you want to do is logically inconsistent. An assert should be used to document a condition that cannot possibly be false. If the false condition ever arises then you know you have a bug; the purpose of an assert is (1) as a kind of comment that describes to the reader what must be true at this point in the code, and (2) a debugging aid that tells you when you have a bug.
Since correct asserts in correct code never fire, there is no way to test an assert firing. The premise of a test is that it produces a possible configuratin of your software and verifies its correctness; but correct code with correct asserts never have a configuration where the assert fires.
It sounds like you are using Assert not to document something that you know is true but rather something that you hope is true or is normally true. Don't use assertions for that. If there is any inputs to the program that cause the assertion to be violated then you need to either remove the assertion, or cause an exception when you get the invalid data so that the assertion never sees it. Assertions are meant to document what must be true, not what is true most of the time.
See also this related question:
Debug.Assert vs Exception Throwing
You should not use Debug.Assert for this.
Debug.Assert is intended to be used only as a debugging aid.
It will not be compiled at all in Release mode.
Instead, you should create your own method, which will show the user a simpler dialog box, and can be configured to always continue for unit testing. (eg, use a public static bool ShowWarnings property)