Using the test tools built into Visual Studio (in my case VS 2013, though I believe this extends to other versions as well), a 'TestMethod' (any method marked with a [TestMethod] attribute) can result in 1 of 3 statuses after being run:
Passed
Skipped/Inconclusive
Failed
A test is passed if it runs to completion with no uncaught exceptions. A test fails it throws an uncaught exception. A test is inconclusive if it throws a special type of uncaught exception: AssertInconclusiveException.
My question is, is it possible (in, say, a [TestCleanup] method) to treat an custom uncaught exception similarly to the AssertInconclusiveException and set the test status to 'Inconclusive' instead of 'Failed'? Or is the recognition of AssertInconclusiveException as a special exception built into Visual Studio itself? What I'm looking for, in rough pseudocode form, is roughly:
[TestCleanup]
public void TestCleanup()
{
if(this.TestContext.CurrentTestOutcome == UnitTestOutcome.Failed
&& this.TestContext.UncaughtException.GetType() == typeof(MyCustomException))
{
this.TestContext.CurrentTestOutcome = UnitTestOutcome.Inconclusive;
}
}
This would be used as part of an integration test suite to mark a test as inconclusive whenever the setup method for the test throws a special SetupFailureException (usually indicative of broken downstream system that is not the one under test).
I've been able to produce the desired behavior in the interim by defining:
public class SetupFailedException : AssertInconclusiveException
{
}
// example test that demonstrates desired behavior
[TestMethod]
public void TestThatFailsAsInconclusiveByThrowingUncaughtSpecialException()
{
throw new SetupFailedException();
}
But this seems messy since the thrown exception has nothing to do with an assertion and prevents custom exceptions which should fail tests as inconclusive from inheriting from any other abstract class.
The Inconclusive recognition is built into the test framework, by the time you get to cleanup, it's too late to change a fail into an inconclusive.
It feels like you should really be catching the exceptions from within your integration tests anyway, since this shows that you're recognising the method your testing can throw a certain type of exception and that the right thing to do is to ignore the result in that situation.
If you need to have multiple exceptions that can be detected like this, I'd be tempted to have them implement an interface, so that the catch block can just check if any exceptions caught implement that interface.
If you wanted to minimize the impact/visibility on the rest of your test code then you could wrap the call that could fail in an action, in which case your code might look something like this:
The interface:
public interface ISomeNonCriticalException {
}
The Caller code:
public void CallPossiblyInconclusiveMethod(Action something) {
try {
something();
}
catch (Exception ex) {
if (ex is ISomeNonCriticalException) Assert.Inconclusive();
else throw;
}
}
And in your test:
CallPossiblyInconclusiveMethod(()=>SomeMethodThatMightFail());
Related
I just started on a rather extensive automation project that uses MS's UnitTestFramework. One thing I noticed is that when there's an error in my code - not the app I test - the framework catches that error and fails the test in a nice happy way that allows for the test iteration to complete. However, I want to be able to see those exceptions & stack trace in my log4net logs, and so far I have found no way to grab them in my test cleanup (or anywhere outside of a try catch block which I have no intention of splattering in every method).
Anyone know how to get these exceptions into my logs?
You could use First-Chance Exception Notifications
via the AppDomain.FirstChanceException Event -
This event is only a notification. Handling this event does not handle
the exception or affect subsequent exception handling in any way.
After the event has been raised and event handlers have been invoked,
the common language runtime (CLR) begins to search for a handler for
the exception. FirstChanceException provides the application domain
with a first chance to examine any managed exception.
So something like this (note it is in a method marked as AssemblyInitialize, which means it runs once per test run, and the code is excluding the AssertFailedException thrown by MSTest when a test fails. You might want to exclude other exceptions as well, as otherwise there could be a lot of 'noise' in the logs.)
[TestClass]
public class Initialize
{
[AssemblyInitialize]
public static void InitializeLogging(TestContext testContext)
{
AppDomain.CurrentDomain.FirstChanceException += (source, e) =>
{
if (e.Exception is AssertFailedException == false)
LogManager.GetLogger("TestExceptions").Error(e.Exception);
};
}
}
If it feasible for you to replace the [TestMethod] attribute then you can define your own attribute MyTestMethod, say, by deriving from the default one like this:
public class MyTestMethodAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
TestResult[] testResults = base.Execute(testMethod);
foreach (var testResult in testResults.Where(e => e.Outcome == UnitTestOutcome.Failed))
testResult.LogOutput += $"Exception `{testResult.TestFailureException.GetType().Name}` with message `{testResult.TestFailureException.Message}`.";
return testResults;
}
}
The following test then produces the expected log message Exception TestFailedException with message Assert.Fail failed. Some exception text. in the standard output panel of Visual Studio's Test Explorer.
[TestClass]
public class Tests
{
[MyTestMethod]
public void Test()
=> Assert.Fail("Some exception text");
}
The approach also works when tests execute in parallel.
Using VS's testing framework, I'm currently writing my tests like this:
[TestMethod]
public void TestSomething()
{
try
{
var someTestValue = _someTestClass.SomeTestMethod();
Assert.IsNotNull(someTestValue);
}
catch (Exception e)
{
Assert.Fail(e.Message);
}
}
My logic is that if an exception is thrown in SomeTestMethod(), I'll immediately terminate the test displaying the exception message through Assert.Fail(e.Message).
The "normal way" of doing things would be:
[TestMethod]
public void TestSomething()
{
var someTestValue = _someTestClass.SomeTestMethod();
Assert.IsNotNull(someTestValue);
}
Is my approach correct, or is the "normal way" correct? Am I writing redundant code ?
I'd say this is redundant, yes. If the exception is an unexpected result then the test fails. Whether it fails by throwing an exception or by failing an Assert is treated the same way by the testing framework. Basically, a failed test is a failed test.
The testing framework will already fail the test if the test method throws an exception. You're adding extra work for no added value.
You're writing redundant code. The Test method already traps exceptions and will fail the test method.
If you are expecting some exception, use this attribute:
[TestMethod]
[ExpectedException(typeof(//Here the exception you expect))] <------------
public void TestSomething()
{
//Your test code
}
This means that if the tests throw this exception, will return as passed the tests.
If you don´t go to tests exceptions, then avoid try/catch blocks inside your tests. You will notice that something went wrong because the tests will fail ;)
I hope this helps
See this tutorial; there is an example unit test using the [expectedException()] attribute:
[Tutorial for ExpectedException Unit tests][1]
[1]: http://msdn.microsoft.com/en-us/library/hh694602%28v=vs.110%29.aspx#BKMK_Writing_your_tests
I have a test which uses an external assembly to access UI features in the application we're testing. This assembly throws an exception of its own custom type if the UI is not in the appropriate state.
I've set up a TestCleanup method that kills the application's process (while a TestInitialize starts it) so that after a test run has been completed, the UI is restarted with a clean state.
This work well under regular conditions, however, whenever an exception from the referenced assembly is thrown, it never gets to the cleanup method and jumps straight ahead to the next test. This doesn't happen with exceptions thrown from the test itself, like AssertFailedException. I even tried throwing a basic Exception from the test, and it got to the cleanup method.
If an exception is uncaught in TestInitialize, TestCleanup won't be called.
http://web.archive.org/web/20140310065725/http://connect.microsoft.com/VisualStudio/feedback/details/694337/testcleanup-method-does-not-run-when-it-should
When MSTest fails in TestInitialize, why doesn't TestCleanup get executed?
Under what circumstances are [ClassCleanup] and [TestCleanup] not run
This unfortunately diverges from the way C# handles exceptions in constructors : when this happens, the finalizer is called.
But you can directly call CleanUp method from the catch block
[TestCleanup]
public void Clenup()
{
..............
}
[TestMethod]
public void Test1()
{
try
{...................}
catch (Exception e)
{
Cleanup();
throw new Exception();
}
}
That was fixed in MsTest v2.
https://github.com/Microsoft/testfx/issues/250
Extract from the above link:
This was a conscious compat break to give unit test writers a choice to cleanup partially
initialized methods.
I am building a TestProject for my client/server setup. I want to verify that a test fails within a certain block of execution (there is no client to send to so the server's Send() method will throw an exception). Since I do not want to have the tests boot up a client and server and have them communicate (which I have had problems doing on a single machine.)
If the code reaches this line, that means that the program's execution flow could only fail within the responsibilities of another test. Is there an easier way to do this other than doing a substring check on the thrown exception's stacktrace? I feel like this method is not very expandable and would require constant attention if class names change.
Is there a way that doesn't even involve manually checking the exception's stacktrace?
If you are using NUnit
Without using DataAnnotations
[Test]
public void Test_XXXXXXX
{
var yourClass = new YourClass();
Assert.That(()=>yourClass.Method(),
.Throws.Exception
.TypeOf<TypeOfYourException>
.With.Property("Message")
.EqualTo("the message you are expecting goes here")
);
}
Using DataAnnotations
[Test]
[ExpectedException(typeof(ExceptionType), ExpectedMessage="your message goes here!")]
public void Test_XXXXXXX
{
var yourClass = new YourClass();
// Call your method in a way that it will fail
yourClass.YourMethod();
}
Is there anything unique about the exception in the class, other than that it's specific to that class?
If it's identifiable by the message you can test it as the other answer has shown, or like this if you're not using NUnit:
try {
myMethod();
Assert.Fail("Expected exception to be thrown.");
} catch (MyException ex) {
Assert.Equals("My Exception Message", ex.Message, "Exception message was formatted incorrectly.");
} catch (Exception) {
Assert.Fail("An exception was thrown, but of the wrong type.");
}
When you are unit-testing some class, there is two sources of exception:
Class which you are testing can throw an exception
Dependency of a class can throw an exception
In second case you usually either handle exception, or wrap it in more high-level exception and throw it to caller. So, how to test all these cases?
System under test throws exception
E.g. throwing exception in case of wrong argument passed to method (NUnit sample):
StockService service = new StockService();
Assert.Throws<ArgumentNullException>(() => service.Connect(null));
You don't need to check stack trace, because it's a class under test who supposed to throw exception.
Dependency throws exception and we handle it
When your class has dependencies, you should mock dependencies in order to test your class in isolation. So, it's very easy to setup mocks to throw exceptions when your class interacts with them. Consider case when service should run on default settings if configuration file not found (Moq sample):
var configMock = new Mock<IStockServiceConfig>();
configMock.Setup(c => c.Load()).Throws<FileNotFoundException>();
StockService service = new StockService(configMock.Object);
service.Connect("google");
configMock.VerifyAll();
Assert.That(service.Port, Is.EqualTo(80));
This test will fail if you will not try to load config, or if you will not handle FileNotFoundException.
Exception stacktrace does not matters here - we don't care whether our direct dependency thrown exception, or it was some other class inside dependency. Actually we don't know if that class exists - we are interacting only with direct dependency. And we should care only about fact that dependency can throw exception, which we can handle.
Dependency throws exception and we wrap it
And last case - wrapping exception in something more high-level. Consider previous sample, but config is really important and you can't start without configuration file. In this case you wrap FileNotFoundException into something more business-specific, which makes sense at higher levels of your application. E.g. StockServiceInitializationException:
var configMock = new Mock<IStockServiceConfig>();
configMock.Setup(c => c.Load()).Throws<FileNotFoundException>();
StockService service = new StockService(configMock.Object);
Assert.Throws<StockServiceInitializationException>(_ => service.Connect("bing"));
configMock.VerifyAll();
As you can see, we also don't care about stacktrace of exception, which throws our dependency. It also could be some wrapper of more low level exception. Expected behavior of service - throw high-level initialization exception if config not found. We are verifying that behavior here.
I'm new to Nunit testing and I wanted to test the following constructor:
public class IngredientDAONHibernate : NutritionLibrary.DAO.IngredientDAO
{
private Configuration config;
private ISessionFactory factory;
public IngredientDAONHibernate()
{
try
{
config = new Configuration();
config.AddClass(typeof(NutritionLibrary.Entity.Ingredient));
config.AddClass(typeof(Entity.Nutrient));
config.AddClass(typeof(Entity.NutrientIngredient));
factory = config.BuildSessionFactory();
}
catch (Exception e)
{
// logger.Error("Exception Occured", e);
}
}
The test stub is as follows:
[TestMethod()]
public void IngredientDAONHibernateConstructorTest()
{
IngredientDAONHibernate target = new IngredientDAONHibernate();
}
Could someone help me with some tips as to how I can get started? Thanks!
I would suggest that you don't catch all the exceptions, but, if there are certain you want to catch and ignore then just do those, otherwise it is harder to tell if you have a problem.
I tend not to test the constructor, unless it is doing quite a bit, and it will be obvious if there is a problem when doing the other unit tests, as, if the constructor fails you should see by the exception, and by the fact that all the tests will be failing.
If you want to test this constructor, limit your exceptions and just ensure that there is no exception when you run the test.
Let me turn the question back on to you.. How do you know if the constructor executed as intended ?
Normally constructors are trivial.. but here it seems that you have some third party lib interfacing code that you need some confidence with.
If you only want to test that there are no exceptions raised from within the constructor... then Extract a logger interface. Now in your test pass in a mock logger (a fake can also suffice), which should help you to sense if an exception was logged.
[TestMethod()]
public void IngredientDAONHibernateConstructorTest()
{
_errorLogged = false;
ILogger logger = this; // make test fixture implement the logger interface ; self-shunt
IngredientDAONHibernate target = new IngredientDAONHibernate(logger);
Assert.IsNotNull(target);
Assert.IsFalse(_errorLogged,
String.Format("ERROR! Constructor has thrown {0}", _loggedException) );
}
bool _errorLogged;
Exception _loggedException;
public void Error(string message, Exception e)
{
_errorLogged = true;
_loggedException = e;
}
Like everyone else, I would caution against catching Exception. The documentation should tell you what exceptions can be expected and you could write specific Catch clauses for each of them, if that made sense.
Leaving that aside, your tests should verify that your objects are in the state you expect. Given what we can see, all you can do is test that it's not null. If you can force an exception, you could test that the logger gets the exception message. If there was a public property that gave access to your factory, you would want to test that (not null, perhaps other things) and likewise with the config field. For that, if you have access to the added classes, you could test not null and count==3. If the exact objects are important, you could test that they are there. It's always tricky to decide how much to trust third-party stuff. Generally, I do unless I get evidence that I shouldn't.
When it comes to testing methods, test for each parameter whatever is possible. So, for a string, test null and emptystring plus relevant (valid and invalid) values. If you're expecting alphanumerics, test special chars as well. If data has boundaries test values on and either side of the boundary(ies) plus typical values. E.g. if you're expecting an int between 0 and 10, test -1, 0, 1, 5, 9, 10, 11. It only takes seconds to write but it can save you hours in two years when you're trying to fix a bug.