I just started on a rather extensive automation project that uses MS's UnitTestFramework. One thing I noticed is that when there's an error in my code - not the app I test - the framework catches that error and fails the test in a nice happy way that allows for the test iteration to complete. However, I want to be able to see those exceptions & stack trace in my log4net logs, and so far I have found no way to grab them in my test cleanup (or anywhere outside of a try catch block which I have no intention of splattering in every method).
Anyone know how to get these exceptions into my logs?
You could use First-Chance Exception Notifications
via the AppDomain.FirstChanceException Event -
This event is only a notification. Handling this event does not handle
the exception or affect subsequent exception handling in any way.
After the event has been raised and event handlers have been invoked,
the common language runtime (CLR) begins to search for a handler for
the exception. FirstChanceException provides the application domain
with a first chance to examine any managed exception.
So something like this (note it is in a method marked as AssemblyInitialize, which means it runs once per test run, and the code is excluding the AssertFailedException thrown by MSTest when a test fails. You might want to exclude other exceptions as well, as otherwise there could be a lot of 'noise' in the logs.)
[TestClass]
public class Initialize
{
[AssemblyInitialize]
public static void InitializeLogging(TestContext testContext)
{
AppDomain.CurrentDomain.FirstChanceException += (source, e) =>
{
if (e.Exception is AssertFailedException == false)
LogManager.GetLogger("TestExceptions").Error(e.Exception);
};
}
}
If it feasible for you to replace the [TestMethod] attribute then you can define your own attribute MyTestMethod, say, by deriving from the default one like this:
public class MyTestMethodAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
TestResult[] testResults = base.Execute(testMethod);
foreach (var testResult in testResults.Where(e => e.Outcome == UnitTestOutcome.Failed))
testResult.LogOutput += $"Exception `{testResult.TestFailureException.GetType().Name}` with message `{testResult.TestFailureException.Message}`.";
return testResults;
}
}
The following test then produces the expected log message Exception TestFailedException with message Assert.Fail failed. Some exception text. in the standard output panel of Visual Studio's Test Explorer.
[TestClass]
public class Tests
{
[MyTestMethod]
public void Test()
=> Assert.Fail("Some exception text");
}
The approach also works when tests execute in parallel.
Related
I would like Visual Studio to break when a handled exception happens (i.e. I don't just want to see a "First chance" message, I want to debug the actual exception).
e.g. I want the debugger to break at the exception:
try
{
System.IO.File.Delete(someFilename);
}
catch (Exception)
{
//we really don't care at runtime if the file couldn't be deleted
}
I came across these notes for Visual Studio.NET:
1) In VS.NET go to the Debug Menu >>
"Exceptions..." >> "Common Language
Runtime Exceptions" >> "System" and
select "System.NullReferenceException"
2) In the bottom of that dialog there
is a "When the exception is thrown:"
group box, select "Break into the
debugger"
3) Run your scenario. When the
exception is thrown, the debugger will
stop and notify you with a dialog that
says something like:
"An exception of type "System.NullReferenceException" has
been thrown.
[Break] [Continue]"
Hit [Break]. This will put you on the
line of code that's causing the
problem.
But they do not apply to Visual Studio 2005 (there is no Exceptions option on the Debug menu).
Does anyone know where the find this options dialog in Visual Studio that the "When the exception is thrown" group box, with the option to "Break into the debugger"?
Update: The problem was that my Debug menu didn't have an Exceptions item. I customized the menu to manually add it.
With a solution open, go to the Debug - Windows - Exception Settings (Ctrl+Alt+E) menu option. From there you can choose to break on Thrown or User-unhandled exceptions.
EDIT: My instance is set up with the C# "profile" perhaps it isn't there for other profiles?
There is an 'exceptions' window in VS2005 ... try Ctrl+Alt+E when debugging and click on the 'Thrown' checkbox for the exception you want to stop on.
Took me a while to find the new place for expection settings, therefore a new answer.
Since Visual Studio 2015 you control which Exceptions to stop on in the Exception Settings Window (Debug->Windows->Exception Settings). The shortcut is still Ctrl-Alt-E.
The simplest way to handle custom exceptions is selecting "all exceptions not in this list".
Here is a screenshot from the english version:
Here is a screenshot from the german version:
From Visual Studio 2015 and onward, you need to go to the "Exception Settings" dialog (Ctrl+Alt+E) and check off the "Common Language Runtime Exceptions" (or a specific one you want i.e. ArgumentNullException) to make it break on handled exceptions.
Step 1
Step 2
Check Managing Exceptions with the Debugger page, it explains how to set this up.
Essentially, here are the steps (during debugging):
On the Debug menu, click Exceptions.
In the Exceptions dialog box, select Thrown for an entire category of exceptions, for example, Common Language Runtime Exceptions.
-or-
Expand the node for a category of exceptions, for example, Common Language Runtime Exceptions, and select Thrown for a specific exception within that category.
A technique I use is something like the following. Define a global variable that you can use for one or multiple try catch blocks depending on what you're trying to debug and use the following structure:
if(!GlobalTestingBool)
{
try
{
SomeErrorProneMethod();
}
catch (...)
{
// ... Error handling ...
}
}
else
{
SomeErrorProneMethod();
}
I find this gives me a bit more flexibility in terms of testing because there are still some exceptions I don't want the IDE to break on.
The online documentation seems a little unclear, so I just performed a little test. Choosing to break on Thrown from the Exceptions dialog box causes the program execution to break on any exception, handled or unhandled. If you want to break on handled exceptions only, it seems your only recourse is to go through your code and put breakpoints on all your handled exceptions. This seems a little excessive, so it might be better to add a debug statement whenever you handle an exception. Then when you see that output, you can set a breakpoint at that line in the code.
There are some other aspects to this that need to be unpacked. Generally, an app should not throw exceptions unless something exceptional happens.
Microsoft's documentation says:
For conditions that are likely to occur but might trigger an exception, consider handling them in a way that will avoid the exception.
and
A class can provide methods or properties that enable you to avoid making a call that would trigger an exception.
Exceptions degrade performance and disrupt the debugging experience because you should be able to break on all exceptions in any running code.
If you find that your debugging experience is poor because the debugger constantly breaks on pointless exceptions, you may need to detect handled exceptions in your tests. This technique allows you to fail tests when code throws unexpected exceptions.
Here are some helper functions for doing that
public class HandledExceptionGuard
{
public static void DoesntThrowException(Action test,
Func<object?, Exception, bool>? ignoreException = null)
{
var errors = new List<ExceptionInformation>();
EventHandler<FirstChanceExceptionEventArgs> handler = (s, e) =>
{
if (e.Exception is AssertFailedException) return;
if (ignoreException?.Invoke(s, e.Exception) ?? false) return;
errors.Add(new ExceptionInformation(s, e.Exception, AppDomain.CurrentDomain.FriendlyName));
};
AppDomain.CurrentDomain.FirstChanceException += handler;
test();
AppDomain.CurrentDomain.FirstChanceException -= handler;
if (errors.Count > 0)
{
throw new ExceptionAssertionException(errors);
}
}
public async static Task DoesntThrowExceptionAsync(Func<Task> test,
Func<object?, Exception, bool>? ignoreException = null)
{
var errors = new List<ExceptionInformation>();
EventHandler<FirstChanceExceptionEventArgs> handler = (s, e) =>
{
if (e.Exception is AssertFailedException) return;
if (ignoreException?.Invoke(s, e.Exception) ?? false) return;
errors.Add(new ExceptionInformation(s, e.Exception, AppDomain.CurrentDomain.FriendlyName));
};
AppDomain.CurrentDomain.FirstChanceException += handler;
await test();
AppDomain.CurrentDomain.FirstChanceException -= handler;
if (errors.Count > 0)
{
throw new ExceptionAssertionException(errors);
}
}
}
If you wrap any code in these methods as below, the test will fail when a handled exception occurs. You can ignore exceptions with the callback. This validates your code against unwanted handled exceptions.
[TestClass]
public class HandledExceptionTests
{
private static void SyncMethod()
{
try
{
throw new Exception();
}
catch (Exception)
{
}
}
private static async Task AsyncMethod()
{
try
{
await Task.Run(() => throw new Exception());
}
catch (Exception)
{
}
}
[TestMethod]
public void SynchronousTest()
{
HandledExceptionGuard.DoesntThrowException(() => SyncMethod());
}
[TestMethod]
public async Task AsyncTest()
{
await HandledExceptionGuard.DoesntThrowExceptionAsync(() => AsyncMethod());
}
}
I have few selenium tests and there are exception handling mechanisms for the most expected exceptions like ElementNotFoundException, ElementNotVisibleException in my global exception handling block. In the below block using the WebDriverWait I was able to handle the known exceptions but sometimes in my webpage there are chances of getting ElementNotInteractble exception and some other unhandled exceptions.
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds ((Constants.MediumWaitTime)));
wait.IgnoreExceptionTypes(
typeof(WebDriverTimeoutException),
typeof(NoSuchElementException),
typeof(ElementNotVisibleException),
typeof(StaleElementReferenceException),
typeof(ElementClickInterceptedException),
typeof(ElementNotSelectableException),
typeof(ElementNotInteractableException));
try
{
Func(TestCaseInput, testCaseStep);
}
catch (WebDriverTimeoutException webDriverTimeoutException)
{
testCaseStep.ErrorMessage = $#"""Exception Occoured in Method:
'{MethodBase.GetCurrentMethod().Name}' with Exception: '{webDriverTimeoutException.StackTrace}'
""WebDriverTimeoutException.InnerException is::"" {webDriverTimeoutException.InnerException}
""Exception occured due to timeout""";
}
Is there any way to capture or listen to unhandled exceptions in MSTest in TestCleanUp. Is there any way to get the exceptions thrown from TestExplorer using some reflection or other mechanisms in c#
NOTE: I do not want to have some try/catch blocks for every method.
I was able to handle every exception raised from selenium using the EventFiringWebDriver. This solves one part of my problem. But there are some exceptions raised from .net framework for example I'm expecting a dropdown to contain some values and select one of the value from dropdown Like
IList accounts = driver.findElements(By.XPath("//a[id='accounts']"));
accounts.FirstOrDefault().Click(). If the accounts list doesn't have any webelements then it throws a "System.InvalidOperationException: Sequence contains no matching element" . I can put an if condition before clicking the element. if(accounts.Any()) but I cannot put such conditions around all such dropdowns.
I went through some links which are helpful in the cases of asp.net or .netcore to log firstexceptions or any exceptions but I couldn't find related links to MSTest unhandled exceptions. This link https://stackify.com/csharp-catch-all-exceptions/ has been useful for asp.net or .netcore
to log every exception:
Try something like the folllowing. Extend the TestMethod and decorate with the new one.
public class SeleniumTestMethodAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
try {
base.Execute(testMethod);
}
catch (WebDriverTimeoutException webDriverTimeoutException)
{
testCaseStep.ErrorMessage = $#"""Exception Occoured in Method:
'{MethodBase.GetCurrentMethod().Name}' with Exception: '{webDriverTimeoutException.StackTrace}'
""WebDriverTimeoutException.InnerException is::"" {webDriverTimeoutException.InnerException}
""Exception occured due to timeout""";
}
}
}
And then decorate with
[SeleniumTestMethodAttribute]
public void TestMethodWithSelenium() {
}
Using the test tools built into Visual Studio (in my case VS 2013, though I believe this extends to other versions as well), a 'TestMethod' (any method marked with a [TestMethod] attribute) can result in 1 of 3 statuses after being run:
Passed
Skipped/Inconclusive
Failed
A test is passed if it runs to completion with no uncaught exceptions. A test fails it throws an uncaught exception. A test is inconclusive if it throws a special type of uncaught exception: AssertInconclusiveException.
My question is, is it possible (in, say, a [TestCleanup] method) to treat an custom uncaught exception similarly to the AssertInconclusiveException and set the test status to 'Inconclusive' instead of 'Failed'? Or is the recognition of AssertInconclusiveException as a special exception built into Visual Studio itself? What I'm looking for, in rough pseudocode form, is roughly:
[TestCleanup]
public void TestCleanup()
{
if(this.TestContext.CurrentTestOutcome == UnitTestOutcome.Failed
&& this.TestContext.UncaughtException.GetType() == typeof(MyCustomException))
{
this.TestContext.CurrentTestOutcome = UnitTestOutcome.Inconclusive;
}
}
This would be used as part of an integration test suite to mark a test as inconclusive whenever the setup method for the test throws a special SetupFailureException (usually indicative of broken downstream system that is not the one under test).
I've been able to produce the desired behavior in the interim by defining:
public class SetupFailedException : AssertInconclusiveException
{
}
// example test that demonstrates desired behavior
[TestMethod]
public void TestThatFailsAsInconclusiveByThrowingUncaughtSpecialException()
{
throw new SetupFailedException();
}
But this seems messy since the thrown exception has nothing to do with an assertion and prevents custom exceptions which should fail tests as inconclusive from inheriting from any other abstract class.
The Inconclusive recognition is built into the test framework, by the time you get to cleanup, it's too late to change a fail into an inconclusive.
It feels like you should really be catching the exceptions from within your integration tests anyway, since this shows that you're recognising the method your testing can throw a certain type of exception and that the right thing to do is to ignore the result in that situation.
If you need to have multiple exceptions that can be detected like this, I'd be tempted to have them implement an interface, so that the catch block can just check if any exceptions caught implement that interface.
If you wanted to minimize the impact/visibility on the rest of your test code then you could wrap the call that could fail in an action, in which case your code might look something like this:
The interface:
public interface ISomeNonCriticalException {
}
The Caller code:
public void CallPossiblyInconclusiveMethod(Action something) {
try {
something();
}
catch (Exception ex) {
if (ex is ISomeNonCriticalException) Assert.Inconclusive();
else throw;
}
}
And in your test:
CallPossiblyInconclusiveMethod(()=>SomeMethodThatMightFail());
I am building a TestProject for my client/server setup. I want to verify that a test fails within a certain block of execution (there is no client to send to so the server's Send() method will throw an exception). Since I do not want to have the tests boot up a client and server and have them communicate (which I have had problems doing on a single machine.)
If the code reaches this line, that means that the program's execution flow could only fail within the responsibilities of another test. Is there an easier way to do this other than doing a substring check on the thrown exception's stacktrace? I feel like this method is not very expandable and would require constant attention if class names change.
Is there a way that doesn't even involve manually checking the exception's stacktrace?
If you are using NUnit
Without using DataAnnotations
[Test]
public void Test_XXXXXXX
{
var yourClass = new YourClass();
Assert.That(()=>yourClass.Method(),
.Throws.Exception
.TypeOf<TypeOfYourException>
.With.Property("Message")
.EqualTo("the message you are expecting goes here")
);
}
Using DataAnnotations
[Test]
[ExpectedException(typeof(ExceptionType), ExpectedMessage="your message goes here!")]
public void Test_XXXXXXX
{
var yourClass = new YourClass();
// Call your method in a way that it will fail
yourClass.YourMethod();
}
Is there anything unique about the exception in the class, other than that it's specific to that class?
If it's identifiable by the message you can test it as the other answer has shown, or like this if you're not using NUnit:
try {
myMethod();
Assert.Fail("Expected exception to be thrown.");
} catch (MyException ex) {
Assert.Equals("My Exception Message", ex.Message, "Exception message was formatted incorrectly.");
} catch (Exception) {
Assert.Fail("An exception was thrown, but of the wrong type.");
}
When you are unit-testing some class, there is two sources of exception:
Class which you are testing can throw an exception
Dependency of a class can throw an exception
In second case you usually either handle exception, or wrap it in more high-level exception and throw it to caller. So, how to test all these cases?
System under test throws exception
E.g. throwing exception in case of wrong argument passed to method (NUnit sample):
StockService service = new StockService();
Assert.Throws<ArgumentNullException>(() => service.Connect(null));
You don't need to check stack trace, because it's a class under test who supposed to throw exception.
Dependency throws exception and we handle it
When your class has dependencies, you should mock dependencies in order to test your class in isolation. So, it's very easy to setup mocks to throw exceptions when your class interacts with them. Consider case when service should run on default settings if configuration file not found (Moq sample):
var configMock = new Mock<IStockServiceConfig>();
configMock.Setup(c => c.Load()).Throws<FileNotFoundException>();
StockService service = new StockService(configMock.Object);
service.Connect("google");
configMock.VerifyAll();
Assert.That(service.Port, Is.EqualTo(80));
This test will fail if you will not try to load config, or if you will not handle FileNotFoundException.
Exception stacktrace does not matters here - we don't care whether our direct dependency thrown exception, or it was some other class inside dependency. Actually we don't know if that class exists - we are interacting only with direct dependency. And we should care only about fact that dependency can throw exception, which we can handle.
Dependency throws exception and we wrap it
And last case - wrapping exception in something more high-level. Consider previous sample, but config is really important and you can't start without configuration file. In this case you wrap FileNotFoundException into something more business-specific, which makes sense at higher levels of your application. E.g. StockServiceInitializationException:
var configMock = new Mock<IStockServiceConfig>();
configMock.Setup(c => c.Load()).Throws<FileNotFoundException>();
StockService service = new StockService(configMock.Object);
Assert.Throws<StockServiceInitializationException>(_ => service.Connect("bing"));
configMock.VerifyAll();
As you can see, we also don't care about stacktrace of exception, which throws our dependency. It also could be some wrapper of more low level exception. Expected behavior of service - throw high-level initialization exception if config not found. We are verifying that behavior here.
In the code snippet below, why wouldn't the attached handler (of AppDomain.CurrentDomain.UnhandledException event) fire up when an exception is thrown in the unit test?
I'm using NUnit 2.5.10 with TestDriven.NET 3.0 on VS2010.
[TestFixture]
public class MyTests {
private void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e) {
Console.WriteLine("Gotcha!");
}
[Test]
public void ExceptionTest1() {
AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException;
throw new Exception("ExceptionInTest");
}
}
Output: (No gotchas)
------ Test started: Assembly: WcfQueue.Test.dll ------
Test 'xxxxx.Test.MyTests.ExceptionTest1' failed: System.Exception : ExceptionInTest
ProgramTests.cs(83,0): at xxxxx.Test.MyTests.ExceptionTest1()
0 passed, 1 failed, 0 skipped, took 1.98 seconds (NUnit 2.5.5).
Update: The purpose of this question is NOT to test .Net framework or NUnit. I just want to find out the reason why, in a unit test, the handler wouldn't fire.
An exception will percolate up the call stack until you reach a try/catch block that can handle that exception, an AppDomain boundary, or the top of the stack (in that order of priorities).
If you're executing within the same AppDomain that NUnit gives you, NUnit will catch your exception. This preempts the AppDomain boundary stuff, which would have called your event.
So your test needs to create a new AppDomain and execute its code there (including the setup, which adds the event handler for AppDomain.UnhandledException). Everything should work as expected there.
I guess, the event isn't fired because exception is handled. By test framework, to generate report.
As far as I know the unhandled exception event is only triggered in the default AppDomain. I think NUnit uses to execute the tests a different AppDomain which is the reason why your event does not get triggered.