I've the following test in Nunit with Moq:
[TestFixture]
public class MessageServiceTests
{
private Mock<IFtpClient> _ftpService;
private IMessageService _messageService;
[SetUp]
public void Setup()
{
_ftpService = new Mock<IFtpClient>();
_messageService = new MessageService(_ftpService.Object);
}
[Test]
public void SendAsync_WithSettings_ConnectsWithCredentials()
{
//act
_messageService.SendAsync(It.IsAny<Stream>(), It.IsAny<String>(), It.IsAny<Settings>());
}
}
and the following method that is tested:
public async Task SendAsync(Stream stream, string fileName, Settings settings)
{
throw new NotImplementedException();
}
Is expect the test to fail, but when I run it in Visual Studio it passes. I can't get my head around it, tests should fail when an unexpected exception is thrown, right? So why does it pass?
SendAsync ran successfully, and returned a Task which contained an exception (its IsFaulted property returns true, and its Exception property contains the NotImplementedException).
However, you're not checking the Task which is returned from SendAsync, so you never realise that it contains an exception.
The easiest way to check the Task for exceptions is using await, and make your test method async Task. This also handles the case where the Task doesn't complete straight away.
[Test]
public async Task SendAsync_WithSettings_ConnectsWithCredentials()
{
//act
await _messageService.SendAsync(It.IsAny<Stream>(), It.IsAny<String>(), It.IsAny<Settings>());
}
Related
XUnit ThrowsAsync<> doesn't seem to be checking the exception type.
I am trying to check the type of the exception thrown in my unit test. However, XUnit does not seem to be not checking the exception type but only that its an Exception.
Surely the below test should fail as the exception thrown is of type NotImplementedException but the test was expecting ArgumentException. It passes as long as any kind of exception is thrown.
public class AssertThrowsTests
{
[Fact]
public void Tests()
{
// Using a method as a delegate
Assert.ThrowsAsync<ArgumentException>(async () => await MethodThatThrows());
}
async Task MethodThatThrows()
{
await Task.Delay(100);
throw new NotImplementedException();
}
}
Is this a bug with XUnit or my code?
You're ignoring the fact that ThrowsAsync is itself asynchronous, and returns a task that you should await. Change your test to:
[Fact]
public async Task Tests()
{
await Assert.ThrowsAsync<ArgumentException>(async () => await MethodThatThrows());
}
I am writing NUnit Test for async methods and am using extent reporting to report the results. The ExtentTest linked to my test get completed as soon as the await step i the test method has completed execution and am no longer able to access the ExtentTest for any logging purpose. Is there any issue with my code or is this expected ?
Here is my Test method:
[Test, RequiresThread]
public async Task GetList()
{
try
{
ReportHelper.ExtentTestInfo("system.readResources() method is called");
Resources resources = await system.readResources();
ReportHelper.ExtentTestInfo("system.readResources() method finished and responded");
//Test Assertions
}
}
Here is my ReportHelper class:
public class ReportHelper
{
private static ExtentReports TestReportHTML = new ExtentReports();
var htmlReporter = new ExtentV3HtmlReporter("Test_Run_Report_" + #".html");
TestReportHTML.AttachReporter(htmlReporter);
[ThreadStatic] private static ExtentTest _extentTest;
_extentTest = TestReportHTML.CreateTest(testName); //testName is passed during [SetUp]
public static void ExtentTestInfo(string testInfo)
{
_extentTest.Info(testInfo);
}
}
Once the await call has been executed the _extentTest status is passed and on the next line I am getting NullReferenceException for the _extentTest
You need to remove the [ThreadStatic] attribute. That stops the variable being shared between threads but await may cause the code to execute the remaining code on a different thread, which is causing your _extentTest to be null after the await.
Consider the following highly simplified viewmodel for fetching and showing a list of projects:
public class ProjectListViewModel
{
private readonly IWebService _webService;
public ICommand RefreshCommand { get; }
// INotifyPropertyChanged implementation skipped for brevity
public ObservableCollection<Project> Projects { get; set; }
public ProjectListViewModel(IWebService serverApi)
{
_serverApi = serverApi;
// ICommand implemented by Xamarin.Forms
RefreshCommand = new Command(async () => await RefreshAsync());
}
private async Task RefreshAsync()
{
try
{
Projects = await _webService.GetProjectsAsync();
}
catch (TaskCanceledException)
{
// Empty (task only cancelled when we are navigating away from page)
}
}
}
Using NUnit and Moq, I'm trying test that when GetProjectsAsync throws a TaskCanceledException, the ViewModel will catch it. The closest I get is this:
[Test]
public void When_Refreshing_Catches_TaskCanceledException()
{
// Arrange
webService = new Mock<IServerApi>();
webService.Setup(mock => mock.GetProjectsAsync())
.ThrowsAsync(new TaskCanceledException());
vm = new ProjectListViewModel(webService.Object);
// Act and assert
Assert.That(() => vm.RefreshCommand.Execute(null), Throws.Nothing);
}
The test passes, but unfortunately it's faulty - it still passes if I throw e.g. Exception instead of TaskCanceledException. As far as I know, the reason is that the exception doesn't bubble up past the command lambda, async () => await RefreshAsync(), so no exception thrown by GetProjectsAsync will ever be detected by the test. (When running the actual app however, the TaskCanceledException will bubble up and crash the app if not caught. I suspect this is related to synchronization contexts, of which I have very limited understanding.)
It works if I debug the test - if I mock it to throw Exception, it will break on the line with the command/lambda definition, and if I throw TaskCanceledException, the test will pass.
Note that the results are the same if I use Throws instead of ThrowsAsync. And in case it's relevant, I'm using the test runner in ReSharper 2016.2.
Using nUnit, is it possible at all to unit test exceptions thrown when executing "async" commands like this? Is it possible without writing a custom Command implementation?
Your problem is here:
new Command(async () => await RefreshAsync())
This async lambda is converted to an async void method by the compiler.
In my article on async best practices, I explain why the exception cannot be caught like this. async methods cannot propagate their exceptions directly (since their stack can be gone by the time the exception happens). async Task methods solve this naturally by placing the exception on their returned task. async void methods are unnatural, and they have nowhere to place the exception, so they raise it directly on the SynchronizationContext that was current at the time the method started.
In your application, this is the UI context, so it's just like it was thrown directly in an event handler. In your unit test, there is no context, so it's thrown on a thread pool thread. I think NUnit's behavior in this situation is to catch the exception and dump it to the console.
Personally, I prefer using my own asynchronous-compatible ICommand such as AsyncCommand in my Mvvm.Async library (also see my article on asynchronous MVVM commands):
new AsyncCommand(_ => RefreshAsync())
which can then be naturally unit tested:
await vm.RefreshCommand.ExecuteAsync(null); // does not throw
Alternatively, you can provide your own synchronization context in the unit test (using, e.g., my AsyncContext):
// Arrange
webService = new Mock<IServerApi>();
webService.Setup(mock => mock.GetProjectsAsync())
.ThrowsAsync(new TaskCanceledException());
vm = new ProjectListViewModel(webService.Object);
// Act/Assert
AsyncContext.Run(() => vm.RefreshCommand.Execute(null));
In this case, if there was an exception, Run would propagate it.
Since async void (which is what the handler to your command is) is basically "fire and forget" and you can't await for it in the test I would suggest unit testing the RefreshAsync() method (you may want to make it internal or public), this can be easily done in NUnit:
if you are asserting exceptions being thrown:
[Test]
public async Task Test_Exception_RefreshAsync(){
try
{
await vm.RefreshAsync();
Assert.Fail("No exception was thrown");
}
catch (NotImplementedException e)
{
// Pass
}
}
or simply
[Test]
public async Task Test_RefreshAsync(){
var vm = new ProjectListViewModel(...);
await vm.RefreshAsync();
//Assertions here
}
or as other answer state you can create your own AsyncCommand that you can await on.
I have an xUnit (2.1.0) test that always hangs/deadlocks. Here the code with the names of classes/methods changed for clarity and confidentiality:
[Fact]
public void DocumentModificationTest()
{
Exception ex = null;
using (TestDependency testDependency = new TestDependency())
{
TestDependencyDocument testDependencyDoc = testDependency.Document;
MockTestDependency fakeDependency = new MockTestDependency();
try
{
DoStuffToTheDocument(testDependencyDoc, "fileName.doc", fakeDependency);
}
catch (Exception e)
{
ex = e;
}
}
Assert.Null(ex);
}
If I set a breakpoint and step over until the assert I can see that ex is null and the test should pass and be done with, but it just hangs and I never see Test Successful on the runner.
Here's what DoStuffToTheDocument looks like:
public static void DoStuffToTheDocument(TestDependencyDocument document, string pFileName, MockTestDependency pContainer)
{
pContainer.CheckInTheDocFirst(async () =>
{
//check some stuff
//test returns early here
//check other stuff(test never gets here)
//await method (thus the async anonymous method)
});
}
And lastly here's what CheckInTheDocFirst looks like:
public void CheckInTheDocFirst(Action pConfirmAction)
{
pConfirmAction(); //since this is a method in a mock class only used for testing we just call the action
}
Any ideas whats happening here? Is there something with my async-await paradigm that is causing this test to hang?
It turns out that this is an issue caused by async void test method support in xUnit: https://github.com/xunit/xunit/issues/866#issuecomment-223477634
While you really should carry async all the way up, sometimes this isn't feasible because of interoperability concerns. You can't always change the signatures of everything you're working with.
One thing to note about your assert, however, is that it will always prove true even if there's an exception thrown in the async void method (the lambda, in this case). This is because exception handling is different for async void:
Async void methods have different error-handling semantics. When an exception is thrown out of an async Task or async Task method, that exception is captured and placed on the Task object. With async void methods, there is no Task object, so any exceptions thrown out of an async void method will be raised directly on the SynchronizationContext that was active when the async void method started. Figure 2 illustrates that exceptions thrown from async void methods can’t be caught naturally.
When you have an async function, you should really be async all the way down. Otherwise you run into issues with sync contexts being blocked which will result in lockups.
pContainer.CheckInTheDocFirst should be async and return a Task (since it's taking an async function that returns a Task object).
DoStuffToDocument should be an async function that returns a Task, since it calls an async function.
And finally, the test itself should also be an async method that returns a task.
If you run the async all the way up the stack, I think you'll find things just work.
I am trying to discover how to apply the async and await keywords to my xUnit tests. I am using xUnit 1.9 and Async CTP 1.3. Here is my test case
I have an interface which specifies one asynchronous method call
public interface IDoStuffAsync
{
Task AnAsyncMethod(string value);
}
I have a class which consumes the interface and calls the async method
public class UseAnAsyncThing
{
private readonly IDoStuffAsync _doStuffAsync;
public UseAnAsyncThing(IDoStuffAsync doStuffAsync)
{
_doStuffAsync = doStuffAsync;
}
public async Task DoThatAsyncOperation(string theValue)
{
await _doStuffAsync.AnAsyncMethod(theValue);
}
}
In my tests I wish to check that the method DoThatAsyncOperation is calling the method with the correct value so I mock the interface and use the Moq to verify the call
[Fact]
public async void The_test_will_pass_even_though_it_should_fail()
{
var mock = new Mock<IDoStuffAsync>();
var sut = new UseAnAsyncThing(mock.Object);
mock.Setup(x => x.AnAsyncMethod(It.IsAny<string>()));
await sut.DoThatAsyncOperation("test");
// This won't throw a Moq.MockExcpetion so the test appears to pass
// However it does not run
mock.Verify(x => x.AnAsyncMethod("fail"));
}
This test is using the async and await keywords. When it runs it erroneously passes as Moq should assert that the verify fails. Any code after the call to sut.DoThatAsyncOperation("test"); does not run
[Fact]
public void This_will_work_and_assert_the_reslt()
{
var mock = new Mock<IDoStuffAsync>();
var sut = new UseAnAsyncThing(mock.Object);
mock.Setup(x => x.AnAsyncMethod(It.IsAny<string>()));
sut.DoThatAsyncOperation("test").ContinueWith(y => { });
// This won't throw a Moq.MockExcpetion so the test appears to pass
// However it does not run
mock.Verify(x => x.AnAsyncMethod("fail"));
}
This test is setup without the await and async keywords and passes fine.
Is this expected behavior for xUnit and Moq?
Update
Thanks for Stephen's comment I managed to fix the first test by making two changes. The test now returns a Task instead of void and the Mock also returns a Task.
[Fact]
public async Task The_test_will_pass_even_though_it_should_fail()
{
var mock = new Mock<IDoStuffAsync>();
var sut = new UseAnAsyncThing(mock.Object);
mock.Setup(x => x.AnAsyncMethod(It.IsAny<string>())).ReturnAsync(true);
await sut.DoThatAsyncOperation("test");
// This now fails as it should
mock.Verify(x => x.AnAsyncMethod("fail"));
}
Change your unit test method to return Task instead of void, and it should work. Support for async void unit tests is being considered for a future release.
I describe in detail why async unit tests don't work by default on my blog. (My blog examples use MSTest, but the same problems existed in every other test runner, including xUnit pre-1.9).
I tried to use the code from your 'Update', but it was stopping at the async method that I was mocking.
var tcs = new TaskCompletionSource<T>();
tcs.SetResult(default(T));
mock.Setup(x => x.AnAsyncMethod(It.IsAny<T>())).Returns(tcs.Task);
So to fix that I had to change the 'Return' method:
mock.Setup(x => x.AnAsyncMethod(It.IsAny<T>())).Returns(()=> { return tcs.Task; } );