I'm trying to learn automation with Selenium Webdriver using c#. I have my custom method Assert. What I did to continue the test after catching an AssertFailedException is using try-catch below is my code
public static void assert(string value, IWebElement element)
{
try
{
Assert.AreEqual(value, element.Text);
}
catch (AssertFailedException e)
{
Console.WriteLine(e.Message.ToString());
}
}
My problem is it catches all AssertFailedException(which is my goal) but the result of the test is PASSED in visual studio. My question is, how do I implement Continue on Failure and Fail the test if the Console contains Exceptions. Thankyou in advance guys!
As far as I understand, you want to do several checks inside your test, and at the very end of it to determine whether any of of them failed. You may need to write some custom code to achieve this. For example, you may introduce a class Assertion:
internal class Assertion
{
private readonly string title;
private readonly object expected;
private readonly object actual;
public Assertion(string title, object expected, object actual)
{
this.title = title;
this.expected = expected;
this.actual = actual;
}
public bool IsMatch()
{
return this.actual == this.expected;
}
public override string ToString()
{
return $"Title: {title}. Expected: {expected}. Actual: {actual}";
}
}
When your test is running, you will create new instances of Assertion class and store them in a list. At the end of the test, you can use the following method:
private static void VerifyAssertions(Assertion[] assertions)
{
var failedAssertions = assertions.Where(a => !a.IsMatch()).ToArray();
if (failedAssertions.Any())
{
throw new AssertFailedException(string.Join<Assertion>("; ", failedAssertions));
}
}
You could try using verify instead of assert for minor checks. Assert by default indicates a major checkpoint and script execution will be terminated on fail and if you catch that exception the reporting will be ignored - this is expected behaviour. However, a verify indicates that the script can continue even on fail - in this case the failed step will be reported and the script will continue.
To put it simply, use assert when you don't want the script to proceed on failure and use verify when you want the script to report the failure and proceed.
Related
I am beginning to look into using White UI testing with XUnit.
The basic structure for my tests is
Open the application
Test Something
Close the application
This works really well when test passes. However, when a test fails, the application is not closed. If multiple tests fail, this leads to lots of opened instances of my application.
To get around this, I use try and finally blocks but it is not very nice.
Is there an alternate option which achieves the same cleanup behaviour but looks a bit nicer? Like a "RunOnAssertFail" method?
[Fact]
public void MainWindowCreated()
{
bool testFailed = false;
Application application = Application.Launch(#"C:\Program\Program.exe");
Window mainWindow = GetWindow(application, "MainWidndow", 500);
try
{
testFailed = true;
mainWindow.Should().NotBe(null, ". Main Widndow could not be found");
testFailed = false;
}
finally
{
if (testFailed)
{
application.Close();
}
}
/*
* Rest of test case
*/
application.Close();
}
private static Window GetWindow(Application application,
string windowName,
int timeoutAfterMilliseconds)
{
Window window = null;
try
{
window = Retry.For(
() => application.GetWindows().First(
windowX => windowX.Title.Trim().Equals(windowName.Trim())),
TimeSpan.FromMilliseconds(timeoutAfterMilliseconds));
}
catch (InvalidOperationException)
{
}
return window;
}
Requires xUnit, White and Fluent Assertions to run.
After playing around I realised the assertions were it was throwing an exception and not actually asserting.
Therefore to help tidy it up, a try catch block is more appropriate
try
{
mainWindow.Should().NotBeNull("because this window is required for the rest of the test");
}
catch(XunitException)
{
application.Close();
throw;
}
However, this is still not ideal.
What about implementing IDisposable on your test class and use that to clean up?
Using MbUnit, I create tests using several StaticTestFactory methods, each having corresponding test setup and teardown methods. A requirement is to log test results to an external system, especially failed ones.
However, I am unable to get the correct test outcome status using TestContext.CurrentContext.Outcome.Status. Using below code, you will see that the test fails, but the Outcome.status is always returned as 'Passed' from FactoryAssignedTearDownMethod, even when both Gallio Icarus and Echo show the test as failed.
Looking for any workaround or fix to get the correct outcome in this scenario.
public class FactoryTest
{
[StaticTestFactory]
public static IEnumerable<Test> CreateStaticTests()
{
var testcase = new TestCase("simpletest" , () =>
{
Assert.Fail("staticfactory created test failed.");
});
testcase.TearDown = FactoryAssignedTearDownMethod;
yield return testcase;
}
public static void FactoryAssignedTearDownMethod()
{
//outcome value is always 'Passed', even when test fails
TestLog.WriteLine("Test Outcome Status from factory assigned method: " + TestContext.CurrentContext.Outcome.Status);
}
}
I worked around this by writing a Gallio TestRunnerExtension. By handling the TestStepFinished event, I can get the proper test result for all tests created with the StaticTestFactory.
I've just started playing around with Moq in my unit tests, but am having an issue where the unit test is passing - and I don't think it should be.
I have two objects, one that despatches data to a queue and another that implements INotifier which is called if the despatcher fails, they look like this (cut down for brevity):
public class EmailNotifier : INotifier
{
public void Notify(string message)
{
// sends the notification by email
}
}
public class Despatcher
{
public void Despatch(int batchNumber, INotifier failureNotifier)
{
try
{
if (batchNumber.Equals(0)) throw new InvalidOperationException("Error message");
}
catch (InvalidOperationException ex)
{
failureNotifier.Notify(ex.ToString());
throw ex;
}
}
}
I am unit testing the Despatcher to specifically verify that Notify is being called on the provided INotifier (which I'm mocking) when it fails (I'm intentionally forcing the Despatcher to fail by passing a 0 batch number). I set up my mocks like this:
[TestMethod]
[ExpectedException(typeof(InvalidOperationException))]
public void Despatcher_notifies_on_failure()
{
var mockNotifier = new Mock<EmailNotifier>();
mockNotifier.Setup(n => n.Notify(It.IsAny<string>())).Verifiable();
var despatcher = new Despatcher();
despatcher.Despatch(0, mockNotifier.Object);
mockNotifier.Verify(n => n.Notify(It.IsAny<string>()), Times.Once());
}
This passes the test fine, which is expected as a 0 batch number raises the exception which causes the INotifier to call Notify (when I step through the test everything works as expected).
So, I go on to comment out the failureNotifier.Notify(ex.ToString()) line and run the test again - passes fine? I'm not sure whether I'm setting up and verifying correctly as I've only been using Moq for about 2 hours now but I thought I was understanding this correctly, but this has just thrown me a bit. I expect this test to fail as I specifically want to make sure Notify is called in the event of a failure - can anyone see anything obviously wrong here? Thanks in advance for your help as always.
Your test never gets to the verification part. Why? This line
despatcher.Despatch(0, mockNotifier.Object);
Throws exception, which is consumed by ExpectedException attribute and test ends. The mockNotifier.Verify line is never executed.
What you want is two unit tests, instead:
one testing that notifier is called upon exception (with .Verify). Note that you'll have to wrap .Despatch call into try { } catch { }, so that exception is ignored.
second, checking that exception is rethrown (with ExpectedException)
I'm writing a unit test for this one method which returns "void". I would like to have one case that the test passes when there is no exception thrown. How do I write that in C#?
Assert.IsTrue(????)
(My guess is this is how I should check, but what goes into "???")
I hope my question is clear enough.
Your unit test will fail anyway if an exception is thrown - you don't need to put in a special assert.
This is one of the few scenarios where you will see unit tests with no assertions at all - the test will implicitly fail if an exception is raised.
However, if you really did want to write an assertion for this - perhaps to be able to catch the exception and report "expected no exception but got this...", you can do this:
[Test]
public void TestNoExceptionIsThrownByMethodUnderTest()
{
var myObject = new MyObject();
try
{
myObject.MethodUnderTest();
}
catch (Exception ex)
{
Assert.Fail("Expected no exception, but got: " + ex.Message);
}
}
(the above is an example for NUnit, but the same holds true for MSTest)
In NUnit, you can use:
Assert.DoesNotThrow(<expression>);
to assert that your code does not throw an exception. Although the test would fail if an exception is thrown even if there was no Assert around it, the value of this approach is that you can then distinguish between unmet expectations and bugs in your tests, and you have the option of adding a custom message that will be displayed in your test output. A well-worded test output can help you locate errors in your code that have caused a test to fail.
I think it's valid to add tests to ensure that your code is not throwing exceptions; for example, imagine you are validating input and need to convert an incoming string to a long. There may be occasions when the string is null, and this is acceptable, so you want to ensure that the string conversion does not throw an exception. There will therefore be code to handle this occasion, and if you haven't written a test for it you will be missing coverage around an important piece of logic.
This helper class scratched my itch with MSTest. Maybe it can scratch yours also.
[TestMethod]
public void ScheduleItsIneligibilityJob_HasValid_CronSchedule()
{
// Arrange
var factory = new StdSchedulerFactory();
IScheduler scheduler = factory.GetScheduler();
// Assert
AssertEx.NoExceptionThrown<FormatException>(() =>
// Act
_service.ScheduleJob(scheduler)
);
}
public sealed class AssertEx
{
public static void NoExceptionThrown<T>(Action a) where T:Exception
{
try
{
a();
}
catch (T)
{
Assert.Fail("Expected no {0} to be thrown", typeof(T).Name);
}
}
}
Don't test that something doesn't happen. It's like assuring that code doesn't break. That's sort of implied, we all strive for non-breaking, bug-less code. You want to write tests for that? Why just one method? Don't you want all your methods being tested that they don't throw some exception? Following that road, you'll end up with one extra, dummy, assert-less test for every method in your code base. It brings no value.
Of course, if your requirement is to verify method does catch exceptions, you do test that (or reversing it a bit; test that it does not throw what it is supposed to catch).
However, the general approach/practices remain intact - you don't write tests for some artificial/vague requirements that are out of scope of tested code (and testing that "it works" or "doesn't throw" is usually an example of such - especially in scenario when method's responsibilities are well known).
To put it simple - focus on what your code has to do and test for that.
I like to see an Assert.Whatever at the end of each test, just for consistency... without one, can I really be sure there's not supposed to be one there?
For me, this is as simple as putting Assert.IsTrue(true);
I know I didn't accidentally put that code in there, and thus I should be confident enough at quick a skim through that this was as intended.
[TestMethod]
public void ProjectRejectsGappedVersioningByDefault() {
var files = new List<ScriptFile>();
files.Add(ScriptProjectTestMocks.GetVersion1to2());
files.Add(ScriptProjectTestMocks.GetVersion3to4());
Assert.Throws<ScriptProject.InvalidProjectFormatException>(() => {
var sut = new ScriptProject(files);
});
}
[TestMethod]
public void ProjectAcceptsGappedVersionsExplicitly() {
var files = new List<ScriptFile>();
files.Add(ScriptProjectTestMocks.GetVersion1to2());
files.Add(ScriptProjectTestMocks.GetVersion3to4());
var sut = new ScriptProject(files, true);
Assert.IsTrue(true); // Assert.Pass() would be nicer... build it in if you like
}
My friend Tim told me about ExpectedException. I really like this b/c it is more succinct, less code, and very explicit that you are testing for an exception.
[TestMethod()]
[ExpectedException(typeof(System.Exception))]
public void DivideTest()
{
int numerator = 4;
int denominator = 0;
int actual = numerator / denominator;
}
You can read way more about it here: ExpectedException Attribute Usage.
With Xunit you can use this:
var exception = Record.Exception(() =>
MethodUnderTest());
Assert.Null(exception);
or for async operations
var exception = await Record.ExceptionAsync(async () =>
await MethodUnderTestAsync());
Assert.Null(exception);
Another way which worked for me is to store it in a variable and check output.
var result = service.Run()
Assert.IsFalse(result.Errors.Any())
using Moq;
using Xunit;
[Fact]
public void UnitTest_DoesNotThrow_Exception()
{
var builder = new Mock<ISomething>().Object;
//Act
var exception = Record.Exception(() => builder.SomeMethod());
//Assert
Assert.Null(exception);
}
I'm using Microsoft WebTest and want to be able to do something similar to NUnit's Assert.Fail(). The best i have come up with is to throw new webTestException() but this shows in the test results as an Error rather than a Failure.
Other than reflecting on the WebTest to set a private member variable to indicate the failure, is there something I've missed?
EDIT: I have also used the Assert.Fail() method, but this still shows up as an error rather than a failure when used from within WebTest, and the Outcome property is read-only (has no public setter).
EDIT: well now I'm really stumped. I used reflection to set the Outcome property to Failed but the test still passes!
Here's the code that sets the Oucome to failed:
public static class WebTestExtensions
{
public static void Fail(this WebTest test)
{
var method = test.GetType().GetMethod("set_Outcome", BindingFlags.NonPublic | BindingFlags.Instance);
method.Invoke(test, new object[] {Outcome.Fail});
}
}
and here's the code that I'm trying to fail:
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
this.Fail();
yield return new WebTestRequest("http://google.com");
}
Outcome is getting set to Oucome.Fail but apparently the WebTest framework doesn't really use this to determine test pass/fail results.
Set the Outcome property to Fail:
Outcome = Outcome.Fail;
There's also an Assert.Fail() in the Microsoft.VisualStudio.QualityTools.UnitTestFramework assembly.
The Outcome property will set the public at vsts 2010 :-)
You make a test always fail by adding a validation rule that always fails. For example, you could write a fail validation rule like this:
public class FailValidationRule : ValidationRule
{
public override void Validate(object sender, ValidationEventArgs e)
{
e.IsValid = false;
}
}
Then attach the new validation rule you your webtest's ValidateResponse event, like so:
public class CodedWebTest : WebTest
{
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
WebTestRequest request1 = new WebTestRequest("http://www.google.com");
FailValidationRule failValidation = new FailValidationRule();
request1.ValidateResponse += new EventHandler<ValidationEventArgs>(failValidation.Validate);
yield return request1;
}
}
A solution that would work in declarative tests (as well as coded) is:
write a Validation Rule that fails when a certain Context Parameter (e.g. 'FAIL') is present in the Context
when you want to trigger the failure, set the Context Parameter and call WebTest.Stop()
add the Validation Rule as a WebTest-level rule (not request-level) so that it runs on all requests
I think that's as concise as it can be done.
First off, I'm working with VB.net, but I also tried to set the outcome to fail before finding out it does not work (which brought me here).
I finally managed to do it by just throwing an exception :
Public Overrides Sub PostRequest(ByVal sender As Object, ByVal e As PostRequestEventArgs)
If YourTest = True Then
Throw New WebTestException("My test Failed")
End If
MyBase.PostRequest(sender, e)
End Sub
I know this topic is old but I hope it helps someone anyway :)
Set the value of Outcome in the PostWebTest event handler.