I'm trying to write a NUnit 3 (3.8.1) extension that lets a test fail if it has a failed Debug.Assert(...) (instead of silently run through or even hang because it shows the failed assertion dialog).
In a NUnit 2 addin, I was able to so by unregistering all debug trace listeners and adding my own that just throws an exception (as for example explained here). However, this doesn't seem to work on NUnit 3 anymore.
I'm able to successfully deploy the extension and it's code is being executed.
[Extension(Description = "Failed Assertions Tracker", EngineVersion = "3.4")]
public class TrackerEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report); // prints -> so I know this method is being called
Debug.Listeners.Clear();
Debug.Listeners.Add(new UnitTestTraceListener());
}
}
However, my unit test unfortunately shows me that still the DefaultTraceListener is installed.
[Test]
public void FailingAssertionShouldNotHang()
{
foreach (object listener in Debug.Listeners)
{
Console.WriteLine(listener.GetType().FullName);
}
Debug.Fail("I'm sorry. I've failed.");
}
And so the test shows the dialog instead of simply failing.
What am I doing wrong? I suspect that the call to the static Listeners collection is ineffective because the actual test is run in a different context (different AppDomain, process, ?). But if this is the case, how can I solve my problem?
It's important to keep in mind that NUnit 3 Extensions, while capable of replacing NUnit 2 Addins in a few cases, are actually entirely different technology. NUnit 3 Extensions extend the Engine, which is separate from the framework.
In this case, your extension is setting up a Trace Listener that will capture any Debug Trace or Assert output produced by the engine. If the engine contained Trace statements (it doesn't) you would get the output. Meanwhile, the framework is happily continuing to run tests on its own.
Any code that will successfully capture Trace has to be part of the actual framework execution of the tests. This gives you two options.
Create a custom attribute that will capture trace. Custom attributes allow you to take actions when a test is being created or executed. They are created by implementing various interfaces supported by the framework. In your case, you would want to supply the attribute at assembly level, in order to capture all output produced by the assembly.
Create the code as part of your tests themself, without extending the framework at all. You would want to capture the Trace output in an assembly-level SetUpFixture using the OneTimeSetUp attribute and release it under the OneTimeTearDown attribute. Since this approach is simpler than creating a custom attribute, it's the one I would use.
Related
I have some tests that do some write-operation on a database. I know that´s not really unit-testing, but let´s leave that asside.
In order to enable every test to work on a clean workspace, I rollback all transactions doe so far. However I randomly get concurrency-errors due to database-locks that cannot be established.
This is my code:
Test1.dll
[TestFixture]
class MyTest1
{
[OneTimeSetup]
public void SetupFixture()
{
myworkspace.StartEditing(); // this will establish a lock on the underlying database
}
[OneTimeTearDow]
public void TearDownFixture()
{
myWorkspace.Rollback();
}
}
The same code also exists within another test-assembly, let's name it Test2.dll. Now when I use the nunit-console-runner using nunit3-console Test1.dll Test2.dll, I get the following error:
System.Runtime.InteropServices.COMException : Table 'GDB_DatabaseLocks' cannot be locked; it is currently used by user 'ADMIN' (this is me) on host 'MyHost'
at ESRI.ArcGIS.Geodatabase.IWorkspaceEdit.StartEditing(Boolean withUndoRedo)
myWorkspace is a COM-object (Arcobjects-interface IWorkspace) that relates to an MS-Access-Database. I assume this is because nunit creates multiple threads that enter the above code at the same time. So I added the NonParalizable-attribute to both assemblies without success. I also tried to add Apartment(ApartmentState.STA) to my assembly in order to execute everything in a single thread, which resulted in the console never finishing.
What drives me nuts is that running my tests using ReSahrpers test-runner works perfectly. However I have no clue how ReSharper starts nunit. It seems ReSharper does not use nunit-console, but the nunit-API instead.
Is there another way to force all my tests to run in a single thread? I use nunit3.10 and ArcGIS 10.8.
By default, the NUnit Console will run multiple test assemblies in parallel. Add --agents=1 to force the two assemblies to run sequentially, under a single agent.
Just to clarify some of the other things you tried as well...
[NonParallelizable] is used to prevent the parallelization of different tests within a single assembly. By default, tests within an assembly do not run in parallel, so adding this attribute when you haven't specifically added [Parallelizable] at a higher level will have no effect.
[Apartments(Apartment.STA)] can be added as an assembly-level attribute, and does not have to be added per test, as mentioned in the comments. Check out the docs here: https://docs.nunit.org/articles/nunit/writing-tests/attributes/apartment.html
I have a testing framework that has been converted to heavily utilize NUnit [Parallelizable]. I used to store contextual test data in the base class of the [TestFixture] which NUnit orchestrates the hooks like [OneTimeSetUp], [TearDown], etc.
For example:
[Test]
public void GoToGoogle()
{
var driver = new ChromeDriver();
// do some stuff
// Would like to pass data outside of test scope
TestContext.CurrentContext.Test.Properties.Set("DriverUrl", driver.Url); // Obviously does not work
Assert.Fail("This test should fail");
}
In the [TearDown] hook, I would like to get certain information about the test contextually. Because not everything is able to be handled nicely in asserts.
[TearDown]
public void TearDown()
{
var url = TestContext.CurrentContext.Test.Properties["DriverUrl"].ToString();
var msg = $"Test encountered an error at URL: {url}"
TestAPI.PushResult(Result.Fail, msg);
}
The code above involving the TestContext does not work for obvious reasons, but I am wondering if there is a best practice that allows for me to pass data in this manner, keeping in mind respect to [Parallelizable] and that I cannot scope test data or dependencies to the [TestFixture]
You say "for obvious reasons" but I'll first spell out the reasons why you cannot effectively set a property on the current test through TestContext. After all, other people just might be reading this. :-)
The Obvious Part
TestContext.CurrentContext.Test does not return the internal representation of a test from inside NUnit. Doing so would allow users to break NUnit in a variety of ways. In particular, TestContext.CurrentContext.Test.Properties returns a copy of the properties used within NUnit.
That copy of the properties is not readonly, so you are able to set properties on it. For that reason, one might expect to be able to set it in the [Test] method and access the value in the [Teardown].
Unfortunately, because of a minor implementation detail, that's not the case. In fact, each time you use TestContext.CurrentContext, an entirely new copy of the context is created. The only reason for this, I'm afraid, is that it was originally implemented that way and is a bit difficult to change in a non-breaking way.
As a result of this implementation detail, we lost an easy way for the three parts (SetUp, Test method, TearDown) of a test to communicate. Prior to the availability of parallel execution, it was possible to pass such information using members of the fixture class. That no longer works once tests are run in parallel.
Workarounds
Use Thread Local Storage to hold the retained information. SetUp, Test and Teardown all run on the same thread. Note that OneTimeSetUp and OneTimeTearDown will not generally use the same thread in a parallel execution environment.
If you are wiling to run fixtures in parallel but not individual test cases, then you can still use class members to retain information. As a further step, apply the SingleThreadedAttribute to your fixture, forcing all the code associated with it (including one-time setup and teardown) to run on the same thread.
If you have many fixtures, which can run in parallel, the second approach may actually give you a better performance trade-off than other approaches. Unfortunately, not everyone can use it - at least not without a major reorganization of their tests. You have to look at what your own tests are doing.
Permanent Solution
That would be to modify NUnit so that properties are both writable and shareable, at least within a single fixture instance. There have already been a few feature requests out there to do that on the NUnit GitHub project. I'm no longer active on the framework project, so I don't know what the plans are. However, I think I can say that it's not likely to happen before a major version change, i.e. NUnit 4.0.
I'm working on refactoring a testing application that uses NUnit 3.6. Most of the application was written by non-developers and is in need of refactoring/re-organization. There are a lot of legacy helper classes that use the standard
Console.Out.WriteLine();
Console.Error.WriteLine();
methods to log errors and test progress. Some effort was made to use the NUnit
TestContext.Progress.WriteLine();
TestContext.Out.WriteLine();
TestContext.Error.WriteLine();
methods to provide "live output" as test run, but it's not consistent. Moving forward, I want all helper classes/methods to use the Console.Out/Error methods so that they can be detached without the NUnit dependency and then redirect the output in the [SetUp] methods for each test.
[SetUp]
public void BaseTestSetupMethod(){
Console.SetError(TestContext.Error);
/* Set Console.Out to TestContext.Out or TestContext.Progress */
}
What is the difference between TestContext.Out and TestContext.Progress?
Which one should I redirect Console.Out to for general messages like printing the name of the current running test and the operations that it's doing?
I read through the documentation for NUnit 3, including the https://github.com/nunit/docs/wiki/TestContext page. While it lists the difference between TestContext.Out and TestContext.Progress, they are not really descriptive and have no examples of why you would use one over the other in practice.
The answer is indeed on the linked documentation page (https://github.com/nunit/docs/wiki/TestContext), but it's been a while since this question was asked so perhaps the documentation page has been improved since then.
At least now it says:
Out
Gets a TextWriter used for sending output to the current test result.
Error
Gets a TextWriter used for sending error output intended for immediate
display.
Progress
Gets a TextWriter used for sending normal (non-error) output intended for immediate display.)
The keyword here is "immediate display" - when something is sent to Error and Progress the intention is that the test runner should display it immediately, while text sent to Out will not be visible until each test case has terminated.
This is because NUnit 3 will catch Console.Out and not output it until the test case finished. In NUnit 2, the regular Console.Out and Error was visible immediately.
I am trying to run my approvals tests from nUnit under TeamCity
[assembly: FrontLoadedReporter(typeof(TeamCityReporter))]
[Test]
[UseReporter(typeof(WinMergeReporter))]
public void Test()
{
}
Unfortunately test fails because approvals is trying to pick up the approved file from C drive.
Test(s) failed. ApprovalTests.Core.Exceptions.ApprovalMissingException : Failed Approval: Approval File "C:\...approved.txt" Not Found.
Is there anyway I can specify right location for my approval files?
It is appeared that TeamCityReporter was hiding real reason of this issue.
Here is result of local run and output of approvals test with listed solutions.
System.Exception : Could Not Detect Test Framework
Either: 1) Optimizer Inlined Test Methods
Solutions: a) Add [MethodImpl(MethodImplOptions.NoInlining)] b) Set
Build->Opitmize Code to False & Build->Advanced->DebugInfo to Full
or 2) Approvals is not set up to use your test framework. It currently
supports [NUnit, MsTest, MbUnit, xUnit.net, xUnit.extensions,
Machine.Specifications (MSpec)]
Solution: To add one use
ApprovalTests.Namers.StackTraceParsers.StackTraceParser.AddParser()
method to add implementation of
ApprovalTests.Namers.StackTraceParsers.IStackTraceParser with support
for your testing framework. To learn how to implement one see
http://blog.approvaltests.com/2012/01/creating-namers.html
It was tricky to catch because usually local run is done under Debug while deployment and tests under Release. Nevertheless I hope the question and answer will be helpful for somebody else.
I'm using the latest NUnit to run Selenium tests. The tests are compiled into a class library DLL file which is then run by NUnit.
My problem is that before the automation begins, I need to run some initialization such as creating a log file, setting up specific parameters, etc. I don't see a way to do this in NUnit - setup() does this but for every Test or Fixture - I just need to run this code once at the start of the application.
Any idea how I can do what I want?
Your help is very appreciated.
J.
Take a look at SetUpFixtureAttribute (more information here). It says:
This is the attribute that marks a class that contains the one-time setup or teardown methods for all the test fixtures under a given namespace. The class may contain at most one method marked with the SetUpAttribute and one method marked with the TearDownAttribute.