We Keep Coding

How to perform global setup/teardown in xUnit and run tests in parallel?

Here is what I want to achieve with xUnit:
Run initialization code.
Run tests in parallel.
Perform teardown.
I have tried [CollectionDefinition]/[Collection]/ICollectionFixture
approach described here but it has disabled the parallel execution, which is critical for me.
Are there any way to run tests in parallel and be able to write global setup/tear-down code in xUnit?
If it is not possible with xUnit, does NUnit or MSUnit support this scenario?

NUnit supports this scenario. For global setup, create a class in one of your root namespaces and add the [SetupFixture] attribute to it. Then add a [OneTimeSetUp] method to that class. This method will get run once for all tests in that namespace and in child namespaces. This allows you to have additional namespace specific onetime setups.
[SetUpFixture]
public class MySetUpClass
{
[OneTimeSetUp]
public void RunBeforeAnyTests()
{
// ...
}
[OneTimeTearDown]
public void RunAfterAnyTests()
{
// ...
}
}
Then to run your tests in parallel, add the [Parallelizable] attribute at the assembly level with the ParallelScope.All. If you have tests that should not be run in parallel with others, you can use the NonParallelizable attribute at lower levels.
[assembly: Parallelizable(ParallelScope.All)]
Running test methods in parallel in NUnit is supported in NUnit 3.7 and later. Prior to that, it only supported running test classes in parallel. I would recommend starting any project with the most recent version of NUnit to take advantages of bug fixes, new features and improvements.

A somewhat basic solution would be static class with a static constructor and subscribing to the AppDomain.CurrentDomain.ProcessExit event.
public static class StaticFixture
{
static StaticFixture()
{
AppDomain.CurrentDomain.ProcessExit += (o, e) => Dispose();
// Initialization code here
}
private static void Dispose()
{
// Teardown code here
}
}
There's no guarantee when the static constructor gets called though, other than at or before first use.

Load testing Visual Studio, start up script / setup

I was wondering if it was possible to have a start-up script before running any load tests? For example, perhaps to seed some data or clear anything down prior to the tests executing.
In my instance I have a mixed bag of designer and coded tests. Put it simply, I have:
Two coded tests
A designer created web test which points to these coded tests
A load test which runs the designer
I have tried adding a class and decorating with the attributes [TestInitialize()], [ClassInitialize()] but this code doesn't seem to get run.
Some basic code to show this in practice (see below). Is there a way of doing this whereby I can have something run only the once before test run?
[TestClass]
public class Setup : WebTest
{
[TestInitialize()]
public static void Hello()
{
// Run some code
}
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
return null;
}
}
Probably should also mention that on my coded tests I have added these attributes and they get ignored. I have come across a workaround which is to create a Plugin.
EDIT
Having done a little more browsing around I found this article on SO which shows how to implement a LoadTestPlugin.

Visual Studio provides a way of running a script before and also after a test run. They are intended for use in deploying data for a test and cleaning up after a test. The scripts are specified on the "Setup and cleanup" page in the ".testsettings" file.
A load test plugin can contain code to run before and after any test cases are executed, also at various stages during test execution. The interface is that events are raised at various points during the execution of a load test. User code can be called when these events occur. The LoadTestStarting event is raised before any test cases run. See here for more info.

If you are willing to use NUnit you have SetUp/TearDown for a per test scope and TestFixtureSetUp/TestFixtureTearDown to do something similar for a class (TestFixture)

Maybe a bit of a hack, but you can place your code inside the static constructor of your test class as it will automatically run exactly once before the first instance is created or any static members are referenced:
[TestClass]
public class Setup : WebTest
{
static Setup()
{
// prepare data for test
}
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
return null;
}
}

Initialize log4Net as early as possible with NUnit

I'm wondering what is the best way to initialize log4Net in a NUnit project. Of course I want to call the init code (ie. XmlConfigurator.Configure()) as soon as I can to get as many early log output as I can. But since my project is run through NUnit, I have little control on its entry point.
According to NUnit documentation, it should call some constructors first, then a method marked with the [SetUp] attribute in a class marked with [TestFixtureSetup].
So, First, I created a static helper class which I can call several times without trouble.
public static class LoggingFacility
{
private static bool _loggerIsUp = false;
public static void InitLogger()
{
if (_loggerIsUp == false)
XmlConfigurator.ConfigureAndWatch(f);
_loggerIsUp = true;
}
}
Then, I made all my [TestFixtureSetup] inherit a single one that does pretty much nothing else than calling LoggingFacility.initLogger(). But that still leaves all constructors that are run earlier, in an order I can only assume random. And moreover, it will probably do some static initializations before I am even able to execute some code.
In fact, as I can see in my log, the first 4 seconds or so of execution are completely unrecorded.
Does it mean I will have to call my InitLogger() in every constructor and forbid the use of any static initializer? That's tough job!
Does somebody know a magic trick with this?

For single initialization point you should use class marked with [SetUpFixture] attribute and method marked with [SetUp], for example:
[SetUpFixture]
public class TestsInitializer
{
[SetUp]
public void InitializeLogger()
{
LoggingFacility.InitLogger();
}
}
Now, this method ([SetUp] InitializeLogger) will run before any test is run, same as one marked with [TearDown] will run once all tests are run. But here's the catch - what does any and all mean in this context? Tests from classes declared in the same namespace as class marked with [SetUpFixture].
For example, assuming hierarchy like this:
- Tests
--- Business
----- TestsInitializer.cs // SetUpFixture class
----- FirstBusinessTests.cs
----- SecondBusinesTests.cs
--- ComplexLogic
----- VeryComplexLogicTests.cs
First and SecondBusinessTests will run after SetUp from TestsInitializer, however VeryComplexLogicTests might run in random order.
According to linked documentation, if you declare SetUpFixture class outside of any namespace, setup and teardown will apply for entire assembly:
Only one SetUpFixture should be created in a given namespace. A SetUpFixture outside of any namespace provides SetUp and TearDown for the entire assembly.

A work mate provided me with the following workaround, that does the job:
In all my classes that require logging, I had the following logger initialization
private static readonly ILog Log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
I simply changed it to a singleton initializer
private static readonly ILog Log = LoggingFacility.GetLoggerWithInit(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
/*** ... ***/
public static class LoggingFacility
{
private static bool _loggerIsUp = false;
public static ILog GetLoggerWithInit(Type declaringType)
{
if (_loggerIsUp == false)
XmlConfigurator.Configure(_log4NetCfgFile);
_loggerIsUp = true;
return LogManager.GetLogger(declaringType);
}
}
Because I have this code in every class, this static initializer has to be called very early by NUnit wile instantiating my test classes.
Next step is to make that thread safe :(

Make MSTest respect [Conditional()] attribute?

I'm using VS2010, I have the following method call:
[Conditional("DEBUG")]
public void VerboseLogging() { }
public void DoSomething() {
VerboseLogging();
Foo();
Bar();
}
Then I have a unit test for the DoSomething method which checks that it emits proper logging.
[Conditional("DEBUG"), TestMethod()]
public void EnsureVerboseLog() {
DoSomething();
VerifyVerboseLoggingCalled(); // <-- fail in release builds since VerboseLogging() calls get eliminated.
}
It seems that MSTest only sees TestMethod and executes it (generating failed test) even though I've marked it with Conditional("DEBUG") and compile it in release mode.
So, is there a way to exclude certain tests depending on compilation constant other than #if?

The ConditionalAttribute does not affect whether or not a method is compiled into an application. It controls whether or not calls to the method are compiled into the application.
There is no call to EnsureVerboseLog in this sample. MSTest just sees a method in the assemlby with the TestMethod attribute and correctly executes it. In order to prevent MSTest from running the method you will need to do one of the following
Not compile it into your application (possible through #if's)
Not annotate it with the TestMethod attribute

So, is there a way to exclude certain tests depending on compilation
constant other than #if?
I'd use the #if - it's readable.
[TestMethod]
#if !DEBUG
[Ignore]
#endif
public void AnyTest()
{
// Will invoke for developer and not in test-server!
}
HTH..

If you want to be able to conditionally run different tests, you will probably want to use the [TestCategory] attribute:
[TestMethod]
[TestCategory("SomeCategory")]
public void SomethingWorks()
{
...
}
And then use the filter parameter to include or exclude the categories:
dotnet test --filter "TestCategory=SomeCategory"
dotnet test --filter "TestCategory!=SomeCategory"

A work around is to set the Priority attribute to -1 to your method. Then run mstest with "minpriority:0" as an argument.
[TestMethod()]
[Priority(-1)]
public void Compute_Foo()
{
This method will not be executed
}

Run a single unit test from a fixture of many in MbUnit

Is there anyway to add an attribute to a [Test] method in a [TestFixture] so that only that method runs? This would be similar to the way the [CurrentFixture] attribute can be used to only run a single fixture. I ask as sometimes when I test the model I want to profile the sql being executed and I only want to focus on a single test. Currently I have to comment out all the other tests in the fixture.
Updated:
The code I'm using to initiate the test follows, I'm really looking for a solution I can weave into this code.
public static void Run(bool currentFixturesOnly) {
using(AutoRunner auto = new AutoRunner()) {
if(currentFixturesOnly) { // for processing [CurrentFixture]s only
auto.Domain.Filter = FixtureFilters.Current;
}
auto.Verbose = true;
auto.Run();
auto.ReportToHtml();
}
}

If you use a test runner like TestDriven.Net, ReSharper or Icarus then you can select the specific test to run and just run that. If you're using the command-line tools, consider using a filter.
eg.
Gallio.Echo MyTestAssembly.dll /f:Name:TheNameOfTheParticularIWantToRun