We Keep Coding

Running tests marked as [Explicit] with dotnet test

Is it at all possible to execute tests marked with the NUnit [Explicit] attribute using dotnet test?
I have some tests which I would like to be ran as a separate build on Bamboo. They take a while to run so didn't want them to be included in the standard test run in VS so marked them with the [Explicit] attribute. I also marked them with [Category("Nightly")] so on Bamboo I hoped to have a step of:
dotnet test --filter "TestCategory=Nightly"
but as expected they still get ignored because of the Explicit property.
I tried explicitly naming, like this:
dotnet test --filter "ClassName=MyTests.Tests.TestClassName"
but still no luck, I've tried all of the possible arguments to dotnet test https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=mstest to refer to the tests specifically but they still get ignored.
Has anyone managed this?
Rich

As per this Github issue this is actually a known bug with NUnit3TestAdapter where tests marked as [Explicit] would not execute regardless of how they were referenced in the arguments to dotnet test. This has been addressed in the latest pre-release version of the nuget package.

AppVeyor ignores explicit assemly name for Unit tests

I'm trying to setup AppVeyor for my project (here) and I can't seem to find a way to only have it run tests from the .NET Core Unit test project.
This is the link for the AppVeyor project: ci.appveyor.com/project/Sergio0694/neuralnetwork-net
I also have a .NET Framework Unit test project that executes some GPU-based tests, so they just fail when run with AppVeyor as they're missing both a CUDA GPU and the required cuDNN files.
This is my configuration:
version: 1.0.{build}
image: Visual Studio 2017
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
assemblies:
only:
- NeuralNetwork.NET.Unit.dll
categories:
except:
- NetworkTest
That NetworkTest category that's skipped is just a group of tests that are very CPU intensive and require some time to run, so I'm just skipping them now to make the builds finish sooner.
The project builds fine, but AppVeyor keeps running the tests in the other NeuralNetwork.NET.Cuda.Unit.dll assembly as well, which fail as expected and cause the whole build to be marked as failing.
So my questions are:
Am I doing something wrong here? Why isn't the assembly constraint respected?
Is there another way to have AppVeyor only run the tests from NeuralNetwork.NET.Unit.dll?
EDIT: I've tried to set the "all assemblies excluding" option and all of these combinations to specify the .NET Framework Unit test project to skip:
NeuralNetwork.NET.Cuda.Unit.dll
**\*.NeuralNetwork.NET.Cuda.Unit.dll
**\NeuralNetwork.NET.Cuda.Unit.dll
The tests keep being executed (failing as expected), am I missing something here?
EDIT #2: This is the workaround I'm using for now, I've manually excluded all the test categories in the .NET Framework project and so far this seems to be working (even if it's not so clean to see):
version: 1.0.{build}
image: Visual Studio 2017
configuration: Release
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
categories:
except:
- CuDnnInceptionLayerTest
- CuDnnLayersTest
- GpuExtensionsTest
- SerializationTest

.NET Core tests detection works differently than detection of classic .NET Framework tests. When searching .NET Core tests, AppVeyor look for .csproj files with certain properties, not for assemblies. Assembly filter does not work for .NET Core tests (Categories filter works though).
Sorry for confusion. Created this issue to implement behavior similar to assembly name filtering, but based on .csproj file names for .NET Core tests auto-detection.
For now workaround is to do a "black list", e.g. list all .NET Framework assemblies you do not want to run under assemblies/except. You can use wildcard syntax to make it more compact as described here.

Generating TestResult with nunit-console failing

According to the SpecFlow website, I can generate the TestResult by executing the following statement:
nunit3-console.exe --labels=All --out=TestResult.txt "--result=TestResult.xml;format=nunit2" bin\Debug\BookShop.AcceptanceTests.dll
I've adapted this statement to the following:
nunit3-console.exe --labels=All --out=C:\temp\TestResult.txt "--result=C:\temp\TestResult.xml;format=nunit2" C:\Projects\DataService.IntegrationTests\bin\Debug\DataService.IntegrationTests.dll
Unfortunately, I get the following errors:
Errors, Failures and Warnings
1) Invalid : C:\Projects\DataService.IntegrationTests\bin\Debug\DataService.IntegrationTests.dll
No suitable tests found in 'C:\Projects\DataService.IntegrationTests\bin\Debug.DataService.IntegrationTests.dll'.
Either assembly contains no tests or proper test driver has not been found.
The .feature file are available in this assembly...
What could be the problem here?
An assumption of mine is, that we're using Specflow.MsTest...
Is there a way to generate the TestResult.xml by using MsTest?
Thanks in advance

NUnit3-console can only run NUnit tests. (Well technically, it can run any tests for which a driver is provided, but it amounts to the same thing.)
If you want to create NUnit output, then use both the NUnit framework and the NUnit console runner. MsTest doesn't do NUnit output.

In app.condig I don't have any unitTestProvider
<specFlow>
<stepAssemblies>
<stepAssembly assembly="otherProject" />
</stepAssemblies>
<runtime detectAmbiguousMatches="true" stopAtFirstError="false" missingOrPendingStepsOutcome="Inconclusive" />
<trace traceSuccessfulSteps="true" traceTimings="false" minTracedDuration="0:0:0.1" />
<!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
</specFlow>

NUnit for monodevelop doesn't see tests

I'm writing some code in c# in monodevelop. I have one project for the code, and another for testing. I've installed nunit for monodevelop, and in the testing project i have referenced the nunit framework i have in the packages. For some reason when i click the "run all tests" button all i get is that one test was ran succesfully, TestCase. I have the following code for instance:
[TestFixture ()]
public class Test
{
[Test ()]
public void AnotherTest()
{
Assert.AreEqual (2, 1);
}
[Test()]
public void JustTesting()
{
Assert.AreEqual (3, 1);
}
}
Obviously I would expect both of these to fail but instead Nunit prints out that one test (named TestCase) was ran successfully, ignoring the tests i wrote (no error message). I know TestCase is the default constructor name for a test class, i deleted it to check if it was still the only test ran, and somehow it is. What could possibly be causing all of this? The closest question i found to mine is this one NUnit doesn't find tests in assembly but sadly none of the answers seem to help. I have Nunit version 3.6.0 but i tried other older versions as well.

Microsoft.Fakes won't run in normal unit test contexts

I'm using a simple proof-of-concept Fakes nUnit test:
[Test]
public void TestFakes()
{
using (var ctx = ShimsContext.Create())
{
System.Fakes.ShimDateTime.NowGet = () => { return new DateTime(2000, 1, 1); };
Assert.That(DateTime.Now.Year, Is.EqualTo(2000));
}
}
This test runs in the Visual Studio Test Explorer, but doesn't run in:
nUnit GUI
nUnit console
The JetBrains test runner (dotCover OR Resharper)
TestDriven.net test runner
In each of these, I receive the following error:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException : Failed to resolve profiler path from COR_PROFILER_PATH and COR_PROFILER environment variables
When I reflect into that assembly, it seems like it's looking for Intellitrace, a VS Ultimate-only feature - I only have Premium installed.
Any suggestions on how to work around this (we use the nUnit runner on our build servers, so this is a blocker to using Fakes)

I don't think you will be able to execute MS Fakes based tests using anything other then MS Test framework.
I believe that the way MS Fakes works causes problems for test runners such as NUnit. Precisely why this is the case, I don't know, since other mocking frameworks such as TypeMock work fine in NUnit, etc. But there is something specific to MS Fakes which make it harder (if not impossible) to run with anything other than MS Test. That's my theory anyway.
Unless the authors of NUnit, xUnit, etc add support for MS Fakes (or there is a crafty workaround), I think you will have to stick with MS Test.
EDIT:
It looks like the latest version of NCrunch v2.5 does work with MS Fakes. I've tried the beta during it's development, and can confirm that MS Fake tests were executed without fail using NCrunch.

Fakes works only with Visual Studio Test Runner(AKA VStest.Console.exe). Even previous Microsoft MSTest runner doesn't support it.

I believe your answer is to use one of following commands:
VSTest.Console.exe UnitTests.dll /UseVsixExtensions
VSTest.Console.exe UnitTests.dll /TestAdapterPath:%LocalAppData%\Microsoft\VisualStudio\12.0\Extensions\<nunit test adapter installation folder>
These commands will run fine on dev machines. In case you want to run unit tests on build server, copy nunit test adapter folder to build server and mention that path.