Is it at all possible to execute tests marked with the NUnit [Explicit] attribute using dotnet test?
I have some tests which I would like to be ran as a separate build on Bamboo. They take a while to run so didn't want them to be included in the standard test run in VS so marked them with the [Explicit] attribute. I also marked them with [Category("Nightly")] so on Bamboo I hoped to have a step of:
dotnet test --filter "TestCategory=Nightly"
but as expected they still get ignored because of the Explicit property.
I tried explicitly naming, like this:
dotnet test --filter "ClassName=MyTests.Tests.TestClassName"
but still no luck, I've tried all of the possible arguments to dotnet test https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=mstest to refer to the tests specifically but they still get ignored.
Has anyone managed this?
Rich
As per this Github issue this is actually a known bug with NUnit3TestAdapter where tests marked as [Explicit] would not execute regardless of how they were referenced in the arguments to dotnet test. This has been addressed in the latest pre-release version of the nuget package.
Related
I am working on a C# project using Visual Studio 2022. The project has a Unit Test assembly, and this works fine. We can run tests in the Test Explorer and also the tests are run automatically in DevOps.
I would like to introduce a test into this assembly that needs some setup with an external utility app. The external app will receive a message, modify it, and send it back to the source, but it needs to be run interactively. It is a useful test to have in terms of product quality, but the tester needs to set up the environment so it can't be part of any automated test.
Is there any way to either:
Specify that the test can only be run manually, and should not be part of an automated test run, or
Detect whether the test is being run manually or automatically, so that it can just indicate success if run automatically?
When using Xunit you can use the Trait attribute on your test to define a category and use this as filter when executing test in your pipeline.
So on a regular unit test you can add this:
[Trait("Category", "Unit")]
And on a more complex test you could add something like this:
[Trait("Category", "Integration")]
And use this as filter in your pipeline. For example to only run unit tests:
- task: DotNetCoreCLI#2
displayName: "dotnet test ProjectWithTests.csproj"
inputs:
command: "test"
projects: "ProjectWithTests.csproj"
arguments: '--no-build --configuration "Release" --filter Category=Unit'
I have a project where I can not run the tests using dotnet test while my colleague can. The error message I get is
System.IO.FileNotFoundException: No test is available in C:\projects\...\bin\Debug\netcoreapp1.1\Test.AnalyzeFirmwareRollout.dll. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk" and framework version settings are appropriate and try again.
The project does have a reference to Microsoft.NET.Test.Sdk and the target framework is
<TargetFramework>netcoreapp1.1</TargetFramework>
in the csproj file. Any Ideas what could be going wrong here?
For running xUnit tests from command line, you will need the xunit.runner.console package.
I'm trying to setup AppVeyor for my project (here) and I can't seem to find a way to only have it run tests from the .NET Core Unit test project.
This is the link for the AppVeyor project: ci.appveyor.com/project/Sergio0694/neuralnetwork-net
I also have a .NET Framework Unit test project that executes some GPU-based tests, so they just fail when run with AppVeyor as they're missing both a CUDA GPU and the required cuDNN files.
This is my configuration:
version: 1.0.{build}
image: Visual Studio 2017
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
assemblies:
only:
- NeuralNetwork.NET.Unit.dll
categories:
except:
- NetworkTest
That NetworkTest category that's skipped is just a group of tests that are very CPU intensive and require some time to run, so I'm just skipping them now to make the builds finish sooner.
The project builds fine, but AppVeyor keeps running the tests in the other NeuralNetwork.NET.Cuda.Unit.dll assembly as well, which fail as expected and cause the whole build to be marked as failing.
So my questions are:
Am I doing something wrong here? Why isn't the assembly constraint respected?
Is there another way to have AppVeyor only run the tests from NeuralNetwork.NET.Unit.dll?
EDIT: I've tried to set the "all assemblies excluding" option and all of these combinations to specify the .NET Framework Unit test project to skip:
NeuralNetwork.NET.Cuda.Unit.dll
**\*.NeuralNetwork.NET.Cuda.Unit.dll
**\NeuralNetwork.NET.Cuda.Unit.dll
The tests keep being executed (failing as expected), am I missing something here?
EDIT #2: This is the workaround I'm using for now, I've manually excluded all the test categories in the .NET Framework project and so far this seems to be working (even if it's not so clean to see):
version: 1.0.{build}
image: Visual Studio 2017
configuration: Release
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
categories:
except:
- CuDnnInceptionLayerTest
- CuDnnLayersTest
- GpuExtensionsTest
- SerializationTest
.NET Core tests detection works differently than detection of classic .NET Framework tests. When searching .NET Core tests, AppVeyor look for .csproj files with certain properties, not for assemblies. Assembly filter does not work for .NET Core tests (Categories filter works though).
Sorry for confusion. Created this issue to implement behavior similar to assembly name filtering, but based on .csproj file names for .NET Core tests auto-detection.
For now workaround is to do a "black list", e.g. list all .NET Framework assemblies you do not want to run under assemblies/except. You can use wildcard syntax to make it more compact as described here.
I upgraded my Nunit to 3.0 from 2.6 and now all of my Unit Tests are failing with the following error:
OneTimeSetUp: SetUpAttribute attribute not allowed in a SetUpFixture
I read through the NUnit documentation and found that there are some "breaking changes" with the new release and I can no longer use a "SetUpAttribute" inside a "SetUpFixture".
I will have to replace this with "OneTimeSetUpAttribute" and I would but my MAIN problem is that my Nunit test fixtures are generated by Specflow.
I have specflow 2.1 and this is generating test fixtures with default "OneTimeSetUpAttribute" regardless of Nunit version.
Did anyone else come across this problem and have a solution for this?
Thanks in advance.
I found this neat way to use NUnit in Powershell. http://elegantcode.com/2009/10/25/integration-test-brought-to-you-by-powershell-nunit-with-a-little-specification-syntax-for-flavoring/
and we are using it many of our tests.
However I want to run these tests in TeamCity.
I want similar behavior when we use a NUnit runner for running C# tests in TeamCity ie the build fails when the execution of tests fail. Has anyone of you achieved this? I suspect the Powershell runner will just execute it as a simple script, without any indication whether the test passes or fails.
Take a look at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity and http://confluence.jetbrains.net/display/TCD7/Build+Failure+Conditions
There is an issue in Powershell runner support http://youtrack.jetbrains.com/issue/TW-21554
I'm not familiar with the approach you're referencing for executing NUnit tests via Powershell in TeamCity. But, we are successfully using PSake for Powershell build scripts, including executing NUnit tests and failing the build appropriately. The same issue exists with PSake and TeamCity with exit codes, but you can get around it by specifying in TeamCity in the Script Source for the Build Step using -Command for Script execution:
import-module .\tools\psake\psake.psm1
$psake.use_exit_on_error = $true
invoke-psake build.ps1
remove-module psake
You can also integrate the Test results into the TeamCity using the Build Feature option in TeamCity Build Steps.