We Keep Coding

VSTest-testAssemblies agent is not running my Category search filter for NUnit framework in Azure Pipelines

I am using in the Test Filter criteria TestCategory=CategoryA
In my Tests I have the below saved and think it should be the only test run but I get success and no tests ran. I have tried with dotnet test and used the same in the arguments and gain it fails with other issues. I can get dot net test to run without adding any arguments for VS Test agent it seems to not want to run even when I do not add the test filter criteria.
[Test]
[Category("CategoryA")]
public async Task ActivateDevice()
2022-02-18T20:14:54.5643568Z ##[section]Starting: VsTest - testAssemblies
2022-02-18T20:14:54.5771277Z ==============================================================================
2022-02-18T20:14:54.5771649Z Task : Visual Studio Test
2022-02-18T20:14:54.5772497Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2022-02-18T20:14:54.5773342Z Version : 2.198.0
2022-02-18T20:14:54.5773587Z Author : Microsoft Corporation
2022-02-18T20:14:54.5773921Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2022-02-18T20:14:54.5774333Z ==============================================================================
2022-02-18T20:14:55.5713320Z SystemVssConnection exists true
2022-02-18T20:14:55.5732257Z SystemVssConnection exists true
2022-02-18T20:14:55.5991193Z Running tests using vstest.console.exe runner.
2022-02-18T20:14:55.5991725Z ======================================================
2022-02-18T20:14:55.5997078Z Test selector : Test assemblies
2022-02-18T20:14:55.5998382Z Test filter criteria : null
2022-02-18T20:14:55.5999337Z Search folder : D:\a\r1\a
2022-02-18T20:14:55.6102675Z Action when minimum tests threshold not met : donothing
2022-02-18T20:14:55.6103140Z Minimum tests expected to be run: 0
2022-02-18T20:14:55.6104476Z VisualStudio version selected for test execution : latest
2022-02-18T20:14:55.6107744Z Attempting to find vstest.console from a visual studio installation with version [17.0,18.0).
2022-02-18T20:14:56.0357719Z Attempting to find vstest.console from a visual studio build tools installation with version [17.0,18.0).
2022-02-18T20:14:56.0642271Z Attempting to find vstest.console from a visual studio installation with version [16.0,17.0).
2022-02-18T20:14:56.0884398Z Attempting to find vstest.console from a visual studio build tools installation with version [16.0,17.0).
2022-02-18T20:14:56.1142098Z Attempting to find vstest.console from a visual studio installation with version [15.0,16.0).
2022-02-18T20:14:56.3498931Z Run in parallel : false
2022-02-18T20:14:56.3500994Z Run in isolation : false
2022-02-18T20:14:56.3502074Z Path to custom adapters : null
2022-02-18T20:14:56.3504871Z Other console options : null
2022-02-18T20:14:56.3505911Z Code coverage enabled : false
2022-02-18T20:14:56.3510216Z Diagnostics enabled : false
2022-02-18T20:14:56.3519771Z SystemVssConnection exists true
2022-02-18T20:14:56.3697972Z Run the tests locally using vstest.console.exe
2022-02-18T20:14:56.3698460Z ========================================================
2022-02-18T20:14:56.5771570Z Source filter: D:\a\r1\a/_Test.AppiumTest/AppiumTesting/AppiumTest
2022-02-18T20:14:56.5973322Z ##[warning]No test sources found matching the given filter

1. Check whether the following latest packages are installed or not:
Microsoft.NET.Test.Sdk
MSTest.TestAdapter
MSTest.TestFramework
NUnit3TestAdapter
2. As answered by Marina Liu , you can configure MSTest or the "Visual Studio Test" task in VSO to fail if there are warnings:
a. Add a PowerShell task after Visual Studio Test task.
b. Then get Visual Studio Test task build information by Timeline:
c. Search for Visual Studio Test task information by task name or task display name.
d. Fail the build result if VS test has the warning you specified.
References: How to Fix this C# issue No test matches the given testcase filter `FullyQualifiedName = , How can I configure MSTest to fail the unit test run on any warnings? and Run NUnit tests in Azure DevOps pipeline

VS2017 and NUnit 3.9 No test is available

I am using the latest VS2017 version 15.6.4, NUnit3TestAdapter 3.10.0 and Nunit version 3.9.0.0.
When I try to run a unit test in Test Explorer the test are grayed out, when I right click and and run selected tests I see the following error: No test is available
Here is how my test class looks
[TestFixture]
public partial class ListViewBOTest
{
[Test]
public void TestSearch_DateTime()
{
Assert.AreEqual(1,0);
}
}
Text from output:
[3/26/2018 10:53:55 AM Informational] ------ Run test started ------
[3/26/2018 10:53:55 AM Informational] NUnit Adapter 3.10.0.21: Test execution started
[3/26/2018 10:53:55 AM Informational] Running all tests in C:\Projects\MVPPlant\DEV\CMMSdg.Plant\CMMSdg.Plant\Sln.2010\CMMSdg.Plant.BusinessObjects.Test\bin\Debug\CMMSdg.Plant.BusinessObjects.Test.dll
[3/26/2018 10:53:56 AM Informational] NUnit failed to load C:\Projects\MVPPlant\DEV\CMMSdg.Plant\CMMSdg.Plant\Sln.2010\CMMSdg.Plant.BusinessObjects.Test\bin\Debug\CMMSdg.Plant.BusinessObjects.Test.dll
[3/26/2018 10:53:56 AM Informational] NUnit Adapter 3.10.0.21: Test execution complete
[3/26/2018 10:53:56 AM Warning] No test is available in C:\Projects\MVPPlant\DEV\CMMSdg.Plant\CMMSdg.Plant\Sln.2010\CMMSdg.Plant.BusinessObjects.Test\bin\Debug\CMMSdg.Plant.BusinessObjects.Test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[3/26/2018 10:53:56 AM Informational] ========== Run test finished: 0 run (0:00:01.0435303) ==========

Make sure you have installed the NUnit3 Test Adapter from here
https://marketplace.visualstudio.com/items?itemName=NUnitDevelopers.NUnit3TestAdapter

Can you check the following steps and see if it works?
In the Visual Studio menu, go to Test > Test Settings > Default Processor Architecture and make a note if X86 is selected or X64
Now go to the Build section in the Properties window of the project where the tests are written. Make sure the Platform target drop-down is selected to either Any CPU or at least it matches the architecture from the above step 1.
Now if you build the solution and try running those tests, you should see that they are now running.

I had the same problem as Amete Blessed and commenting out other Test methods made Test Explorer work and run my test

I found that my Build Events were wrong. Invalid copy command blew all my tests and half a day:
Copy C:\repo\Architecture\*.json $(ProjectPath)/Y
Copy C:\repo\Architecture\*.json $(TargetPath) /Y
instead of
Copy C:\repo\Architecture\*.json $(ProjectDir)/Y
Copy C:\repo\Architecture\*.json $(TargetDir) /Y

The messages in the Test Output window ("NUnit failed to load [assembly]", "No test is available...", etc.) can hide the underlying issue that's causing the runner to not load the test assembly. This includes hiding failures to load dependencies of the test assembly or the item under test.
If there's a test assembly that's showing up in the Test Explorer window, but the tests refuse to run, it's worth temporarily enabling fusion logging to see if any assembly binding errors occur when trying to run the tests.

After installing NUnit through nuget tests have appeared into Test Explorer
but when I ran them I got "No test is available".
Installing NUnit Test Adapter fixed the issue.

I had similar problem when using Xamarin.Forms.
The solution was to install NUnit.XForms from NuGet and add
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
to csproj file of the project where the tests are.

What worked for me was to delete the ComponentModelCache folder located at %localappdata%\Microsoft\VisualStudio\<version>

You may have been a casualty of this problem that was (theoretically) resolved in 15.6.3. According to this answer, try deleting your %temp%\VisualStudioTestExplorerExtensions folder. That has resolved the issue for some other users.

Rather than using the Test Explorer, can you right click on the solution and Run Unit Tests from there?

Had the same problem. In my case I found that the NUnit test adapter will not be used by Test Explorer if your test project contains a reference to MSTest. Typically the Microsoft.VisualStudio.TestPlatform.TestFramework.dll but also check your .csproj file for "MSTest" and your packages.config file.
I discovered this by first enabling diagnostic logging for Visual Studio tests. This is found under "Tools" -> "Options" -> "Tests" -> "Logging Level".
In my log I found this entry:
[22/11/2018 10:36:42 Diagnostic] Project C:\Git\myProject\src\myProject.Tests\myProject.Tests.csproj references test adapter: MSTest.TestAdapter, version 1.1.18

This is caused due to memory problem.
Clean object after test execution
Sample Code
private TestController testController;
[OneTimeSetUp]
public void TestSetup()
{
testController= new TestController();
}
[OneTimeTearDown]
public void TestCleanup()
{
testController= null;
}
Change test execution to 64 bits in settings
Test -> Test Settings -> Default Processors Architecture -> x64.

I have also encountered the same issue.
Steps to resolve this issue -
1. Install/Reinstall NUnit3TestAdapter package
2. Delete Debug folder from the Bin
3. Clean --> Build the project
Note - Also ensure all the packages are installed properly.
Now try to execute the tests.

I had the same issue, but it was fixed after installing "NUnit 3 TestAdapter version 3.16.1" via NuGet to my project.

AppVeyor ignores explicit assemly name for Unit tests

I'm trying to setup AppVeyor for my project (here) and I can't seem to find a way to only have it run tests from the .NET Core Unit test project.
This is the link for the AppVeyor project: ci.appveyor.com/project/Sergio0694/neuralnetwork-net
I also have a .NET Framework Unit test project that executes some GPU-based tests, so they just fail when run with AppVeyor as they're missing both a CUDA GPU and the required cuDNN files.
This is my configuration:
version: 1.0.{build}
image: Visual Studio 2017
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
assemblies:
only:
- NeuralNetwork.NET.Unit.dll
categories:
except:
- NetworkTest
That NetworkTest category that's skipped is just a group of tests that are very CPU intensive and require some time to run, so I'm just skipping them now to make the builds finish sooner.
The project builds fine, but AppVeyor keeps running the tests in the other NeuralNetwork.NET.Cuda.Unit.dll assembly as well, which fail as expected and cause the whole build to be marked as failing.
So my questions are:
Am I doing something wrong here? Why isn't the assembly constraint respected?
Is there another way to have AppVeyor only run the tests from NeuralNetwork.NET.Unit.dll?
EDIT: I've tried to set the "all assemblies excluding" option and all of these combinations to specify the .NET Framework Unit test project to skip:
NeuralNetwork.NET.Cuda.Unit.dll
**\*.NeuralNetwork.NET.Cuda.Unit.dll
**\NeuralNetwork.NET.Cuda.Unit.dll
The tests keep being executed (failing as expected), am I missing something here?
EDIT #2: This is the workaround I'm using for now, I've manually excluded all the test categories in the .NET Framework project and so far this seems to be working (even if it's not so clean to see):
version: 1.0.{build}
image: Visual Studio 2017
configuration: Release
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
categories:
except:
- CuDnnInceptionLayerTest
- CuDnnLayersTest
- GpuExtensionsTest
- SerializationTest

.NET Core tests detection works differently than detection of classic .NET Framework tests. When searching .NET Core tests, AppVeyor look for .csproj files with certain properties, not for assemblies. Assembly filter does not work for .NET Core tests (Categories filter works though).
Sorry for confusion. Created this issue to implement behavior similar to assembly name filtering, but based on .csproj file names for .NET Core tests auto-detection.
For now workaround is to do a "black list", e.g. list all .NET Framework assemblies you do not want to run under assemblies/except. You can use wildcard syntax to make it more compact as described here.

NUnit 3 console runner can't assert that collection is ordered

I am running a CI build using Travis CI. I am running NUnit tests via the nunit3-console.exe. I have several tests that attempt to assert that a collection is ordered:
[Test]
public void FeatsAreSorted()
{
var result = controller.Generate() as JsonResult;
dynamic data = result.Data;
Assert.That(data.character.Ability.Feats, Is.Ordered.By("Name"));
}
When I run this test in Visual Studio, the test passes fine. However, when I run the test via nunit3-console.exe in Travis CI, I get the following error:
1) Error : DNDGenSite.Tests.Unit.Controllers.CharacterControllerTests.GenerateSortsCharacterFeats
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : `NUnit.Framework.Assert.That<System.Linq.OrderedEnumerable<CharacterGen.Common.Abilities.Feats.Feat,string>>(System.Linq.OrderedEnumerable<CharacterGen.Common.Abilities.Feats.Feat,string>, NUnit.Framework.Constraints.IResolveConstraint)' is inaccessible due to its protection level
This is my .travis.yml:
language: csharp
solution: DNDGenSite.sln
install:
- nuget restore DNDGenSite.sln
- nuget install NUnit.Runners -OutputDirectory testrunner
- nuget install Chutzpah -OutputDirectory testrunner
script:
- xbuild DNDGenSite.sln /p:TargetFrameworkVersion="v4.5.1" /p:Configuration=Release
- mono ./testrunner/NUnit.Console.*/tools/nunit3-console.exe ./Tests/bin/Release/DNDGenSite.Tests.dll
- mono ./testrunner/Chutzpah.*/tools/chutzpah.console.exe ./Tests/Unit/Scripts
Any thoughts?
UPDATE: If I run the tests in the git bash, everything passes correctly, in both Debug and Release build modes. So, there is something different about the environment in which Travis CI builds the console runner.

In the end, I found that asserting the order off of the dynamic object was causing the error in Travis. If instead, I verified it equaled a different object and checked the properties on tat object, the test passes fine.

TFS 2012 Build Server NUnit Exception

I have a NUnit test project that I'm trying to run on my TFS 2012 Build server.
I'm getting the following error:
Exception System.InvalidOperationException, Exception thrown
executing tests
No test is available in C:\Builds\1\Projects\NUNIT
Test\bin\NUnitTestProject.dll C:\Builds\1\Projects\NUNIT
Test\bin\NUnitTestProject.dll. Make sure that installed test
discoverers & executors, platform & framework version settings are
appropriate and try again.
I Have the NUnit.VisualStudio.TestAdapter 1.1.0.0 installed from NuGet and referenced in the project.
I also have set the Version control path to custom assemblies set on the controller, which contains (they basically are a copy of the NuGet package):
nunit.core.dll
nunit.core.interfaces.dll
nunit.util.dll
NUnit.VisualStudio.TestAdapter.dll
For test purposes, I'm only building the NUnit test project.
My test class is very simple and is the following:
using NUnit.Framework;
namespace NUnitTestProject
{
[TestFixture]
public class MySuperTests
{
[Test]
public void MyFirstUnitTestMethod_ReturnsTrue()
{
Assert.IsTrue(true);
}
}
}
Is there something I haven't done correctly?
Edit 1:
Using TFS 2012 Update 4 on my build server
Edit 2:
The problem I believe lies in the 1.1.0.X version.
Updating to 1.1.0.7 I did not have the exception anymore, however, I did have the tests not found problem.
There are some issues going on about that on github https://github.com/nunit/nunit-vs-adapter/
Edit 3:
The problem has been fixed in 1.1.0.8 of the NUnit test adapter!

Do you have your test runner set correctly in the build defiintion?
Image shows TFS2013 but 2012 has a different option for 2010 compatability which may be selected. The Visual Studio Test Runner should support multiple test frameworks, whereas the 2010 compatability one only supports MSTest