We Keep Coding

dotnet test: List all tests and their test attributes/properties

When running dotnet test my-tests.dll --list-tests from command line, the utility outputs:
The following tests are available:
MyTest1
MyTest2
However, these tests contain meta-data that I would also like displayed:
[TestFixture]
public class MyTests
{
[Test, Property("some_key", "some_value1")]
public void MyTest1()
{
// ...
}
[Test, Property("some_key", "some_value2")]
public void MyTest2()
{
// ...
}
}
Is there any way that I can display the Property key/values while using the --list-tests flag in dotnet test?

No -- according to the dotnet test CLI documentation, the --list-tests flag lists the discovered test names rather than running them.
If your goal is for your test names to reflect the fact that they are being run against certain values, [TestCase] may be where you want to look.
For example, the following:
public class Tests
{
[TestCase("some_value1")]
[TestCase("some_value2")]
public void MyTestWithSomeKeyOf(string value)
{
Assert.Pass();
// ...
}
}
would produce the following output from dotnet test --list-tests:
PS C:\Users\SeanK\Repositories\2023_01_NunitRepro> dotnet test --list-tests
Determining projects to restore...
All projects are up-to-date for restore.
Repro -> C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll
Test run for C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
The following Tests are available:
MyTestWithSomeKeyOf("some_value1")
MyTestWithSomeKeyOf("some_value2")

ScriptCS 0.17.01 Error Unexpected named argument

I tried to run my program using Visual Studio's Coderunner extension as well as from terminal with the scriptcs command.
My code is as follows:
using System;
namespace HelloWorldApplication {
class HelloWorld {
static void Main(string[] args) {
Console.WriteLine("hellowol");
}
}
}
The error message reads:
Unexpected named argument: Users/jfitz/Projects/C#/Projtest/test.cs

As mentioned in scriptcs/scriptcs issue 1188, this is from a scriptcs bug, which will be fixed in the next release (PR 1289 and commit 9e49b72)
In the meantime, pending the next 0.18 scriptcs release:
The workaround is the following:
instead of doing
mono scriptcs.exe /path/to/foo.csx
do:
mono scriptcs.exe -script /path/to/foo.csx
Jonas suggests in the comments:
For Visual Studio Code, add this to settings.json:
"code-runner.executorMap": { "csharp": "scriptcs -script" }

How to filter NUnit tests by category using "dotnet test"

I have a project that has a
[TestFixture, Category("Oracle")]
and a
[TestFixture, Category("OracleOdbc")]
with a couple of tests which I would like to execute separately using dotnet test.
Here's what I tried after some Googling:
dotnet test MyProject.csproj --where "cat==Oracle" but this switch does not exists anymore.
dotnet test MyProject.csproj --filter Category="Oracle" yields 0 applicable tests: No test is available in ....
Then, I've stumbled over this article and although it describes MSTest (and NUnit has the CategoryAttribute and not a TestCategoryAttribute), I've tried
dotnet test MyProject.csproj --filter TestCategory="Oracle"
Bingo. This time all "Oracle" tests were executed. But now comes the confusing part. If I run dotnet test MyProject.csproj --filter TestCategory="OracleOdbc", all tests are being executed, including "Oracle" and "OracleOdbc". This makes me wonder if TestCategroy is the proper way to go for NUnit or if this is a bug.
I'm using .NET Command Line Tools (2.1.2) and the project references are:
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
<PackageReference Include="NUnit" Version="3.8.1" />
<PackageReference Include="NUnit3TestAdapter" Version="3.9.0" />
<PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.7" />
BTW, I don't know if it matters but my test project is multi-targeting netcoreapp2.0 and net462.

This might not be very helpful, but it seems to be working for me correctly. I created the projects using the dotnet-cli.
First I installed the NUnit3 test adapter instructions from here. This only needs to be run once on each machine so you don't need to do it again if you have already run it.
dotnet new -i NUnit3.DotNetNew.Template
Then I created my solution, created my test project and added the test project to the solution.
dotnet new sln -n Solution
dotnet new nunit -n TestProject -o tests\TestProject
dotnet sln add tests\TestProject\TestProject.csproj
Then I updated UnitTest1.cs to include two test fixtures, one with the category Oracle and one with the category OracleOdbc.
using NUnit.Framework;
namespace Tests
{
[TestFixture]
[Category("Oracle")]
public class OracleTests
{
[Test]
public void OracleTest()
{
Assert.Fail();
}
}
[TestFixture]
[Category("OracleOdbc")]
public class OracleOdbcTests
{
[Test]
public void OracleOdbcTest()
{
Assert.Fail();
}
}
}
Then I can specify which category I choose to run.
dotnet test tests/TestProject/TestProject.csproj --filter TestCategory="Oracle"
or
dotnet test tests/TestProject/TestProject.csproj --filter TestCategory="OracleOdbc"
both run only one test and the message shows it is the correct test that fails.
Using DotNet-Cli version 2.1.4 and NUnit3TestAdapter version 3.9.0

In Nunit Framework, Category Attribute can be on the level of the Method.
Example:
public class NUnitTest
{
[Test]
[Category("CategoryA")]
public void TestMethod1()
{
}
[Test]
[Category("CategoryB")]
public void TestMethod2()
{
}
}
and command is:
dotnet test --filter TestCategory=CategoryA #Runs tests which are annotated with [Category("CategoryA")].
Also, there are many options on the level of method and others
For more details: read

There are now two options for filtering tests by category using dotnet test. You can use either dotnet.exe's built-in test filtering syntax, or the NUnit filtering syntax.
First add the NUnit3TestAdapter to your project using NuGet:
install-package nunit3testadapter -proj YourProject
You can then filter tests either like this:
dotnet.exe test .\ --test-adapter-path:. --filter TestCategory=Foo
or like this:
dotnet.exe test .\ --test-adapter-path:. -- NUnit.Where="cat=Foo"
This blog post goes into further detail.

You can also place a filter inside your test if you like enums like me:
[Test]
public void Test_Foo()
{
// filter test
switch (GlobalDefinitions.Category)
{
// OK Test
case Category.Oracle:
case Category.SQL:
break;
// Do NOT test
case Category.MongoDb:
Assert.Inconclusive();
// Error
default:
Assert.Fail("Not implemented case");
}
// perform test...
}
Have the variable GlobalDefinitions.Category obtain the value from a resource file or whatever works best for you.
Edit make the same code shorter with Flags
Create your categories
[Flags] // <-------------------- Important to shorten code
public enum Category: int
{
None = 0,
Oracle = 1 << 0,
SQL = 1 << 1,
MongoDb = 1 << 2,
// future categories
ALL = -1
}
// create your filter method
public static void Filter(Category category)
{
if(GlobalDefinitions.Category.HasFlag(category) == false)
Assert.Inconclusive(); // do not perform test
// else perform test
}
// then create your test as:
[Test]
public void Test_Foo()
{
Filter(Category.SQL | Category.MongoDb); // place filter (in this test we are testing for SQL and MongoDb
// perform your test ...
}

How to programmatically run unit tests with Gallio and MBUnit?

I'm trying to programmatically check my unit tests are passing as part of my deployment process. The application uses MBunit and Gallio for it's unit testing framework.
Here's my code:
var setup = new Gallio.Runtime.RuntimeSetup();
setup.AddPluginDirectory(#"C:\Program Files\Gallio\bin");
using (TextWriter tw = new StreamWriter(logFilename))
{
var logger = new Gallio.Runtime.Logging.TextLogger(tw);
RuntimeBootstrap.Initialize(setup, logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern(dllToRunFilename);
TestLauncherResult result = launcher.Run();
}
Here's the test which is contained in the DLL I'm loading (I've validated this works with the Icarus test runner):
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
When I run the application I get the following values in results
Which is incorrect as there are indeed tests to run! The log file has the following in it
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin enable
condition was not satisfied. Please note that this is the intended
behavior for plugins that must be hosted inside third party
applications in order to work. Enable condition:
'${process:DEVENV.EXE_V9.0} or ${process:VSTESTHOST.EXE_V9.0} or
${process:MSTEST.EXE_V9.0} or ${framework:NET35}'. Disabled plugin
'Gallio.VisualStudio.Tip90': The plugin depends on another disabled
plugin: 'Gallio.VisualStudio.Shell90'.
How do I resolve this issue and find the results to the tests?

This works for me, note I used this GallioBundle nuget to get gallio and mbunit, so perhaps there is some difference to what you have installed.
The log messages regarding plugins are expected, those plugins wont work if you are self-hosting the Gallio runtime.
using System;
using System.IO;
using Gallio.Runner;
using Gallio.Runtime;
using Gallio.Runtime.Logging;
using MbUnit.Framework;
public static class Program
{
public static void Main()
{
using (TextWriter tw = new StreamWriter("RunTests.log"))
{
var logger = new TextLogger(tw);
RuntimeBootstrap.Initialize(new RuntimeSetup(), logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern("RunTests.exe");
TestLauncherResult result = launcher.Run();
Console.WriteLine(result.ResultSummary);
}
}
}
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
Tested like this:
› notepad RunTests.cs
› nuget.exe install -excludeversion GallioBundle
Installing 'GallioBundle 3.4.14.0'.
Successfully installed 'GallioBundle 3.4.14.0'.
› cd .\GallioBundle\bin
› csc ..\..\RunTests.cs /r:Gallio.dll /r:MbUnit.dll
Microsoft (R) Visual C# Compiler version 12.0.21005.1
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.
› .\RunTests.exe
2 run, 1 passed, 1 failed, 0 inconclusive, 0 skipped

These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.
Firstly, install the NUnit Test Adapter extension (yes, NUnit)
Tools > Extension and Updates > Online > search for NUnit > install
NUnit Test Adapter.
You may need to restart the Visual Studio IDE.
Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...
//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;
namespace MyTests
{
[TestFixture]
public class UnitTest1
{
[Test, NuTest]
public void myTest()
{
//this will pass
}
}
}
You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit

sonar with gallio and opencover, code coverage: 0%

I'm using sonar to check my c# project. I would like to measure code coverage that's why i installed gallio and opencover. When I run soner-runner everything works fine, my unit test is performed, ... but the code coverage is 0% on the sonar web UI.
do you know the reason why the code coverage is 0%?
My solution, project and classes:
(S) SonarTestSolution
(P) ClassLibrary1
(C) Class1.cs
(P) ClassLibrary1NUnitTest
(C) Class1NUnitTest.cs
content of Class1.cs:
public class Class1 {
public String getTestString() { return "abc";}
}
content of Class1NUnitTest.cs:
[TestFixture]
public class Class1NUnitTest {
[Test]
public void createServiceFromFactoryTest() {
Class1 c = new Class1();
Assert.That(c.getTestString(), Is.EqualTo("abc"));
}}
sonar-project.properties:
sonar.projectKey=cs_sonar_test
sonar.projectVersion=1.0-SNAPSHOT
sonar.projectName=C# Sonar Test Project
sonar.sourceEncoding=UTF-8
sources=.
sonar.language=cs
sonar.donet.visualstudio.testProjectPattern=*NUnitTest*
If you want I can include the log og sonar-runner, and the generated coverage-report.xml and gallio-report.xml files

solved - the project was build another machine, this was the problem. Building the solution in the same folder solves the proble.