We Keep Coding

Identify if Visual Studio tests are being run manually or automatic

I am working on a C# project using Visual Studio 2022. The project has a Unit Test assembly, and this works fine. We can run tests in the Test Explorer and also the tests are run automatically in DevOps.
I would like to introduce a test into this assembly that needs some setup with an external utility app. The external app will receive a message, modify it, and send it back to the source, but it needs to be run interactively. It is a useful test to have in terms of product quality, but the tester needs to set up the environment so it can't be part of any automated test.
Is there any way to either:
Specify that the test can only be run manually, and should not be part of an automated test run, or
Detect whether the test is being run manually or automatically, so that it can just indicate success if run automatically?

When using Xunit you can use the Trait attribute on your test to define a category and use this as filter when executing test in your pipeline.
So on a regular unit test you can add this:
[Trait("Category", "Unit")]
And on a more complex test you could add something like this:
[Trait("Category", "Integration")]
And use this as filter in your pipeline. For example to only run unit tests:
- task: DotNetCoreCLI#2
displayName: "dotnet test ProjectWithTests.csproj"
inputs:
command: "test"
projects: "ProjectWithTests.csproj"
arguments: '--no-build --configuration "Release" --filter Category=Unit'

xUnit does not produce results file with lots of tests

Issue
When I run xUnit tests on a project containing a lot of tests, it does not produce the xml result file. If I reduce the number of tests in the same project to a smaller set of tests it does produce the result file. Why is this happening?
Details
My project has about 3200 xUnit tests running in a Jenkins build. I am using the following command to run the tests.
dotnet test --logger:"xunit;LogFilePath=/results/integrationtest_results.xml"
However, this is not creating the integrationtest_results.xml file. I have confirmed the tests did run even though it did not produce the result file.
When I use the following command to filter down the tests to less than 1000 tests it DOES create the integrationtest_results.xml result file.
dotnet test --logger:"xunit;LogFilePath=/results/integrationtest_results.xml" --filter:"someAttribute=someValue"
In the same Jenkins build, I have another project that runs 500 unit tests with the following command.
dotnet test --logger:"xunit;LogFilePath=/results/unittest_results.xml"
This has always produced the expected unittest_results.xml.
In the cases where it does not produce the result file, I do not see any logs stating an error occurred relating to the xunit results file.
Environment
C# dotnet core 6.0
Xunit v2.4.1
Test logger: XunitXml.TestLogger v3.0.70

Generate both code coverage report and Unit test results using a single command (if possible)

I have a Windows service written in .net framework 4.6. I'm trying to run Sonar analysis for this service. My requirement is to generate both code coverage result and Unit test case report either by using MStest.exe or vstest.console.exe. I have written test cases using MStest for my service.
Using MSTest, I have written the below command:
MSTest /testcontainer:.\SolutionTests\bin\Release\SolutionTests.dll /resultsfile:"C:\SonarQube\Solution.trx"
Using vstest.console.exe, I have written the below command:
vstest.console.exe SolutionTests\bin\Release\SolutionTests.dll /Enablecodecoverage /Logger:trx;LogFileName="C:\SonarQube\Solution.trx"
In both the cases only Unit test report is generated (.trx file) as I have set the filename explicitly in the command.
Is there any way I can generate .coverage file as well, with in the same command by adding other parameters. I read in few articles which said MSTest command generates both the reports (result.trx and data.coverage), but no where it is written the exact command how to do it. I ran the above command, it did not generate data.coverage file for me.

This can be achieved using a package called Coverlet. Basically you can define the output format of the tests and generate the .trx file for Bamboo. The command will look something like:
executable test --logger "trx;LogFileName=testResults.trx" /p:CollectCoverage=true /p:CoverletOutputFormat=preferredformat.
Check out the documentation for how to specifically integrate with VSTest.

SonarQube project not Code Coverage as blank when integrated with Atlassian Bamboo

I have an Atlassian Bamboo plan referring to my C# project. The tasks configured are as follows:
a) MSBuild task to build my C# solution file.
b) NUnit Runner and NUnit Parser tasks to run the Unit tests.
c) OpenCover task to execute the NUnit Unit tests and generate the metrics like Lines covered, Branch coverage etc.
d) SonarQube Runner task to generate the code coverage metrics on SonarQube project.
The metrics are successfully generated at step c) and results are written out to .xml file as output of OpenCover command. But after step d), SonarQube runner executes successfully and shows the results on Sonar project in terms of Number of unit tests, tests passed, tests skipped, tests failed. But the Code Coverage column shows as "-" blank.
Please note that my C# solution as multiple projects and I have the sonar-project.properties file at the root (Solution) level.
I use the following Sonar-project.properties file:
sonar.projectKey=<KeyName> (Key name of C# project in SonarQube)
sonar.projectName=<ProjectName> (Project name in SonarQube)
sonar.projectVersion=1.0
sonar.sources=.
sonar.language=cs
sonar.cs.nunit.reportsPaths=ABC.xml(Output of Nunit task)
sonar.cs.opencover.reportsPath=XYZ.xml(Output of OpenCover task)
Please guide me in this regard, any help will be greatly appreciated.
Thanks

Very late answer but it might help someone in the future:
You misspelled reportsPaths in your line
sonar.cs.opencover.reportsPath=XYZ.xml

Using VSTest to run unit test cases instead of MSTest

I have an x64 platform C# solution(VS2012) on a TFS2010 server. I have attached a unit test project (also x64) to this solution and created a build definition. When I queue the build, it succeeds but the unit test cases will not be executed. This is because MSTest is a 32 bit application. So, I decided to customize the default build process template (DefaultTemplate.xaml) to invoke VSTest(VSTest.console.exe) instead of MSTest. This is quite complex and I am unable to add a build activity to the toolbox for VSTest.
Has anyone done this kind of customization? I have also considered other approaches like configuring .runsettings file. Do we have a VSTest adapter interface that can be added in the .runsettings file ?

Executing unit tests through VSTest and publishing the test results through MSTest gave me a successful outcome. Given below is the Powershell script:
# Get the UnitTest Binaries
$files = Get-ChildItem $TestAssembliesDir\*est*.dll
# VSTest.console.exe path
$VSTestPath = 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe'
# MSTest path
$MSTestpath = "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe"
# Loop through the test assemblies and add them to the VSTestFiles
$VSTestFiles = ''
foreach($file in $files)
{
$VSTestFiles += "`"$file`""
$VSTestFiles += " "
}
# Run the UnitTests using VSTest
&$VSTestPath $vstestplatform "/Framework:Framework45" "/InIsolation" $VSTestFiles "/logger:trx"
# Get TRX files
$TrxFilesList = Get-ChildItem $TestResDir\*.trx
$TrxFiles = ''
$loop = 1
foreach($file in $TrxFilesList)
{
$TrxFiles = "$file"
# copy the trx file into a space-less named trx
$newTrxFileName = "$BuildUri" + "_" + "$loop" + ".trx"
copy-item $TrxFiles -destination $TestResDir\$newTrxFileName
$loop = $loop + 1
$newTrxFile += "$TestResDir\$newTrxFileName"
$newTrxFile += " "
}
# specify MSTest arguments
$mspubl = "/publish:"+$TeamProjColUri
$msteampr = "/teamproject:" + $TeamProj
$mspublbuild = "/publishbuild:" +$BuildUri
$mspubresfile = "/publishresultsfile:" +"`"$newTrxFile`""
#Publish test results through MSTest
&$MSTestpath $mstestplatform $flavor $mspubl $msteampr $mspublbuild $mspubresfile

I too have the exact same need for using VSTest.Console.exe instead of MSTest.exe for a TFS2010 build process that compiles a VS2012/.NET 4.5 x64 application, while waiting for the upgrade to TFS2012 to commence.
The approach I have taken was to edit the build script XAML, deleted the existing workflow for unit tests and replaced it with a customised workflow that builds up the VSTest.Console.exe parameters and then executes VSTest.Console.exe via InvokeProcess. I then ensured that in the Finally block that regardless of test result that we publish the test results and code coverage to TFS using MSTest.exe from a VS2012 installation on the build server.
Unfortunately I cannot post the XAML in the answer as it exceeds character length, but I do have a text file consisting of the snippet to be replaced in DefaultTemplate.xaml and what to replace it with. The file can be found here. Please note that although this approach works it is a hack.
Another alternative would be to use NUnit instead of MSTest or VSTest.Console as this support 64-bit binaries. This article explains how to integrate NUnit in a TFS2010 build script, and has links to tools and resources required to make this happen. The only issues with NUnit are code coverage (need yet another tool plus work out how to publish these results to TFS) and MSTest-style integration tests using attributes such as DeploymentItem and properties such as TestContext, which is why where I work we opted with the VSTest.Console.exe approach.
And from what I have read TFS2012 offers easy integration to VSTest.Console.exe from build scripts, so if you do ever upgrade to TFS2012 the VSTest.Console.exe hack that I have documented may not be required.

This does not directly answer you question, but it might help. I did a similar thing for TeamCity. I used command-line to call vstest.console.exe and created a .runsettings file.
I used this Microsoft template for the runsettings file. Note however that on my machine, the path mentioned in the comment in Line 5 is relative to the .runsettings location, not the .sln.
If you use /logger:trx option of vstest.console.exe, it will generate output in the same format as MSTest (good for result visualization).