I am working on a C# project using Visual Studio 2022. The project has a Unit Test assembly, and this works fine. We can run tests in the Test Explorer and also the tests are run automatically in DevOps.
I would like to introduce a test into this assembly that needs some setup with an external utility app. The external app will receive a message, modify it, and send it back to the source, but it needs to be run interactively. It is a useful test to have in terms of product quality, but the tester needs to set up the environment so it can't be part of any automated test.
Is there any way to either:
Specify that the test can only be run manually, and should not be part of an automated test run, or
Detect whether the test is being run manually or automatically, so that it can just indicate success if run automatically?
When using Xunit you can use the Trait attribute on your test to define a category and use this as filter when executing test in your pipeline.
So on a regular unit test you can add this:
[Trait("Category", "Unit")]
And on a more complex test you could add something like this:
[Trait("Category", "Integration")]
And use this as filter in your pipeline. For example to only run unit tests:
- task: DotNetCoreCLI#2
displayName: "dotnet test ProjectWithTests.csproj"
inputs:
command: "test"
projects: "ProjectWithTests.csproj"
arguments: '--no-build --configuration "Release" --filter Category=Unit'
Related
I am using in the Test Filter criteria TestCategory=CategoryA
In my Tests I have the below saved and think it should be the only test run but I get success and no tests ran. I have tried with dotnet test and used the same in the arguments and gain it fails with other issues. I can get dot net test to run without adding any arguments for VS Test agent it seems to not want to run even when I do not add the test filter criteria.
[Test]
[Category("CategoryA")]
public async Task ActivateDevice()
2022-02-18T20:14:54.5643568Z ##[section]Starting: VsTest - testAssemblies
2022-02-18T20:14:54.5771277Z ==============================================================================
2022-02-18T20:14:54.5771649Z Task : Visual Studio Test
2022-02-18T20:14:54.5772497Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2022-02-18T20:14:54.5773342Z Version : 2.198.0
2022-02-18T20:14:54.5773587Z Author : Microsoft Corporation
2022-02-18T20:14:54.5773921Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2022-02-18T20:14:54.5774333Z ==============================================================================
2022-02-18T20:14:55.5713320Z SystemVssConnection exists true
2022-02-18T20:14:55.5732257Z SystemVssConnection exists true
2022-02-18T20:14:55.5991193Z Running tests using vstest.console.exe runner.
2022-02-18T20:14:55.5991725Z ======================================================
2022-02-18T20:14:55.5997078Z Test selector : Test assemblies
2022-02-18T20:14:55.5998382Z Test filter criteria : null
2022-02-18T20:14:55.5999337Z Search folder : D:\a\r1\a
2022-02-18T20:14:55.6102675Z Action when minimum tests threshold not met : donothing
2022-02-18T20:14:55.6103140Z Minimum tests expected to be run: 0
2022-02-18T20:14:55.6104476Z VisualStudio version selected for test execution : latest
2022-02-18T20:14:55.6107744Z Attempting to find vstest.console from a visual studio installation with version [17.0,18.0).
2022-02-18T20:14:56.0357719Z Attempting to find vstest.console from a visual studio build tools installation with version [17.0,18.0).
2022-02-18T20:14:56.0642271Z Attempting to find vstest.console from a visual studio installation with version [16.0,17.0).
2022-02-18T20:14:56.0884398Z Attempting to find vstest.console from a visual studio build tools installation with version [16.0,17.0).
2022-02-18T20:14:56.1142098Z Attempting to find vstest.console from a visual studio installation with version [15.0,16.0).
2022-02-18T20:14:56.3498931Z Run in parallel : false
2022-02-18T20:14:56.3500994Z Run in isolation : false
2022-02-18T20:14:56.3502074Z Path to custom adapters : null
2022-02-18T20:14:56.3504871Z Other console options : null
2022-02-18T20:14:56.3505911Z Code coverage enabled : false
2022-02-18T20:14:56.3510216Z Diagnostics enabled : false
2022-02-18T20:14:56.3519771Z SystemVssConnection exists true
2022-02-18T20:14:56.3697972Z Run the tests locally using vstest.console.exe
2022-02-18T20:14:56.3698460Z ========================================================
2022-02-18T20:14:56.5771570Z Source filter: D:\a\r1\a/_Test.AppiumTest/AppiumTesting/AppiumTest
2022-02-18T20:14:56.5973322Z ##[warning]No test sources found matching the given filter
1. Check whether the following latest packages are installed or not:
Microsoft.NET.Test.Sdk
MSTest.TestAdapter
MSTest.TestFramework
NUnit3TestAdapter
2. As answered by Marina Liu , you can configure MSTest or the "Visual Studio Test" task in VSO to fail if there are warnings:
a. Add a PowerShell task after Visual Studio Test task.
b. Then get Visual Studio Test task build information by Timeline:
c. Search for Visual Studio Test task information by task name or task display name.
d. Fail the build result if VS test has the warning you specified.
References: How to Fix this C# issue No test matches the given testcase filter `FullyQualifiedName = , How can I configure MSTest to fail the unit test run on any warnings? and Run NUnit tests in Azure DevOps pipeline
I've been running Selenium tests in Visual Studio using C# and everything works fine locally. I have an automated build process in TFS that runs unit tests in my solution. That task (Test Assemblies) is finding the new Selenium tests but failing to run them. However, when I created a separate build definition that deployed a test agent (successfully) and then attempts to run functional tests, I get this message:
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: Test Run Discovery Aborted . Test run id : 1600
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: Error : No tests were discovered from the specified test sources
I have searched the DTALog, and found that the test sources are being found successfully, just no actual tests within them. Any ideas what I am doing wrong? I have removed the 'Owner' decoration from the tests.
Please try below things to narrow down the issue:
Make sure the appropriate test adapter getting deployed along with
the test assemblies as Daniel mentioned.
If not deployed, you would need to copy the appropriate adapter from your local vs machine (\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions) to your test agent box (\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions)
Try run directly from vstest.console.exe on test agent machine with
specifing test adapter path using /testadapterpath flag and see if
vstest.console works.
eg:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "F:\Test****TestAutomation.dll" /TestAdapterPath:F:\Test*\
Check if you have included the dependencies with the test dlls.
eg: manually add reference of Microsoft.VisualStudio.QualityTools.UnitTestFramework in the test project.
If the isse persist, please share the logs for furthre troubleshoot.
I have a web application using Angular 1.5 with bower/npm/gulp coded in Typescript to do our build. Our back end is a c# .net WebApi2. Both are built and deployed on TFS2015. My c# nUnit tests are easy to integrate as part of the build process. The Typescript jasmine unit tests however are more difficult to integrate. How do I get my Typescript jasmine unit tests to run as part of the TFS build and if they fail, fail the build? We have them running through a Jasmine Spec runner and also Karma but not integrated.
I have read the many posts on StackOverflow integrating Javascript unit tests and each avenue took me through an overly complex solution that didn't work. These include Powershell scripts, Chutzpah amoungst others.
Rather than try to recreate the Specrunner via Chutzpah on the build server, which I found difficult to configure and get working. The aim was to get karma to output the running tests in the 'trx' test format that TFS recognises and then publish them to the build. Please note I am using PhantomJs to run my tests through Karma but won't cover that here as it is well covered elsewhere.
1) install the karma-trx-reporter plugin via npm into your web project (or similar plugin)
2) Configure the Karma.config to include the trx reporter
reporters: ['dots', 'trx'],
trxReporter: { outputFile: 'test-results.trx' },
// notify karma of the available plugins
plugins: [
'karma-jasmine',
'karma-phantomjs-launcher',
'karma-trx-reporter',
],
3) Create a Gulp (or grunt) task to run the karma tests if you don't already have one. Run the task locally and check it creates the 'test-results.trx' specified above. (It doesn't matter where the file is created on the build server):
gulp.task('test', function () {
return gulp.src(['tests/*.js']).pipe(karma({
configFile: __dirname + '/Testing/karma.config.js',
singleRun: true
}));
});
4) Add a Gulp (or Grunt) TFS build task to run the karma tests created in the previous step and output the trx file.
5) Add a Gulp (or Grunt) TFS build task to Publish the test results and merge them into the build. Note that the "Test Result Files" path is a wild card **/*.trx to find any trx files in the build path (i.e. finds our previously created file). "Merge Test results" is checked to Merge both our Jasmine test run and our c# test run into the same session. "Continue on error" is unticked to ensure any jasmine test failures break the build.
You will notice two sets of tests that have been run and included as part of the build!
I found this neat way to use NUnit in Powershell. http://elegantcode.com/2009/10/25/integration-test-brought-to-you-by-powershell-nunit-with-a-little-specification-syntax-for-flavoring/
and we are using it many of our tests.
However I want to run these tests in TeamCity.
I want similar behavior when we use a NUnit runner for running C# tests in TeamCity ie the build fails when the execution of tests fail. Has anyone of you achieved this? I suspect the Powershell runner will just execute it as a simple script, without any indication whether the test passes or fails.
Take a look at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity and http://confluence.jetbrains.net/display/TCD7/Build+Failure+Conditions
There is an issue in Powershell runner support http://youtrack.jetbrains.com/issue/TW-21554
I'm not familiar with the approach you're referencing for executing NUnit tests via Powershell in TeamCity. But, we are successfully using PSake for Powershell build scripts, including executing NUnit tests and failing the build appropriately. The same issue exists with PSake and TeamCity with exit codes, but you can get around it by specifying in TeamCity in the Script Source for the Build Step using -Command for Script execution:
import-module .\tools\psake\psake.psm1
$psake.use_exit_on_error = $true
invoke-psake build.ps1
remove-module psake
You can also integrate the Test results into the TeamCity using the Build Feature option in TeamCity Build Steps.
I am using Team City as the build server, and I have a msbuild build script to build and run integration tests for my project. However, Team City when running my tests decides to ignore the WorkingDirectory attribute on the NUnit task, and as a result my tests fail to initialize:
[17:46:54]: [Project "MyProject.msbuild.xml.teamcity.patch.tcprojx" (ficc-build target(s)):] C:\dev\BuildAgent\work\30decc96a6997d21\MyProject\MyProject.msbuild.xml Value from NUnit task attribute OutputXmlFile has been ignored because TeamCity NUnit task is used
[17:46:54]: [Project "MyProject.msbuild.xml.teamcity.patch.tcprojx" (ficc-build target(s)):] C:\dev\BuildAgent\work\30decc96a6997d21\MyProject\MyProject.msbuild.xml Value from NUnit task attribute WorkingDirectory has been ignored because TeamCity NUnit task is used
[17:46:54]: [Project "MyProject.msbuild.xml.teamcity.patch.tcprojx" (ficc-build target(s)):] Value from NUnit task attribute ToolPath has been ignored because TeamCity NUnit task is used
How can I force Team City to run these tests from my working directory? I don't want to execute nunit-console directly, because I want to benefit from the built-in reporting on test failures, etc.
I can't see a way to change TeamCity's working directory using the MSBuild task, but you could take the middle road and use TeamCity's own console runner, which you could start from within the proper working directory. Another option is to run the standard NUnit console, with TeamCity's NUnit Addin loaded.