We Keep Coding

Problem manually calling dotnet test on playwright tests generated by specflow

I'm working on a C# project that's using specflow with playwright to handle most of our testing, and I'm trying to set up a way to generate playwright trace files only for tests that have failed. Right now I'm printing the name of each test that fails to a .txt file, then using a shell script to manually call dotnet test on each test in that file at the end of our test run with playwright trace generation enabled. This works fine for tests that have a simple name, such as "FillInAllFieldsOnPage", but if a test that is generated from a scenario outline using examples fails it gives me an error. I believe this is due to the way that dotnet test is parsing the name of the test, as specflow generates the name in the format
CheckSortingAndDataInHoverMenu("C, User",null)
where "C, User" is one of the examples given to the Scenario Outline. I've tried adding a line that reformats the string to be more in line with what I've seen recommended elsewhere, so that the end result looks like this:
CheckSortingAndDataInHoverMenu(\"C%2C User\"%2Cnull)
but I still get the following error:
Running dotnet test on "CheckSortingAndDataInHoverMenu(\"C%2C User\"%2Cnull)"
MSBUILD : error MSB1009: Project file does not exist.
Switch: User"%2Cnull)
I've verified that the .csproj file does exist, and other tests are able to be run from that command so I know it's being called from the correct directory. It looks like dotnet test thinks that "User"%2Cnull)" is a name for a .csproj file that I'm passing it as an arg, and I don't know how to get it to properly process the full test name.

The solution I found to this was to simply remove the part of the test name in parenthesis, which caused the dotnet test command to run every test with a matching name. I then added a random guid to the name of each test trace that was generated in order to prevent them from overwriting eachother.

Generate both code coverage report and Unit test results using a single command (if possible)

I have a Windows service written in .net framework 4.6. I'm trying to run Sonar analysis for this service. My requirement is to generate both code coverage result and Unit test case report either by using MStest.exe or vstest.console.exe. I have written test cases using MStest for my service.
Using MSTest, I have written the below command:
MSTest /testcontainer:.\SolutionTests\bin\Release\SolutionTests.dll /resultsfile:"C:\SonarQube\Solution.trx"
Using vstest.console.exe, I have written the below command:
vstest.console.exe SolutionTests\bin\Release\SolutionTests.dll /Enablecodecoverage /Logger:trx;LogFileName="C:\SonarQube\Solution.trx"
In both the cases only Unit test report is generated (.trx file) as I have set the filename explicitly in the command.
Is there any way I can generate .coverage file as well, with in the same command by adding other parameters. I read in few articles which said MSTest command generates both the reports (result.trx and data.coverage), but no where it is written the exact command how to do it. I ran the above command, it did not generate data.coverage file for me.

This can be achieved using a package called Coverlet. Basically you can define the output format of the tests and generate the .trx file for Bamboo. The command will look something like:
executable test --logger "trx;LogFileName=testResults.trx" /p:CollectCoverage=true /p:CoverletOutputFormat=preferredformat.
Check out the documentation for how to specifically integrate with VSTest.

dotnet test - exit code 0 when test project doesn't compile

I'm using TeamCity to build my .NET Core project and dotnet test with xunit to run tests.
My build is configured to fail if any tests fail, and this works fine if a test fails, but when the test project doesn't compile the exit code ends up being zero.
I believe this is because I run dotnet test in the folder that contains the test-directories like this: for /f %%%a in ('dir /b /s project.json') do dotnet test %%%a.
Looking at the log I can see the individual jobs that do not compile return with exit code 1, but the build step itself returns with exit code 0.
How can I make the exit code from the failed compilation attempts propagate down to the build step?

You may create a separate step in TC to just compile target test project, and then run tests without compilation (using no-build option) in next step (if all previous steps were successful).
--no-build
Does not build the test project prior to running it.

How do I run Typescript tests for Jasmine\Karma through a TFS Build Process

I have a web application using Angular 1.5 with bower/npm/gulp coded in Typescript to do our build. Our back end is a c# .net WebApi2. Both are built and deployed on TFS2015. My c# nUnit tests are easy to integrate as part of the build process. The Typescript jasmine unit tests however are more difficult to integrate. How do I get my Typescript jasmine unit tests to run as part of the TFS build and if they fail, fail the build? We have them running through a Jasmine Spec runner and also Karma but not integrated.
I have read the many posts on StackOverflow integrating Javascript unit tests and each avenue took me through an overly complex solution that didn't work. These include Powershell scripts, Chutzpah amoungst others.

Rather than try to recreate the Specrunner via Chutzpah on the build server, which I found difficult to configure and get working. The aim was to get karma to output the running tests in the 'trx' test format that TFS recognises and then publish them to the build. Please note I am using PhantomJs to run my tests through Karma but won't cover that here as it is well covered elsewhere.
1) install the karma-trx-reporter plugin via npm into your web project (or similar plugin)
2) Configure the Karma.config to include the trx reporter
reporters: ['dots', 'trx'],
trxReporter: { outputFile: 'test-results.trx' },
// notify karma of the available plugins
plugins: [
'karma-jasmine',
'karma-phantomjs-launcher',
'karma-trx-reporter',
],
3) Create a Gulp (or grunt) task to run the karma tests if you don't already have one. Run the task locally and check it creates the 'test-results.trx' specified above. (It doesn't matter where the file is created on the build server):
gulp.task('test', function () {
return gulp.src(['tests/*.js']).pipe(karma({
configFile: __dirname + '/Testing/karma.config.js',
singleRun: true
}));
});
4) Add a Gulp (or Grunt) TFS build task to run the karma tests created in the previous step and output the trx file.
5) Add a Gulp (or Grunt) TFS build task to Publish the test results and merge them into the build. Note that the "Test Result Files" path is a wild card **/*.trx to find any trx files in the build path (i.e. finds our previously created file). "Merge Test results" is checked to Merge both our Jasmine test run and our c# test run into the same session. "Continue on error" is unticked to ensure any jasmine test failures break the build.
You will notice two sets of tests that have been run and included as part of the build!

Running automated Powershell-NUnit tests on TeamCity

I found this neat way to use NUnit in Powershell. http://elegantcode.com/2009/10/25/integration-test-brought-to-you-by-powershell-nunit-with-a-little-specification-syntax-for-flavoring/
and we are using it many of our tests.
However I want to run these tests in TeamCity.
I want similar behavior when we use a NUnit runner for running C# tests in TeamCity ie the build fails when the execution of tests fail. Has anyone of you achieved this? I suspect the Powershell runner will just execute it as a simple script, without any indication whether the test passes or fails.

Take a look at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity and http://confluence.jetbrains.net/display/TCD7/Build+Failure+Conditions
There is an issue in Powershell runner support http://youtrack.jetbrains.com/issue/TW-21554

I'm not familiar with the approach you're referencing for executing NUnit tests via Powershell in TeamCity. But, we are successfully using PSake for Powershell build scripts, including executing NUnit tests and failing the build appropriately. The same issue exists with PSake and TeamCity with exit codes, but you can get around it by specifying in TeamCity in the Script Source for the Build Step using -Command for Script execution:
import-module .\tools\psake\psake.psm1
$psake.use_exit_on_error = $true
invoke-psake build.ps1
remove-module psake
You can also integrate the Test results into the TeamCity using the Build Feature option in TeamCity Build Steps.