Running automated Powershell-NUnit tests on TeamCity - c#

I found this neat way to use NUnit in Powershell. http://elegantcode.com/2009/10/25/integration-test-brought-to-you-by-powershell-nunit-with-a-little-specification-syntax-for-flavoring/
and we are using it many of our tests.
However I want to run these tests in TeamCity.
I want similar behavior when we use a NUnit runner for running C# tests in TeamCity ie the build fails when the execution of tests fail. Has anyone of you achieved this? I suspect the Powershell runner will just execute it as a simple script, without any indication whether the test passes or fails.

Take a look at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity and http://confluence.jetbrains.net/display/TCD7/Build+Failure+Conditions
There is an issue in Powershell runner support http://youtrack.jetbrains.com/issue/TW-21554

I'm not familiar with the approach you're referencing for executing NUnit tests via Powershell in TeamCity. But, we are successfully using PSake for Powershell build scripts, including executing NUnit tests and failing the build appropriately. The same issue exists with PSake and TeamCity with exit codes, but you can get around it by specifying in TeamCity in the Script Source for the Build Step using -Command for Script execution:
import-module .\tools\psake\psake.psm1
$psake.use_exit_on_error = $true
invoke-psake build.ps1
remove-module psake
You can also integrate the Test results into the TeamCity using the Build Feature option in TeamCity Build Steps.

Related

Generate both code coverage report and Unit test results using a single command (if possible)

I have a Windows service written in .net framework 4.6. I'm trying to run Sonar analysis for this service. My requirement is to generate both code coverage result and Unit test case report either by using MStest.exe or vstest.console.exe. I have written test cases using MStest for my service.
Using MSTest, I have written the below command:
MSTest /testcontainer:.\SolutionTests\bin\Release\SolutionTests.dll /resultsfile:"C:\SonarQube\Solution.trx"
Using vstest.console.exe, I have written the below command:
vstest.console.exe SolutionTests\bin\Release\SolutionTests.dll /Enablecodecoverage /Logger:trx;LogFileName="C:\SonarQube\Solution.trx"
In both the cases only Unit test report is generated (.trx file) as I have set the filename explicitly in the command.
Is there any way I can generate .coverage file as well, with in the same command by adding other parameters. I read in few articles which said MSTest command generates both the reports (result.trx and data.coverage), but no where it is written the exact command how to do it. I ran the above command, it did not generate data.coverage file for me.
This can be achieved using a package called Coverlet. Basically you can define the output format of the tests and generate the .trx file for Bamboo. The command will look something like:
executable test --logger "trx;LogFileName=testResults.trx" /p:CollectCoverage=true /p:CoverletOutputFormat=preferredformat.
Check out the documentation for how to specifically integrate with VSTest.

TFS Functional Testing is not finding my tests

I've been running Selenium tests in Visual Studio using C# and everything works fine locally. I have an automated build process in TFS that runs unit tests in my solution. That task (Test Assemblies) is finding the new Selenium tests but failing to run them. However, when I created a separate build definition that deployed a test agent (successfully) and then attempts to run functional tests, I get this message:
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: Test Run Discovery Aborted . Test run id : 1600
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
2017-11-03T18:49:43.1345753Z ##[warning]DistributedTests: Error : No tests were discovered from the specified test sources
I have searched the DTALog, and found that the test sources are being found successfully, just no actual tests within them. Any ideas what I am doing wrong? I have removed the 'Owner' decoration from the tests.
Please try below things to narrow down the issue:
Make sure the appropriate test adapter getting deployed along with
the test assemblies as Daniel mentioned.
If not deployed, you would need to copy the appropriate adapter from your local vs machine (\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions) to your test agent box (\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions)
Try run directly from vstest.console.exe on test agent machine with
specifing test adapter path using /testadapterpath flag and see if
vstest.console works.
eg:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "F:\Test****TestAutomation.dll" /TestAdapterPath:F:\Test*\
Check if you have included the dependencies with the test dlls.
eg: manually add reference of Microsoft.VisualStudio.QualityTools.UnitTestFramework in the test project.
If the isse persist, please share the logs for furthre troubleshoot.

How do I run Typescript tests for Jasmine\Karma through a TFS Build Process

I have a web application using Angular 1.5 with bower/npm/gulp coded in Typescript to do our build. Our back end is a c# .net WebApi2. Both are built and deployed on TFS2015. My c# nUnit tests are easy to integrate as part of the build process. The Typescript jasmine unit tests however are more difficult to integrate. How do I get my Typescript jasmine unit tests to run as part of the TFS build and if they fail, fail the build? We have them running through a Jasmine Spec runner and also Karma but not integrated.
I have read the many posts on StackOverflow integrating Javascript unit tests and each avenue took me through an overly complex solution that didn't work. These include Powershell scripts, Chutzpah amoungst others.
Rather than try to recreate the Specrunner via Chutzpah on the build server, which I found difficult to configure and get working. The aim was to get karma to output the running tests in the 'trx' test format that TFS recognises and then publish them to the build. Please note I am using PhantomJs to run my tests through Karma but won't cover that here as it is well covered elsewhere.
1) install the karma-trx-reporter plugin via npm into your web project (or similar plugin)
2) Configure the Karma.config to include the trx reporter
reporters: ['dots', 'trx'],
trxReporter: { outputFile: 'test-results.trx' },
// notify karma of the available plugins
plugins: [
'karma-jasmine',
'karma-phantomjs-launcher',
'karma-trx-reporter',
],
3) Create a Gulp (or grunt) task to run the karma tests if you don't already have one. Run the task locally and check it creates the 'test-results.trx' specified above. (It doesn't matter where the file is created on the build server):
gulp.task('test', function () {
return gulp.src(['tests/*.js']).pipe(karma({
configFile: __dirname + '/Testing/karma.config.js',
singleRun: true
}));
});
4) Add a Gulp (or Grunt) TFS build task to run the karma tests created in the previous step and output the trx file.
5) Add a Gulp (or Grunt) TFS build task to Publish the test results and merge them into the build. Note that the "Test Result Files" path is a wild card **/*.trx to find any trx files in the build path (i.e. finds our previously created file). "Merge Test results" is checked to Merge both our Jasmine test run and our c# test run into the same session. "Continue on error" is unticked to ensure any jasmine test failures break the build.
You will notice two sets of tests that have been run and included as part of the build!

Using VSTest to run unit test cases instead of MSTest

I have an x64 platform C# solution(VS2012) on a TFS2010 server. I have attached a unit test project (also x64) to this solution and created a build definition. When I queue the build, it succeeds but the unit test cases will not be executed. This is because MSTest is a 32 bit application. So, I decided to customize the default build process template (DefaultTemplate.xaml) to invoke VSTest(VSTest.console.exe) instead of MSTest. This is quite complex and I am unable to add a build activity to the toolbox for VSTest.
Has anyone done this kind of customization? I have also considered other approaches like configuring .runsettings file. Do we have a VSTest adapter interface that can be added in the .runsettings file ?
Executing unit tests through VSTest and publishing the test results through MSTest gave me a successful outcome. Given below is the Powershell script:
# Get the UnitTest Binaries
$files = Get-ChildItem $TestAssembliesDir\*est*.dll
# VSTest.console.exe path
$VSTestPath = 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe'
# MSTest path
$MSTestpath = "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe"
# Loop through the test assemblies and add them to the VSTestFiles
$VSTestFiles = ''
foreach($file in $files)
{
$VSTestFiles += "`"$file`""
$VSTestFiles += " "
}
# Run the UnitTests using VSTest
&$VSTestPath $vstestplatform "/Framework:Framework45" "/InIsolation" $VSTestFiles "/logger:trx"
# Get TRX files
$TrxFilesList = Get-ChildItem $TestResDir\*.trx
$TrxFiles = ''
$loop = 1
foreach($file in $TrxFilesList)
{
$TrxFiles = "$file"
# copy the trx file into a space-less named trx
$newTrxFileName = "$BuildUri" + "_" + "$loop" + ".trx"
copy-item $TrxFiles -destination $TestResDir\$newTrxFileName
$loop = $loop + 1
$newTrxFile += "$TestResDir\$newTrxFileName"
$newTrxFile += " "
}
# specify MSTest arguments
$mspubl = "/publish:"+$TeamProjColUri
$msteampr = "/teamproject:" + $TeamProj
$mspublbuild = "/publishbuild:" +$BuildUri
$mspubresfile = "/publishresultsfile:" +"`"$newTrxFile`""
#Publish test results through MSTest
&$MSTestpath $mstestplatform $flavor $mspubl $msteampr $mspublbuild $mspubresfile
I too have the exact same need for using VSTest.Console.exe instead of MSTest.exe for a TFS2010 build process that compiles a VS2012/.NET 4.5 x64 application, while waiting for the upgrade to TFS2012 to commence.
The approach I have taken was to edit the build script XAML, deleted the existing workflow for unit tests and replaced it with a customised workflow that builds up the VSTest.Console.exe parameters and then executes VSTest.Console.exe via InvokeProcess. I then ensured that in the Finally block that regardless of test result that we publish the test results and code coverage to TFS using MSTest.exe from a VS2012 installation on the build server.
Unfortunately I cannot post the XAML in the answer as it exceeds character length, but I do have a text file consisting of the snippet to be replaced in DefaultTemplate.xaml and what to replace it with. The file can be found here. Please note that although this approach works it is a hack.
Another alternative would be to use NUnit instead of MSTest or VSTest.Console as this support 64-bit binaries. This article explains how to integrate NUnit in a TFS2010 build script, and has links to tools and resources required to make this happen. The only issues with NUnit are code coverage (need yet another tool plus work out how to publish these results to TFS) and MSTest-style integration tests using attributes such as DeploymentItem and properties such as TestContext, which is why where I work we opted with the VSTest.Console.exe approach.
And from what I have read TFS2012 offers easy integration to VSTest.Console.exe from build scripts, so if you do ever upgrade to TFS2012 the VSTest.Console.exe hack that I have documented may not be required.
This does not directly answer you question, but it might help. I did a similar thing for TeamCity. I used command-line to call vstest.console.exe and created a .runsettings file.
I used this Microsoft template for the runsettings file. Note however that on my machine, the path mentioned in the comment in Line 5 is relative to the .runsettings location, not the .sln.
If you use /logger:trx option of vstest.console.exe, it will generate output in the same format as MSTest (good for result visualization).

How to execute NUnit test cases from command prompt

How can I execute a test case from Command Console using NUnit? I had set of Selenium Tests written in C# based on NUnit framework. I need to execute the test cases simply by running from command console.
In JUnit we can run test case from cmd as
java junit.swingui.TestRunner test.Run
How can we do above in NUnit?
Use nunit-console.exe to run tests from the command line.
For example:
nunit-console.exe /xml:results.xml path/to/test/assembly.dll
This will run the unit tests and save the results in the results.xml file, which you can work with easily.
See the documentation for all of the various command line switches that are available.
I would like to add a few words about the latest version of NUnit. The name of the console application has changed to nunit3-console.exe in NUnit 3. Information about all possible options can be found in the official documentation. For example, run all tests in the assembly (the results are saved into the TestResult.xml file by default).
nunit3-console.exe path/to/test/assembly.dll
I've just find another nice solution:
Adding the following command to the "Build Events" / "Post-Build Events", will run the tests in Nunit-Gui automatically after the project has been built.
I hope this can be useful:
"C:\Program Files (x86)\NUnit 2.6.3\bin\nunit-x86.exe" $(TargetPath) /run
Visual Studio: 2017, 2019(Preview)
On Mac use below command:
nunit-console <path/to/project>/<project-name>/bin/Debug/<project-solution-name>.dll
For example:
nunit-console /Users/pratik/Projects/selenium-mac13/selenium-test/bin/Debug/selenium-test.dll
nunit3-console.exe "path of the testfile (dll)"
Working on a Windows 10 Desktop with Visual Studio
I had a set of tests in C# where I'd set the test method with Category==API.
To run the tests (Nunit3-console) remotely via Bamboo, I added this Bamboo Powershell script:
Invoke-Command -Credential $credentials -ComputerName $Server -ScriptBlock{
$pathToDdrive = "D:"
$pathtoDLL = Join-Path $pathToDdrive -ChildPath "RestOfThePathToDLL"
cd D:\...\NUnit.ConsoleRunner.3.10.0\tools
.\nunit3-console.exe $pathToDLL --where "cat=='API'"
}

Categories