Is there a way to export the result of a unit test?
When I run NUnit, can I get formal output saying what methods have been tested and out of those, which one failed and which passed?
You can specify the test results to be outputted to a file:
Redirecting Text Output
Output created by the test, which is normally shown on the console, may be redirected to a file. The following command redirects standard output to the file TestResult.txt:
nunit-console nunit.tests.dll /out:TestResult.txt
The following command redirects standard error output to the StdErr.txt file.
nunit-console nunit.tests.dll /err:StdErr.txt
Note:This option only redirects output produced by the tests, together with selected NUnit output that is interspersed with the test output. It does not redirect all console output. If you want to redirect all output to a file, you should use command line redirection as supported by the shell you are using. This option exists for the purpose of separating test output from other output, such as the NUnit summary report.
From the NUnit documentation.
I just tested against one of my test assemblies by completing the following steps:
Downloaded and installed latest version of NUnit
Added 'C:\Program Files (x86)\NUnit 2.6.4\bin' to System Path
Navigated to where my assembly was in the command prompt
Ran nunit-console MyTests.dll /out:TestResult.txt
The results were then written to TestResult.txt in the same directory.
Related
I'm working on a C# project that's using specflow with playwright to handle most of our testing, and I'm trying to set up a way to generate playwright trace files only for tests that have failed. Right now I'm printing the name of each test that fails to a .txt file, then using a shell script to manually call dotnet test on each test in that file at the end of our test run with playwright trace generation enabled. This works fine for tests that have a simple name, such as "FillInAllFieldsOnPage", but if a test that is generated from a scenario outline using examples fails it gives me an error. I believe this is due to the way that dotnet test is parsing the name of the test, as specflow generates the name in the format
CheckSortingAndDataInHoverMenu("C, User",null)
where "C, User" is one of the examples given to the Scenario Outline. I've tried adding a line that reformats the string to be more in line with what I've seen recommended elsewhere, so that the end result looks like this:
CheckSortingAndDataInHoverMenu(\"C%2C User\"%2Cnull)
but I still get the following error:
Running dotnet test on "CheckSortingAndDataInHoverMenu(\"C%2C User\"%2Cnull)"
MSBUILD : error MSB1009: Project file does not exist.
Switch: User"%2Cnull)
I've verified that the .csproj file does exist, and other tests are able to be run from that command so I know it's being called from the correct directory. It looks like dotnet test thinks that "User"%2Cnull)" is a name for a .csproj file that I'm passing it as an arg, and I don't know how to get it to properly process the full test name.
The solution I found to this was to simply remove the part of the test name in parenthesis, which caused the dotnet test command to run every test with a matching name. I then added a random guid to the name of each test trace that was generated in order to prevent them from overwriting eachother.
console app with a batch file which will hit my Automation testing application and runs selected test cases. I have test case with code coverage and it runs from my visual studio . and now on top of it I have to create a console app which will keep some time interval and hit my VS test case and execute it.Any links will be helpful.
I expect the test case pass and fails status
Here's the batch file I use to run my Selenium tests with multiple runsettings files.
#ECHO OFF
IF NOT EXIST Results MKDIR Results
SETLOCAL
SET PATH=%PATH%;"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\Extensions\TestPlatform"
SET VSTEST=vstest.console.exe
SET TESTS=.\bin\Debug\something.something.Tests.dll
IF "%~1"=="" (
FOR %%J IN (*.runsettings) DO CALL :func %%J
) ELSE (
CALL :func "%~1"
)
GOTO :EOF
:func
ECHO.
ECHO %~1 **********
ECHO.
"%VSTEST%" %TESTS% /Settings:"%~1" /Logger:trx /ResultsDirectory:Results
GOTO :EOF
You could use NUNit's console runner to run tests from the command line, or on a build server. I've had great success with running my tests this way.
First, you need to install the NUnit.ConsoleRunner Nuget package onto your project.
Then, navigate to the NUnit.ConsoleRunner directory under your packages folder that exists in the project directory.
Open NUnit.ConsoleRunner > tools folder to get into the same directory as the .exe itself.
Then, you can run:
nunit3-console {Path to your project's .dll} --testlist={Path to .txt testlist}
With valid parameters, it looks something like this:
nunit3-console C:\Users\christine.harbour\Repository\AutomationTestSuite\AutomationTestSuite.dll --testlist=C:\Users\christine.harbour\Repository\AutomationTestSuite\MyTestList.txt
Your testlist should contain namespaces of the test cases you wish to run, separated by a line break. For example:
AutomationTestSuite.Tests.MyTestClass_1.MyTest
AutomationTestSuite.Tests.MyTestClass_2.MyOtherTest
After you run the tests, the results will be saved in the NUnit.ConsoleRunner > tools directory. The results are in XML format and can be programatically parsed to push your test results to another tool.
There are plenty of arguments you can pass into the ConsoleRunner, including build configuration and framework version, all which are specified on NUnit's documentation.
NUnit console runner also integrates with Cake, which is a build scripting tool for C# projects. So, you could hypothetically clean / build your project, restore missing package references, and run your tests, all from the console.
More info on NUnit console runner can be found here: https://github.com/nunit/docs/wiki/Console-Command-Line
I have a Windows service written in .net framework 4.6. I'm trying to run Sonar analysis for this service. My requirement is to generate both code coverage result and Unit test case report either by using MStest.exe or vstest.console.exe. I have written test cases using MStest for my service.
Using MSTest, I have written the below command:
MSTest /testcontainer:.\SolutionTests\bin\Release\SolutionTests.dll /resultsfile:"C:\SonarQube\Solution.trx"
Using vstest.console.exe, I have written the below command:
vstest.console.exe SolutionTests\bin\Release\SolutionTests.dll /Enablecodecoverage /Logger:trx;LogFileName="C:\SonarQube\Solution.trx"
In both the cases only Unit test report is generated (.trx file) as I have set the filename explicitly in the command.
Is there any way I can generate .coverage file as well, with in the same command by adding other parameters. I read in few articles which said MSTest command generates both the reports (result.trx and data.coverage), but no where it is written the exact command how to do it. I ran the above command, it did not generate data.coverage file for me.
This can be achieved using a package called Coverlet. Basically you can define the output format of the tests and generate the .trx file for Bamboo. The command will look something like:
executable test --logger "trx;LogFileName=testResults.trx" /p:CollectCoverage=true /p:CoverletOutputFormat=preferredformat.
Check out the documentation for how to specifically integrate with VSTest.
I have a solution with mstest unit testcases and using Opencover for testcoverage.
TestCoverage results are coming fine in Sonarqube but it is not displaying the result from generated TestResult.trx file in Sonar. It only displays the unit test cases count but not testcases names, which one is pass and which one is fail.
I have followed the same steps mentioned in the link below:
https://docs.sonarqube.org/pages/viewpage.action?pageId=6389772#UnitTestExecutionResultsImport(C#,VB.NET)-MSTest
TRX file is also generated correctly but no data in sonar.
Please let me know if this possible or any alternative.
Below are the command I am using:
>MSBuild.SonarQube.Runner.exe begin /k:rent /n:rent /v:1.0.0 /d:sonar.host.url=http://127.0.0.1:9000/ /d:sonar.exclusions=**/Files/**/* /d:sonar.cs.opencover.reportsPaths=E:\MSTest\rent\opencover.xml /d:sonar.cs.vstest.reportsPaths=E:\MSTest\rent\TestResults.trx
>MSBuild.exe /p:Configuration=debug /t:Publish /p:TargetProfile=Local E:\rent\rent.sln
>OpenCover.Console.exe -output:"E:\MSTest\rent\opencover.xml" -register:user -target:"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" -targetargs:"/testcontainer:E:\rent\rent.Tests\bin\debug\rent.Tests.dll /resultsfile:E:\MSTest\rent\TestResults.trx" -excludebyattribute:*.ExcludeFromCodeCoverage*
>MSBuild.SonarQube.Runner.exe end
I'm currently changing things to our Sonar setup since Gallio isn't supported anymore by C# Ecosystem 3. I already successfully imported the unit test coverage by using OpenCover with following command.
"C:\Program Files (x86)\OpenCover\OpenCover.Console.exe" -register -target:"c:\Program Files (x86)\NUnit 2.6.3\bin\nunit-console-x86.exe" -targetargs:"d:\Sonar\MyTest.dll /noshadow /framework=net-4.0" -output:"d:\Deploy\Sonar\Coverage.xml" "-filter:+[*]* -[*]Test"
With this command I'm only getting the Unit Test Coverage but I would also like to see the number of failed and successful tests. I think I can achieve this with the /result option like
"C:\Program Files (x86)\OpenCover\OpenCover.Console.exe" -register -target:"c:\Program Files (x86)\NUnit 2.6.3\bin\nunit-console-x86.exe" -targetargs:"d:\Deploy\Sonar\MyTest.dll /noshadow /framework=net-4.0 /result=tests.xml"
This command returns an xml with information about the tests that ran but is there any way to import this xml into SonarQube? Or isn't this supported?
Currently I'm importing the Coverage.xml file with the following command:
sonar.cs.opencover.reportsPaths=D:/Deploy/Sonar/Coverage.xml
Is there a similar property to import the tests.xml file with the test results?
OpenCover is now officially supported by SonarQube, please see the SonarQube documentation. You can pass the location of the XML report OpenCover generated with the command line parameter
/d:sonar.cs.opencover.reportsPaths="%path%\%report_file%"
where %path% is the folder your report file is generated in and %report_file% is the file name of your OpenCover XML report.
From the documentation at http://docs.codehaus.org/display/SONAR/C%23+Plugin, it looks like you can import unit test execution reports of MSTest format only.
Thanks to the following site http://www.codewrecks.com/blog/index.php/2009/07/19/integrate-nunit-test-into-a-tfs-build/ I was able to convert the output of OpenCover test results to a .trx format using an XSLT transformation. After the transformation I just used the default import functionality
sonar.cs.vstest.reportsPaths=MyTestRestultFile.trx