I'm using TeamCity to build my .NET Core project and dotnet test with xunit to run tests.
My build is configured to fail if any tests fail, and this works fine if a test fails, but when the test project doesn't compile the exit code ends up being zero.
I believe this is because I run dotnet test in the folder that contains the test-directories like this: for /f %%%a in ('dir /b /s project.json') do dotnet test %%%a.
Looking at the log I can see the individual jobs that do not compile return with exit code 1, but the build step itself returns with exit code 0.
How can I make the exit code from the failed compilation attempts propagate down to the build step?
You may create a separate step in TC to just compile target test project, and then run tests without compilation (using no-build option) in next step (if all previous steps were successful).
--no-build
Does not build the test project prior to running it.
Related
I am working on a C# project using Visual Studio 2022. The project has a Unit Test assembly, and this works fine. We can run tests in the Test Explorer and also the tests are run automatically in DevOps.
I would like to introduce a test into this assembly that needs some setup with an external utility app. The external app will receive a message, modify it, and send it back to the source, but it needs to be run interactively. It is a useful test to have in terms of product quality, but the tester needs to set up the environment so it can't be part of any automated test.
Is there any way to either:
Specify that the test can only be run manually, and should not be part of an automated test run, or
Detect whether the test is being run manually or automatically, so that it can just indicate success if run automatically?
When using Xunit you can use the Trait attribute on your test to define a category and use this as filter when executing test in your pipeline.
So on a regular unit test you can add this:
[Trait("Category", "Unit")]
And on a more complex test you could add something like this:
[Trait("Category", "Integration")]
And use this as filter in your pipeline. For example to only run unit tests:
- task: DotNetCoreCLI#2
displayName: "dotnet test ProjectWithTests.csproj"
inputs:
command: "test"
projects: "ProjectWithTests.csproj"
arguments: '--no-build --configuration "Release" --filter Category=Unit'
Issue
When I run xUnit tests on a project containing a lot of tests, it does not produce the xml result file. If I reduce the number of tests in the same project to a smaller set of tests it does produce the result file. Why is this happening?
Details
My project has about 3200 xUnit tests running in a Jenkins build. I am using the following command to run the tests.
dotnet test --logger:"xunit;LogFilePath=/results/integrationtest_results.xml"
However, this is not creating the integrationtest_results.xml file. I have confirmed the tests did run even though it did not produce the result file.
When I use the following command to filter down the tests to less than 1000 tests it DOES create the integrationtest_results.xml result file.
dotnet test --logger:"xunit;LogFilePath=/results/integrationtest_results.xml" --filter:"someAttribute=someValue"
In the same Jenkins build, I have another project that runs 500 unit tests with the following command.
dotnet test --logger:"xunit;LogFilePath=/results/unittest_results.xml"
This has always produced the expected unittest_results.xml.
In the cases where it does not produce the result file, I do not see any logs stating an error occurred relating to the xunit results file.
Environment
C# dotnet core 6.0
Xunit v2.4.1
Test logger: XunitXml.TestLogger v3.0.70
console app with a batch file which will hit my Automation testing application and runs selected test cases. I have test case with code coverage and it runs from my visual studio . and now on top of it I have to create a console app which will keep some time interval and hit my VS test case and execute it.Any links will be helpful.
I expect the test case pass and fails status
Here's the batch file I use to run my Selenium tests with multiple runsettings files.
#ECHO OFF
IF NOT EXIST Results MKDIR Results
SETLOCAL
SET PATH=%PATH%;"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\Extensions\TestPlatform"
SET VSTEST=vstest.console.exe
SET TESTS=.\bin\Debug\something.something.Tests.dll
IF "%~1"=="" (
FOR %%J IN (*.runsettings) DO CALL :func %%J
) ELSE (
CALL :func "%~1"
)
GOTO :EOF
:func
ECHO.
ECHO %~1 **********
ECHO.
"%VSTEST%" %TESTS% /Settings:"%~1" /Logger:trx /ResultsDirectory:Results
GOTO :EOF
You could use NUNit's console runner to run tests from the command line, or on a build server. I've had great success with running my tests this way.
First, you need to install the NUnit.ConsoleRunner Nuget package onto your project.
Then, navigate to the NUnit.ConsoleRunner directory under your packages folder that exists in the project directory.
Open NUnit.ConsoleRunner > tools folder to get into the same directory as the .exe itself.
Then, you can run:
nunit3-console {Path to your project's .dll} --testlist={Path to .txt testlist}
With valid parameters, it looks something like this:
nunit3-console C:\Users\christine.harbour\Repository\AutomationTestSuite\AutomationTestSuite.dll --testlist=C:\Users\christine.harbour\Repository\AutomationTestSuite\MyTestList.txt
Your testlist should contain namespaces of the test cases you wish to run, separated by a line break. For example:
AutomationTestSuite.Tests.MyTestClass_1.MyTest
AutomationTestSuite.Tests.MyTestClass_2.MyOtherTest
After you run the tests, the results will be saved in the NUnit.ConsoleRunner > tools directory. The results are in XML format and can be programatically parsed to push your test results to another tool.
There are plenty of arguments you can pass into the ConsoleRunner, including build configuration and framework version, all which are specified on NUnit's documentation.
NUnit console runner also integrates with Cake, which is a build scripting tool for C# projects. So, you could hypothetically clean / build your project, restore missing package references, and run your tests, all from the console.
More info on NUnit console runner can be found here: https://github.com/nunit/docs/wiki/Console-Command-Line
I am running a CI build using Travis CI. I am running NUnit tests via the nunit3-console.exe. I have several tests that attempt to assert that a collection is ordered:
[Test]
public void FeatsAreSorted()
{
var result = controller.Generate() as JsonResult;
dynamic data = result.Data;
Assert.That(data.character.Ability.Feats, Is.Ordered.By("Name"));
}
When I run this test in Visual Studio, the test passes fine. However, when I run the test via nunit3-console.exe in Travis CI, I get the following error:
1) Error : DNDGenSite.Tests.Unit.Controllers.CharacterControllerTests.GenerateSortsCharacterFeats
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : `NUnit.Framework.Assert.That<System.Linq.OrderedEnumerable<CharacterGen.Common.Abilities.Feats.Feat,string>>(System.Linq.OrderedEnumerable<CharacterGen.Common.Abilities.Feats.Feat,string>, NUnit.Framework.Constraints.IResolveConstraint)' is inaccessible due to its protection level
This is my .travis.yml:
language: csharp
solution: DNDGenSite.sln
install:
- nuget restore DNDGenSite.sln
- nuget install NUnit.Runners -OutputDirectory testrunner
- nuget install Chutzpah -OutputDirectory testrunner
script:
- xbuild DNDGenSite.sln /p:TargetFrameworkVersion="v4.5.1" /p:Configuration=Release
- mono ./testrunner/NUnit.Console.*/tools/nunit3-console.exe ./Tests/bin/Release/DNDGenSite.Tests.dll
- mono ./testrunner/Chutzpah.*/tools/chutzpah.console.exe ./Tests/Unit/Scripts
Any thoughts?
UPDATE: If I run the tests in the git bash, everything passes correctly, in both Debug and Release build modes. So, there is something different about the environment in which Travis CI builds the console runner.
In the end, I found that asserting the order off of the dynamic object was causing the error in Travis. If instead, I verified it equaled a different object and checked the properties on tat object, the test passes fine.
I found this neat way to use NUnit in Powershell. http://elegantcode.com/2009/10/25/integration-test-brought-to-you-by-powershell-nunit-with-a-little-specification-syntax-for-flavoring/
and we are using it many of our tests.
However I want to run these tests in TeamCity.
I want similar behavior when we use a NUnit runner for running C# tests in TeamCity ie the build fails when the execution of tests fail. Has anyone of you achieved this? I suspect the Powershell runner will just execute it as a simple script, without any indication whether the test passes or fails.
Take a look at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity and http://confluence.jetbrains.net/display/TCD7/Build+Failure+Conditions
There is an issue in Powershell runner support http://youtrack.jetbrains.com/issue/TW-21554
I'm not familiar with the approach you're referencing for executing NUnit tests via Powershell in TeamCity. But, we are successfully using PSake for Powershell build scripts, including executing NUnit tests and failing the build appropriately. The same issue exists with PSake and TeamCity with exit codes, but you can get around it by specifying in TeamCity in the Script Source for the Build Step using -Command for Script execution:
import-module .\tools\psake\psake.psm1
$psake.use_exit_on_error = $true
invoke-psake build.ps1
remove-module psake
You can also integrate the Test results into the TeamCity using the Build Feature option in TeamCity Build Steps.