I am running a collection of unit tests in an Azure DevOps Pipeline. The unit testing framework in use is MSTest.
Some unit tests load files that are inside the repo. Therefore the Property Copy to Output Directory is set to Copy Always.
Down below you can see the property being set inside the csproj file.
<Project Sdk="Microsoft.NET.Sdk">
<!-- shortened csproj file -->
<ItemGroup>
<None Update="TestFiles\sample.xml">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
Test Method
Inside the test, the file gets loaded as seen in the next code block.
var path = Path.Join("TestFiles", "sample.xml");
var fileStream = File.OpenRead(path);
Local Run
I set the verbosity to detailed in order to see if the file gets copied or not.
Running dotnet test .\Project\Project.csproj --logger trx --verbosity detailed locally, everything works as expected with the test results.
Same applies for running the tests in Visual Studio.
Pipeline
I execute the tests inside an Azure DevOps Pipeline with the DotNetCoreCLI task.
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- task: DotNetCoreCLI#2
displayName: Run Unit Tests
inputs:
command: test
projects: Project/Project.csproj
arguments: --verbosity detailed
I extracted the generated command from the pipeline logs: /usr/bin/dotnet test /home/vsts/work/1/s/Project/Project.csproj --logger trx --results-directory /home/vsts/work/_temp --verbosity detailed. I don't see any significant differences between my local command compared to the one being generated by the pipeline.
MSBuild
Indeed inspecting the local and Azure DevOps Pipeline logs, the file gets copied as expected.
Copying file from "/home/vsts/work/1/s/Project/TestFiles/sample.xml" to "/home/vsts/work/1/s/Project/bin/Debug/net6.0/TestFiles/sample.xml".
Test Error
The path matches the one from the MSBuild log /home/vsts/work/1/s/Project/bin/Debug/net6.0/TestFiles/sample.xml. Then again, I get a FileNotFoundException:
System.IO.FileNotFoundException: Could not find file '/home/vsts/work/1/s/Project/bin/Debug/net6.0/TestFiles/sample.xml'.
Pipeline debugging
I tried to list all files inside the output directory adding the continueOnError key to the test task.
# ...
tasks:
# ...
- script: ls -la /home/vsts/work/1/s/Project/bin/Debug/net6.0
- script: ls -la /home/vsts/work/1/s/Project/bin/Debug/net6.0/TestFiles
The first script action already fails with the following error message:
ls: cannot access '/home/vsts/work/1/s/Project/bin/Debug/net6.0': No such file or directory
At this point I'm just lost and don't know what to do about this seemingly simple problem... Any suggestions how I get the files containing test data to the unit test? Where is my mistake?
Update 1
I tried to switch to xUnit. But it gave me the same FileNotFoundException as before. I don't think it's a problem with the test itself but with the Azure DevOps Pipeline environment.
After some further debugging of the pipeline, I found a mistake I caused myself.
When debugging the pipeline I used the - script: ls -la /home/vsts/work/1/s/Project/... task. The path I specified was the one from this post (which I abstracted from the real thing cuz its from work yk) NOT the actual one of my test project.
Listing a non existing directory does not work for obvious reasons...
Comparing the paths from the MSBuild log message and the actual test run showed that they were almost identical. The actual file had a lowercase extension and the file name inside the test project was uppercase.
Running the pipeline in the ubuntu-latest environment means that we are working with a case sensitive file system (unlike on my local machine which is a Windows 11 machine).
tl;dr - Everything was correct, I just got to match the actual file name and file path inside the consuming test 100% correctly or run it in a windows environment.
Related
Hi I've got a solution that has multiple projects held within it where I'm hoping to automate the build and deployment of the webapplication within the project.
on my HD the dir looks like C:\Users\mleip\source\repos\edocs.utils.bluebeam-api\edocs.utils.bluebeam-api.Caller\edocs.utils.CDEComms , however, when I run the below yaml
i get
MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
buildConfiguration: 'Release'
steps:
- script: dotnet restore
- powershell: |
Write-Host "Your parameter is now: $env:Param"
Write-Host "When outputting standard variable build id: $(Build.BuildId)"
Write-Host "When outputting standard variable build id via env: $env:BuildNumber"
Write-Host "The repo name is: $(Build.Repository.Name)"
Write-Host "The build definition name is: $(Build.DefinitionName)"
- script: dotnet build --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: '**/edocs.utils.bluebeam-api.Caller/edocs.utils.CDEComms/edocs.utils.CDEComms.csproj'
So far I have tried several combinations of dir for where azure should locate the build file, however, each time it fails. The solution has several projects within it. CDEComms is the only project within that folder so I'm left quite confused as to what's going on.
I'm quite new to building pipelines and any help as to why this is failing would be greatly appreciated.
In order to narrow down the issue, try to simply specify **/*.csproj in the YAML file to check the build result. And set variable system.debug to True, to check whether the path is correct in C:\hostedtoolcache\windows\dotnet\dotnet.exe build {path}. Also, clone the repo to your local machine to run build it locally, to check the result.
From the description you have provided, you will first have to push your code/solution i.e. the content of your C:\ drive to your repository.
console app with a batch file which will hit my Automation testing application and runs selected test cases. I have test case with code coverage and it runs from my visual studio . and now on top of it I have to create a console app which will keep some time interval and hit my VS test case and execute it.Any links will be helpful.
I expect the test case pass and fails status
Here's the batch file I use to run my Selenium tests with multiple runsettings files.
#ECHO OFF
IF NOT EXIST Results MKDIR Results
SETLOCAL
SET PATH=%PATH%;"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\Extensions\TestPlatform"
SET VSTEST=vstest.console.exe
SET TESTS=.\bin\Debug\something.something.Tests.dll
IF "%~1"=="" (
FOR %%J IN (*.runsettings) DO CALL :func %%J
) ELSE (
CALL :func "%~1"
)
GOTO :EOF
:func
ECHO.
ECHO %~1 **********
ECHO.
"%VSTEST%" %TESTS% /Settings:"%~1" /Logger:trx /ResultsDirectory:Results
GOTO :EOF
You could use NUNit's console runner to run tests from the command line, or on a build server. I've had great success with running my tests this way.
First, you need to install the NUnit.ConsoleRunner Nuget package onto your project.
Then, navigate to the NUnit.ConsoleRunner directory under your packages folder that exists in the project directory.
Open NUnit.ConsoleRunner > tools folder to get into the same directory as the .exe itself.
Then, you can run:
nunit3-console {Path to your project's .dll} --testlist={Path to .txt testlist}
With valid parameters, it looks something like this:
nunit3-console C:\Users\christine.harbour\Repository\AutomationTestSuite\AutomationTestSuite.dll --testlist=C:\Users\christine.harbour\Repository\AutomationTestSuite\MyTestList.txt
Your testlist should contain namespaces of the test cases you wish to run, separated by a line break. For example:
AutomationTestSuite.Tests.MyTestClass_1.MyTest
AutomationTestSuite.Tests.MyTestClass_2.MyOtherTest
After you run the tests, the results will be saved in the NUnit.ConsoleRunner > tools directory. The results are in XML format and can be programatically parsed to push your test results to another tool.
There are plenty of arguments you can pass into the ConsoleRunner, including build configuration and framework version, all which are specified on NUnit's documentation.
NUnit console runner also integrates with Cake, which is a build scripting tool for C# projects. So, you could hypothetically clean / build your project, restore missing package references, and run your tests, all from the console.
More info on NUnit console runner can be found here: https://github.com/nunit/docs/wiki/Console-Command-Line
First of all please keep in mind that I am just discovering how "things" work in the DevOps world and my knowledge is pretty limited - especially in scripting (PS and C#).
So I have to figure it out how to set up some Javascript unit tests for an app - specifically to get the "Test Results Analyzer" widget up and running on the Jenkins website.
I am currently recycling code from another Jenkins pipeline (def) and using the already existing "run unit tests" stage in the makefile.include of the project which (from what the main Dev of the project told me) it never worked - was never tested - to try and make the tests run.
I thought the best way (without breaking anything on the main app) is to test locally via cygwin.
I have mapped the project as a workspace in Perforce and tried to run the command from the makefile.include
$ make runRoutingPlannerCSTests
Which is working about half way and after it prompts me with some missing files errors which I was not able to find anywhere on my local project folder.
CSC : error CS0006: Metadata file 'D:\src\project\ServicePlanning\BackendAPI\Comtec.Portal.OSPBackend.ApiModel\bin\Release\Comtec.Portnd [D:\src\project\ServicePlanning\Frontend_RoutingPlanner\Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests\Comtec.ServicePlanning
CSC : error CS0006: Metadata file 'D:\src\project\ServicePlanning\Frontend_RoutingPlanner\Comtec.ServicePlanning.Frontend.RoutingPlannngPlanner.dll' could not be found [D:\src\project\ServicePlanning\Frontend_RoutingPlanner\Comtec.ServicePlanning.Frontend.RoutingPlanntingPlanner.Tests.csproj]
CSC : error CS0006: Metadata file 'D:\src\project\ServicePlanning\Frontend_RoutingPlanner\Comtec.ServicePlanning.Frontend.RoutingPlannFrontend.RoutingPlanner.Model.dll' could not be found [D:\src\project\ServicePlanning\Frontend_RoutingPlanner\Comtec.ServicePlanning.Flanning.Frontend.RoutingPlanner.Tests.csproj]
Makefile.include:121: recipe for target `buildRoutingPlannerCSTests' failed
make: *** [buildRoutingPlannerCSTests] Error 1
and also
$ make runTests
Which gets stuck somewhere after running the Specrunner.html
######################################################
# run unit tests
######################################################
runTests: runRoutingPlannerJasmineTests runRoutingPlannerCSTests
runRoutingPlannerJasmineTests:
$(RoutingPlanner_TESTS_UTILITIES_DIR)/bin/phantomjs.exe $(RoutingPlanner_TESTS_DIR)/Scripts/jasmine.js $(RoutingPlanner_TESTS_DIR)/jsTests/SpecRunner.html
buildRoutingPlannerCSTests:
$(MSBUILD) `cygpath -a -w $(RoutingPlanner_TESTS_DIR)/Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests.csproj` $(MSBUILD_OPTIONS) /t:Build /property:Configuration=$(VS_CONFIG) /fileloggerparameters:LogFile=`cygpath -a -w $(RoutingPlanner_TESTS_DIR)/Comtec.ServicePlanning.Tests-Build-$(VS_CONFIG).log`
runRoutingPlannerCSTests: buildRoutingPlannerCSTests
#rm -f '$(shell cygpath -aw '$(RoutingPlanner_TESTS_DIR)/bin/$(VS_CONFIG)/Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests.trx')'
$(DOTNET_UNIT_MSTEST) /nologo '/testcontainer:$(shell cygpath -aw '$(RoutingPlanner_TESTS_DIR)/bin/$(VS_CONFIG)/Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests.dll')' '/resultsfile:$(shell cygpath -aw '$(RoutingPlanner_TESTS_DIR)/bin/$(VS_CONFIG)/Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests.trx')'
This section has an additional section at the begging of the script where the paths have been added.
RoutingPlanner_FRONTEND_API_DIR ?= $(RoutingPlanner_FRONTEND_DIR)/Comtec.ServicePlanning.Frontend.RoutingPlanner
RoutingPlanner_TESTS_DIR = $(RoutingPlanner_FRONTEND_DIR)/Comtec.ServicePlanning.Frontend.RoutingPlanner.Tests
RoutingPlanner_TESTS_UTILITIES_DIR = $(RoutingPlanner_TESTS_DIR)/Utilities/phantomjs-2.0.0-windows
RoutingPlanner_UTILITIES_DIR = $(PREPARESETUPDIR)/RoutingPlannerPortalUtilities
To be honest I am not even sure if I am following the right path to this since the steps I have found on the internet are different from what I found in the project and what I believe should be done.
For example: https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin
Mainly I believe this might be a simple "wrong path" issue but since my scripting skills are close to zero I am not sure what to modify and where.
I do not expect the solution at hand but I would really like to understand what the issues is or the proper way to actually make this work.
Thank you in advance.
I have a web application using Angular 1.5 with bower/npm/gulp coded in Typescript to do our build. Our back end is a c# .net WebApi2. Both are built and deployed on TFS2015. My c# nUnit tests are easy to integrate as part of the build process. The Typescript jasmine unit tests however are more difficult to integrate. How do I get my Typescript jasmine unit tests to run as part of the TFS build and if they fail, fail the build? We have them running through a Jasmine Spec runner and also Karma but not integrated.
I have read the many posts on StackOverflow integrating Javascript unit tests and each avenue took me through an overly complex solution that didn't work. These include Powershell scripts, Chutzpah amoungst others.
Rather than try to recreate the Specrunner via Chutzpah on the build server, which I found difficult to configure and get working. The aim was to get karma to output the running tests in the 'trx' test format that TFS recognises and then publish them to the build. Please note I am using PhantomJs to run my tests through Karma but won't cover that here as it is well covered elsewhere.
1) install the karma-trx-reporter plugin via npm into your web project (or similar plugin)
2) Configure the Karma.config to include the trx reporter
reporters: ['dots', 'trx'],
trxReporter: { outputFile: 'test-results.trx' },
// notify karma of the available plugins
plugins: [
'karma-jasmine',
'karma-phantomjs-launcher',
'karma-trx-reporter',
],
3) Create a Gulp (or grunt) task to run the karma tests if you don't already have one. Run the task locally and check it creates the 'test-results.trx' specified above. (It doesn't matter where the file is created on the build server):
gulp.task('test', function () {
return gulp.src(['tests/*.js']).pipe(karma({
configFile: __dirname + '/Testing/karma.config.js',
singleRun: true
}));
});
4) Add a Gulp (or Grunt) TFS build task to run the karma tests created in the previous step and output the trx file.
5) Add a Gulp (or Grunt) TFS build task to Publish the test results and merge them into the build. Note that the "Test Result Files" path is a wild card **/*.trx to find any trx files in the build path (i.e. finds our previously created file). "Merge Test results" is checked to Merge both our Jasmine test run and our c# test run into the same session. "Continue on error" is unticked to ensure any jasmine test failures break the build.
You will notice two sets of tests that have been run and included as part of the build!
I have an x64 platform C# solution(VS2012) on a TFS2010 server. I have attached a unit test project (also x64) to this solution and created a build definition. When I queue the build, it succeeds but the unit test cases will not be executed. This is because MSTest is a 32 bit application. So, I decided to customize the default build process template (DefaultTemplate.xaml) to invoke VSTest(VSTest.console.exe) instead of MSTest. This is quite complex and I am unable to add a build activity to the toolbox for VSTest.
Has anyone done this kind of customization? I have also considered other approaches like configuring .runsettings file. Do we have a VSTest adapter interface that can be added in the .runsettings file ?
Executing unit tests through VSTest and publishing the test results through MSTest gave me a successful outcome. Given below is the Powershell script:
# Get the UnitTest Binaries
$files = Get-ChildItem $TestAssembliesDir\*est*.dll
# VSTest.console.exe path
$VSTestPath = 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe'
# MSTest path
$MSTestpath = "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe"
# Loop through the test assemblies and add them to the VSTestFiles
$VSTestFiles = ''
foreach($file in $files)
{
$VSTestFiles += "`"$file`""
$VSTestFiles += " "
}
# Run the UnitTests using VSTest
&$VSTestPath $vstestplatform "/Framework:Framework45" "/InIsolation" $VSTestFiles "/logger:trx"
# Get TRX files
$TrxFilesList = Get-ChildItem $TestResDir\*.trx
$TrxFiles = ''
$loop = 1
foreach($file in $TrxFilesList)
{
$TrxFiles = "$file"
# copy the trx file into a space-less named trx
$newTrxFileName = "$BuildUri" + "_" + "$loop" + ".trx"
copy-item $TrxFiles -destination $TestResDir\$newTrxFileName
$loop = $loop + 1
$newTrxFile += "$TestResDir\$newTrxFileName"
$newTrxFile += " "
}
# specify MSTest arguments
$mspubl = "/publish:"+$TeamProjColUri
$msteampr = "/teamproject:" + $TeamProj
$mspublbuild = "/publishbuild:" +$BuildUri
$mspubresfile = "/publishresultsfile:" +"`"$newTrxFile`""
#Publish test results through MSTest
&$MSTestpath $mstestplatform $flavor $mspubl $msteampr $mspublbuild $mspubresfile
I too have the exact same need for using VSTest.Console.exe instead of MSTest.exe for a TFS2010 build process that compiles a VS2012/.NET 4.5 x64 application, while waiting for the upgrade to TFS2012 to commence.
The approach I have taken was to edit the build script XAML, deleted the existing workflow for unit tests and replaced it with a customised workflow that builds up the VSTest.Console.exe parameters and then executes VSTest.Console.exe via InvokeProcess. I then ensured that in the Finally block that regardless of test result that we publish the test results and code coverage to TFS using MSTest.exe from a VS2012 installation on the build server.
Unfortunately I cannot post the XAML in the answer as it exceeds character length, but I do have a text file consisting of the snippet to be replaced in DefaultTemplate.xaml and what to replace it with. The file can be found here. Please note that although this approach works it is a hack.
Another alternative would be to use NUnit instead of MSTest or VSTest.Console as this support 64-bit binaries. This article explains how to integrate NUnit in a TFS2010 build script, and has links to tools and resources required to make this happen. The only issues with NUnit are code coverage (need yet another tool plus work out how to publish these results to TFS) and MSTest-style integration tests using attributes such as DeploymentItem and properties such as TestContext, which is why where I work we opted with the VSTest.Console.exe approach.
And from what I have read TFS2012 offers easy integration to VSTest.Console.exe from build scripts, so if you do ever upgrade to TFS2012 the VSTest.Console.exe hack that I have documented may not be required.
This does not directly answer you question, but it might help. I did a similar thing for TeamCity. I used command-line to call vstest.console.exe and created a .runsettings file.
I used this Microsoft template for the runsettings file. Note however that on my machine, the path mentioned in the comment in Line 5 is relative to the .runsettings location, not the .sln.
If you use /logger:trx option of vstest.console.exe, it will generate output in the same format as MSTest (good for result visualization).