I'm putting together a new stack of unit tests to be run together as a CI job. I'm using vstest.console.exe instead of mstest.exe mainly for its ability to run tests from several frameworks, but right now the focus is a few xUnit dlls. The jobs are run as part of a Jenkins pipeline.
I have tested everything successfully on a couple of dev boxes, but annoyingly test discovery is not working on any of the CI build boxes so far. This is after the addition of the 0.99.8 xUnit test adapter vsix (also tested with 0.99.7). The xUnit dlls are being built against 4.5 with the 2.0.0.2378 beta nuget version of xUnit.
I have reproduced the symptoms with the simplest possible dll, with a single public test method, working fine on my own box, not working on any build boxes. The deployment enviroment is pretty straight forward, with a VS2012 install and the xUnit test adapter, on Windows 2012.
I have enabled the TpTrace logging via the vstest exe config files and everything looks fine. I guess I'm looking for a way to troubleshoot the issue further (maybe tracing for the xUnit discovery process) or a workaround the problem. I'd prefer to retain the use of vstest console for the simplicity of running several frameworks.
I have written up this issue via the xUnit codeplex site as well.
I've checked out this SO post but none of the suggested solutions make sense here.
It takes me quite some time to find the solution of how to use vstest.console.exe and xunit for VS2013, so I think it worth the time to explain here how I did for everyone...
The first step is to follow what is explained here to install the pre-release xunit.runner.visualstudio nuget package in the xunit projects that need it to be able to run the xunit test from Visual Studio.
Then, when you run your vstest.console.exe command, you MUST use the parameter /TestAdapterPath.
Your command line should look like something (the path toward the xunit adapter is here relative so, you could put it in absolute or adapt depending on the active directory):
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:".\packages\xunit.runner.visualstudio.0.99.9-build1021\build\_common\"
edit: Because the adapter dlls are copied to the output folder, we could simplify the command line giving the path "." to the /TestAdapterPath option:
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:"."
For information, it works also with NUnit, nuget package NUnitTestAdapter and the command:
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" c:\path\to\your\assembly.to.test.dll /TestAdapterPath:"."
OK so problem solved, but after some frustrating troubleshooting that I will cover in case its useful for someone. The problem was that xunit.execution.dll was not available in the same folder as the dlls containing tests. This is required for xunit discovery. I only got here by:
Setting HKCU\Software\Outercurve Foundation\xUnit.net\Visual Studio Test Plugin\MessageDisplay = Diagnostic (this should be possible via the runsettings file but is not being picked up, and not possible via the VS tools options xunit page because its failing to open)
vstest now spits out 'Skipping xunitTests.dll (no reference to xUnit.net)'
This message actually means that xunit.dll and xunit.execution.dll weren't found in the folder
Problem solved by making sure that dll gets copied into that folder on the build boxes.
Related
I followed the tutorials in the docs for azure devops with regards to running Selenium tests as part of the build.
My local tests run fine with the browsers I have webdrivers installed for, but when the VSTest task runs, it doesn't unpack any tests from anything under NUnit attributes.
Now a few things to point out with my project structure:
The test project started out as a basic Visual Studio 2017 Unit Test template, with MS Test.
I preferred NUnit, so I am writing/rewriting the tests using that framework instead of the MSTest it was created with.
My current azure devops pipeline config:
Working local tests setup:
Now, I can easily add a new project using the NUnit template which for whatever reason allows the azure build to see these tests, but that causes my project structure to change and it defaults to netcoreapp2.1 instead of 4.6, and then the azure build can't find all sorts of stuff.
Please follow the below steps may it resolve your issue
There is no need to add TestAdapter.dll in Test files
uncheck "Test mix contains UI tests"
Select test platform using "version" as "Installed by Tools Installer"
or
You might need to search the test dll into $(System.DefaultStagingDirectory) folder. sometimes azure agent place the dlls in staging directory instead of defaultworkingdirectory.
When adding a new NUnit project by the visual studio template like so
It defaults the framework version to netcoreapp2.1
If I change this to my build framework (net461), although the build states the same "These dlls will be skipped" message, the NUnit3Adapter is somehow now recognized and picks up the NUnit tests:
I'm trying to establish continuous integration with TeamCity and stuck at running unit tests step with Visual Studio Tests runner.
Loading [02:42:07]C:\TeamCity\buildAgent\work\f7a05248f49c57a1\LodCore\UserPresentationTests\bin\Release\UserPresentationTests.dll...
[02:42:07][Step 3/3] Starting execution...
[02:42:07][Step 3/3] No tests to execute.
All tests are discoverable, when I build test assemblies from Visual Studio.
I've downloaded UserPresentationTests.dll from build agent to my computer and tried to execute same command with my local MSTest.exe, and it failed with the same issue.
I've decompiled it with ILSpy and found no difference with dll built from Visual Studio.
I've tried different build configurations on MSBuild, but it is not working.
My TeamCity version: Professional 2017.2 (build 50574)
If you are using MSTest v2, then you should use VSTest instead of MSTest:
Can you check that there aren't any differences between the environments in terms of installed libraries, permissions, etc.
We have a couple pointers to troubleshoot the differences between a working local build and a failing TeamCity one, you might want to follow those steps: https://confluence.jetbrains.com/display/TCD10/Common+Problems#CommonProblems-BuildfailsorbehavesdifferentlyinTeamCitybutnotlocally
With this in mind, while the step 3 is the one pointing at the failure, I'd also check the build log of step 2, as it might have skipped some of the build parts for some reason.
Also it would be helpful if you could also add to the OP the TeamCity version you are using and the type of runner you are using in step 3.
In addition of build configuration your solution you should create new configuration with Runner type MSTest.
Example here
I'm working on a suite of application that interact with many websites and api's. I have created unit tests that test this code.
As it's beyond my control, I want to know as soon as sites / api's change and the code stops working so I can fix and would like to run the tests periodically against the assemblies running. How can I run the tests outside of VS?
I'd like the option of testing on Linux mono too.
You can use MsTest from the command line:
Example:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\mstest" /testcontainer:C:\MyTestProject\bin\debug\MyTestProject.dll /resultsfile:C:\TestResults\results.trx
(Full list of command line switches on the link above)
This is set to Visual Studio 2015, for 2017 just make sure you have the right version number, I believe 15.0
MSTest has been deprecated, you'll want to invoke VSTest from the command line
I am using vstest.console.exe to run a suite of integration tests. I am following the instructions here [http://blogs.msdn.com/b/visualstudioalm/archive/2012/12/06/publishing-test-results-through-command-line-test-runner.aspx][1]
Here is my command line (with added newlines for readability):
"E:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
/Tests:UserIsAbleToLogIn
"E:\IntegrationTesting\Ads.Slms.IntegrationTesting.Web -Ver_20150206.1\Ads.Slms.IntegrationTesting.Web.Smartfill.dll"
/logger:TfsPublisher;Collection=http://xxxxx:8080/tfs/DefaultCollection;TeamProject=SLMS;BuildName="Ads.Slms.IntegrationTesting.Web -Ver_20150206.1"
However, despite the tool running fine, and reporting that the results have been published, I don't see any results appearing in TFS. I am using TFS2013. I have Visual Studio 2013 Professional installed on the server where my tests are being run.
Is there something I'm missing? Why would vstest report "Publish completed successfully", if I can't see the results in TFS?
Late to the party, but for those who experience the problem in the future, here's what worked for me.
Use the Platform and Flavor in the /Logger argument. Those will tell TFS which built project to associate your test with. By default Platform is 'Any CPU' and Flavor is 'Debug' - in all likelihood your build definition doesn't use that configuration.
How can I include Specflow tests into the Build process template for nightly build.
I use VS 2012 and our nightly build run on Server 2008,
And I have some PowerShell script for it, but I don`t know how use it.
What you are describing sounds incredibly vague. Does your powershell script run the build and test parts of your build?
Specflow will be run automatically as part of your nunit or mstest test runs. If you already have these and you are presenting the assemblies to the runner, it just happens.
personally I would suggest the easiest way is to download a copy of TeamCity, which if I remember correctly will be a free licence for just a single project. Just add your source code repo details, point a Visual studio step at your solution, a unit test step at your test dlls and a trigger for whenever you checkin and will all happen automatically.