We've been running unit tests in parallel for several years (both with and without code coverage), following the instructions here:
How to run unit tests (MSTest) in parallel?
and here:
Running unit tests in parallel
However, recent updates to Visual Studio 2017, (versions 15.5.0 or 15.5.1 and later) seem to have broken this, and I get the following error now whenever I try to use a custom test settings file and also do a code coverage analysis:
[12/29/2017 1:35:10 PM Error] Microsoft.VisualStudio.TestPlatform.ObjectModel.SettingsException: Data collectors configured via run settings are not supported with embedded test settings. Please see https://aka.ms/vstest-configure-datacollector for more information. Run settings:
I have updated, rebooted, and verified this issue on another system.
When reverting to default settings (no custom test settings file) and using the 'Run Tests in Parallel' button introduced in VS 2015 update 2, it seems to simply ignore the button setting and run the tests serially (whether you do code coverage or not).
I looked into the link provided in the error (https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md), but it did little to help clarify the issue, other than possibly indicate that yes, a change was made.
If I turn off the custom settings file, the tests run just fine, but they take about 7x as long to run, which is very frustrating.
How does one get parallel unit test execution working in VS 2017 15.5+?
Related
I have a project, composed of a 3 C#/.NET Core class libraries, targeting .NET Standard 2.0. In addition, I have a XUnit test project, where I use FluentAssertions.
When working on the project, I use Visual Studio 2017 Enterprise, with Resharper Ultimate installed.
When running the full suite of tests, my SQL Server 2016 Express database is hit around 8100 times.
If I leave my profiler running while working in Visual Studio, I can see that the full suite of tests is executed, whenever I change code and do a build.
It's a bit of a nuisance, since it takes a good 1-1½ minute for the tests to finish, and often, I will only want to run a specific subset of the tests, when making incremental changes.
My problem: I can't seem to stop the tests running from running on build.
What puzzles me the most is, that I can't cancel the tests from Test explorer or Resharpers Test window, as they appear not to be running at all.
Also, the Test pane of the output window does not show anything except "....Discovering unittests" without progressing any further.
My impressions is, that from Visual Studios perspective, the tests are not running at all.
Getting the primary suspects out of the way:
The "Run tests after build" button in test explorer is not toggled
on.
Live unit testing appears to be disabled (The only option from the
menu is "Start")
I've spent hours browsing the net trying to figure out what is going on, with no luck. I'm hoping someone here can help me find out why the tests are running, and how to stop them.
This is my first ever StackOverflow question. I hope my questions meets the requirements for asking. If not, please let me know what to do better.
First I would advise you update VS. There was an issue in test discovery taking long time that was fixed in update 15.7 (See here)
If that doesn't do the trick I advise you try disabling this option in Tools > Options > Test:
As this might be attempting to analyse all assemblies in build output folder to discover tests.
I'm assisting another developer who's using NUnit with Visual Studio 2017. She's running unit tests and it appears that they're all getting run but for some reason on some of the tests, the ambient test indicators simply don't show up.
I am not totally sure of the version of NUnit but I think it's 2.6.x. The tests are showing up in the test explorer (I've seen that issue before with VS 2017) and, as I say, it looks like they're actually running.
It looks like the ambient indicator doesn't show up for tests decorated with the [TestCase] attribute. Is this a known issue? She just wants to make sure this isn't indicating some other problem.
EDIT: I can reproduce this behavior with NUnit (3.8.1) and NUnit3TestAdapter (3.8.0) both installed via NuGet. VS Professional 2017 v 15.3.5
Neither the integrated test environment nor Resharper won't run any unit tests.
When starting Visual Studio the Test Output Pane shows 10+ lines stating:
An exception was thrown while initializing part "Microsoft.VisualStudio.TestWindow.Controller.TestPlatformProvider".
This is what I've tried so far:
Clear folder %LocalAppData%\Microsoft\VisualStudio\14.0\ComponentModelCache
Create a simple test project with one class library, one class and one test library containing one unit test for that class
Uninstallation and new installation of Visual Studio 2015
All actions didn't fix the problem. Even the simple test won't run.
Running tests from command line using MSTest does work.
Ensure that your anti-virus programs are not quarantining Visual Studio components.
I had this problem when our corporate IT department changed our Sophos Protection Quarantine policy. VS2015 Unit testing stopped working for everyone, with a similar error.
This question is very similar to this one, but I'm experiencing it in Visual Studio 2010 SP1. I am running my tests locally (there is no remote test controller).
I recently started working on a client's .NET 4.0 project. It has a suite of Visual Studio Unit Testing Framework unit tests. When I launch the unit tests, IIS Express automatically starts. I can run the unit tests dozens of times without any issues to see whether they pass or fail. However, when I attempt to debug the unit tests (by right-clicking on a test in Test View and clicking Debug Selection), it almost always fails with the following error message:
Error 3/19/2014 2:42:44 PM Failed to queue test run 'ThoughtProcess#HOSTNAME 2014-03-19-14:42:40': Process with an Id of [12345] is not running.
I say "almost" because every 20 or so clean starts of VS, it will successfully attach the debugger. But I haven't been able to consistently reproduce this behavior.
I tried the suggestions from the Visual Studio 2012 post, but those didn't work in my situation. Has anyone else encountered this issue?
In the spirit of this xkcd comic, I did find a solution to this issue by installing ReSharper and using its test runner instead of the built-in Visual Studio test runner.
The ReSharper runner consistently and repeatably lets me run and debug unit tests without any issues. I'm still not sure that I understand the underlying cause of why the VS runner can't debug, but hopefully this will save someone else future frustration.
I have looked around SO and the web for quite some time now and I am struggling to find any information for how to setup my C# project for unit tests.
I started on the NUnit main site http://www.nunit.org, the download includes a GUI to run your unit tests, sounds mint. Only when I try and point the GUI to my .exe, I get an error immediately saying that NUnit is not designed to be run with this framework. That got me thinking that maybe NUnit doesn't work with .net 4. Apparently it does, plenty of people online seem to state that they are running NUnit unit tests with .net 4 .... of course they don't seem to go on to say how.
Clicking around google I found what appears to be one of the core members of the NUnit development is a dude called Charlie Poole. I found he had a launchpad here: https://launchpad.net/nunit-xtp. The version numbers are newer than on the main NUnit site. When I downloaded the project however, it is just the codebase and solution for the NUnit framework with no GUI. There is a command line runner included but I really need something visual.
We do have jetbrains teamcity as a BMS so I am now going down the path of maybe trying to integrate NUnit into the build pipeline somehow. If anyone can shed any light on how they got NUnit working with an express edition of c# that would be wonderfully helpful.
Edit : I have the teamcity BMS running the tests now, so that part is no longer a problem. However, waiting for the build each time just to find a NullException error or something doesn't really cut it. I have to get the GUI runner working. I found this question NUnit isn't running Visual Studio 2010 code but even that isn't fixing it for me. The error has now updated to read version v4.0.30319, so the config changes have taken effect. Same old error though :(
Right I sussed it out. there is a very similar question here:
NUnit isn't running Visual Studio 2010 code
Only what it doesn't mention is that there are multiple config files for NUnit. I just assumed it would suffice to edit nunit.exe.config. This did indeed update the settings as I saw it in the error message. However, the nunit gui runner must make use of one of the other multiple .config files. I simply went through them all and added the two required settings.
Make sure that you run the correct .exe. One is called "nunit.exe" and it seems like the correct one to use. The other one is called "nunit-x86.exe" and if your app is 32bit then you have to run this one. If "nunit.exe" was names "nunit64.exe" I think I would have noticed that much sooner.
You must also make sure that you check the version number of the CLR matches the setting you are pasting into those config files. Use this link to work out your current CLR version number. How do I find out what CLR i am using? I am Microsoft Visual C# 2010 Express and my CLR version number is v4.0.30319
The unit tests should be in a seperate assembly (of type class library). If you point NUnit to that assembly, it will run. Same applies to the command line runner (which is normaly used in automated build processes).
The default NUnit distro includes a GUI runner called 'nunit.exe'. If using it, I set my debug target on my test assembly to load the GUI runner with the appropriate config.