I'm trying to setup AppVeyor for my project (here) and I can't seem to find a way to only have it run tests from the .NET Core Unit test project.
This is the link for the AppVeyor project: ci.appveyor.com/project/Sergio0694/neuralnetwork-net
I also have a .NET Framework Unit test project that executes some GPU-based tests, so they just fail when run with AppVeyor as they're missing both a CUDA GPU and the required cuDNN files.
This is my configuration:
version: 1.0.{build}
image: Visual Studio 2017
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
assemblies:
only:
- NeuralNetwork.NET.Unit.dll
categories:
except:
- NetworkTest
That NetworkTest category that's skipped is just a group of tests that are very CPU intensive and require some time to run, so I'm just skipping them now to make the builds finish sooner.
The project builds fine, but AppVeyor keeps running the tests in the other NeuralNetwork.NET.Cuda.Unit.dll assembly as well, which fail as expected and cause the whole build to be marked as failing.
So my questions are:
Am I doing something wrong here? Why isn't the assembly constraint respected?
Is there another way to have AppVeyor only run the tests from NeuralNetwork.NET.Unit.dll?
EDIT: I've tried to set the "all assemblies excluding" option and all of these combinations to specify the .NET Framework Unit test project to skip:
NeuralNetwork.NET.Cuda.Unit.dll
**\*.NeuralNetwork.NET.Cuda.Unit.dll
**\NeuralNetwork.NET.Cuda.Unit.dll
The tests keep being executed (failing as expected), am I missing something here?
EDIT #2: This is the workaround I'm using for now, I've manually excluded all the test categories in the .NET Framework project and so far this seems to be working (even if it's not so clean to see):
version: 1.0.{build}
image: Visual Studio 2017
configuration: Release
before_build:
- cmd: dotnet restore
build:
verbosity: minimal
test:
categories:
except:
- CuDnnInceptionLayerTest
- CuDnnLayersTest
- GpuExtensionsTest
- SerializationTest
.NET Core tests detection works differently than detection of classic .NET Framework tests. When searching .NET Core tests, AppVeyor look for .csproj files with certain properties, not for assemblies. Assembly filter does not work for .NET Core tests (Categories filter works though).
Sorry for confusion. Created this issue to implement behavior similar to assembly name filtering, but based on .csproj file names for .NET Core tests auto-detection.
For now workaround is to do a "black list", e.g. list all .NET Framework assemblies you do not want to run under assemblies/except. You can use wildcard syntax to make it more compact as described here.
Related
I'm writing some unit tests with MSTest, using C#, MSVS 2015 and .Net 4.6.1.
This line:
Assert.ThrowsException<ArgumentOutOfRangeException>( () =>
select.AllSelectedOptions[0]
);
Fails with this compile error:
CS0117 'Assert' does not contain a definition for 'ThrowsException'
My namespace is Microsoft.VisualStudio.TestTools.UnitTesting (the default when you create a unit test project in MSVS).
According to the documentation, Assert.ThrowsException(Action) should exist. But I don't see it in Intellisense ... and I'm getting the compile error.
I've tried a couple of different versions of MSVS (MSVS 2015 and MSVS 2019) and a couple of different versions of MSTest.
Q: Any ideas what might be wrong?
As Clint said below, I need to install MSTest v2 from NuGet in order to use Assert.ThrowsException<T>() in MSVS 2015.
But after doing this, MSVS isn't finding any of my tests anymore:
MSVS > Test > Run All (or "Test > Debug > All Tests"):
------ Discover test started ------
========== Discover test finished: 0 found (0:00:01.127375) ==========
Any suggestions?
You need to use [MSTest V2] to be able to Assert.ThrowsException
Starting with VS2017, the in-box Unit Test Project templates use only MSTest V2.
Now that you're on VS2015 you can install this package MSTest.Test from Nuget but make sure to remove the old test references like Microsoft.VisualStudio.QualityTools.UnitTestFramework before upgrading to this package
Add > New Test project > Select MSTest project type
After this you should be able to use Assert.ThrowsException<ArgumentOutOfRangeException>(
To discover and execute tests also ensure to install MSTest.TestAdapter.
Further Reading
The build for my solution, which contains multiple projects, suddenly appears to be broken and the cause of the issue isn't clear. All of the projects in the solution target the same framework (4.5.1), however, some of the projects' dependencies might be targeting an earlier version of the framework. As of last week, we were able to successfully build solutions that fall under this scenario. Starting last Thursday, the builds appear to be broken without any changes to the build server or the solution.
To illustrate and remove some of the complexities with our production setup, I created a sample solution which mimics the behavior.
Project structure:
CBI Solution
CBI website
CBI Implementation Library
CBI Core library, defining the interfaces
Target Framework Version for the website and class libraries is 4.5.1
Replication Steps
Solution 1
Github repo: https://github.com/NikitaGolovko/Solution1.git
Build, using MSBuild 14. My command is:
msbuild.exe "%SOLUTION1PATH%\CBILite.sln" /p:platform="Any CPU" /p:configuration="Release" /v:d
Output
C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe -v /localhost_61806 -p cbi\ -u -f PrecompiledWeb\lo
calhost_61806\
Microsoft (R) ASP.NET Compilation Tool version 4.7.2046.0
Utility to precompile an ASP.NET application
Copyright (C) Microsoft Corporation. All rights reserved.
Done executing task "AspNetCompiler".
Task "Message" skipped, due to false condition; (!((false) or ('$(AspNetConfiguration)' == 'Debug') or ('$(AspNetConfig
uration)' == 'Release'))) was evaluated as (!((false) or ('Release' == 'Debug') or ('Release' == 'Release'))).
Done building target "Build" in project "cbi.metaproj".
Done Building Project "D:\Work\DotNet\Nikita\Solution1\cbi.metaproj" (default targets).
Done executing task "MSBuild".
Done building target "Build" in project "CBILite.sln".
Done Building Project "D:\Work\DotNet\Nikita\Solution1\CBILite.sln" (default targets).
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:03.10
Full output is located here: https://github.com/NikitaGolovko/Solution1/blob/master/MSBuild_Output.txt
Solution 2
Github Repo: https://github.com/NikitaGolovko/Solution2.git
The solution is nearly identical to the Solution1, with the only exception being a dependency for Unity in CBI Implementation library. To simplify the process and eliminate the nuget restore step, I have included nuget packages with the solution.
Build, using MSBuild 14. My command is:
msbuild.exe "%SOLUTION2PATH%\CBILite.sln" /p:platform="Any CPU" /p:configuration="Release" /v:d
Output
C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe -v /localhost_61806 -p cbi\ -u -f PrecompiledWeb\lo
calhost_61806\
Microsoft (R) ASP.NET Compilation Tool version 4.7.2046.0
Utility to precompile an ASP.NET application
Copyright (C) Microsoft Corporation. All rights reserved.
D:\Work\DotNet\Nikita\Solution2\cbi\Index.aspx.vb(5): error BC30002: Type 'CBILite.Calculator' is not defined. [D:\Work
\DotNet\Nikita\Solution2\cbi.metaproj]
The command exited with code 1.
Done executing task "AspNetCompiler" -- FAILED.
Done building target "Build" in project "cbi.metaproj" -- FAILED.
Done Building Project "D:\Work\DotNet\Nikita\Solution2\cbi.metaproj" (default targets) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "Build" in project "CBILite.sln" -- FAILED.
Done Building Project "D:\Work\DotNet\Nikita\Solution2\CBILite.sln" (default targets) -- FAILED.
Build FAILED.
Full output is located here: https://github.com/NikitaGolovko/Solution2/blob/master/MSBuild_Output.txt
Observations
Bin folder
When looking in the bin folder of the website, I notice that CBILite.dll is missing after the build for Solution2, but present in Solution1.
TargetFrameworkVersion
Passing TargetFrameworkVersion in the MSBuild arguments does not appear to influence the build. I have attempted to pass 4.5, 4.5.1 to no avail. HOWEVER passing 3.5 results in the successful build. This is extremely strange.
Metaproj
When comparing Metaproj files generated for both solutions, the only observable and seemingly important difference is the lack of the TargetFrameworkVersion element. Solution1 (without dependencies), contains v.4.0 element. Solution2 (with the dependency) does not.
I didn't include metaproj files in the repo but can share them if needed.
Visual Studio
Building the solution in Visual Studio works just fine.
Additional thoughts
The issue manifested itself rather suddenly and seems to be affecting most of our solutions that have mixed projects (C#/VB + website). Web Projects continue to function normally.
I've attempted to use MSBuild 15, but it resulted in the same behavior.
There are a few workarounds
Retaining the metaproj file with the solution and modifying it manually by adding v4.5.1 element.
Adding an additional build step to manually copy CBILib.dll into the website project (via the batch file or other means).
Adding a refresh file for the website pointing to CBILib/bin/Release folder
While all of these solutions might work, they're certainly hacks and will present problems in the future.
I'm hoping someone else has a better suggestion on how to handle this.
Make sure you have installed the following NuGet packages:
Microsoft.NET.Test.Sdk
MSTest.TestAdapter
MSTest.TestFramework
Microsoft.NETCore.App (if you use .NET Core)
My situation is similar to user2768132's question (VS2010 pro not able to start Moles host), however, I have a couple differences. I also attempted the suggestion by SouthShoreAK but it didn't resolve my problem. Pardon the similar post.
I'm new to this project at work as well as Moles so I might be missing something simple/obvious. We are also using Moles framework to write unit test cases but are not able to either debug or run a unit test that involves Moling a public static class.
System - Win-7 Professional SP1
.NET - .NET v4.0.30319 SP1 Rel
VS - VS2010 Professional v10.0.40219.1
Moles - v0.94.51023.0
The solution builds successfully. The 6 simple unit tests (i.e. not Moling static classes) pass but the one unit test that requires Moles to deal with a static class aborts.
Error for the aborted test:
Error 3/26/2014 2:26:06 PM System.InvalidOperationException: Could not start Moles host. Please review the Test Run Errors for more information.
at Microsoft.Moles.VsHost.Agent.HostTestAdapterDriver.EnsureHostAdapter()
at Microsoft.Moles.VsHost.Agent.HostTestAdapterDriver.Microsoft.VisualStudio.TestTools.Execution.IBaseAdapter.Run(ITestElement testElement, ITestContext testContext)
at Microsoft.Moles.VsHost.Agent.MolesAgentAdapter.Run(ITestElement testElement, ITestContext testContext)
My fuslogvw was empty and did not identify any errors during assembly loading.
I found the same blog as user2768132 that mentioned removing the .exe.config file from privateassemblies folder under VS2010 IDE folder. I did that and it didn't fix my problem either.
I believe the tests are 32 bit. With that said, I attempted SouthShoreAK's suggestion of editing the test settings to 64 bit and adding the bitness line to the bottom of the AssemblyInfo.cs file. Unfortunately, that didn't solve my problem either. The simple tests that originally passed would fail and the unit test requiring Moles would error saying that Moles cannot be loaded because the key 'Moles' cannot be found. I undid these changes and I'm back to my starting point.
Anyone have any ideas/suggestions?
Thanks,
Steve
Answer was to make sure the "Platform target" parameter was set to x86 in the Build tab of the test project's Properties.
We're asked to move from NUnit to MSTest and now have to convert all the existing tests to the new platform. Most of it converted fine but we have an issue with parameterised tests. We found the following web site showing how to write an extension and downloaded the code for it.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/09/04/extending-the-visual-studio-unit-test-type-part-2.aspx
However, although it compiles fine, the RowTestClass test all fail to run with the following error message:
"Failed to initialize the unit test extension 'urn:RunAsTestClassAttribute': A unit test extension is not registered for the following attribute: Microsoft.VisualStudio.Test.Sample.UnitTestTypeExtension.RunAsSample.RunAsTestClassAttribute."
We're using VS2010 (10.0.40219.1 SP1Rel) (Microsoft .NET Framework Version 4.0.30319 SP1Rel)
I wonder if it has something to do with the following libraries:
Microsoft.VisualStudio.QualityTools.Common.dll
Runtime Version: v2.0.50727
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0\Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Runtime Version: v4.0.30319
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Vsip.dll
v4.0.30319
C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.Vsip\v4.0_10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.Vsip.dll
Any help to get this working greatly appreciated.
Have you registered the new types (in the Windows Registry)? There's a section on how to do it in the tutorial - have a look at Row Test Test Type Extension: Registering your test type extension.
If you're having trouble, try downloading the complete code + registry keys from the MSDN code gallery.
You may want to look into using SpecFlow to make your tests more parameterized. It is traditionally a BDD tool, but it will allow you to make re-usable steps that take parameters.
For example you can create a step saying
"Given the user 'admin' exists."
and on the c# side you get a method like this:
[Given(#"the user '(.*)' exists"]
public void GivenTheUserExists(string name)
{
//create the user
}
Going forward you can reuse the step in any "feature file" without having to rewrite the implementation of that step.
Just refer to the blog post Extending Unit Tests in Visual Studio. And see what you might have missed. I tried the same, and it works well for me.
I have dll with tests for gallio.
I specified in "Project Properties \ Debug \ Start External Command": Gallio.Echo.exe
I also specfied "Command Line Arguments": /runner:IsolatedAppDomain Full\Path\To\My\Dll\With\Tests.dll
I specified /runner:IsolatedAppDomain because I want to run tests in the same process which should allow MS VS debugger to automatically connect to tests which are in the same process as Gallio.Echo.exe. (At least for C++ projects it works)
When I execute "Debug \ Start debugging" I obtain the following warning:
[warning] File 'Tests.dll' is not supported by any installed test framework.
It will be ignored.
Location: Full\Path\To\My\Dll\With\Tests.dll
[ignored] Unsupported Tests.dll
Detected a probable test framework assembly version mismatch.
Referenced test frameworks: 'MbUnit, Version=3.1.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e'.
Supported test frameworks: 'MbUnit, Version=3.3.0.0'.
I installed lastest gallio version GallioBundle-3.3.458.0.
It seems that Gallio should have the same framework version as Tests.dll.
But Gallio is 2.0, and I use 4.0.
How to fix the issue?
P.S. Now I use TestDriven.Net. But it is not exactly what I want, it is not convenient to use when I work with one test case but several source files.
TestDriven.Net has option in context menu "Repeat Test Run"