I'm trying to programmatically check my unit tests are passing as part of my deployment process. The application uses MBunit and Gallio for it's unit testing framework.
Here's my code:
var setup = new Gallio.Runtime.RuntimeSetup();
setup.AddPluginDirectory(#"C:\Program Files\Gallio\bin");
using (TextWriter tw = new StreamWriter(logFilename))
{
var logger = new Gallio.Runtime.Logging.TextLogger(tw);
RuntimeBootstrap.Initialize(setup, logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern(dllToRunFilename);
TestLauncherResult result = launcher.Run();
}
Here's the test which is contained in the DLL I'm loading (I've validated this works with the Icarus test runner):
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
When I run the application I get the following values in results
Which is incorrect as there are indeed tests to run! The log file has the following in it
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin enable
condition was not satisfied. Please note that this is the intended
behavior for plugins that must be hosted inside third party
applications in order to work. Enable condition:
'${process:DEVENV.EXE_V9.0} or ${process:VSTESTHOST.EXE_V9.0} or
${process:MSTEST.EXE_V9.0} or ${framework:NET35}'. Disabled plugin
'Gallio.VisualStudio.Tip90': The plugin depends on another disabled
plugin: 'Gallio.VisualStudio.Shell90'.
How do I resolve this issue and find the results to the tests?
This works for me, note I used this GallioBundle nuget to get gallio and mbunit, so perhaps there is some difference to what you have installed.
The log messages regarding plugins are expected, those plugins wont work if you are self-hosting the Gallio runtime.
using System;
using System.IO;
using Gallio.Runner;
using Gallio.Runtime;
using Gallio.Runtime.Logging;
using MbUnit.Framework;
public static class Program
{
public static void Main()
{
using (TextWriter tw = new StreamWriter("RunTests.log"))
{
var logger = new TextLogger(tw);
RuntimeBootstrap.Initialize(new RuntimeSetup(), logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern("RunTests.exe");
TestLauncherResult result = launcher.Run();
Console.WriteLine(result.ResultSummary);
}
}
}
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
Tested like this:
› notepad RunTests.cs
› nuget.exe install -excludeversion GallioBundle
Installing 'GallioBundle 3.4.14.0'.
Successfully installed 'GallioBundle 3.4.14.0'.
› cd .\GallioBundle\bin
› csc ..\..\RunTests.cs /r:Gallio.dll /r:MbUnit.dll
Microsoft (R) Visual C# Compiler version 12.0.21005.1
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.
› .\RunTests.exe
2 run, 1 passed, 1 failed, 0 inconclusive, 0 skipped
These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.
Firstly, install the NUnit Test Adapter extension (yes, NUnit)
Tools > Extension and Updates > Online > search for NUnit > install
NUnit Test Adapter.
You may need to restart the Visual Studio IDE.
Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...
//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;
namespace MyTests
{
[TestFixture]
public class UnitTest1
{
[Test, NuTest]
public void myTest()
{
//this will pass
}
}
}
You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit
Related
When running dotnet test my-tests.dll --list-tests from command line, the utility outputs:
The following tests are available:
MyTest1
MyTest2
However, these tests contain meta-data that I would also like displayed:
[TestFixture]
public class MyTests
{
[Test, Property("some_key", "some_value1")]
public void MyTest1()
{
// ...
}
[Test, Property("some_key", "some_value2")]
public void MyTest2()
{
// ...
}
}
Is there any way that I can display the Property key/values while using the --list-tests flag in dotnet test?
No -- according to the dotnet test CLI documentation, the --list-tests flag lists the discovered test names rather than running them.
If your goal is for your test names to reflect the fact that they are being run against certain values, [TestCase] may be where you want to look.
For example, the following:
public class Tests
{
[TestCase("some_value1")]
[TestCase("some_value2")]
public void MyTestWithSomeKeyOf(string value)
{
Assert.Pass();
// ...
}
}
would produce the following output from dotnet test --list-tests:
PS C:\Users\SeanK\Repositories\2023_01_NunitRepro> dotnet test --list-tests
Determining projects to restore...
All projects are up-to-date for restore.
Repro -> C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll
Test run for C:\Users\SeanK\Repositories\2023_01_NunitRepro\Repro\bin\Debug\net7.0\Repro.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
The following Tests are available:
MyTestWithSomeKeyOf("some_value1")
MyTestWithSomeKeyOf("some_value2")
I am checking TestContext.CurrentTestOutcome in my TestCleanup method in order to perform an action if the test did not pass (in this case, the tests are using Selenium to exercise a website and I am saving a screenshot if the test does not pass).
private static TestContext _testContext;
private static IWebDriver _driver;
[ClassInitialize]
public static void SetupTests(TestContext testContext)
{
_testContext = testContext;
_driver = new FirefoxDriver();
}
[TestCleanup]
public void TeardownTest()
{
if (_testContext.CurrentTestOutcome != UnitTestOutcome.Passed)
{
var fileName = Path.Combine(
Environment.CurrentDirectory,
string.Format("{0}.{1}.gif", _testContext.FullyQualifiedTestClassName, _testContext.TestName));
((ITakesScreenshot)driver).GetScreenshot().SaveAsFile(fileName, ImageFormat.Gif);
Console.WriteLine("Test outcome was {0}, saved image of page to '{1}'", _testContext.CurrentTestOutcome, fileName);
}
}
This works well when run on a local development PC using ReSharper, but on our build server (which uses TeamCity) the UnitTestOutcome is always Unknown, although TeamCity reports them as passed.
The documentation on MSDN is not very helpful. What can cause this value to be set to Unknown?
According to http://confluence.jetbrains.com/display/TCD8/MSTest+Support TeamCity does not support on-the-fly reporting of individual test results, it parses the tests results file to provide the results to the build step.
That would explain how TeamCity is able to report the tests as passed even though UnitTestOutcome may be unknown at the time an individual test has completed.
The link above mentions "specifics of MSTest tool" as the reason for non-on-the-fly test result reporting so I can only theorize that the same specifics may mean that TestContext is unavailable when running from your build server.
Also, the MSDN documentation for TestContext.CurrentTestOutcome does mention that Full Trust for the immediate caller is required. TeamCity could be executing the tests in a manner that is only partially trusted and therefore causing the test outcome to be Unknown.
A quick way to check if MSTest is your problem would be to switch to NUnit using:
#if NUNIT
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using IgnoreAttribute = NUnit.Framework.IgnoreAttribute;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
using IgnoreAttribute = Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute;
#endif
source http://www.anotherchris.net/tools/using-team-city-for-staging-and-test-builds-with-asp-net-and-selenium/
You would have to do something similar in your TeardownTest method to use the NUnit TestContext.CurrentContext.Result.Status though.
The fix for this issue is to use a public property for TestContext, rather than using the parameter passed to the [ClassInitialize] method.
i.e.
public TestContext TestContext { get; set; }
The test runner will automatically set the property.
(This is related to another question I posted on SO)
To use the debug mode in NUnit I added an online template "NUnit Test application". So when I add a new project I choose NUnit test application instead of a class library. When the project gets created two .cs files gets added automatically. I added a simple program to check the debug mode and it shows an error. How to rectify this error? Thanks.
TypeInitializationException was unhandled.
Error occurs at
int returnCode = NUnit.ConsoleRunner.Runner.Main(my_args);
The automatically added files are
Program.cs
namespace NUnitTest1
{
class Program
{
[STAThread]
static void Main(string[] args)
{
string[] my_args = { Assembly.GetExecutingAssembly().Location };
int returnCode = NUnit.ConsoleRunner.Runner.Main(my_args);
if (returnCode != 0)
Console.Beep();
}
}
}
TestFixture.cs
namespace NUnitTest1
{
[TestFixture]
public class TestFixture1
{
[Test]
public void TestTrue()
{
Assert.IsTrue(true);
}
// This test fail for example, replace result or delete this test to see all tests pass
[Test]
public void TestFault()
{
Assert.IsTrue(false);
}
}
}
I added a new item class to it and tried to debug
namespace NUnitTest1
{
[TestFixture]
public class Class1
{
IWebDriver driver = null;
[SetUp]
public void setup()
{
//set the breakpoint here
driver = new FirefoxDriver();
}
[Test]
public void test1()
{
driver.Navigate().GoToUrl("http://www.google.com/");
}
[TearDown]
public void quit()
{
driver.Quit();
}
}
}
As already mentioned by #Arran, you really don't need to do all this. But you can make it even easier to debug NUnit tests.
Using F5 in Visual Studio to debug unit tests
Instead of executing NUnit runner and attaching to the process using Visual Studio, it's better to configure yout test project to start the NUnit test runner and debug your tests. All you have to do is to follow these steps:
Open test project's properties
Select Debug tab
Set Start action to Start external program and point to NUnit runner
Set Command line arguments
Save project properties
And you're done. Hit F5 and your test project will start in debug mode executed by NUnit runner.
You can read about this in my blog post.
You don't need to do all this at all.
Open the NUnit GUI, open up your compiled tests. In Visual Studio, use the Attach to Process feature to attach the nunit-agent.exe.
Run the tests in the NUnit GUI. The VS debugger will take it from there.
You're going through way too much effort to get this done.
What I usually do is to go and create a new "Class Library" project. I then add a reference to the nunin-framework.dll onto my project.
You can define your class as follows:
[TestFixture]
public class ThreadedQuery
{
[Test]
public void Query1()
{
}
}
The TestFixture attribute is described here
You can then go ahead and create multiple Tests with public methods as above.
There are 3 things that are quite important to get this to work then.
You need to set your debugger on your project file to an external executable, ie nunint.exe
The arguments that are passed need to be the name of your assembly.
If you're making use of .net 4.0, you need to specify that in your nunint.exe.config
If you do not do this, you will not be able to debug using VS. See snippet of config below:
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<!--<supportedRuntime version="v2.0.50727" />-->
<supportedRuntime version="v4.0.30319" />
<supportedRuntime version="4.0" />
</startup>
Hope this is helpful
When i try to run my tests from Visual Studio 2012 Ultimate i get this output
------ Discover test started ------
========== Discover test finished: 0 found (0:00:05.8242806) ==========
Here is the code:
[Then(#"the submitter company list is in alphabetical order")]
public void ThenTheSubmitterCompanyListIsInAlphabeticalOrder()
{
List<string> submitterCompanyList = _currentFilingPage.SubmitterCompanyList;
submitterCompanyList.Should().BeInAscendingOrder();
}
I have created a .bat file and from there i can run my tests. Please assits me with a way to run my tests from visual studio. (Extra Information: I can't see my tests on Test Explorer)
All the Tests must have the TestMethodAttribute so MSTest/Visual Studio can find them.
see: MSDN Anatomy of a Unit Test
So edit your code and add the TestMethod to your method so it can be found as a Test. Also your method must be added inside a TestClass:
[TestClass]
public class TestClass
{
[TestMethod]
public void ThenTheSubmitterCompanyListIsInAlphabeticalOrder()
{
}
}
I'm using sonar to check my c# project. I would like to measure code coverage that's why i installed gallio and opencover. When I run soner-runner everything works fine, my unit test is performed, ... but the code coverage is 0% on the sonar web UI.
do you know the reason why the code coverage is 0%?
My solution, project and classes:
(S) SonarTestSolution
(P) ClassLibrary1
(C) Class1.cs
(P) ClassLibrary1NUnitTest
(C) Class1NUnitTest.cs
content of Class1.cs:
public class Class1 {
public String getTestString() { return "abc";}
}
content of Class1NUnitTest.cs:
[TestFixture]
public class Class1NUnitTest {
[Test]
public void createServiceFromFactoryTest() {
Class1 c = new Class1();
Assert.That(c.getTestString(), Is.EqualTo("abc"));
}}
sonar-project.properties:
sonar.projectKey=cs_sonar_test
sonar.projectVersion=1.0-SNAPSHOT
sonar.projectName=C# Sonar Test Project
sonar.sourceEncoding=UTF-8
sources=.
sonar.language=cs
sonar.donet.visualstudio.testProjectPattern=*NUnitTest*
If you want I can include the log og sonar-runner, and the generated coverage-report.xml and gallio-report.xml files
solved - the project was build another machine, this was the problem. Building the solution in the same folder solves the proble.