I am checking TestContext.CurrentTestOutcome in my TestCleanup method in order to perform an action if the test did not pass (in this case, the tests are using Selenium to exercise a website and I am saving a screenshot if the test does not pass).
private static TestContext _testContext;
private static IWebDriver _driver;
[ClassInitialize]
public static void SetupTests(TestContext testContext)
{
_testContext = testContext;
_driver = new FirefoxDriver();
}
[TestCleanup]
public void TeardownTest()
{
if (_testContext.CurrentTestOutcome != UnitTestOutcome.Passed)
{
var fileName = Path.Combine(
Environment.CurrentDirectory,
string.Format("{0}.{1}.gif", _testContext.FullyQualifiedTestClassName, _testContext.TestName));
((ITakesScreenshot)driver).GetScreenshot().SaveAsFile(fileName, ImageFormat.Gif);
Console.WriteLine("Test outcome was {0}, saved image of page to '{1}'", _testContext.CurrentTestOutcome, fileName);
}
}
This works well when run on a local development PC using ReSharper, but on our build server (which uses TeamCity) the UnitTestOutcome is always Unknown, although TeamCity reports them as passed.
The documentation on MSDN is not very helpful. What can cause this value to be set to Unknown?
According to http://confluence.jetbrains.com/display/TCD8/MSTest+Support TeamCity does not support on-the-fly reporting of individual test results, it parses the tests results file to provide the results to the build step.
That would explain how TeamCity is able to report the tests as passed even though UnitTestOutcome may be unknown at the time an individual test has completed.
The link above mentions "specifics of MSTest tool" as the reason for non-on-the-fly test result reporting so I can only theorize that the same specifics may mean that TestContext is unavailable when running from your build server.
Also, the MSDN documentation for TestContext.CurrentTestOutcome does mention that Full Trust for the immediate caller is required. TeamCity could be executing the tests in a manner that is only partially trusted and therefore causing the test outcome to be Unknown.
A quick way to check if MSTest is your problem would be to switch to NUnit using:
#if NUNIT
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using IgnoreAttribute = NUnit.Framework.IgnoreAttribute;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
using IgnoreAttribute = Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute;
#endif
source http://www.anotherchris.net/tools/using-team-city-for-staging-and-test-builds-with-asp-net-and-selenium/
You would have to do something similar in your TeardownTest method to use the NUnit TestContext.CurrentContext.Result.Status though.
The fix for this issue is to use a public property for TestContext, rather than using the parameter passed to the [ClassInitialize] method.
i.e.
public TestContext TestContext { get; set; }
The test runner will automatically set the property.
(This is related to another question I posted on SO)
Related
I am working with Selenium WebDriver in C# and I have to create a service for an applicant. I have done this already but after I confirm the service goes to a List ( Services that need to be confirmed from another user ) which is increased by 1 in read mode. Is there any way how to assert these values that are increased by 1 every time a new service is added?
You need to use a test framework to do this -- selenium itself cannot assert for you.
If you are using C#, I recommend installing NUnit. You can find this under NuGet Package manager, and you'll also want to install NUnitTestAdapter if you are using Visual Studio.
Once you have installed a test framework on your project, you can use [Test] flags to designate entry point methods for test cases, and use Assert statements which are part of the NUnit namespace.
You can find documentation here: https://github.com/nunit/docs/wiki/NUnit-Documentation
Selenium's built-in assert functionality only exists in SeleniumIDE, the point-and-click browser add-on available for Chrome and Firefox.
If you are going to write your tests in C#, as Christine said, you need to use a unit testing framework. For example, I'm using Xunit, and a simple test looks like this:
using Xunit; // Testing framework. NuGet package
using OpenQA.Selenium.Firefox; // Driver for Firefox
using Xunit.Priority; // NuGet add-on to Xunit that allows you to order the tests
using OpenQA.Selenium; // NuGet package
using System.Diagnostics; // Can Debug.Print when running tests in debug mode
namespace Test_MyWebPage
{
[TestCaseOrderer(PriorityOrderer.Name, PriorityOrderer.Assembly)] // Set up ordering
public class Test_BasicLogin : IDisposable
{
public static IWebDriver driver = new FirefoxDriver(#"path\to\geckodriver");
// Here be the tests...
[Fact, Priority(0)]
public void Test_LaunchWebsite()
{
// Arrange
var url = "https://yourserver.yourdomain/yourvirtualdir";
// Act
// Sets browser to maximized, allows 1 minute for the page to
// intially load, and an implicit time out of 1 minute for elements
// on the page to render.
driver.Manage().Window.Maximize();
driver.Manage().Timeouts().PageLoad = new TimeSpan(0, 1, 0);
driver.Manage().Timeouts().ImplicitWait = new TimeSpan(0, 1, 0);
driver.url = url; // Launches the browser and opens the page
/* Assuming your page has a login prompt
/* we'll try to locate this element
/* and perform an assertion to test that the page comes up
/* and displays a login prompt */
var UserNamePrompt = driver.FindElement(By.Id("userLogin_txtUserName"));
// Assert
Assert.NotNull(UserNamePrompt); // Bombs if the prompt wasn't found.
Debug.Print("Found User Name Prompt successfully.");
}
public void Dispose()
{
// Properly close the browser when the tests are done
try
{
driver.Quit();
}
catch (Exception ex)
{
Debug.WriteLine($"Error disposing driver: {ex.Message}");
}
}
}
}
As you can see, there is considerably more work in setting up tests for the Selenium WebDriver than in setting up simple smoke tests with the SeleniumIDE. I haven't addressed how you should store your configs properly (hardcoding as in the example is bad) and you will have to tailor your driver.find() statements to fit your precise situation. I am using the Xunit.Priority package so I can make sure the tests don't all run in parallel; I need to test one thing at a time in a progression. Your needs might be met by putting all of the steps into a single Test_* method. Each method appears as a separate test in the Visual Studio Test Explorer window. Right-clicking a test in the Test Explorer and selecting 'Debug selected tests' will allow you to set breakpoints and also enable your Debug.Print (or Debug.Write/Writeline) methods to display in the Tests section of the VS output window.
Another gotcha is in setting up the IWebDriver: don't put the complete path including the executable in the path, just the path to the folder that contains it.
Good luck and happy testing!
This code should takes a screenshot when test fail:
[TestClass]
public class UnitTest1
{
[OneTimeTearDown]
public void TestFail()
{
IWebDriver driver = new ChromeDriver();
if (NUnit.Framework.TestContext.CurrentContext.Result.Outcome != ResultState.Success)
{
string screensLocation = #"D:\";
string testName = NUnit.Framework.TestContext.CurrentContext.Test.Name;
var screenshot = ((ITakesScreenshot)driver).GetScreenshot();
screenshot.SaveAsFile(screensLocation + testName + ".png");
}
}
[TestMethod]
public void TestMethod1()
{
// my code, here test is failed
}
}
But it is not working. I don't have any screen in location D:\
Otherwise is there a way to debug code under OneTimeTearDown Attribute? Because when the test is fail, debugging ends and I don't know what's going on in the method TestFail().
Thanks for your help.
OneTimeTearDownAttribute is a feature of NUnit.
Although your tag says "nunit", your code is not actually using it. TestClassAttribute and TestMethodAttribute are features of MS Test. If you tried to run this test with NUnit, it would not recognize the tests at all.
Obviously, your test assembly does reference the NUnit framework, since it would not otherwise compile.
So... bottom line, your test code references two different frameworks in such a way that it cannot be run successfully by either runner!!! You have to choose which of the two you want to use, remove the other reference and use a runner for the framework you choose to keep.
I have the following basic setup for a uni-test: it tests a class that is responsible for indexing files in a directory, and keep giving the correct one on demand.
To do this I use a mock file system provided by system.io.abstractions library. The basic setup is as follows:
[TestFixture]
public class ImageDataTests
{
private static MockFileSystem fs;
private const int TESTNUM = 3;
[SetUp]
public static void Init() {
var allnamegroups = TestSettings.NameGroupFactory(TESTNUM);
fs = new MockFileSystem();
foreach (string[] namegroup in allnamegroups) {
var dname = ImageData.MakeDirectoryName(namegroup[0], namegroup[1]);
TestSettings.AddTestFilesToMockFileSystem(fs, dname);
}
}
}
Now each test works on a case for this, and to test (say) a "reload" function I add the following method to above class:
public static IEnumerable<ImageData> ImgDatSource() {
while (true) {
yield return new ImageData("test", fs);
}
}
[Test, Sequential]
public void GetAllOfTypeTest(
[ValueSource(nameof(ImgDatSource))] ImageData img,
[Values("foo", "bar", "this")]string type,
[Values(1, 2, 0)]int ex) {
Assert.That(() => img.GetAllOfType(type).Count(), Is.EqualTo(ex));
}
This should (right now) list all files starting with respectively "foo", "bar" and "this". - It fails, stating that the directory is not found. - Even before the first test is run: at the initializing of Imagedata.
Debugging verifies what I thought: the Init() is not run yet when the ValueSource tries to initialize the input.
Is there a way to do "such" a test, short of initializing the Imagedata inside the test body? (And either losing the parametric construction + test indexing, or having to write many similar tests manually).
Notice that I really NEED to run the filesystem generator each test: while above test is simple and doesn't do much. Some tests will adapt/change the filesystem to test the class against more edge-cases. - Also during the test.
NUnit's execution has two phases: loading (discovering) tests and running tests. OneTimeSetUp, SetUp, test method execution, TearDown and OneTimeTearDown are all part of running the test.
Test loading takes place, as you might expect, before running the tests. Your data sources (TestCaseSource or ValueSource) are all run during the load phase.
This leads to two conclusions:
1. It is not possible for anything in your SetUp and other test execution methods to have an effect on the data that is used for a test.
2. Your data sources need to provide data parameters that will allow you to create objects at runtime, not the objects themselves.
In your example, I would change the test method to take four parameters, replacing img with "test" and fs. I would add a line to the start of each test to create the imagedata by calling a helper method modeled after your Init.
I have a suite of SpecFlow tests that are using the MSTest framework. I run them via Microsoft Test manager. What I want to know is if I can get it to attach a file to the run.
My code generates a HTML file that i'd like attached so that users can go into the results for test in MTM and simply open it up.
I saw a previous question had:
TestContext.AddResultFile(testPassedFile);
But when I tried to add TestContext to my "[AfterScenario]" method it doesnt have a method called AddResultFile.
Does anyone know how I might be able to achieve adding this HTML file to the results.
AFAIK there is no way of accessing the TestContext from within the StepBindings:
Access TestContext in SpecFlow Step Binding class
The only way I see is to write your own generator for the tests so that the generated test-code writes the TestContext for example to SpecFlow's ScenarioContext, so that you can access it from the step bindings.
If you want to take all that hassle, you might take a look at https://jessehouwing.net/specflow-custom-unit-test-generator/.
In SpecFlow 3, you can get the TestContext via content injection, and store a reference to it, which can then be accessed in the step definitions:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TechTalk.SpecFlow;
[Binding]
public class Context
{
private static TestContext test_context;
[BeforeScenario]
private static void SetContext(ScenarioContext context)
{
test_context = context.ScenarioContainer.Resolve<TestContext>();
}
public static void Attach(string file_name)
{
test_context.AddResultFile(file_name);
}
}
I'm trying to programmatically check my unit tests are passing as part of my deployment process. The application uses MBunit and Gallio for it's unit testing framework.
Here's my code:
var setup = new Gallio.Runtime.RuntimeSetup();
setup.AddPluginDirectory(#"C:\Program Files\Gallio\bin");
using (TextWriter tw = new StreamWriter(logFilename))
{
var logger = new Gallio.Runtime.Logging.TextLogger(tw);
RuntimeBootstrap.Initialize(setup, logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern(dllToRunFilename);
TestLauncherResult result = launcher.Run();
}
Here's the test which is contained in the DLL I'm loading (I've validated this works with the Icarus test runner):
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
When I run the application I get the following values in results
Which is incorrect as there are indeed tests to run! The log file has the following in it
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin enable
condition was not satisfied. Please note that this is the intended
behavior for plugins that must be hosted inside third party
applications in order to work. Enable condition:
'${process:DEVENV.EXE_V9.0} or ${process:VSTESTHOST.EXE_V9.0} or
${process:MSTEST.EXE_V9.0} or ${framework:NET35}'. Disabled plugin
'Gallio.VisualStudio.Tip90': The plugin depends on another disabled
plugin: 'Gallio.VisualStudio.Shell90'.
How do I resolve this issue and find the results to the tests?
This works for me, note I used this GallioBundle nuget to get gallio and mbunit, so perhaps there is some difference to what you have installed.
The log messages regarding plugins are expected, those plugins wont work if you are self-hosting the Gallio runtime.
using System;
using System.IO;
using Gallio.Runner;
using Gallio.Runtime;
using Gallio.Runtime.Logging;
using MbUnit.Framework;
public static class Program
{
public static void Main()
{
using (TextWriter tw = new StreamWriter("RunTests.log"))
{
var logger = new TextLogger(tw);
RuntimeBootstrap.Initialize(new RuntimeSetup(), logger);
TestLauncher launcher = new TestLauncher();
launcher.AddFilePattern("RunTests.exe");
TestLauncherResult result = launcher.Run();
Console.WriteLine(result.ResultSummary);
}
}
}
public class Tests
{
[Test]
public void Pass()
{
Assert.IsTrue(true);
}
[Test]
public void Fail()
{
Assert.Fail();
}
}
Tested like this:
› notepad RunTests.cs
› nuget.exe install -excludeversion GallioBundle
Installing 'GallioBundle 3.4.14.0'.
Successfully installed 'GallioBundle 3.4.14.0'.
› cd .\GallioBundle\bin
› csc ..\..\RunTests.cs /r:Gallio.dll /r:MbUnit.dll
Microsoft (R) Visual C# Compiler version 12.0.21005.1
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.
› .\RunTests.exe
2 run, 1 passed, 1 failed, 0 inconclusive, 0 skipped
These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.
Firstly, install the NUnit Test Adapter extension (yes, NUnit)
Tools > Extension and Updates > Online > search for NUnit > install
NUnit Test Adapter.
You may need to restart the Visual Studio IDE.
Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...
//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;
namespace MyTests
{
[TestFixture]
public class UnitTest1
{
[Test, NuTest]
public void myTest()
{
//this will pass
}
}
}
You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit