I am using Microsoft's WinAppDriver in conjunction with Visual Studio 2015 unit tests to automate testing of Windows applications. These tests are being run from the command line using VSTest.exe because I can pass a .runsettings file as a parameter to specify certain test settings that I may have to change in the future. I want to be able to reference the .runsettings file directly from my test methods without needing to create a bunch of global variables at the beginning of my code in the Setup method. Although I am using multiple classes, I am more or less doing it like so:
protected static string basicFile;
[ClassInitialize]
public static void Setup(TestContext context)
{
var basicFile = context.Properties["basic"].ToString();
}
[TestMethod]
public void BasicTest(){
OpenFile(basicFile);
}
Where context.Properties[] references the key in my .runsettings file.
The reason I cannot simply do
[TestMethod]
public void BasicTest(TestContext context){
var basicFile = context.Properties["basic"].ToString();
OpenFile(basicFile);
}
is because test methods cannot accept any parameters. So, is there any way I can directly reference the .runsettings file within the test method without using context.Properties?
I've just switched to C# project from Java and providing .runsettings in our test automation project also gave me a hard time. I also faced the problem of passing TestContext parameter through several classes.
So, my solution may be not so accurate but it worked:
[TestClass]
public class Operations : TestBase
{
public void CreateRelationship()
{
// Add test logic here
}
}
Test Base class:
[TestClass]
public abstract class TestBase
{
[AssemblyInitialize]
public static void ContextInitialize(TestContext context)
{
DriverUtils.Initialize(context);
}
}
And DriverUtils in turn:
public static class DriverUtils
{
private static IWebDriver driver;
private static TestContext testContext;
private static string testEnvironment = string.Empty;
public static void Initialize(TestContext context)
{
testContext = context;
testEnvironment = Convert.ToString(testContext.Properties["TestEnvironmentUrl"]);
}
}
The .runsettings file looks exactly like in the examples, but I left "TestEnvironmentUrl" parameter blank. Then I added .runsettings file to TFS artifacts and later in TFS release config added path to the file in 'Run Functional UI Tests' section. Then I was able to override 'TestEnvironmentUrl' with the real Server URL.
Still haven't found the implementation of [AssemblyInitialize], but I suppose it makes possible to pass the test context from child test classes to parent TestBase.
Related
I've added an integration test project to my solution and I'd like to not run them by default when using the test explorer.
The only solution I've come across to isolate these tests is to manually categorize them and choose not to run tests with a particular trait from test explorer. Ideally, I could exclude them so that people on my project don't have to make this choice explicitly.
Thanks!
There is a special attribute in NUnit to mark your tests that should not be run automatically.
[Explicit]
The Explicit attribute causes a test or test fixture to be skipped unless it is explicitly selected for running.
https://github.com/nunit/docs/wiki/Explicit-Attribute
You just put it on the class or method:
[TestFixture]
[Explicit]
public class IntegrationTests
{
// ...
}
[TestFixture]
public class UnitTests
{
[Test]
public void ShouldNotFail()
{
// This will run
}
[Test]
[Explicit]
public void ManualTest()
{
// This will be ignored
}
}
This is the result:
I'm currently refactoring a medium size application due to some terminology changes in the business terms. We have around 121 SpecFlow feature files that would need to be changed.
I like how you can "deprecate" an API in C#, first as warnings:
[Obsolete("Use MyClass.NewMethod instead")]
public void OldMethod() { }
Then compiler errors:
[Obsolete("Use MyClass.NewMethod instead", true)]
public void OldMethod() { }
It would be nice to have this sort of functionality for SpecFlow steps:
[When("I old foo", Obsolete = true)]
[When("I new foo")]
public void WhenIFoo() { }
Is there any way to mark steps in SpecFlow as obsolete so other developers know the steps need to be changed in their Feature files, but not prevent them from authoring and running tests? And as an added bonus, Is there a way to optionally cause compiler or test run failures?
Starting with version v2.4, SpecFlow respects [Obsolete] attributes on step bindings.
For your example you would probably use it like this:
[When("I old foo")]
[Obsolete("you should use WhenIFoo instead")]
public void WhenIOldFoo() { }
[When("I new foo")]
public void WhenIFoo() { }
As the default behavior for using such steps a warning is issued. Utilizing the obsoleteBehavior attribute on SpecFlow's <runtime> configuration, you can change that, esp. failing test execution:
<specFlow>
...
<runtime obsoleteBehavior="Error" />
</specFlow>
In the meantime the Wiki-page for the Configuration has been updated to document that feature.
I still haven't found a solution directly in SpecFlow, but I did Get It To Work⢠using an extension method to the ScenarioContext:
Note: This uses MsTest, but I'm sure other unit testing frameworks have a way of marking tests as "pending" or inconclusive with a message.
using TechTalk.SpecFlow;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace MyTestProject
{
public static class ScenarioContextExtensions
{
public static void MarkStepObsolete(this ScenarioContext scenario, string newStep, params object[] newStepFormatArgs)
{
var message = string.Format(#"This step is obsolete. Use '" + scenario.CurrentScenarioBlock + " " + newStep + "' instead.", newStepFormatArgs);
Assert.Inconclusive(message);
}
}
}
Now in your step definitions you make the old steps "obsolete" and they show up as inconclusive tests which give you a hint to a replacement step:
[When(#"I click the ""(.*)"" anchor")]
public void WhenIClickTheAnchor(string anchorText)
{
ScenarioContext.Current.MarkStepObsolete(#"I click the ""{0}"" link", anchorText);
}
[When(#"I click the ""(.*)"" link")]
public void WhenIClickTheLink(string linkText)
{
// ...
}
And it shows up in the Test Explorer panel:
Assert.Inconclusive Failed - This step is obsolete. Use 'When I click the "Home" link' instead.
This at least tells other developers how to fix the new tests they were writing while I was changing the terminology in the existing scenarios.
There is a way to to do it with steps Scopes:
[When("I old foo", Scope(Tag = "Obsolete"))]
[When("I new foo")]
public void WhenIFoo() { }
Ensure the tag(its a regex) is unique because the idea is to not match any of the scenarios tagged.
What will happen is that your test will be marked as Not Run and more importantly the scenarios using this step will get a pink step so it will be visible compile time.
If you want to mark as obsolete the whole class then just simply tag the class as follows:
[Binding, Scope(Tag = "Obsolete")]
public class Steps
Reference:
https://github.com/techtalk/SpecFlow/wiki/Scoped-bindings#scoping-tips--tricks
I have a suite of SpecFlow tests that are using the MSTest framework. I run them via Microsoft Test manager. What I want to know is if I can get it to attach a file to the run.
My code generates a HTML file that i'd like attached so that users can go into the results for test in MTM and simply open it up.
I saw a previous question had:
TestContext.AddResultFile(testPassedFile);
But when I tried to add TestContext to my "[AfterScenario]" method it doesnt have a method called AddResultFile.
Does anyone know how I might be able to achieve adding this HTML file to the results.
AFAIK there is no way of accessing the TestContext from within the StepBindings:
Access TestContext in SpecFlow Step Binding class
The only way I see is to write your own generator for the tests so that the generated test-code writes the TestContext for example to SpecFlow's ScenarioContext, so that you can access it from the step bindings.
If you want to take all that hassle, you might take a look at https://jessehouwing.net/specflow-custom-unit-test-generator/.
In SpecFlow 3, you can get the TestContext via content injection, and store a reference to it, which can then be accessed in the step definitions:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TechTalk.SpecFlow;
[Binding]
public class Context
{
private static TestContext test_context;
[BeforeScenario]
private static void SetContext(ScenarioContext context)
{
test_context = context.ScenarioContainer.Resolve<TestContext>();
}
public static void Attach(string file_name)
{
test_context.AddResultFile(file_name);
}
}
I am checking TestContext.CurrentTestOutcome in my TestCleanup method in order to perform an action if the test did not pass (in this case, the tests are using Selenium to exercise a website and I am saving a screenshot if the test does not pass).
private static TestContext _testContext;
private static IWebDriver _driver;
[ClassInitialize]
public static void SetupTests(TestContext testContext)
{
_testContext = testContext;
_driver = new FirefoxDriver();
}
[TestCleanup]
public void TeardownTest()
{
if (_testContext.CurrentTestOutcome != UnitTestOutcome.Passed)
{
var fileName = Path.Combine(
Environment.CurrentDirectory,
string.Format("{0}.{1}.gif", _testContext.FullyQualifiedTestClassName, _testContext.TestName));
((ITakesScreenshot)driver).GetScreenshot().SaveAsFile(fileName, ImageFormat.Gif);
Console.WriteLine("Test outcome was {0}, saved image of page to '{1}'", _testContext.CurrentTestOutcome, fileName);
}
}
This works well when run on a local development PC using ReSharper, but on our build server (which uses TeamCity) the UnitTestOutcome is always Unknown, although TeamCity reports them as passed.
The documentation on MSDN is not very helpful. What can cause this value to be set to Unknown?
According to http://confluence.jetbrains.com/display/TCD8/MSTest+Support TeamCity does not support on-the-fly reporting of individual test results, it parses the tests results file to provide the results to the build step.
That would explain how TeamCity is able to report the tests as passed even though UnitTestOutcome may be unknown at the time an individual test has completed.
The link above mentions "specifics of MSTest tool" as the reason for non-on-the-fly test result reporting so I can only theorize that the same specifics may mean that TestContext is unavailable when running from your build server.
Also, the MSDN documentation for TestContext.CurrentTestOutcome does mention that Full Trust for the immediate caller is required. TeamCity could be executing the tests in a manner that is only partially trusted and therefore causing the test outcome to be Unknown.
A quick way to check if MSTest is your problem would be to switch to NUnit using:
#if NUNIT
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using IgnoreAttribute = NUnit.Framework.IgnoreAttribute;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
using IgnoreAttribute = Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute;
#endif
source http://www.anotherchris.net/tools/using-team-city-for-staging-and-test-builds-with-asp-net-and-selenium/
You would have to do something similar in your TeardownTest method to use the NUnit TestContext.CurrentContext.Result.Status though.
The fix for this issue is to use a public property for TestContext, rather than using the parameter passed to the [ClassInitialize] method.
i.e.
public TestContext TestContext { get; set; }
The test runner will automatically set the property.
(This is related to another question I posted on SO)
To use the debug mode in NUnit I added an online template "NUnit Test application". So when I add a new project I choose NUnit test application instead of a class library. When the project gets created two .cs files gets added automatically. I added a simple program to check the debug mode and it shows an error. How to rectify this error? Thanks.
TypeInitializationException was unhandled.
Error occurs at
int returnCode = NUnit.ConsoleRunner.Runner.Main(my_args);
The automatically added files are
Program.cs
namespace NUnitTest1
{
class Program
{
[STAThread]
static void Main(string[] args)
{
string[] my_args = { Assembly.GetExecutingAssembly().Location };
int returnCode = NUnit.ConsoleRunner.Runner.Main(my_args);
if (returnCode != 0)
Console.Beep();
}
}
}
TestFixture.cs
namespace NUnitTest1
{
[TestFixture]
public class TestFixture1
{
[Test]
public void TestTrue()
{
Assert.IsTrue(true);
}
// This test fail for example, replace result or delete this test to see all tests pass
[Test]
public void TestFault()
{
Assert.IsTrue(false);
}
}
}
I added a new item class to it and tried to debug
namespace NUnitTest1
{
[TestFixture]
public class Class1
{
IWebDriver driver = null;
[SetUp]
public void setup()
{
//set the breakpoint here
driver = new FirefoxDriver();
}
[Test]
public void test1()
{
driver.Navigate().GoToUrl("http://www.google.com/");
}
[TearDown]
public void quit()
{
driver.Quit();
}
}
}
As already mentioned by #Arran, you really don't need to do all this. But you can make it even easier to debug NUnit tests.
Using F5 in Visual Studio to debug unit tests
Instead of executing NUnit runner and attaching to the process using Visual Studio, it's better to configure yout test project to start the NUnit test runner and debug your tests. All you have to do is to follow these steps:
Open test project's properties
Select Debug tab
Set Start action to Start external program and point to NUnit runner
Set Command line arguments
Save project properties
And you're done. Hit F5 and your test project will start in debug mode executed by NUnit runner.
You can read about this in my blog post.
You don't need to do all this at all.
Open the NUnit GUI, open up your compiled tests. In Visual Studio, use the Attach to Process feature to attach the nunit-agent.exe.
Run the tests in the NUnit GUI. The VS debugger will take it from there.
You're going through way too much effort to get this done.
What I usually do is to go and create a new "Class Library" project. I then add a reference to the nunin-framework.dll onto my project.
You can define your class as follows:
[TestFixture]
public class ThreadedQuery
{
[Test]
public void Query1()
{
}
}
The TestFixture attribute is described here
You can then go ahead and create multiple Tests with public methods as above.
There are 3 things that are quite important to get this to work then.
You need to set your debugger on your project file to an external executable, ie nunint.exe
The arguments that are passed need to be the name of your assembly.
If you're making use of .net 4.0, you need to specify that in your nunint.exe.config
If you do not do this, you will not be able to debug using VS. See snippet of config below:
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<!--<supportedRuntime version="v2.0.50727" />-->
<supportedRuntime version="v4.0.30319" />
<supportedRuntime version="4.0" />
</startup>
Hope this is helpful