I tried extending extending ICommandWrapper, following this article: https://www.skyrise.tech/blog/tech/extending-nunit-3-with-command-wrappers/. I figured out that I can also extend TestAttribute and it just works, then I tried extending TestCaseAttribute:
[AttributeUsage(AttributeTargets.Method), AllowMultiple = true]
public class MyTestCaseAttribute : TestCaseAttribute, IWrapSetUpTearDown
{
private object[] _args;
public MyTestCaseAttribute(params object[] args) : base(args)
{
_args = args;
}
public TestCommand Wrap(TestCommand command)
{
return new MyTestCommand(command, _args);
}
}
MyTestCommand extends DelegatingTestCommand, just like in the article.
The problem is, if I add multiple MyTestCaseAttributes to a test method, the test method gets wrapped by MyTestCommand.Execute's code multiple times.
[EDIT] Example:
Suppose MyTestCommand looks like this:
public abstract class MyCommandDecorator : DelegatingTestCommand
{
public override TestResult Execute(TestExecutionContext context)
private object[] _testCaseArgs;
protected TestCommandDecorator(TestCommand innerCommand, params object[] args) : base(innerCommand)
{
_testCaseArgs = args;
}
public override TestResult Execute(TestExecutionContext context)
{
DoSomething(_testCaseArgs);
return context.CurrentResult = innerCommand.Execute(context);
}
}
Suppose I decorate a test method with two [MyTestCase] attributes:
[MyTestCase(1)]
[MyTestCase(2)]
public void MyTest(int foo)
{
//...
}
The desired behaviour is something like:
DoSomething(1);
MyTest(1);
DoSomething(2);
MyTest(2);
But actual behaviour is:
DoSomething(2)
DoSomething(1)
MyTest(1)
DoSomething(2)
DoSomething(1)
MyTest(1)
The key to your problem is this... C# allows you to decorate a method or a class with an attribute. But an individual test case doesn't exist outside of NUnit - there is no C# equivalent - so you can't decorate it.
IOW your two Attributes apply to the method and cause NUnit to use that method to generate two test cases. However, your attributes also implement ICommandWrapper, which causes NUnit to wrap any test cases it generates. One part of NUnit is looking for test cases to create another part is looking for attributes to wrap test cases. Those two parts are entirely separated.
That's why NUnit uses properties on the test case method to indicate things like Ignoring the case. It can't use an attribute because an attribute would apply to every test case generated by that method.
Hopefully, that explains what's happening.
To get past the problem, your command wrapper should only apply itself to a test that was generated by that particular instance of the attribute. That means you have to get involved in the creation of the test, at least to the extent that your attribute remembers the reference to the test it created. This is a bit complicated, but you should look at the code for TestCaseAttribute to see how the test case is created.
Figured it out.
Instead of extending TestCaseAttribute, I can extend TestAttribute and obtain the arguments to pass to the wrapper class from standard TestCaseAttributes using command.Test.Arguments.
[AttributeUsage(AttributeTargets.Method), AllowMultiple = true]
public class MyTestAttribute : TestAttribute, IWrapSetUpTearDown
{
public TestCommand Wrap(TestCommand command)
{
return new MyTestCommand(command, command.Test.Arguments);
}
}
[TestCase(1)]
[TestCase(2)]
[MyTest]
public void MyTest(int foo)
{
//...
}
Related
I have the following hook in my test project:
[AfterScenario]
[Scope(Tag = "Tag1"), Scope(Tag = "Tag2")]
public static void AfterScenarioMethod()
{
}
Based on the SpecFlow documentation, I should expect to enter AfterFeatureMethod() only if the ScenarioContext.Current.ScenarioInfo.Tags includes both of the required tags. However, the method is executed even when only Tag1 is available.
Am I missing something here?
First AfterScenario is not static.
If you define multiple properties on one attribute, they are combined with AND.
If you have multiple attributes, they are combined with OR.
From documentation: https://specflow.org/documentation/Scoped-Bindings/
If multiple [Scope] attributes are defined for the same method or class, the attributes are combined with OR, i.e. at least one of the [Scope] attributes needs to match.
To check for multiple tags you have to do following:
[Binding]
public class Bindings
{
private ScenarioContext _scenarioContext;
public Bindings(ScenarioContext scenarioContext)
{
_scenarioContext = scenarioContext;
}
[AfterScenario]
public static void AfterScenarioMethod()
{
if (_scenarioContext.ScenarioInfo.Tags.Contains("Tag1") &&
_scenarioContext.ScenarioInfo.Tags.Contains("Tag2") {
//do you stuff
}
}
}
Code is written from memory, I didn't tried it out.
I heavily use the Autofixture AutoData Theories for creating my data and mocks. However this prevents me from using the InlineData Attributes from XUnit to pipe in a bunch of different data for my tests.
So I am basically looking for something like this:
[Theory, AutoMoqDataAttribute]
[InlineData(3,4)]
[InlineData(33,44)]
[InlineData(13,14)]
public void SomeUnitTest([Frozen]Mock<ISomeInterface> theInterface, MySut sut, int DataFrom, int OtherData)
{
// actual test omitted
}
Is something like this possible?
You'll have to create your own InlineAutoMoqDataAttribute, similar to this:
public class InlineAutoMoqDataAttribute : InlineAutoDataAttribute
{
public InlineAutoMoqDataAttribute(params object[] objects) : base(new AutoMoqDataAttribute(), objects) { }
}
and you'd use it like this:
[Theory]
[InlineAutoMoqData(3,4)]
[InlineAutoMoqData(33,44)]
[InlineAutoMoqData(13,14)]
public void SomeUnitTest(int DataFrom, int OtherData, [Frozen]Mock<ISomeInterface> theInterface, MySut sut)
{
// actual test omitted
}
Note that the inlined data, the ints in this case, must be the first parameters of the test method.
All the other parameters will be provided by AutoFixture.
With the latest AutoFixture, you can use Inline AutoData Theories
Uses the InlineData values for the the first method arguments, and then uses AutoData for the rest (when the InlineData values run out).
Is it possible to skip all tests from a specific class like in NUnit
[TestFixture]
[Ignore("Reason")]
public class TestClass {
}
No - there is no such facility at present, and the last time it was requested it was considered too low value to add,
One quick way of achieving the effect in xUnit is to comment out the public - private classes are not reflected over (obviously it won't appear on the skip list that way though).
UPDATE: Another way is to put a TraitAttribute on the class and then (assuming you're using the xunit.console runner) filter it out by running with /-trait traitName. (e.g. you can achieve ExplicitAttribute, some aspects of the BDD frameworky technique of Pending tests and similar semantics that way - of course the big problem is they don't show up in any reports when using any of these filtering techniques)
UPDATE 2: You can do
const string skip = "Class X disabled";
[Fact(Skip=skip)]
void Test() {}
Then you can change to to const string skip = null to undo the skip. The (dis)advantage of this is that the test is still shown as a Skipped test in the test list, generally with a reason included in the test run report (vs making it private which makes it likely to be forgotten)
Here is my hack to avoid error xUnit1000: Test classes must be public (checked on single Fact, I think Theories can be hacked this way, too).
// Uncomment to enable tests
//public class FactSwitch : FactAttribute { } // public! ahh, a bug!
// Uncomment to disable tests
internal class FactSwitch : Attribute { }
public class MyTests
{
[FactSwitch]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
(3 years later)
While searching for the same solution I found there are better ways to do the same.
Let's rewrite the example above in a way Ruben Bartelink suggested (continuation of his idea).
public class MyTests
{
//const string SkipOrNot = null; // Run all tests
const string SkipOrNot = "reason"; // Skip all tests
[Fact(Skip = SkipOrNot)]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
Nathan Cooper suggested a good improvement for my idea:
public class MyTests
{
// Uncomment to disable tests
//private class FactAttribute : Attribute { }
[Fact]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
So I like both ideas from Ruben and Nathan. There is a subtle difference between using Skip="something" (Ruben) and not using Skip at all. Using "Skip" will put all your tests in a "Skipped tests" warning zone, while "FactAttribute : Attribute" will hide them.
I've found yet another way of temporary disabling entire class without compiler warning.
Disabled:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
/*
public /**/class DatabaseTests
{
}
to enable move the /* one line up (i.e. using alt+up):
/*
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
public /**/class DatabaseTests
{
}
Note that using full namespace path for SupressMessage does not mess up with your usings.
You need to set the your class access level as as internal and surpress message as #Miq did:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]
internal class MyClassThatIsNotATestClass
{ ... }
You can create LocalOnlyFactAttribute
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => base.Skip = value; }
}
As far as I know, the simplest way to dynamically skip a whole xUnit test class at runtime is to use the TestFrameworkAttribute at the assembly level, to point to a class that implements the ITestFramework interface (or inherits from XunitTestFramework, which is simpler) and which overrides the CreateDiscoverer() method to return another class, that implements the ITestFrameworkDiscoverer interface (or inherits from XunitTestFrameworkDiscoverer, which is simpler), where you can finally override the IsValidTestClass() method, to decide whether a class should be skipped or not.
Here is some sample code:
[assembly: TestFramework("MyNamespace.Xunit.MyTestFramework", "MyAssembly")]
namespace MyNamespace.Xunit
{
public class MyTestFramework : XunitTestFramework
{
public MyTestFramework(IMessageSink messageSink)
: base(messageSink)
{
}
protected override ITestFrameworkDiscoverer CreateDiscoverer(
IAssemblyInfo assemblyInfo)
=> new MyTestFrameworkDiscoverer(
assemblyInfo,
SourceInformationProvider,
DiagnosticMessageSink);
}
public class MyTestFrameworkDiscoverer : XunitTestFrameworkDiscoverer
{
public MyTestFrameworkDiscoverer(
IAssemblyInfo assemblyInfo,
ISourceInformationProvider sourceProvider,
IMessageSink diagnosticMessageSink,
IXunitTestCollectionFactory collectionFactory = null)
: base(
assemblyInfo,
sourceProvider,
diagnosticMessageSink,
collectionFactory)
{
}
protected override bool IsValidTestClass(ITypeInfo type)
=> base.IsValidTestClass(type) &&
FilterType(type);
protected virtual bool FilterType(ITypeInfo type)
{
// Insert your custom filter conditions here.
return true;
}
}
}
Tested with xUnit 2.4.1.
We are using it in Pomelo.EntityFrameworkCore.MySql (see AssemblyInfo.cs and MySqlXunitTestFrameworkDiscoverer.cs) (a bit more complex than the sample code here).
You could achieve this through a custom ITestClassCommand.
See http://mariangemarcano.blogspot.be/2010/12/xunitnet-running-tests-testcategory.html
Here's another hack that requires minimal changes to code
using FactAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
using TheoryAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
Any compatible attribute can be used for the replacement.
If you also use the InlineDataAttribute then you'll need to define a replacement as I don't think there's an existing compatible attribute.
using InlineDataAttribute = DummyDataAttribute;
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
internal class DummyDataAttribute : Attribute
{
public DummyDataAttribute(params object[] data)
{
}
}
Adding a reason almost after one year after the initial question. I have a set of tests which are calling real server apis, and I would like to run then on demand. With nUnit, it has Ignore attribute : with that set, test runner will skip those tests, but I can still manually run it.
xUnit has no such feature. The nearest one is setting such a class level attribute, and comment it out when I want to run it.
Consider creating LocalOnlyFactAttribute, which can be reused across multiple test files.
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => this.Skip = value; }
}
When I try to use approvals with my unit test decorated with [Theory] attribute it says:
System.Exception: System.Exception : Approvals is not set up to use your test framework.
It currently supports [NUnit, MsTest, MbUnit, xUnit.net]
To add one use ApprovalTests.StackTraceParsers.StackTraceParser.AddParser() method to add implementation of ApprovalTests.StackTraceParsers.IStackTraceParser with support for your testing framework.
To learn how to implement one see http://blog.approvaltests.com/2012/01/creating-namers.html
at ApprovalTests.StackTraceParsers.StackTraceParser.Parse(StackTrace stackTrace)
at ApprovalTests.Namers.UnitTestFrameworkNamer..ctor()
at ApprovalTests.Approvals.GetDefaultNamer()
at ApprovalTests.Approvals.Verify(IApprovalWriter writer)
at ApprovalTests.Approvals.Verify(Object text)
Seems like it recognizes [Fact] attributes only. I tried to follow the link from stacktrace but there is nothing about how to plug in your own namer/parser into approvals.
Is there any entry point where I can add my own namer/parser? Itself it seems to be trivial, the only question is how to use it:
public class TheoryNamer : AttributeStackTraceParser
{
protected override string GetAttributeType()
{
return typeof(TheoryAttribute).FullName;
}
public override string ForTestingFramework
{
get { return "xUnit Extensions"; }
}
}
There are a couple parts to this answer & question.
How to add
The Namer
Dealing with data driven tests in approvaltests
1) How to add
Adding is simple (if a bit rough)
The method mentioned should have been static, but it works none the less.
To add one use
ApprovalTests.StackTraceParsers.StackTraceParser.AddParser() method to
add implementation of
ApprovalTests.StackTraceParsers.IStackTraceParser with support for
your testing framework.
so you'll need to do a
new StackTraceParser().AddParser(new TheoryNamer());
I apologize for this, and it will be static in the next version (v.21)
2) The Namer
The Namer is suppose to generate a unique name for each approved/received file. This is normally done on the name of the method, however the name here will not be unique as a theory based test will be data driven and therefore have multiple calls to the same method.
Naming: classname.methodname(optional: .additionalInformation).received.extension
As such, you will probably have to include additional information in the method it's self
public class StringTests1
{
[Theory,
InlineData("goodnight moon"),
InlineData("hello world")]
public void Contains(string input)
{
NamerFactory.AdditionalInformation = input; // <- important
Approvals.Verify(transform(input));
}
}
3) Dealing with data driven tests in approvaltests
To be honest, in most cases, the data driven method of approach in Approval Tests isn't thru parameters in the Method Decorators. It is usually thru the VerifyAll with a lambda transform. For Example the above might look like
[Fact]
public void UpperCase()
{
var inputs = new[]{"goodnight moon","hello world"};
Approvals.VerifyAll(inputs, i => "{0} => {1}".FormatWith(i, i.ToUpperInvariant()));
}
Which would create the received file:
goodnight moon => GOODNIGHT MOON
hello world => HELLO WORLD
It's better to inherit TheoryNamer class from XUnitStackTraceParser.
It works perfect!
I think it would be cool to add such class into ApprovalTests.StackTraceParsers namespace :)
public class XUnitTheoryStackTraceParser : XUnitStackTraceParser
{
public const string TheoryAttribute = "Xunit.Extensions.TheoryAttribute";
protected override string GetAttributeType()
{
return TheoryAttribute;
}
}
public class ApproveTheoryTest
{
static ApproveTheoryTest()
{
StackTraceParser.AddParser(new XUnitTheoryStackTraceParser());
}
[Theory]
[UseReporter(typeof(DiffReporter))]
[InlineData("file1.txt")]
[InlineData("file2.txt")]
public void approve_file(string fileName)
{
NamerFactory.AdditionalInformation = fileName;
Approvals.Verify("sample text");
}
}
I'm trying to use selenium to run tests; seems like there isn't a good way to run the same set of unit tests on multiple browsers.
I read this post about running tests in parallel:
http://slmoloch.blogspot.com/2009/12/design-of-selenium-tests-for-aspnet_19.html
However, I'm using the visual studio unit testing framework.
I can create a proxy class like this:
public class SeleniumProxy {
private List<DefaultSelenium> targets;
public SeleniumProxy() {
targets = new List<DefaultSelenium>();
targets.Add(new DefaultSelenium(... "firefox"...));
targets.Add(new DefaultSelenium(... "iexplore"...));
}
public void Open(String url) {
foreach (var i in targets) {
i.Open(url);
}
}
...
}
My question is this? How can I do it without having to rewrite the entire class as a proxy?
I thought maybe passing a lamda in to map arguments, or passing by a function that takes the name of the method to invoke, but these all seem like pretty lame ideas.
What I really want is to add a member like:
public class SeleniumProxy {
public dynamic proxy;
....
}
And invoke this like:
var selenium = getProxy();
selenium.proxy.Open("...");
Does c# allow this kind of syntax for dynamic objects?
Or some kind of meta-handler for classes that lets them catch no-such-method exceptions and handle them manually?
Basically:
How can I create a proxy object that dynamically invokes methods on an internal member of the class?
(Edit: perhaps... using reflection on the DefaultSelenium object and creating function stubs on the dynamic proxy object for each entry..?)
If I understand what you're attempting, I think you can extend DynamicObject to achieve this.
class Proxy : System.Dynamic.DynamicObject
{
public Proxy(object someWrappedObject) { ... }
public override bool TryInvokeMember(System.Dynamic.InvokeMemberBinder binder, object[] args, out object result)
{
// Do whatever, binder.Name will be the called method name
}
}
//Do whatever... would become some code that pokes at some other object's internal members (via reflection, presumably) using binder.Name as part of the lookup process.
There are TryGetMember and TryGetIndex methods to override if you need to wrap up anything fancier that simple method invokes.
You will have to cast instances of Proxy to dynamic after construction to make arbitrary calls, just like when dealing with ExpandoObject.
You could leverage inheritance and have your tests defined in an abstract base class with a factory method to create your selenium instance, then inherit this for each type of browser you want to model. The tests will then be run for each inherited class with the appropriate browser. Using NUnit as an example:
public abstract class AbstractTests
{
protected abstract DefaultSelenium CreateSelenium();
[Test]
public void TestSomethingA()
{
DefaulSelenium selenium = CreateSelenium();
//Do some testing with selenium.
}
}
[TestFixture]
public class IETests : AbstractTests
{
protected override DefaultSelenium CreateSelenium()
{
return new DefaultSelenium("iexplore");
}
}
[TestFixture]
public class FirefoxTests : AbstractTests
{
protected override DefaultSelenium CreateSelenium()
{
return new DefaultSelenium("firefox");
}
}