I am looking at setting up SpecFlow for various levels of tests, and as part of that I want to be able to filter which tests run.
For example, say I want to do a full GUI test run, where I build up the dependencies for GUI testing on a dev environment and run all the specs tagged #gui, with the steps executed through the gui. Also from the same script I want to run only the tests tagged #smoke, and set up any dependencies needed for a deployed environment, with the steps executed through the api.
I'm aware that you can filter tags when running through the specflow runner, but I need to also change the way each test works in the context of the test run. Also I want this change of behaviour to be switched with a single config/command line arg when run on a build server.
So my solution so far is to have build configuration for each kind of test run, and config transforms so I can inject behaviour into specflow when the test run starts up. But I am not sure of the right way to filter by tag as well.
I could do somethig like this:
[BeforeFeature]
public void CheckCanRun()
{
if(TestCannotBeRunInThisContext())
{
ScenarioContext.Current.Pending();
}
}
I think this would work (it would not run the feature) but the test would still come up on my test results, which would be messy if I'm filtering out most of the tests with my tag. If there a way I can do this which removes the feature from running entirely?
In short, no I don't think there is anyway to do what you want other than what you have outlined above.
How would you exclude the tests from being run if they were just normal unit tests?
In ReSharper's runner you would probably create a test session with only the tests you wanted to run in. On the CI server you would only run tests in a specific dll or in particular categories.
Specflow is a unit test generation tool. It generates unit tests in the flavour specified in the config. The runner still has to decide which of those tests to run, so the same principles of choosing the tests to run above applies to specflow tests.
Placing them into categories and running only those categories is the simplest way, but having a more fine grained programmatic control of that is not really applicable. What you are asking to do is basically like saying 'run this test, but let me decide in the test if I want it to run' which doesn't really make sense.
Related
Is it possible to determine how many tests have been selected to execute before the test runner executes them? This would be helpful for local testing since our logic generates test configuration for every test suite configuration at a time, having the ability to figure out which tests have been selected to execute could allow me to create logic to only create test data configuration for those tests.
This would be useful when writing tests and testing that they work. Since we only want to generate test data for the selected test.
Right now we have to comment out code to disable it from executing the test data configuration.
Thanks.
I think you are overthinking this a bit. Your setup can be done with a combination of splitting the setup work between the whole test assembly setup that only runs once, namespace wide setup that runs once before any test in the namespace is run, the constructor for a test fixture, the start of an actual test, etc.
If you are reusing a docker instance and app pool for all the tests then initialize it in the whole assembly setup so that it is only done once. Then each test can just add whatever data it needs before it starts. If some of that data is shared between tests then just setup global flags to indicate what has been done already and if some data a test needs hasn't been setup then just do the incremental additional setup needed before continuing with that test, but this generally isn't required if you organize your tests into namespaces properly and just use the namespace wide setup for fixtures.
I'm using NUnit.ConsoleRunner.3.8.0 to run NUnit 3.10.1 tests.
The problem is: if there is specific tests in run filter, I should properly configure my SUT. It is quite painful process, so I would like to do it only if some specific test should be ran.
Is any way to receive list of tests to be ran by console runner, ideally in SetUpFixture?
If any tests in the same namespace (or descendants) as the SetUpFixture are selected, the the SetUpFixture will be run. If none are selected, then it will not be run.
Since this is how SetUpFixtures work, you should organize your tests so that only those that need this configuration step are in the namespaces covered by the SetUpFixture.
In my experience working with teams, I have found that they are sometimes hampered by standards (imposed or self-chosen) that require the test namespaces to conform to a particular design. This is a bad idea when using a system like NUnit that depends on the namespace structure to control how tests are executed.
I like the convenience of test runners to run a single test method. Is there a way to run any method without making it a test?
The reason being I'm tired of writing Console apps with a CLI menu or a Windows Form with a button per method only for the sake of running various utility code that doesn't need to be realized in a UI. A parameterized Console app is too much effort too.
I have multiple methods. They don't rely on each other. Each completes a task that I as a developer run.
Ideally I'd like to have a Visual Studio .csproj file with classes of individually runnable methods that would not be picked up by the test framework, but that I could choose to run from a tool like a test runner.
You can just do a method invocation in the Interactive Window in Visual Studio. You'll need type the fully qualified name out, like MyNamespace.MyType.MyMethod("some-arg"). If it's not static then you'll need to new it up first of course.
If you want more convenience and don't mind paying for it then ReSharper has the ability to right click and run/debug static methods
. Alive also has this feature.
Finally, if you want to use a test runner, but don't want these to be run with your "regular" tests, NUnit has the Explicit attribute, which means the tests only run when you explicitly run them. Other testing frameworks have concepts that are close.
I'm implementing tests written in Gherkin using SpecFlow. The tests are very high level, for example:
Given Youtube's watch Page
When I click play
Then I see an ad
which can be implemented into a UI test or a unit test.
The end result is that in the same project I have a mix of UI tests and unit tests. This creates a problem when running the project, due to the UI tests being very slow compared to unit tests.
My question is, how should specification by example be implemented in the UI tests VS Unit tests context?
You should always have this kind of tests in its own project and they are strictly integration tests. This gives you option to run real unit tests all the time and run these tests frequently but not every time. And these tests are by definition integration tests, i.e. going through multiple layers without mocking.
It does not mean for every single scenario you need to go through a browser (or a headless browser) though.
I have three unit tests that cannot pass when run from the build server—they rely on the login credentials of the user who is running the tests.
Is there any way (attribute???) I can hide these three tests from the build server, and run all the others?
Our build-server expert tells me that generating a vsmdi file that excludes those tests will do the trick, but I'm not sure how to do that.
I know I can just put those three tests into a new project, and have our build-server admin explicitly exclude it, but I'd really love to be able to just use a simple attribute on the offending tests.
You can tag the tests with a category, and then run tests based on category.
[TestCategory("RequiresLoginCredentials")]
public void TestMethod() { ... }
When you run mstest, you can specify /category:"!RequiresLoginCredentials"
There is an IgnoreAttribute. The post also lists the other approaches.
Others answers are old.
In modern visual studio (2012 and above), tests run with vstest and not mstest.
New command line parameter is /TestCaseFilter:"TestCategory!=Nightly"
as explained in this article.
Open Test->Windows->Test List Editor.
There you can include / hide tests
I figured out how to filter the tests by category in the build definition of VS 2012. I couldn't find this information anywhere else.
in the Test Case Filter field under Test Source, under Automated Tests, under the Build process parameters, in the Process tab you need to write TestCategory=MyTestCategory (no quotes anywhere)
Then in the test source file you need to add the TestCategory attribute. I have seen a few ways to do this but what works for me is adding it in the same attribute as the TestMethod as follows.
[TestCategory("MyTestCategory"), TestMethod()]
Here you do need the quotes
When I run unit tests from VS build definition (which is not exactly MSTest), in the Criteria tab of Automated Tests property I specify:
TestCategory!=MyTestCategory
All tests with category MyTestCategory got skipped.
My preferred way to do that is to have 2 kinds of test projects in my solution : one for unit tests which can be executed from any context and should always pass and another one with integration tests that require a particular context to run properly (user credential, database, web services etc.). My test projects are using a naming convention (ex : businessLogic.UnitTests vs businessLogic.IntegrationTests) and I configure my build server to only run unit tests (*.UnitTests). This way, I don't have to comment IgnoreAttributes if I want to run the Integration Tests and I found it easier than editing test list.