I've got Visual Studio 2010, and we have two VS solutions we work with. The first is the web application, and the second is strictly for SpecFlow tests. Having two instances of Visual Studio running at the same time just to run SpecFlow features is eating all the available RAM causing things to slow down.
I've done some searching on Google and here on StackOverflow, plus perused the MS documentation on the MSTest command line tool, but I haven't found the answer. The full SpecFlow test suite takes ~45 minutes to complete, and I really only need to run a few scenarios.
I was wondering if there is a way to run individual SpecFlow features, and even individual scenarios, from the command line using MSTest?
Behind the scene specflow tests are just regular mstest unit tests. So you should be able to run them the same way using something like:
To run a specific scenario:
mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff
To run a several specific scenario you can use the /test flag multiple times:
mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff /test:GivenMyScenarioWhenIDoSomemthingElse
To run a feature
mstest /testcontainer:tests.dll /test:MyFeatureName
If you add tags on your scenarios using #MyTag for example, you could also use the option
/category:MyTag to filter down the scenarios to run.
Please have a look to the generated code behind of your feature files to get a clue of how things actually work, if you are familliar with mstest it should be pretty straightforward.
Now that SpecFlow 3.0 has been released we can use SpecFlow with .NET Core. The CLI tool for .NET Core is dotnet and tests are run like this if you use MSTest (vstest):
dotnet test
If the tests are in a specific project you can specify the project like this
dotnet test TestProject
where TestProject is the name of the project. You can skip the project name if you want to, but specifying it will make dotnet look in only that project. To list all the tests in the project you can use the -t flag:
dotnet test TestProject -t
To run only specific tests you can use the --filter flag:
dotnet test TestProject --filter ShouldBeSuccess_1
where ShouldBeSuccess_1 is the name of the test. The argument after --filter is an expression, and not necessary the name of the test If you had a test called ShouldBeSuccess_12 it would also run. You can see the rules for --filter here.
To only run the tests in a specific category you can use TestCategory:
dotnet test TestProject --filter TestCategory=ci
where ci is the category name. To add a test to a category you use tags.
To create the results file you have to use the --logger flag:
dotnet test TestProject --logger trx
Here it's used to create a trx results file.
There is a nuget package named "Specrun.Specflow" download that. And it will change your app.config and set unitTestProvider name="SpecRun", so you can remove unitTestProvider name="MSTest" or "NUnit", now on saving App.config changes, visual studio prompts you to regenerate your feature files, click on Yes and now build a solution, What you will see is your test files have been regenerated.
Now in your Command Prompt go to C:\Users\\Documents\Visual Studio 2015\Projects\ and type runtests.cmd , it should trigger all your Feature files directly.
With MSTest v2, you can´t use mstest. You can use vstest.console.exe instead.
Example:
vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll"
https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019
If you want to run all scenarios in a single feature file, then add the /trait flag:
vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll" /trait:"My Feature"
And this runs all scenarios in the feature files that begin with:
Feature: My Feature
In order to ...
As a ...
I want to ...
Scenario: 1
...
Scenario: 2
...
I tried the tags technique but it didn't work, I'm using an older version of SpecFlow.
So, I went to the .feature.cs file related to a feature file and searched for TestMethodAttribute()
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
I added the TestCategory attribute on top of this, just like the following:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategory("MyCat")]
Build and compile and the command worked like a charm with
/Category:MyCat
I hope someone will find the answer useful.
Related
I feel like this should be incredibly simple but I am using xunit.runner to run my specflow tests. I have added smoke and regression tags. I know I can run via the vstest console and use something like
dotnet test --filter <Smoke>
I can't see an option to filter via the xunit.runner.json can't see anything to suggest where I could filter from. I know SpecRunner allows you to do it
Cheers
After being pointed in the right direction by #Amirreza Hashemi
i created a .runsettings file and added
<RunConfiguration>
<TestCaseFilter>Category=smoke</TestCaseFilter>
</RunConfiguration>
I have an NUnit test project called Tests. Within this project I have a number of files with their own class that run separate tests, including UnitTests.cs, IntegrationTests.cs, etc.
When I run dotnet test project I want only the unit tests within UnitTests.cs to be run, not the IntegrationTests.cs.
I want to avoid using command line arguments to ensure this. Is there anything I can write in the source code of IntegrationTests.cs to exclude it from dotnest test, whilst still being able to run them manually from the IDE?
I am running C# specflow tests from visual studio. When I run I am seeing tests are getting executed parallel. How can I make them to run one after another ? I am attaching snapshot which shows in cyan highlighted. I am looking a way to run them one after another. I am using Specflow.AssistDynamic and Techtalk.Specflow
I am able to do this by adding a property to AssemblyInfo.cs
[assembly: CollectionBehavior(DisableTestParallelization = true)]
SpecFlow generates unit test code. so you need to check which xUnit framework you're using in that project.
See this link: https://github.com/techtalk/SpecFlow/wiki/Parallel-Execution
If you are using NUnit, then remove/comment this line from Assembly.cs:
[assembly: Parallelizable(ParallelScope.Fixtures)]
Also, as I remember, in a Visual Studio, in Test explorer, present button - Run tests parallely. Unclick it.
In my project, to do this, I removed this line, and also called test scenarios like: A_A, A_B, in alphabet. In this case they are going one after one.
I have programmed an console application with c# for ranorex automation.
Within the exe i have build i would like to kick off the specflow feature files manually.
I believe I need to create a TestRunnerManager and trigger the run.
Can anyone help?
Although not familiar with nunit, I guess like with mstest, attributes are used to mark methods in an assembly as tests. You could use reflection to find these methods in the assembly and invoke them
This question is similar in spirit to one I asked a while back about MsTest (How do you run SpecFlow scenarios from the command line using MSTest?).
Remember that SpecFlow feature files become C# classes. Scenarios within a feature file become test methods. You can use the nunit-console command line utility to run these:
nunit-console /fixture:Your.Test.Project Your.Test.Project.dll
Which should run all the tests in the Your.Test.Project namespace.
When annotating scenarios using #CategoryName:
#Feature1
Scenario: Some cool feature
Given ...
You should be able to run those from the command line as well:
nunit-console /include:Feature1 Your.Test.Project.dll
Note: This is from an older version of NUnit. Current documentation: https://github.com/nunit/docs/wiki/Console-Command-Line
I use MsTest with Specflow, so my examples might not be correct, but this should put you on the right path. Just look at the *.feature.cs files generated by the .feature file to give you some hints.
No need to create your own console application to run these tests. Worst case scenario, create a batch file or PowerShell script to kick off the tests you want.
I am trying to run a feature file with NUnit Console. I tried googling it and checked NUnit3 help also. But I am unable to find any help.
I want to run either single feature file or any scenario in a feature file which has tag assigned. I am using specflow with specrun. I tried NUnit console command for where "test == path of feature file" but it is not executing test. However I am able to execute all test cases by giving project dll file path. But I just want to execute a single feature file or single scenario in a feature file. Please let me know how can I do this so that I will be able to generate NUnit testresult.xml file.
Thanks.
if you are using SpecRun then you can use the command line of SpecRun to run the tests.
If you really want to use NUnit then you first have to make sure that the project containing the feature files has been compiled. Once you have a test dll this will contain the NUnit tests just like any other and the test categories will be set based on the tags, so you can execute them all by telling NUNit to run tests in the test dll, or you can run tests which have a Tag by telling Nunit to run tests which are in a category matching the Tag.
Running just a feature will be more tricky as there is nothing which groups the scenarios by feature in the tests I don't think, though I may be wrong.
instead of test==featurefile use name==FeatureName