I am trying to run a feature file with NUnit Console. I tried googling it and checked NUnit3 help also. But I am unable to find any help.
I want to run either single feature file or any scenario in a feature file which has tag assigned. I am using specflow with specrun. I tried NUnit console command for where "test == path of feature file" but it is not executing test. However I am able to execute all test cases by giving project dll file path. But I just want to execute a single feature file or single scenario in a feature file. Please let me know how can I do this so that I will be able to generate NUnit testresult.xml file.
Thanks.
if you are using SpecRun then you can use the command line of SpecRun to run the tests.
If you really want to use NUnit then you first have to make sure that the project containing the feature files has been compiled. Once you have a test dll this will contain the NUnit tests just like any other and the test categories will be set based on the tags, so you can execute them all by telling NUNit to run tests in the test dll, or you can run tests which have a Tag by telling Nunit to run tests which are in a category matching the Tag.
Running just a feature will be more tricky as there is nothing which groups the scenarios by feature in the tests I don't think, though I may be wrong.
instead of test==featurefile use name==FeatureName
Related
I am running C# specflow tests from visual studio. When I run I am seeing tests are getting executed parallel. How can I make them to run one after another ? I am attaching snapshot which shows in cyan highlighted. I am looking a way to run them one after another. I am using Specflow.AssistDynamic and Techtalk.Specflow
I am able to do this by adding a property to AssemblyInfo.cs
[assembly: CollectionBehavior(DisableTestParallelization = true)]
SpecFlow generates unit test code. so you need to check which xUnit framework you're using in that project.
See this link: https://github.com/techtalk/SpecFlow/wiki/Parallel-Execution
If you are using NUnit, then remove/comment this line from Assembly.cs:
[assembly: Parallelizable(ParallelScope.Fixtures)]
Also, as I remember, in a Visual Studio, in Test explorer, present button - Run tests parallely. Unclick it.
In my project, to do this, I removed this line, and also called test scenarios like: A_A, A_B, in alphabet. In this case they are going one after one.
I am using MSTest for unit test in jenkins. I am able to successfully integrate MSTest plugin with Jenkins. it's performing Unit tests and generating a TestResults.trx file at specific location.
Now, I want to pass that file to SonarQube. So, I can see code coverage result on SonarQube. Currently I have Sonar configured without code coverage. It's getting all data without unit tests result. To pass the file to sonar, i found this details on Sonar documents here https://docs.sonarqube.org/pages/viewpage.action?pageId=6389772
I tried to pass arguments /d:sonar.cs.vstest.reportsPaths=%CD%\MSTestResults.trx in Sonar. It didn't work. I have tried multiple times by changing directory and other stuff, It didn't help.
Does anyone ever did this before? Why SonarQube plugin in Jenkins is not reading this arguments!!!
I can see in build details that it's reading the agrument which i am passing and it's looking for a file at right location. Still it's not doing anything with that file. as per build details below,
[Test] $ D:\jenkins\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\SonarQubeScanner_MSBuild\MSBuild.SonarQube.Runner.exe begin /k:Test_Jenkins /n:Test_Jenkins /v:1.0 /d:sonar.host.url=http://xxxxxxx:9000 ******** /d:sonar.cs.vstest.reportsPaths=D:\jenkins\workspace\Test\TestResults.trx
SonarQube Scanner for MSBuild 2.3.2
For second option
I have also tried to use Visual Studio Code metrics ( https://wiki.jenkins-ci.org/display/JENKINS/Visual+Studio+Code+Metrics+Plugin ) to get code coverage. I am able to setup it in jenkins. i can see Code coverage result in jenkins, when i tried to pass that file to sonar, it didn't work.
I am getting metrics.xml file as output with coverage result. I tried to feed it to Sonar by passing argument /d:sonar.cs.vscoveragexml.reportesPaths="${WORKSPACE}\metrics.xml
still it's not working. I am getting result of code coverage as you can see below by visual studio code metrics plugin.
Please advice!
Thanks
I think you are mixing two concepts. In SonarQube, you can provide test execution report, so that you track in SonarQube some statistics about tests execution (number of tests, % success, duration, ...). This is done by passing one of the properties sonar.cs.vstest.reportsPaths, sonar.cs.nunit.reportsPaths or sonar.cs.xunit.reportsPaths.
To be honest this is an historical feature, with a very limited interest.
More interesting is to provide test coverage report to SonarQube. This is done by passing one of the properties sonar.cs.ncover3.reportsPaths, sonar.cs.opencover.reportsPaths, sonar.cs.dotcover.reportsPaths or sonar.cs.vscoveragexml.reportsPaths depending on the tool you are using to measure coverage.
I have programmed an console application with c# for ranorex automation.
Within the exe i have build i would like to kick off the specflow feature files manually.
I believe I need to create a TestRunnerManager and trigger the run.
Can anyone help?
Although not familiar with nunit, I guess like with mstest, attributes are used to mark methods in an assembly as tests. You could use reflection to find these methods in the assembly and invoke them
This question is similar in spirit to one I asked a while back about MsTest (How do you run SpecFlow scenarios from the command line using MSTest?).
Remember that SpecFlow feature files become C# classes. Scenarios within a feature file become test methods. You can use the nunit-console command line utility to run these:
nunit-console /fixture:Your.Test.Project Your.Test.Project.dll
Which should run all the tests in the Your.Test.Project namespace.
When annotating scenarios using #CategoryName:
#Feature1
Scenario: Some cool feature
Given ...
You should be able to run those from the command line as well:
nunit-console /include:Feature1 Your.Test.Project.dll
Note: This is from an older version of NUnit. Current documentation: https://github.com/nunit/docs/wiki/Console-Command-Line
I use MsTest with Specflow, so my examples might not be correct, but this should put you on the right path. Just look at the *.feature.cs files generated by the .feature file to give you some hints.
No need to create your own console application to run these tests. Worst case scenario, create a batch file or PowerShell script to kick off the tests you want.
I've got Visual Studio 2010, and we have two VS solutions we work with. The first is the web application, and the second is strictly for SpecFlow tests. Having two instances of Visual Studio running at the same time just to run SpecFlow features is eating all the available RAM causing things to slow down.
I've done some searching on Google and here on StackOverflow, plus perused the MS documentation on the MSTest command line tool, but I haven't found the answer. The full SpecFlow test suite takes ~45 minutes to complete, and I really only need to run a few scenarios.
I was wondering if there is a way to run individual SpecFlow features, and even individual scenarios, from the command line using MSTest?
Behind the scene specflow tests are just regular mstest unit tests. So you should be able to run them the same way using something like:
To run a specific scenario:
mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff
To run a several specific scenario you can use the /test flag multiple times:
mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff /test:GivenMyScenarioWhenIDoSomemthingElse
To run a feature
mstest /testcontainer:tests.dll /test:MyFeatureName
If you add tags on your scenarios using #MyTag for example, you could also use the option
/category:MyTag to filter down the scenarios to run.
Please have a look to the generated code behind of your feature files to get a clue of how things actually work, if you are familliar with mstest it should be pretty straightforward.
Now that SpecFlow 3.0 has been released we can use SpecFlow with .NET Core. The CLI tool for .NET Core is dotnet and tests are run like this if you use MSTest (vstest):
dotnet test
If the tests are in a specific project you can specify the project like this
dotnet test TestProject
where TestProject is the name of the project. You can skip the project name if you want to, but specifying it will make dotnet look in only that project. To list all the tests in the project you can use the -t flag:
dotnet test TestProject -t
To run only specific tests you can use the --filter flag:
dotnet test TestProject --filter ShouldBeSuccess_1
where ShouldBeSuccess_1 is the name of the test. The argument after --filter is an expression, and not necessary the name of the test If you had a test called ShouldBeSuccess_12 it would also run. You can see the rules for --filter here.
To only run the tests in a specific category you can use TestCategory:
dotnet test TestProject --filter TestCategory=ci
where ci is the category name. To add a test to a category you use tags.
To create the results file you have to use the --logger flag:
dotnet test TestProject --logger trx
Here it's used to create a trx results file.
There is a nuget package named "Specrun.Specflow" download that. And it will change your app.config and set unitTestProvider name="SpecRun", so you can remove unitTestProvider name="MSTest" or "NUnit", now on saving App.config changes, visual studio prompts you to regenerate your feature files, click on Yes and now build a solution, What you will see is your test files have been regenerated.
Now in your Command Prompt go to C:\Users\\Documents\Visual Studio 2015\Projects\ and type runtests.cmd , it should trigger all your Feature files directly.
With MSTest v2, you can´t use mstest. You can use vstest.console.exe instead.
Example:
vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll"
https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019
If you want to run all scenarios in a single feature file, then add the /trait flag:
vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll" /trait:"My Feature"
And this runs all scenarios in the feature files that begin with:
Feature: My Feature
In order to ...
As a ...
I want to ...
Scenario: 1
...
Scenario: 2
...
I tried the tags technique but it didn't work, I'm using an older version of SpecFlow.
So, I went to the .feature.cs file related to a feature file and searched for TestMethodAttribute()
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
I added the TestCategory attribute on top of this, just like the following:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategory("MyCat")]
Build and compile and the command worked like a charm with
/Category:MyCat
I hope someone will find the answer useful.
In my program, I use SevenZipSharp to generate zip files. SevenZipSharp is a managed DLL which loads another DLL, 7z.dll. I am manually setting SevenZipSharp's path to 7z.dll using SevenZipCompressor.SetLibraryPath.
When I execute my program in Debug mode, this all works fine, and it generates the zip file as nice as you please. However, when I execute my unit tests with mstest, SevenZipSharp always gives me the following error:
Test method threw exception:
SevenZip.SevenZipLibraryException: Can not load 7-zip library or
internal COM error! Message: failed to load library..
I suspect that MSTest might be doing something that is preventing SevenZipSharp from being able to load 7z.dll, like running in a security-tight sandbox (or something. I'm new to C# and MSTest...)
Does anyone have an idea about what might be happening?
Thank you!
Though the question posits a questionable scenario, the general problem of MSTest not loading required DLLs seems to be a common one, deserving of a less dismissive answer.
By default MSTest will copy the assemblies it believes are required by the test container to the Out folder of the default results folder, which changes for each run.
MSTest does not always automatically infer the necessary assemblies correctly; if there is no explicit direct reference to an assembly it won't be copied. Also, native DLLs are typically not detected.
I am not aware of a direct option to set the MSTest search path. You can determine the search path using procmon.exe as suggested above (it is basically the standard Windows DLL search).
Unintuitively, the default search path does not include the launch directory and I think this is a cause of confusion. When the tests are running the current directory is the test results "Out" directory, not the MSTest launch directory.
However, it is possible to control MSTest search behaviour (and the copying behaviour) with a test settings file. You can easily create and edit these through Visual Studio (see the Test menu) and then specify the created settings file on the MSTest command line. You can use different settings files for Visual Studio and MSTest.
By this means you can control exactly what DLLs are copied to your test directory.
See Create Test Settings to Run Automated Tests from Visual Studio to get more information on this.
Of course, DLL load failures may be due to missing dependencies, and the DLL mentioned in the error message may itself be present. You can use the dependency viewer or procmon to pick up unexpected dependencies in DLLs.
Consider using Process Monitor (aka procmon.exe) from the excellent SysInternals tools to monitor your test harness (MSTest). It will show you where the executable is looking for 7z.dll.
Visual Studio 2017 (and possibly 2015) provides two new ways to indicate that a native dll or other file is required by your tests without the need for a test settings file:
1: Add a link to the dll to your test project and tell VS to copy it to the output directory. Right-click the project in Solution Explorer and choose Add > Existing Item. Browse to 7z.dll, click the down arrow next to the Add button, and choose Add as Link. Then select the new 7z.dll item in Solution Explorer, alt-Enter to bring up Properties, and set "Copy to Output Directory" to "Copy if newer" (or "Copy always").
2: Attach a DeploymentItemAttribute to your test class. This attribute's constructor takes a single string argument, which is the path to the file you want to make available to tests in the class. It is relative to the test project's output directory.
Are you sure you want your unit tests to involve external libraries? Ideally you should have mechanisms to replace external stuff with e.g. mock objects, because testing external libraries like that actually turns your test into an integration test.
For popular libraries such as SevenZipSharp you can assume that it is properly tested, and you can have manual integration tests to verify that it performs correctly in your program.
I would consider getting rid of that dependency through dependency injection, mocking frameworks, etc, and let your unit tests solely test your own code.
Investigate the factory method or abstract factory design patterns for tips on how to easily replace such dependencies.
A good start would be to create your own ICustomZipInterface, and use the wrapper pattern to encapsulate your zip logic for production code. In your unit tests, replace that wrapper class with a dummy implementation. The dummy implementation might for example record how you access your zip component and you could use that information to validate your code, rather than checking if a zip file is actually created.