Using Specflow attributes
I am trying to get a piece of code to run at the very end, so when all tests are finished.
All tests are set to run in parallel
There might be multiple features and scenarios running and I need this to only run once and at the very end when everything is finished, not after each feature because placed in there it will run multiple times.
Use the AfterTestRun hook.
For more specification about the hooks:
https://github.com/techtalk/SpecFlow/wiki/Hooks
Related
We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
I'm setting up automated acceptance tests for an Asp.Net MVC project using Gherkin scenarios, Specflow steps and Selenium remote webdriver. The aim is to run the tests from Jenkins in parallel on multiple nodes using the selenium server hub.
At the moment I'm trying to get them to run in parallel locally and having some difficulties. Is it even possible to run those tests in parallel with the Gherkin/Specflow setup or does the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently ?
If it's not impossible how exactly can it be done ? And if it is impossible what else could be done to speed up the testing process ?
It has nothing to do with
the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently
It's totally possible to run your MSTest cases in parallel. As far as I'm aware the max number of hung tests shouldn't exceed 5. However I've used custom tool for this purpose, spawning every test in a separate thread, it's easy achievable with TPL.
You'll need to use the selenium grid to do this, and I would recommend putting the grid on another computer or virtual machine if you can.
You can run the same test on multiple browsers using a combination of 1) the task parallel library and 2) the dynamic object class. I've written about the specifics here http://blog.dmbcllc.com/running-selenium-in-parallel-with-any-net-unit-testing-tool/
Is there a way of running CodedUI steps outside a test project?
I want to use them to automate some actions in an application.
The program mstest.exe can be used to invoke Coded UI tests. Its /test:{test name}option allows a specific test (ie activity) to be executed, thus several different activities (ie tests) to be combined into one source file but only the desired activity is executed. Calling mstest.exe from a batch or Powershell script allows the activity to be executed without needing to type a long command each time.
If you already use Coded UI then there is no reason why it cannot be used for automating a series of GUI actions.
An example: For one project we needed to set a database from a backup before each series of tests. Manually that took 5 minutes and sometimes we did it wrong and so wasted time. With Coded UI it always worked and it ran quickly.
There's a significant amount of overhead involved in coded ui that you may not need in your automation task. To execute a coded ui test (and therefore run your automation), you'll need a full Visual Studio Professional or Test Controller/Test Agent installed on every machine that will be running the test/automation, and the machine will have to have a UI that is always available, I.E., a virtual machine configured so the desktop is always available and will not have interactions from another user.
Since your question was rather vague about what you want to automate, I can't really suggest anything in place of Coded UI, but it should be enough to say that you should use the tool that's best suited for the job at hand. Sure, you could use it to run your automation, but why would you want to? (insert imagery of a Corvette pulling a camper here)
I've been assigned a task of setting up a build server (jenkins) and running automated tests after the build agent completes the build.
We are using NUnit and selenium to run automated tests.
The main concern is wait time. Suppose several users check in their sources, a build is run and automated tests are run afterwards (there could be several hundred of these). What's the best way to set this up so that each user does NOT have to wait in queue for tests results. Also, I'm to consider things like test result reports etc.
Where do I start? What do I even google?
I'm very new at this stuff and any info on doing this would be greatly appreciated. thanks
The first thing you'll want to do is to separate your unit tests from your integration tests.
Unit tests should be fast. Integration tests will obviously be slower since you're interacting with external components.
As far as configuring your environment, to do what you're trying to do properly, you'll need to research using Jenkins in a Master/multiple-Slave configuration. This isn't terribly complex, but can take some time to set up.
What you'll likely end up doing is setting up a number of Jobs within Selenium to handle each part of your build process. ie, one job to do the compilation, at least one job to run the unit tests, and at least one job to run the integration tests (and then maybe packaging or deployment jobs depending on how far you want to take this..).
Depending on how slow your overall build process is, you could easily have one job for each component's integration tests and run these concurrently on different slave machines. A parent job could would then aggregate the results and determine whether or not the chick-in passed.
For reporting, you'll want to install the HTML Publisher Plugin, and the NUnit Plugin. These plugins will allow you to bundle the reports produced with the rest of the build artifacts.
In order to give feedback to your team, you'll also want to look at the Wall Display Plugin to display the status of the jobs.
Has anyone found a way to run Selenium RC / Selenium Grid tests, written in C# in parallel?
I've currently got a sizable test suite written using Selenium RC's C# driver. Running the entire test suite takes a little over an hour to complete. I normally don't have to run the entire suite so it hasn't been a concern up to now, but it's something that I'd like to be able to do more regularly (ie, as part of an automated build)
I've been spending some time recently poking around with the Selenium Grid project whose purpose essentially is to allow those tests to run in parallel. Unfortunately, it seems that the TestDriven.net plugin that I'm using runs the tests serially (ie, one after another). I'm assuming that NUnit would execute the tests in a similar fashion, although I haven't actually tested this out.
I've noticed that the NUnit 2.5 betas are starting to talk about running tests in parallel with pNUnit, but I haven't really familiarized myself enough with the project to know for sure whether this would work.
Another option I'm considering is separating my test suite into different libraries which would let me run a test from each library concurrently, but I'd like to avoid that if possible since I'm not convinced this is a valid reason for splitting up the test suite.
I am working on this very thing and have found Gallio latest can drive mbUnit tests in parallel. You can drive them against a single Selenium Grid hub, which can have several remote control servers listening.
I'm using the latest nightly from Gallio to get the ParallelizableAttribute and DegreeOfParallelismAttribute.
Something things I've noticed is I cannot rely on TestSet and TestTeardown be isolated the parallel tests. You'll need the test to look something like this:
[Test] public void Foo(){
var s = new DefaultSelenium("http://grid", 4444, "*firefox",
"http://server-under-test");
s.Start();
s.Open("mypage.aspx");
// Continue
s.Stop();
}
Using the [SetUp] attribute to start the Selenium session was causing the tests to not get the remote session from s.Start().
I wrote PNUnit as an extension for NUnit almost three years ago and I'm happy to see it was finally integrated into NUnit.
We use it on a daily basis to test our software under different distros and combinations. Just to give an example: we've a test suite of heavy tests (long ones) with about 210 tests. Each of them sets up a server and runs a client in command line running several operations (up to 210 scenarios).
Well, we use the same suite to run the tests on different Linux combinations and windows variations, and also combined ones like a windows server with a linux client, windows xp, vista, then domain controller, out of domain, and so on. We use the same binaries and then just have "agents" launched at several boxes.
We use the same platform for: balancing load test load -> I mean, running in chunks faster. Running several combinations at the same time, and what I think is more interesting: defining multi client scenarios: two clients wait for the server to start up, then launch operations, synch with each other and so on. We also use PNUnit for load testing (hundreds of boxes against a single server).
So, if you have any questions about how to set it up (which is not simple yet, I'm afraid), don't hesitate to ask.
Also I wrote an article long ago about it at DDJ: http://www.ddj.com/architect/193104810
Hope it helps
I don't know if no answer counts as an answer but I'd say you have researched everything and you really came up with the 2 possible solutions...
Test Suite runs tests in parallel
Split the test suite up
I am at a loss for any thing else.