I hope somebody can help me.
I have a lot of UnitTest to my C# application in VS2010, and therefore I want to execute them in parallel so I can benefit of my four core machine.
This is "easily" done by adding parallelTestCount="0" to the execution in the Local.testsettings.
But some of my UnitTest (around 50) are not thread safe and instead of reworking them, I just want them to be run in not parallel mode.
Is that possible and if so how to do it?
You can't change the parallelism on a single test or set of tests. You could try to create a second test settings file and a second assembly containing your "unsafe" tests; you can define the folder that test assemblies are loaded from under the "Unit test" tab of the test settings dialog.
That said, your tests should be thread-safe. A unit test should be able to run in any order, in any environment, and always pass -- unless, of course, something changed in the code they're testing.
Related
I understand NUnit runs tests in parallel, which isn't what this question is about exactly. In a single test, I can have two local socket connections open using TcpClient on different ports, which can communicate with one another in the same unit test. The problem is the address space is of course shared by the thread executing the unit test, so it doesn't simulate communication through a network how I want.
If I have a test called Test1 with one TcpClient, and whilst it's running, I run Test2 with the other TcpClient, in these two tests I can have both the unit tests talking to one another and communicating - testing the protocol I've designed with separate address spaces, which does give me the simulation of a network I desire, and allows me to do the proper test assertions.
Now the question is, is there any way to do this automatically? I'd like to have a single test that can run Test1 & Test2, where both these tests communicate via their respective sockets, instead of starting them both up manually in parallel.
NUnit is not designed to run tests in different processes (nor is any other unit testing framework I'm aware of). It's designed to start as many tests as possible in the same process, to increase performance (creating processes is expensive).
Of course, that doesn't mean it's impossible to do this, but it's not a domain of NUnit itself. The question doesn't give much information about how the tests are structure, but I'm assuming there's a test assembly for the server and another test assembly for the client. Only when they run at the same time, the tests work.
I would go ahead and create a third test assembly that just starts the other two, really creating processes manually, e.g. launching
dotnet test ServerUnitTests.dll
dotnet test ClientUnitTests.dll
["Integration Tests" would be the better term here, but that's a detail]
I do not think that is possible.you can decide which test should run after another, so basically order. but usually, tests are not required to be run in parallel.
We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
I decided to switch from current solution (using modified NUnit by our team some years ago NDistribUnit which run tests on VirtualMachines and then gather results on hub server) to Selenium Grid 2.
Option with ParallelizableAttribute was tried.
Unfortunately I noticed that IWebDriver was stored in Global variable (puhh). This caused tests to start several browser instances, but tests used single IWebDriver -> tests execution happened in single browser -> tests were run under single process, but with several 'worker' threads. It was tried using 2 VMs as 'nodes' and local PC as Hub.
I know best solution is to change invalid idea to store driver in global variable, but it'll take too much time: there are 3k+ heavy UI tests to be updated; many static methods expected to have driver as global var to be updated also.
As well NUnit 3.0 provides Option to run several assemblies in parallel. To run several test projects it's good, but currently we have 1 assembly per one application.
It would be nice to run tests for one application (one assembly) in parallel.
Is there other ways to use GRID + NUnit 3 here to make it work?
Finally, existed solution were refactored: now each test during execution has own driver. Due to this change a lot of code was re-written (it appears that too much methods expected to have IwebDriver as global variable)
Actually, there are 2 options to do that:
Refactoring - it's done for one test project.
Along with removing static variables (initial refactoring purpose) other code was changed as well.
A great minus - significant effort was required.
Using TeamCity agents for parallel run.
I forgot to mention finally tests are being executed on TeamCity, but by single agent.
For left 'old' tests (where Driver instance was stored in static variable) several TC agents were configured to run only several classes from tests solution.
This option is very 'fast' and doesn't require big code changes.
Is there a way of running CodedUI steps outside a test project?
I want to use them to automate some actions in an application.
The program mstest.exe can be used to invoke Coded UI tests. Its /test:{test name}option allows a specific test (ie activity) to be executed, thus several different activities (ie tests) to be combined into one source file but only the desired activity is executed. Calling mstest.exe from a batch or Powershell script allows the activity to be executed without needing to type a long command each time.
If you already use Coded UI then there is no reason why it cannot be used for automating a series of GUI actions.
An example: For one project we needed to set a database from a backup before each series of tests. Manually that took 5 minutes and sometimes we did it wrong and so wasted time. With Coded UI it always worked and it ran quickly.
There's a significant amount of overhead involved in coded ui that you may not need in your automation task. To execute a coded ui test (and therefore run your automation), you'll need a full Visual Studio Professional or Test Controller/Test Agent installed on every machine that will be running the test/automation, and the machine will have to have a UI that is always available, I.E., a virtual machine configured so the desktop is always available and will not have interactions from another user.
Since your question was rather vague about what you want to automate, I can't really suggest anything in place of Coded UI, but it should be enough to say that you should use the tool that's best suited for the job at hand. Sure, you could use it to run your automation, but why would you want to? (insert imagery of a Corvette pulling a camper here)
I've been assigned a task of setting up a build server (jenkins) and running automated tests after the build agent completes the build.
We are using NUnit and selenium to run automated tests.
The main concern is wait time. Suppose several users check in their sources, a build is run and automated tests are run afterwards (there could be several hundred of these). What's the best way to set this up so that each user does NOT have to wait in queue for tests results. Also, I'm to consider things like test result reports etc.
Where do I start? What do I even google?
I'm very new at this stuff and any info on doing this would be greatly appreciated. thanks
The first thing you'll want to do is to separate your unit tests from your integration tests.
Unit tests should be fast. Integration tests will obviously be slower since you're interacting with external components.
As far as configuring your environment, to do what you're trying to do properly, you'll need to research using Jenkins in a Master/multiple-Slave configuration. This isn't terribly complex, but can take some time to set up.
What you'll likely end up doing is setting up a number of Jobs within Selenium to handle each part of your build process. ie, one job to do the compilation, at least one job to run the unit tests, and at least one job to run the integration tests (and then maybe packaging or deployment jobs depending on how far you want to take this..).
Depending on how slow your overall build process is, you could easily have one job for each component's integration tests and run these concurrently on different slave machines. A parent job could would then aggregate the results and determine whether or not the chick-in passed.
For reporting, you'll want to install the HTML Publisher Plugin, and the NUnit Plugin. These plugins will allow you to bundle the reports produced with the rest of the build artifacts.
In order to give feedback to your team, you'll also want to look at the Wall Display Plugin to display the status of the jobs.