I understand NUnit runs tests in parallel, which isn't what this question is about exactly. In a single test, I can have two local socket connections open using TcpClient on different ports, which can communicate with one another in the same unit test. The problem is the address space is of course shared by the thread executing the unit test, so it doesn't simulate communication through a network how I want.
If I have a test called Test1 with one TcpClient, and whilst it's running, I run Test2 with the other TcpClient, in these two tests I can have both the unit tests talking to one another and communicating - testing the protocol I've designed with separate address spaces, which does give me the simulation of a network I desire, and allows me to do the proper test assertions.
Now the question is, is there any way to do this automatically? I'd like to have a single test that can run Test1 & Test2, where both these tests communicate via their respective sockets, instead of starting them both up manually in parallel.
NUnit is not designed to run tests in different processes (nor is any other unit testing framework I'm aware of). It's designed to start as many tests as possible in the same process, to increase performance (creating processes is expensive).
Of course, that doesn't mean it's impossible to do this, but it's not a domain of NUnit itself. The question doesn't give much information about how the tests are structure, but I'm assuming there's a test assembly for the server and another test assembly for the client. Only when they run at the same time, the tests work.
I would go ahead and create a third test assembly that just starts the other two, really creating processes manually, e.g. launching
dotnet test ServerUnitTests.dll
dotnet test ClientUnitTests.dll
["Integration Tests" would be the better term here, but that's a detail]
I do not think that is possible.you can decide which test should run after another, so basically order. but usually, tests are not required to be run in parallel.
Related
I'm setting up automated acceptance tests for an Asp.Net MVC project using Gherkin scenarios, Specflow steps and Selenium remote webdriver. The aim is to run the tests from Jenkins in parallel on multiple nodes using the selenium server hub.
At the moment I'm trying to get them to run in parallel locally and having some difficulties. Is it even possible to run those tests in parallel with the Gherkin/Specflow setup or does the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently ?
If it's not impossible how exactly can it be done ? And if it is impossible what else could be done to speed up the testing process ?
It has nothing to do with
the fact the Gherkin scenarios re-use the same Specflow steps for similar scenarios make it impossible to run them concurrently
It's totally possible to run your MSTest cases in parallel. As far as I'm aware the max number of hung tests shouldn't exceed 5. However I've used custom tool for this purpose, spawning every test in a separate thread, it's easy achievable with TPL.
You'll need to use the selenium grid to do this, and I would recommend putting the grid on another computer or virtual machine if you can.
You can run the same test on multiple browsers using a combination of 1) the task parallel library and 2) the dynamic object class. I've written about the specifics here http://blog.dmbcllc.com/running-selenium-in-parallel-with-any-net-unit-testing-tool/
I've been assigned a task of setting up a build server (jenkins) and running automated tests after the build agent completes the build.
We are using NUnit and selenium to run automated tests.
The main concern is wait time. Suppose several users check in their sources, a build is run and automated tests are run afterwards (there could be several hundred of these). What's the best way to set this up so that each user does NOT have to wait in queue for tests results. Also, I'm to consider things like test result reports etc.
Where do I start? What do I even google?
I'm very new at this stuff and any info on doing this would be greatly appreciated. thanks
The first thing you'll want to do is to separate your unit tests from your integration tests.
Unit tests should be fast. Integration tests will obviously be slower since you're interacting with external components.
As far as configuring your environment, to do what you're trying to do properly, you'll need to research using Jenkins in a Master/multiple-Slave configuration. This isn't terribly complex, but can take some time to set up.
What you'll likely end up doing is setting up a number of Jobs within Selenium to handle each part of your build process. ie, one job to do the compilation, at least one job to run the unit tests, and at least one job to run the integration tests (and then maybe packaging or deployment jobs depending on how far you want to take this..).
Depending on how slow your overall build process is, you could easily have one job for each component's integration tests and run these concurrently on different slave machines. A parent job could would then aggregate the results and determine whether or not the chick-in passed.
For reporting, you'll want to install the HTML Publisher Plugin, and the NUnit Plugin. These plugins will allow you to bundle the reports produced with the rest of the build artifacts.
In order to give feedback to your team, you'll also want to look at the Wall Display Plugin to display the status of the jobs.
Recently I've been tasked with writing and running unit tests in C# for a desktop application. In the past, I've written mainly automated functional/GUI tests. These get run on dedicated QA machines because these tests can cause crashes, data loss, Blue Screen of Death, etc. Should a similar setup be done for running unit tests? Are there any risks with running unit tests on my local machine?
Unit tests should run on your local machine AND on the build server. They are an invaluable resource as feedback for the developer. The DEV should write the unit tests. Before checking in, he should run the unit tests to make sure he does not break anything. The build server will then run the unit tests again, in order to make sure nothing it actually broken, and that the integration with other code, if any, succeeded.
When the unit tests have run, preferably, the build server should then run integration tests and automated UI tests. When these processes have finished, a build has been produced with no known defects - at least none that is covered by the tests. In other words a green build means the software integration went well, and the developers can keep checking in - or at some point start doing manual testing before actually releasing a piece of software.
Software testing nirvana happens when enough tests has been automated, so that the developers and the business can feel safe releasing a product based only on the success of this stream of automated tests. In reality, only a few products will be this mature - and some amount of manual testing is required.
Unit tests does not touch external systems or interfaces. Therefore, there are no risks by running them locally.
Unit tests should be so isolated that they can be run on every machine without any further setup. Unit tests by definition test only functional units on a very granular level without touching external systems (i.e. database, file system, etc.)
Unit Testing is typically done by the Developers while developing. So Run it on your Development environment. Probably your Local machine.
well if your local machine is a dev machine I see no risk since the machine is not a production end user machine. You can also consider TFS Lab management (in case you have Microsoft TFS as ALM / source control system). in that case you can create virtual or physical test environment and the application gets deployed and tested in those defined environments.
not sure what the names of similar test products are for non TFS scenarios.
I hope somebody can help me.
I have a lot of UnitTest to my C# application in VS2010, and therefore I want to execute them in parallel so I can benefit of my four core machine.
This is "easily" done by adding parallelTestCount="0" to the execution in the Local.testsettings.
But some of my UnitTest (around 50) are not thread safe and instead of reworking them, I just want them to be run in not parallel mode.
Is that possible and if so how to do it?
You can't change the parallelism on a single test or set of tests. You could try to create a second test settings file and a second assembly containing your "unsafe" tests; you can define the folder that test assemblies are loaded from under the "Unit test" tab of the test settings dialog.
That said, your tests should be thread-safe. A unit test should be able to run in any order, in any environment, and always pass -- unless, of course, something changed in the code they're testing.
Has anyone found a way to run Selenium RC / Selenium Grid tests, written in C# in parallel?
I've currently got a sizable test suite written using Selenium RC's C# driver. Running the entire test suite takes a little over an hour to complete. I normally don't have to run the entire suite so it hasn't been a concern up to now, but it's something that I'd like to be able to do more regularly (ie, as part of an automated build)
I've been spending some time recently poking around with the Selenium Grid project whose purpose essentially is to allow those tests to run in parallel. Unfortunately, it seems that the TestDriven.net plugin that I'm using runs the tests serially (ie, one after another). I'm assuming that NUnit would execute the tests in a similar fashion, although I haven't actually tested this out.
I've noticed that the NUnit 2.5 betas are starting to talk about running tests in parallel with pNUnit, but I haven't really familiarized myself enough with the project to know for sure whether this would work.
Another option I'm considering is separating my test suite into different libraries which would let me run a test from each library concurrently, but I'd like to avoid that if possible since I'm not convinced this is a valid reason for splitting up the test suite.
I am working on this very thing and have found Gallio latest can drive mbUnit tests in parallel. You can drive them against a single Selenium Grid hub, which can have several remote control servers listening.
I'm using the latest nightly from Gallio to get the ParallelizableAttribute and DegreeOfParallelismAttribute.
Something things I've noticed is I cannot rely on TestSet and TestTeardown be isolated the parallel tests. You'll need the test to look something like this:
[Test] public void Foo(){
var s = new DefaultSelenium("http://grid", 4444, "*firefox",
"http://server-under-test");
s.Start();
s.Open("mypage.aspx");
// Continue
s.Stop();
}
Using the [SetUp] attribute to start the Selenium session was causing the tests to not get the remote session from s.Start().
I wrote PNUnit as an extension for NUnit almost three years ago and I'm happy to see it was finally integrated into NUnit.
We use it on a daily basis to test our software under different distros and combinations. Just to give an example: we've a test suite of heavy tests (long ones) with about 210 tests. Each of them sets up a server and runs a client in command line running several operations (up to 210 scenarios).
Well, we use the same suite to run the tests on different Linux combinations and windows variations, and also combined ones like a windows server with a linux client, windows xp, vista, then domain controller, out of domain, and so on. We use the same binaries and then just have "agents" launched at several boxes.
We use the same platform for: balancing load test load -> I mean, running in chunks faster. Running several combinations at the same time, and what I think is more interesting: defining multi client scenarios: two clients wait for the server to start up, then launch operations, synch with each other and so on. We also use PNUnit for load testing (hundreds of boxes against a single server).
So, if you have any questions about how to set it up (which is not simple yet, I'm afraid), don't hesitate to ask.
Also I wrote an article long ago about it at DDJ: http://www.ddj.com/architect/193104810
Hope it helps
I don't know if no answer counts as an answer but I'd say you have researched everything and you really came up with the 2 possible solutions...
Test Suite runs tests in parallel
Split the test suite up
I am at a loss for any thing else.