How do you execute millions of unit test quickly, meaning 20 to 30 minutes?
Here is the scenario:
You are releasing certain hardware and you have, let's say, 2000 unit tests.
You are releasing new hardware and you have additional 1000 tests for that.
Each new hardware will include tests, but also you have to run and execute every previous one, and the number gets bigger as does execution time.
During development, this is solved by categorizing, using the TestCategory attribute and running only what you need to.
The CI, however, must run every single test. As the number increases, executing time is slower and sometimes times out. The .testrunconfig is already set for parallelTestCount execution, but over time this does not solve the issue permanently.
How would you solve this?
It seems like with each update on Visual Studio 2017, execution time changes. We currently have over 6000 tests, of which 15 to 20% are unit tests, and the rest are integration tests.
The bottleneck seemed to be the CI server itself, running on a single machine. 70% to 80% of the tests are asynchronous, and analysis showed that there are no blocking I/O operations. Besides IO, we do not use databases, caching so there is that.
Now, we are in the process of migrating to Jenkins and using its Parallel Test Executor Plugin to parallelize the tests across multiple nodes instead of a single machine. Initial testing showed that timing for executing 6000+ tests varies from 10 to 15 minutes versus the old CI which took 2 hours or stopped sometimes.
Related
I am using azure pipeline to execute API tests, the tests are executed serially and I have hit the point where my job runs for over 1 hour - this means agent fails because 1 hour is the maximum job execution time. I have started to read how to execute tests in parallel, and in xunit tests should run in parallel by default when they are not in the same collection. In azure however, following https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops, to run tests parallelly it is required to use more than one agent and apparently more than 1 job. There are two cons in my case:
we have limited amount of agents and other people want to use them too, I do not want to occupy more agents than necessary;
I need to configure two firewall for cosmosdb for each job (1 job = 1 agent and each agent has new ip i need to add to firewall), this tasks takes 8 minutes and i need to configure 2 databases (16 minutes), each new job would mean i need to speend additional 16 minutes configuring firewalls.
So my question is - is it possible to run tests concurrent rather than parallel by utilizing same core? I have many thread.sleeps (but in my opinion i use them the correct way as i wait for some result for some time like 60 seconds, but i thread sleep for 2 seconds and every 2 seconds i'm polling in example databse if results are there) - thread.sleep means thread is sleeping and core should be available for other threads to run.
Apparently I have posted question too soon, found out that all I need to do is to set [assembly: CollectionBehavior(MaxParallelThreads = n)] to number I want (source: https://xunit.net/docs/running-tests-in-parallel). Default is number of logical processor and in case of azure agents this number is equal to 1.
I made a simple WebPerformanceTest in VS2012 which is just logging in and doing some basic jobs on the website. When ran, this test is only running for about a second or two.
Now, I made a load test, containing only this one webtest with a Constant load for 5 minutes and the mode is based on the number of virtual users. And here comes the funny part: No matter how much users I assign to this load test the number of tests executed is always the same as the number of users assigned. I also tried raising the load test execution time giving me the same result: one test = 5 minutes per user, whereas it only took the webtest about 1-2 seconds to execute.
Does anyone have any idea why the test is taking way longer (300 times) in the load test? What am I doing wrong here?
Edit: The machine is a Windows Server 2008 R2 with 4 cores # 3.00ghz and 8 GB RAM
Here's some images of the settings:
Notice the Run Duration: 5:00
You've set each test to run for five minutes. So basically it runs through the test then sits their idle for the rest of the time, closes the tests and then reports back to you that it took 5 minutes. You won't see any difference unless you change this setting or create a test that could take longer than 5 minutes to run.
I've been assigned a task of setting up a build server (jenkins) and running automated tests after the build agent completes the build.
We are using NUnit and selenium to run automated tests.
The main concern is wait time. Suppose several users check in their sources, a build is run and automated tests are run afterwards (there could be several hundred of these). What's the best way to set this up so that each user does NOT have to wait in queue for tests results. Also, I'm to consider things like test result reports etc.
Where do I start? What do I even google?
I'm very new at this stuff and any info on doing this would be greatly appreciated. thanks
The first thing you'll want to do is to separate your unit tests from your integration tests.
Unit tests should be fast. Integration tests will obviously be slower since you're interacting with external components.
As far as configuring your environment, to do what you're trying to do properly, you'll need to research using Jenkins in a Master/multiple-Slave configuration. This isn't terribly complex, but can take some time to set up.
What you'll likely end up doing is setting up a number of Jobs within Selenium to handle each part of your build process. ie, one job to do the compilation, at least one job to run the unit tests, and at least one job to run the integration tests (and then maybe packaging or deployment jobs depending on how far you want to take this..).
Depending on how slow your overall build process is, you could easily have one job for each component's integration tests and run these concurrently on different slave machines. A parent job could would then aggregate the results and determine whether or not the chick-in passed.
For reporting, you'll want to install the HTML Publisher Plugin, and the NUnit Plugin. These plugins will allow you to bundle the reports produced with the rest of the build artifacts.
In order to give feedback to your team, you'll also want to look at the Wall Display Plugin to display the status of the jobs.
i'm unit testing in vs2008 and every time i run now it says
Executing the current test run will produce a new test run result, which will exceed
the limit of 25 that is currently specified. If you continue, older test run results and
their associated deployments will be deleted from the hard drive...
what does this mean, and how do i clear the older test run results? why is this important message?
This message basically means that the Unit Test project has saved/recorded 25 (your threshold) results of previously run unit tests.
By proceeding, it'll remove one from those 25 to include the results of your next run.
You can modify the 'alert' here in the Tools->Options dialog:
You can modify this number here:
Has anyone found a way to run Selenium RC / Selenium Grid tests, written in C# in parallel?
I've currently got a sizable test suite written using Selenium RC's C# driver. Running the entire test suite takes a little over an hour to complete. I normally don't have to run the entire suite so it hasn't been a concern up to now, but it's something that I'd like to be able to do more regularly (ie, as part of an automated build)
I've been spending some time recently poking around with the Selenium Grid project whose purpose essentially is to allow those tests to run in parallel. Unfortunately, it seems that the TestDriven.net plugin that I'm using runs the tests serially (ie, one after another). I'm assuming that NUnit would execute the tests in a similar fashion, although I haven't actually tested this out.
I've noticed that the NUnit 2.5 betas are starting to talk about running tests in parallel with pNUnit, but I haven't really familiarized myself enough with the project to know for sure whether this would work.
Another option I'm considering is separating my test suite into different libraries which would let me run a test from each library concurrently, but I'd like to avoid that if possible since I'm not convinced this is a valid reason for splitting up the test suite.
I am working on this very thing and have found Gallio latest can drive mbUnit tests in parallel. You can drive them against a single Selenium Grid hub, which can have several remote control servers listening.
I'm using the latest nightly from Gallio to get the ParallelizableAttribute and DegreeOfParallelismAttribute.
Something things I've noticed is I cannot rely on TestSet and TestTeardown be isolated the parallel tests. You'll need the test to look something like this:
[Test] public void Foo(){
var s = new DefaultSelenium("http://grid", 4444, "*firefox",
"http://server-under-test");
s.Start();
s.Open("mypage.aspx");
// Continue
s.Stop();
}
Using the [SetUp] attribute to start the Selenium session was causing the tests to not get the remote session from s.Start().
I wrote PNUnit as an extension for NUnit almost three years ago and I'm happy to see it was finally integrated into NUnit.
We use it on a daily basis to test our software under different distros and combinations. Just to give an example: we've a test suite of heavy tests (long ones) with about 210 tests. Each of them sets up a server and runs a client in command line running several operations (up to 210 scenarios).
Well, we use the same suite to run the tests on different Linux combinations and windows variations, and also combined ones like a windows server with a linux client, windows xp, vista, then domain controller, out of domain, and so on. We use the same binaries and then just have "agents" launched at several boxes.
We use the same platform for: balancing load test load -> I mean, running in chunks faster. Running several combinations at the same time, and what I think is more interesting: defining multi client scenarios: two clients wait for the server to start up, then launch operations, synch with each other and so on. We also use PNUnit for load testing (hundreds of boxes against a single server).
So, if you have any questions about how to set it up (which is not simple yet, I'm afraid), don't hesitate to ask.
Also I wrote an article long ago about it at DDJ: http://www.ddj.com/architect/193104810
Hope it helps
I don't know if no answer counts as an answer but I'd say you have researched everything and you really came up with the 2 possible solutions...
Test Suite runs tests in parallel
Split the test suite up
I am at a loss for any thing else.