I have created a number of long running integration tests. They take around 5-10 minutes to complete. I don't want them to run Everytime the developers run tests from VS as it will take a long time. They are purely ad hoc tests
Is there a way to mark a test as ignored but still allow developers to run it locally if they explicitly select and run the test without removing ignore tag each time? Also i want these tests to run during overnight build via TFS and MS test runner
Suggest you look into using Test Categories.
By categorising your long running tests separately, you can choose what to include on test runs, e.g.
[TestMethod]
[TestCategory("Integration")]
public void MyLongRunningTest()
{
// Test Code
}
And to run:
mstest /testcontainer:TestProject.dll /category:"Integration"
Related
I am trying to run my Unity Unit test with the following command:
"C:\Program Files\Unity\Hub\Editor\2019.4.3f1\Editor\Unity.exe" -runTests -quit -batchmode -projectPath "X:\MyProject" -logFile ./log.txt -testResults ./results.xml
However no reports are generated or message is given to the CMD console.
Here is what I have in the log.txt:
https://pastebin.com/aaZBkmUT
Why can't I run the tests? What am I doing wrong?
P.S.
What does that mean ERROR Failed to connect to local IPC ? May this be the cause of my issue somehow?
Try it eihter without -quit
They don't have it in the example. The thing is that afaik Tests run async so you might just be shutting down Unity before the results are available
Alternatively I think you could add the -runSynchronously
If included, the test run will run tests synchronously, guaranteeing that all tests runs in one editor update call. Note that this is only supported for EditMode tests, and that tests which take multiple frames (i.e. [UnityTest] tests, or tests with [UnitySetUp] or [UnityTearDown] scaffolding) will be filtered out.
to make sure your editor stays alive until the tests are done synchronously
I decided to switch from current solution (using modified NUnit by our team some years ago NDistribUnit which run tests on VirtualMachines and then gather results on hub server) to Selenium Grid 2.
Option with ParallelizableAttribute was tried.
Unfortunately I noticed that IWebDriver was stored in Global variable (puhh). This caused tests to start several browser instances, but tests used single IWebDriver -> tests execution happened in single browser -> tests were run under single process, but with several 'worker' threads. It was tried using 2 VMs as 'nodes' and local PC as Hub.
I know best solution is to change invalid idea to store driver in global variable, but it'll take too much time: there are 3k+ heavy UI tests to be updated; many static methods expected to have driver as global var to be updated also.
As well NUnit 3.0 provides Option to run several assemblies in parallel. To run several test projects it's good, but currently we have 1 assembly per one application.
It would be nice to run tests for one application (one assembly) in parallel.
Is there other ways to use GRID + NUnit 3 here to make it work?
Finally, existed solution were refactored: now each test during execution has own driver. Due to this change a lot of code was re-written (it appears that too much methods expected to have IwebDriver as global variable)
Actually, there are 2 options to do that:
Refactoring - it's done for one test project.
Along with removing static variables (initial refactoring purpose) other code was changed as well.
A great minus - significant effort was required.
Using TeamCity agents for parallel run.
I forgot to mention finally tests are being executed on TeamCity, but by single agent.
For left 'old' tests (where Driver instance was stored in static variable) several TC agents were configured to run only several classes from tests solution.
This option is very 'fast' and doesn't require big code changes.
I was running tests in VS2013 with Resharper 8, which integrated with an external process and redirected its output. I'm not seeking guidance on what qualifies as a unit test, because I find the test runner to be a nicer driver for any test than a console app.
This process was a database and so my SetUp and TearDown methods unsurprisingly did the opposites of each other and would absolutely step on each other if allowed to run concurrently.
They did not get to step on each other, because whenever two tests were started by the running simultaneously, they would lock up.
Latest Version 7 of Resharper did not have this issue.
Tinkering with the unit test settings in Resharper > Options > Unit Testing did not help.
How to run NUnit test fixtures serially? -- is close to working.
On my test fixture base (and I always run base.SetUp() at the beginning of derived SetUp), I added the following:
[SetUp]
protected virtual void SetUp()
{
FixtureHandle.Wait();
}
[TearDown]
protected virtual void TearDown()
{
FixtureHandle.Set();
}
FixtureHandle is a ManualResetEvent. This prevents NUnit from being too damn good and fast.
The above answer, from #nik.shornikov, will work, but in Resharper 2016.2 and later (and probably also in earlier versions) you can do this via an option, without writing any code. Please see my answer to Is there a way to *prevent* ReSharper from running an assembly's unit tests in parallel? for full details.
I've been assigned a task of setting up a build server (jenkins) and running automated tests after the build agent completes the build.
We are using NUnit and selenium to run automated tests.
The main concern is wait time. Suppose several users check in their sources, a build is run and automated tests are run afterwards (there could be several hundred of these). What's the best way to set this up so that each user does NOT have to wait in queue for tests results. Also, I'm to consider things like test result reports etc.
Where do I start? What do I even google?
I'm very new at this stuff and any info on doing this would be greatly appreciated. thanks
The first thing you'll want to do is to separate your unit tests from your integration tests.
Unit tests should be fast. Integration tests will obviously be slower since you're interacting with external components.
As far as configuring your environment, to do what you're trying to do properly, you'll need to research using Jenkins in a Master/multiple-Slave configuration. This isn't terribly complex, but can take some time to set up.
What you'll likely end up doing is setting up a number of Jobs within Selenium to handle each part of your build process. ie, one job to do the compilation, at least one job to run the unit tests, and at least one job to run the integration tests (and then maybe packaging or deployment jobs depending on how far you want to take this..).
Depending on how slow your overall build process is, you could easily have one job for each component's integration tests and run these concurrently on different slave machines. A parent job could would then aggregate the results and determine whether or not the chick-in passed.
For reporting, you'll want to install the HTML Publisher Plugin, and the NUnit Plugin. These plugins will allow you to bundle the reports produced with the rest of the build artifacts.
In order to give feedback to your team, you'll also want to look at the Wall Display Plugin to display the status of the jobs.
i'm unit testing in vs2008 and every time i run now it says
Executing the current test run will produce a new test run result, which will exceed
the limit of 25 that is currently specified. If you continue, older test run results and
their associated deployments will be deleted from the hard drive...
what does this mean, and how do i clear the older test run results? why is this important message?
This message basically means that the Unit Test project has saved/recorded 25 (your threshold) results of previously run unit tests.
By proceeding, it'll remove one from those 25 to include the results of your next run.
You can modify the 'alert' here in the Tools->Options dialog:
You can modify this number here: