Is it possible to let Resharper (or NUnit?) know that I want each test to look for an App.config under it's own project, even when running all tests in a solution together?
Background:
I'm using NUnit and the test-runner that ships with Resharper, and I've got several test-projects in the same solution. Some of my tests depend on config-files located under their respective projects.
When I run a test-project by itself, it will use it's internal App.config, and everything works fine. When I try to run all the tests in the solution, or use the shortcut to run all tests in current test session however, no config-file will be selected, and any test depending on a config will fail by default.
For this reason, I typically end up running all tests in the solution once first, then right clicking the nodes in the test-runner for each of the config-dependent projects, and running them separately afterwards.
Solved:
Apparently assemblies containing NUnit-tests can be run in separate processes or domains using command line options.
For the testrunner under Resharper, this setting can be found under Resharper > Options > Unit Testing.
There is an option "Use separate AppDomain for each assempbly with tests". Checking that solved my problem.
Related
I have some Specflow(3.9.40)-based UI test using Selenium Webdriver and NUnit(3.13.2) Framework.
I would like these tests to not run when I hit 'Run All' in the Test Explorer.
In my StepDefinition class I have tried adding the Explicit Attribute:
[TestFixture, Explicit("Only run when called explicitly")]
[Binding]
public class LoginStepDefinitions
{ ...
, but for some reason it does not seem to work - The UI Tests still run.
I imagine there may be something I can add to the [BeforeScenario] in my HookInitialization class, but I am stumped:
[BeforeScenario]
public void FirstBeforeScenario()
{
var seleniumDriver = new SeleniumDriver(_scenarioContext);
_scenarioContext.Set(seleniumDriver, "SeleniumDriver");
}
Is there a way for me to tell Visual Studio Test Explorer to ignore the UI Tests or its folder ?
The basic problem is that NUnit's Explicit setting is unique among the frameworks supported by Test Explorer, which knows nothing of this concept of "explicitness."
If Test Explorer calls the NUnit adapter telling it to run all, then NUnit will honor the Explicit setting. However, in some situations, it simply passes a list of all the tests to NUnit. In that case, all tests appear (to NUnit) to have been run explicitly.
Which way Test Explorer calls NUnit is dependent on the version you are running and it's only possible to tell how it calls by debugging or by observing the outcome. To add to the confusion, some versions of the NUnit3 adapter use heuristics to try to guess whether an explicit test was called explicitly. This doesn't solve your problem but should make it easier to understand what is going on.
The best workaround, if you are using TestExplorer, is to avoid Explicit and to set up categories, which control the running of the tests. Fortunately, TestExplorer and NUnit have pretty much the same understanding of categories.
You can filter tests first in Test Explorer, then the "Run All" button is scoped to only those tests in the Test Explorer panel. If you desire to run all tests except those in a certain project, enter this into the test filter search box in the Test Explorer panel:
-project:NameOfTheTestProject
If your test project has space characters in the name:
-project:"Name of the test project"
Note the - character before the word project. This is crucial. This character tells Visual Studio Test Explorer to invert your filter criteria — it is used to exclude tests in a certain project.
If you have these UI tests intermingled with other kinds of tests, obviously filtering out an entire project is not desirable. Consider placing the UI tests in a specific folder and filter on the FullName instead. For example, if your SpecFlow tests exist in a folder called "Specs", then omit those tests using:
-FullName:Specs
Again, include the - character (with no spaces) before the FUllName criteria to exclude tests in that namespace. This works with SpecFlow because the namespaces for C# classes generated from .feature files follow the folder structure of your test project, just like adding a .cs file does.
This technique should work with any unit test provider, not just NUnit.
I am running C# specflow tests from visual studio. When I run I am seeing tests are getting executed parallel. How can I make them to run one after another ? I am attaching snapshot which shows in cyan highlighted. I am looking a way to run them one after another. I am using Specflow.AssistDynamic and Techtalk.Specflow
I am able to do this by adding a property to AssemblyInfo.cs
[assembly: CollectionBehavior(DisableTestParallelization = true)]
SpecFlow generates unit test code. so you need to check which xUnit framework you're using in that project.
See this link: https://github.com/techtalk/SpecFlow/wiki/Parallel-Execution
If you are using NUnit, then remove/comment this line from Assembly.cs:
[assembly: Parallelizable(ParallelScope.Fixtures)]
Also, as I remember, in a Visual Studio, in Test explorer, present button - Run tests parallely. Unclick it.
In my project, to do this, I removed this line, and also called test scenarios like: A_A, A_B, in alphabet. In this case they are going one after one.
What I have
I have a unit test Visual Studio C# Solution (which runs webdriver tests, not that that's necessarily relevant). It runs via TeamCity. Currently the environment is hard-coded to "Dev" in one of the .cs files, and I manually change the code locally to run elsewhere when required.
What I want
A way to setup two projects in TeamCity - one to run on "Dev" environment and the other on "Test" environment. Obviously I can't use hard-coded values so I need some sort of set of configuration files that can be chosen at runtime, or possibly some sort of build parameters - but I have no clue how to do this or what will work.
(I didn't mention TeamCity in the question as it is not 100% relevant / just provides context --- as long as I can run the unit tests eg from the command prompt with parameters that can be passed in, that would do the trick.)
What I've tried
From what I've asked around, I don't believe I can use web.config as it's not a web solution but a unit test solution. I believe there is a mechanism to tell Configuration Manager what web.config file to use, so I'm hoping there's a similar mechanism that can be used for Unit Test projects. I've tried hunting down information on "build configurations" on "unit test projects" and a range of other searches, but it's a nightmare finding anything relevant.
Can someone point me in the right direction? I'm good with my basic programming, but if it requires messing around with configurations or build parameters, then I might need a more explicit 'how-to' from you.
Thanks in advance.
Check this extension for Visual Studio
https://visualstudiogallery.msdn.microsoft.com/579d3a78-3bdd-497c-bc21-aa6e6abbc859
This allows you to create different config files for different build type. You need to create different build types each with a config file specific to that build type.
Ok as per my comments above, I tried a solution here: https://visualstudiogallery.msdn.microsoft.com/69023d00-a4f9-4a34-a6cd-7e854ba318b5
IT WORKED.
I'm now able to create App.Config transforms that automatically link to a Build Configuration. I can then specify the build configuration inside TeamCity.
The only trick was that the "Rebuild" target didn't work if there were no code changes but there was a difference in Configuration. (They use the same directory of the same Agent, and a rebuild is necessary). The workaround for this is to tick the 'Clean all files before build' option in TeamCity Version Control settings.
I'm working with Unit Test using VS2013 Professional. In particular I'm using the NUnit framework (NUnit TestAdapter for VS2013). My problem is that when I run my tests then VS starts building all the projects inside the solution. Currently the Unit Test project does not reference any solution project.
If I simply code a single test method like:
[Test]
public void SimpleTestMethod(){
Assert.That("a", Is.EqualTo("a"));
}
and the Unit Test project is in a Solution with N project, when I run my test then VS will build all N-1 project... In my case this behavior is boring because it takes too much time (the solution contains many projects) and some projects contain errors.
Is there a way to run my SimpleTestMethod() without complete solution building?
Break your test project to multiple projects that only reference a subset of the solution's projects.
This is also good test housekeeping - have a separate unit test project for each solution project instead of one huge project with dependencies to anything else. There are several advantages to this:
Tests run faster
It's a lot easier to isolate test cases, especially configuration settings
You can version projects and their test cases together
A good naming practice is to name your test projects the same as their target projects with a .Tests suffix. You can also create a solution folder (not a real folder) called "Tests" and move the test projects in it.
As for the why: Test runners use the Unit Test assembly and its dependencies to run their tests. If any of the assembly's dependencies change, the assembly and the dependencies have to be rebuilt. Visual Studio doesn't know what the external tool will call so it has to build all changed assemblies and their dependents.
If the build fails, there are no valid assemblies for the test runner to use so VS has to rebuild the entire solution before the runner can work. In this case, the obvious solution is to fix the error.
There are some stopgap options you can use until you can fix the error:
Temporarily remove the broken project from the build configuration
Split the solution so that you have a solution that can be built and tested
I struggled with this for a very long time as well. I actually hated the automatic build process, even when everything was successful.
I started to run tests through the command line. No build process is necessary. You can write your own .bat files and keep logs of test results. There are plenty of command line parameters that can be added to customize for what you are looking for.
https://msdn.microsoft.com/en-us/library/jj155796.aspx
I have the NUnit GUI runner to run tests on my unit test assembly every time that it is built. The problem is that when I try to build, the .dll in the Debug folder is in use by NUnit, which prevents it from being built, which prevents the automatic test run. Is there a way around this?
Can I suggest a slightly different approach but along the same lines as #UvarajGopu....
As long as you are using Visual Studio higher than "Express" version, and if you write your UnitTests in a separate project (typically alongside the project being tested, with a ".UnitTests" suffix), then do this:
Set your UnitTests project as being the StartUp project (right-click in Solution Explorer, "Set as Startup Project")
In the Project Properties for that project, in the "Debug" tab, choose "Start External Program", and choose the NUnit GUI executable. Put the name of UnitTests assembly in "Command line arguments".
Now you can simply press F5 (to start debugging) which will build your projects, and launch the NUnit GUI for you. This has the added advantage that if your tests fail, you can add breakpoints, and step through using the debugger (without needing to attach the debugger manually).
Whenever you are building your solution. I suggest you to close the Nunit, which is loaded with your .dll. After unloading try to build, then your build will succeed.
As Nunit is using you .dll. It won't allow your build to be success.
I had a similar problem a while ago and my final solution was to resign from using NUnit GUI runner and just run them directly from code.
In my project I referenced to NUnit libraries needed by those freshly built dlls plus nunit.core and nunit.utils. The code itself is very simple:
TestResult ExecuteTests(string testAssemblyPath) {
CoreExtensions.Host.InitializeService();
TestPackage testPackage = new TestPackage(testAssemblyPath);
testPackage.BasePath = Path.GetDirectoryName(testAssemblyPath);
RemoteTestRunner testRunner = new RemoteTestRunner();
testRunner.Load(testPackage);
TestResult testResult = testRunner.Run(new NullListener(), TestFilter.Empty, true, LoggingThreshold.Warn);
testRunner.Unload();
CoreExtensions.Host.UnloadService();
return testResult;
}
The TestResult object is very powerful. Among other things, it contains all the results, subresults, tests themselves etc. In order to analyze them, you can either create a simple parser or use one of the possibilities given by NUnit libraries. My favorite is XmlResultWriter but there are others available as well. All of them can be found in nunit.util.dll.
Unfortunately, that would still block the loaded dll from being rebuilt. I avoided this problem by running this in a separate AppDomain and unloading this domain after the tests are finished running. Then dll is nicely freed and you can do with it whatever you want.