I'm working with Unit Test using VS2013 Professional. In particular I'm using the NUnit framework (NUnit TestAdapter for VS2013). My problem is that when I run my tests then VS starts building all the projects inside the solution. Currently the Unit Test project does not reference any solution project.
If I simply code a single test method like:
[Test]
public void SimpleTestMethod(){
Assert.That("a", Is.EqualTo("a"));
}
and the Unit Test project is in a Solution with N project, when I run my test then VS will build all N-1 project... In my case this behavior is boring because it takes too much time (the solution contains many projects) and some projects contain errors.
Is there a way to run my SimpleTestMethod() without complete solution building?
Break your test project to multiple projects that only reference a subset of the solution's projects.
This is also good test housekeeping - have a separate unit test project for each solution project instead of one huge project with dependencies to anything else. There are several advantages to this:
Tests run faster
It's a lot easier to isolate test cases, especially configuration settings
You can version projects and their test cases together
A good naming practice is to name your test projects the same as their target projects with a .Tests suffix. You can also create a solution folder (not a real folder) called "Tests" and move the test projects in it.
As for the why: Test runners use the Unit Test assembly and its dependencies to run their tests. If any of the assembly's dependencies change, the assembly and the dependencies have to be rebuilt. Visual Studio doesn't know what the external tool will call so it has to build all changed assemblies and their dependents.
If the build fails, there are no valid assemblies for the test runner to use so VS has to rebuild the entire solution before the runner can work. In this case, the obvious solution is to fix the error.
There are some stopgap options you can use until you can fix the error:
Temporarily remove the broken project from the build configuration
Split the solution so that you have a solution that can be built and tested
I struggled with this for a very long time as well. I actually hated the automatic build process, even when everything was successful.
I started to run tests through the command line. No build process is necessary. You can write your own .bat files and keep logs of test results. There are plenty of command line parameters that can be added to customize for what you are looking for.
https://msdn.microsoft.com/en-us/library/jj155796.aspx
Related
I'm struggling to get my unit tests to work. I've wrestled with this issue for several hours now, and I have no explanation as to why things don't work. I had a fairly major refactor on my codebase and have since gone through and fixed all the unit tests. The test project builds, it outputs a new unit test dll. However, when I go to run the tests in the test explorer, I get this message:
[2/27/2019 5:08:05 PM Warning] [MSTest][Discovery][C:\pathtotest.dll] Failed to discover tests from assembly C:\pathtotest.dll. Reason:Could not load file or assembly 'System.Runtime, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
[2/27/2019 5:08:05 PM Warning] No test matches the given testcase filter FullyQualifiedName=<namespace.namespace...testmethod> in C:\pathtotest.dll
Here is what I know:
I recently updated Visual Studio (within the past two weeks, don't remember exactly when I did it).
All packages in the test project have been updated and are running the latest versions of things.
I have another unit test project that is .Net Core 2.1, this is .Net Framework 4.7.2. The other project works.
Some suggestions other posts give are to make sure your test architecture is correct and to delete a folder in %TEMP% (don't recall the exact name, except is was something about VisualStudioExtensions). The folder they suggest to delete was not found in %TEMP% and I tried running my tests on both architectures with the same result.
So the next step was to take a sanity test and make sure the built test dll exists. It does.
At this point I'm about ready to just start a new test project and copy paste over all the tests one by one and see if maybe one is throwing a silent error. I can't find any useful information with my own Google-fu skills and I'm hoping someone has some useful insight or tricks to try.
From the comment above:
Using this C:\pathtotest.dll
Check also if it 32bit or 64bit runtime.
Most cases are using 32bit dlls.
Hope it helps
I had a similar issue with a brand new test project in an existing solution - all of my other test projects compiled and tested correctly, but the new one repeatedly came up with the error:
No test matches the given testcase filter...
The answer came from this post VSTest: A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter. No test is available which suggested switching off the testsettings file, which somehow had been selected on the new project.
As soon as I deselected the Test -> Test Settings -> c:...\Repos...testsettings option in Visual Studio the tests were runnable.
In my case, the issue ended up being a mix of two different test engines. I inadvertently decorated my test methods with [TestMethod] (MSTest) when I was using NUnit. Once I changed my test methods to just [Test] and used the proper test runner, I was finally in business.
I had a similar issue. In my case, the problem was the return type.
The test method needed to use async but the return type was void instead of Task. Simple and small think that might eat a lot of time.
Is it possible to let Resharper (or NUnit?) know that I want each test to look for an App.config under it's own project, even when running all tests in a solution together?
Background:
I'm using NUnit and the test-runner that ships with Resharper, and I've got several test-projects in the same solution. Some of my tests depend on config-files located under their respective projects.
When I run a test-project by itself, it will use it's internal App.config, and everything works fine. When I try to run all the tests in the solution, or use the shortcut to run all tests in current test session however, no config-file will be selected, and any test depending on a config will fail by default.
For this reason, I typically end up running all tests in the solution once first, then right clicking the nodes in the test-runner for each of the config-dependent projects, and running them separately afterwards.
Solved:
Apparently assemblies containing NUnit-tests can be run in separate processes or domains using command line options.
For the testrunner under Resharper, this setting can be found under Resharper > Options > Unit Testing.
There is an option "Use separate AppDomain for each assempbly with tests". Checking that solved my problem.
I have the NUnit GUI runner to run tests on my unit test assembly every time that it is built. The problem is that when I try to build, the .dll in the Debug folder is in use by NUnit, which prevents it from being built, which prevents the automatic test run. Is there a way around this?
Can I suggest a slightly different approach but along the same lines as #UvarajGopu....
As long as you are using Visual Studio higher than "Express" version, and if you write your UnitTests in a separate project (typically alongside the project being tested, with a ".UnitTests" suffix), then do this:
Set your UnitTests project as being the StartUp project (right-click in Solution Explorer, "Set as Startup Project")
In the Project Properties for that project, in the "Debug" tab, choose "Start External Program", and choose the NUnit GUI executable. Put the name of UnitTests assembly in "Command line arguments".
Now you can simply press F5 (to start debugging) which will build your projects, and launch the NUnit GUI for you. This has the added advantage that if your tests fail, you can add breakpoints, and step through using the debugger (without needing to attach the debugger manually).
Whenever you are building your solution. I suggest you to close the Nunit, which is loaded with your .dll. After unloading try to build, then your build will succeed.
As Nunit is using you .dll. It won't allow your build to be success.
I had a similar problem a while ago and my final solution was to resign from using NUnit GUI runner and just run them directly from code.
In my project I referenced to NUnit libraries needed by those freshly built dlls plus nunit.core and nunit.utils. The code itself is very simple:
TestResult ExecuteTests(string testAssemblyPath) {
CoreExtensions.Host.InitializeService();
TestPackage testPackage = new TestPackage(testAssemblyPath);
testPackage.BasePath = Path.GetDirectoryName(testAssemblyPath);
RemoteTestRunner testRunner = new RemoteTestRunner();
testRunner.Load(testPackage);
TestResult testResult = testRunner.Run(new NullListener(), TestFilter.Empty, true, LoggingThreshold.Warn);
testRunner.Unload();
CoreExtensions.Host.UnloadService();
return testResult;
}
The TestResult object is very powerful. Among other things, it contains all the results, subresults, tests themselves etc. In order to analyze them, you can either create a simple parser or use one of the possibilities given by NUnit libraries. My favorite is XmlResultWriter but there are others available as well. All of them can be found in nunit.util.dll.
Unfortunately, that would still block the loaded dll from being rebuilt. I avoided this problem by running this in a separate AppDomain and unloading this domain after the tests are finished running. Then dll is nicely freed and you can do with it whatever you want.
I've written couple of unit tests in seperate project. While developing, I loaded the dll into NUnit.exe each time to check the results. Given that I am done with writing unit tests, how to organize and attach unit tests to the solution.
I'd tried creating "tools" folder in the solution directory and then placed all NUnit related libraries and in postbuild event of the test project I hooked up the nunit like below and it works.
"$(SolutionDir)tools\nunit\nunit-console.exe" "$(TargetPath)"
But in one of my tests, I refer to a sqlite DB. If I hardcode its location it works fine but I intend to place it in the tests project under "App_Data" folder. So, I tried with below code to get relative path but it is not working. But when I copied the file to "tools" folder it is working fine, I'd guess execution context is from NUnit folder.
Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location),"App_Data\\test.txt")
If you don't need specifically nunit, you could as well use built-in visual studio support for unit testing... It has been incorporated to the newest Express editions as well and you don't have to struggle with problems like this ;)
Either way, I believe you've misunderstood the basics of unit testing. We never test against an actual database, that's a very bad practice. You should do some research on the topic and see the theory behind mocking stuff like this.
How do you usually go about separating your codebase and associated unit tests? I know people who create a separate project for unit tests, which I personally find confusing and difficult to maintain. On the other hand, if you mix up code and its tests in a single project, you end up with binaries related to your unit test framework (be it NUnit, MbUnit or whatever else) and your own binaries side by side.
This is fine for debugging, but once I build a release version, I really do not want my code to reference the unit testing framework any more.
One solution I found is to enclose all your unit tests within #if DEBUG -- #endif directives: when no code references an unit testing assembly, the compiler is clever enough to omit the reference in the compiled code.
Are there any other (possibly more comfortable) options to achieve a similar goal?
I definitely advocate separating your tests out to a separate project. It's the only way to go in my opinion.
Yes, as Gary says, it also forces you to test behavior through public methods rather than playing about with the innards of your classes
As the others point out, a seperate test project (for each normal project) is a good way to do it. I usually mirror the namespaces and create a test class for each normal class with 'test' appended to the name. This is supported directly in the IDE if you have Visual Studio Team System which can automatically generate test classes and methods in another project.
One thing to remember if you want to test classes and methods with the 'internal' accessor is to add the following line to the AssemblyInfo.cs file for each project to be tested:
[assembly: InternalsVisibleTo("UnitTestProjectName")]
The .Net framework after v2 has a useful feature where you can mark an assembly with the InternalsVisibleTo attribute that allows the assembly to be accessed by another.
A sort of assembly tunnelling feature.
Yet another alternative to using compiler directives within a file or creating a separate project is merely to create additional .cs files in your project.
With some magic in the project file itself, you can dictate that:
nunit.framework DLLs are only referenced in a debug build, and
your test files are only included in debug builds
Example .csproj excerpt:
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
...
<Reference Include="nunit.framework" Condition=" '$(Configuration)'=='Debug' ">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\debug\nunit.framework.dll</HintPath>
</Reference>
...
<Compile Include="Test\ClassTest.cs" Condition=" '$(Configuration)'=='Debug' " />
...
</Project>
I would recommend a separate project for unit tests (and yet more projects for integration tests, functional tests etc.). I have tried mixing code and tests in the same project and found it much less maintainable than separating them into separate projects.
Maintaining parallel namespaces and using a sensible naming convention for tests (eg. MyClass and MyClassTest) will help you keeping the codebase maintainable.
As long as your tests are in a seperate project, the tests can reference the codebase, but the codebase never has to reference the tests. I have to ask, what's confusing about maintaining two projects? You can keep them in the same solution for organization.
The complicated part, of course, is when the business has 55 projects in the solution and 60% of them are tests. Count yourself lucky.
I put the tests in a separate project but in the same solution. Granted, in big solutions there might be a lot of projects but the solution explorer is good enough on separating them and if you give everything reasonable names I don't really think it's an issue.
One thing yet to be considered is versions of VisualStudio prior to 2005 did not allow EXE assembly projects to be referenced from other projects. So if you are working on a legacy project in VS.NET your options would be:
Put unit tests in the same project and use conditional compilation to exclude them from release builds.
Move everything to dll assemblies so your exe is just an entry point.
Circumvent the IDE by hacking the project file in a text editor.
Of the three conditional compilation is the least error prone.
I've always keep my unit tests in a seperate project so it compiles to it's own assembly.
For each project there is a corresponding .Test project that contains tests on it.
E.g. for the assembly called, say "Acme.BillingSystem.Utils", there would be a test assembly called "Acme.BillingSystem.Utils.Test".
Exclude it from the shipping version of your product by not shipping that dll.
If the #if(DEBUG) tag allows for a clean "release" version, why would you need a separate project for tests. The nunit LibarryA/B example (yeah, I know its a example) does this. Currently wrestling with the scenario. Had been using a separate project, but this seems to possibly allow for some productivity improvements. Still hummin and hawin.
I definitely agree with everyone else that you should separate the tests from your production code. If you insist on not, however, you should define a conditional comiplation constant called TEST, and wrap all of your unit test classes with a
#if TEST
#endif
first to ensure that the test code does not compile in a production scenario. Once that is done, you should either be able to exclude the test dlls from your production deployment, or even better (but higher maintenance), create an NAnt or MSBuild for production that compiles without the references to the test dlls.
I always create a separate Acme.Stuff.Test project that is compiled separately.
The converse argument is: Why do you want to take the tests out? Why not deliver the test? If you deliver the tests along with a test runner you have some level of acceptance test and self test delivered with the product.
I've heard this argument a few times and thought about it but I personally still keep tests in a separate project.