How do you usually go about separating your codebase and associated unit tests? I know people who create a separate project for unit tests, which I personally find confusing and difficult to maintain. On the other hand, if you mix up code and its tests in a single project, you end up with binaries related to your unit test framework (be it NUnit, MbUnit or whatever else) and your own binaries side by side.
This is fine for debugging, but once I build a release version, I really do not want my code to reference the unit testing framework any more.
One solution I found is to enclose all your unit tests within #if DEBUG -- #endif directives: when no code references an unit testing assembly, the compiler is clever enough to omit the reference in the compiled code.
Are there any other (possibly more comfortable) options to achieve a similar goal?
I definitely advocate separating your tests out to a separate project. It's the only way to go in my opinion.
Yes, as Gary says, it also forces you to test behavior through public methods rather than playing about with the innards of your classes
As the others point out, a seperate test project (for each normal project) is a good way to do it. I usually mirror the namespaces and create a test class for each normal class with 'test' appended to the name. This is supported directly in the IDE if you have Visual Studio Team System which can automatically generate test classes and methods in another project.
One thing to remember if you want to test classes and methods with the 'internal' accessor is to add the following line to the AssemblyInfo.cs file for each project to be tested:
[assembly: InternalsVisibleTo("UnitTestProjectName")]
The .Net framework after v2 has a useful feature where you can mark an assembly with the InternalsVisibleTo attribute that allows the assembly to be accessed by another.
A sort of assembly tunnelling feature.
Yet another alternative to using compiler directives within a file or creating a separate project is merely to create additional .cs files in your project.
With some magic in the project file itself, you can dictate that:
nunit.framework DLLs are only referenced in a debug build, and
your test files are only included in debug builds
Example .csproj excerpt:
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
...
<Reference Include="nunit.framework" Condition=" '$(Configuration)'=='Debug' ">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\debug\nunit.framework.dll</HintPath>
</Reference>
...
<Compile Include="Test\ClassTest.cs" Condition=" '$(Configuration)'=='Debug' " />
...
</Project>
I would recommend a separate project for unit tests (and yet more projects for integration tests, functional tests etc.). I have tried mixing code and tests in the same project and found it much less maintainable than separating them into separate projects.
Maintaining parallel namespaces and using a sensible naming convention for tests (eg. MyClass and MyClassTest) will help you keeping the codebase maintainable.
As long as your tests are in a seperate project, the tests can reference the codebase, but the codebase never has to reference the tests. I have to ask, what's confusing about maintaining two projects? You can keep them in the same solution for organization.
The complicated part, of course, is when the business has 55 projects in the solution and 60% of them are tests. Count yourself lucky.
I put the tests in a separate project but in the same solution. Granted, in big solutions there might be a lot of projects but the solution explorer is good enough on separating them and if you give everything reasonable names I don't really think it's an issue.
One thing yet to be considered is versions of VisualStudio prior to 2005 did not allow EXE assembly projects to be referenced from other projects. So if you are working on a legacy project in VS.NET your options would be:
Put unit tests in the same project and use conditional compilation to exclude them from release builds.
Move everything to dll assemblies so your exe is just an entry point.
Circumvent the IDE by hacking the project file in a text editor.
Of the three conditional compilation is the least error prone.
I've always keep my unit tests in a seperate project so it compiles to it's own assembly.
For each project there is a corresponding .Test project that contains tests on it.
E.g. for the assembly called, say "Acme.BillingSystem.Utils", there would be a test assembly called "Acme.BillingSystem.Utils.Test".
Exclude it from the shipping version of your product by not shipping that dll.
If the #if(DEBUG) tag allows for a clean "release" version, why would you need a separate project for tests. The nunit LibarryA/B example (yeah, I know its a example) does this. Currently wrestling with the scenario. Had been using a separate project, but this seems to possibly allow for some productivity improvements. Still hummin and hawin.
I definitely agree with everyone else that you should separate the tests from your production code. If you insist on not, however, you should define a conditional comiplation constant called TEST, and wrap all of your unit test classes with a
#if TEST
#endif
first to ensure that the test code does not compile in a production scenario. Once that is done, you should either be able to exclude the test dlls from your production deployment, or even better (but higher maintenance), create an NAnt or MSBuild for production that compiles without the references to the test dlls.
I always create a separate Acme.Stuff.Test project that is compiled separately.
The converse argument is: Why do you want to take the tests out? Why not deliver the test? If you deliver the tests along with a test runner you have some level of acceptance test and self test delivered with the product.
I've heard this argument a few times and thought about it but I personally still keep tests in a separate project.
Related
I am currently playing around with the latest Visual Studio 2017 Release Candidate by creating a .NET Standard 1.6 library. I am using xUnit to unit test my code and was wondering if you can still test internal methods in VS2017.
I remember that you could add a line in AssemblyInfo.cs class in VS2015 that would enable specified projects to see internal methods:
[assembly:InternalsVisibleTo("MyTests")]
As there is no AssemblyInfo.cs class in VS2017 .NET Standard projects, I was wondering if you can still unit test internal methods?
According to .NET docs for the InternalsVisibleToAttribute:
The attribute is applied at the assembly level. This means that it can be included at the beginning of a source code file, or it can be included in the AssemblyInfo file in a Visual Studio project.
In other words, you can simply place it in your own arbitrarily named .cs file, and it should work fine:
// some .cs file included in your project
using System.Runtime.CompilerServices;
[assembly:InternalsVisibleTo("MyTests")]
As described here:
https://blog.sanderaernouts.com/make-internals-visible-with-new-csproj-format
It is possible to add the internal visible attribute within the project file by adding another ItemGroup:
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>$(AssemblyName).Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
or even:
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>$(MSBuildProjectName).Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
I like that solution because the project file seems to be the right place for defining such concerns.
While the first answer is perfectly fine. If you feel you still want to do this in the original AssemblyInfo you can always choose to not auto generate the file and add it manually.
<PropertyGroup>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
For more information:
https://stackoverflow.com/a/47075759/869033
The "InternalsVisibleTo" attribute is key to any sort of "white-box" (the term of the decade, I guess) testing for .Net. It can be placed in any c# file with the "assembly" attribute on the front. Note that MS DOCs say that the assembly name must be qualified by the public key token, if it is signed. Sometimes that does not work and one must use the full public key in it's place. Access to internals is key to testing concurrent systems and in many other situations. See https://www.amazon.com/xUnit-Test-Patterns-Refactoring-Code/dp/0131495054. In this book, Meszaros describes a variety of coding styles that basically constitute a "Design For Test" approach to program development. At least that's the way I've used it over the years.
ADDED:
Sorry, I haven't been on here for a while. One approach is called the "testing subclass" approach by Meszaros. Again, one has to use "internalsvisableto" to access the base class's internals. This is a great solution, but it doesn't work for sealed classes. When I teach "Design For Test", I suggest that it's one of the things that are required to be "pre-engineered" into the base classes in order to provide testability. It has to become almost a cultural thing. Design a "base" base class that is unsealed. Call it UnsealedBaseClass or something uniformly recognizable. This is the class to be subclassed for testing. It is also subclassed to build the production sealed class, which often only differs in the constructors it exposes. I work in the nuclear industry and the testing requirements are taken VERY seriously. So, I have to think about these things all the time. By the way, leaving testing hooks in production code is not considered a problem in our field, as long as they are "internal" in a .Net implementation. The ramifications of NOT testing something can be quite profound.
Another way is to use a 'wrapper' TestMyFoo public class inside the target project that has public methods and inherits from the class you need to test (e.g. MyFoo). These public methods simply call through onto the base class you want to test.
It is not 'ideal' as you end up shipping a test hook in your target project. But consider modern reliable cars ship with diagnostic ports and modern reliable electronics ship with a JTAG connection. But nobody is silly enough to drive their car using the diagnostic port.
I'm working with Unit Test using VS2013 Professional. In particular I'm using the NUnit framework (NUnit TestAdapter for VS2013). My problem is that when I run my tests then VS starts building all the projects inside the solution. Currently the Unit Test project does not reference any solution project.
If I simply code a single test method like:
[Test]
public void SimpleTestMethod(){
Assert.That("a", Is.EqualTo("a"));
}
and the Unit Test project is in a Solution with N project, when I run my test then VS will build all N-1 project... In my case this behavior is boring because it takes too much time (the solution contains many projects) and some projects contain errors.
Is there a way to run my SimpleTestMethod() without complete solution building?
Break your test project to multiple projects that only reference a subset of the solution's projects.
This is also good test housekeeping - have a separate unit test project for each solution project instead of one huge project with dependencies to anything else. There are several advantages to this:
Tests run faster
It's a lot easier to isolate test cases, especially configuration settings
You can version projects and their test cases together
A good naming practice is to name your test projects the same as their target projects with a .Tests suffix. You can also create a solution folder (not a real folder) called "Tests" and move the test projects in it.
As for the why: Test runners use the Unit Test assembly and its dependencies to run their tests. If any of the assembly's dependencies change, the assembly and the dependencies have to be rebuilt. Visual Studio doesn't know what the external tool will call so it has to build all changed assemblies and their dependents.
If the build fails, there are no valid assemblies for the test runner to use so VS has to rebuild the entire solution before the runner can work. In this case, the obvious solution is to fix the error.
There are some stopgap options you can use until you can fix the error:
Temporarily remove the broken project from the build configuration
Split the solution so that you have a solution that can be built and tested
I struggled with this for a very long time as well. I actually hated the automatic build process, even when everything was successful.
I started to run tests through the command line. No build process is necessary. You can write your own .bat files and keep logs of test results. There are plenty of command line parameters that can be added to customize for what you are looking for.
https://msdn.microsoft.com/en-us/library/jj155796.aspx
I am doing an AI project that currently had the following namespace/package structure:
My ideia is to have some Unit/Integration tests for each one of the modules (probably not for all of them. Startup probably won't be tested).
Each one of these "main" packages is in a different physical file. I'd like to know what you think would be better:
Having for each physical file X.dll a X.Tests.dll
Having a Tests physical file that has the whole lot of tests (doesn't seem like a good idea).
Incorporating Breakthrough's tests in Breakthrough.dll, AI.GeneticAlgorithms in AI.GeneticAlgorithms.dll, etc. I kind of like this solution, as then I won't have my Visual Studio solution explorer cluttered with the the whole set of Test projects.
Thanks!
I would have one Test.dll for each .dll.
It's the way I'm used to work, and I think it's a good method. You now where what is tested.
As Liam already pointed out, the implementation and the tests shouldn't be in the same dll. It increases the size of the delivered dll unnecessarily and you have to deliver all the dependencies of the tests (e. g. unit testing framework, mocking etc.)
Personally, I would have one project/assembly for startup, one for tests, and one for everything else.
Build time increases significantly with multiple projects, so the less projects the better. Unless of course you need to be able to deploy the different packages separately.
Tests are generally not included in the project they are testing because it requires the assembly to take a dependency on any testing frameworks you are using.
Is there a Test Stripper (as defined in xUnit Test Patterns) available that supports removing classes containing methods tagged as [Fact]s etc. plus the dependency on xunit.dll from binaries [as part of a build process] ?
Further details of the full requirements and context are at this xUnit CodePlex post.
Failing that (something that removes the tests and the reference to the DLL), does anyone have a utility/proven clean approach to removing the xunit.dll dependecy without too much monkeying around in .vcproj files etc.
Open source preferred. A cleanly written one for NUnit might potentially serve as a base.
NB: I'm aware of the negatives of embedding code into assemblies - this question isnt about that (unless you feel after reading the codeplex thread that some key points are being missed).
If your unit tests are placed in a project resp. assembly of their own there should be no need to strip test classes prior to deployment. What's wrong about this approach?
As noted in the codeplex post where I specified the requirements, I implemented a basic stripper with Mono.Cecil but ended up leveraging the fact that unused references get optimised out in a release build to 'strip' tests out by excluding them based on a #define, which I control from outside via the MSBuild invocation
When writing unit tests, do you place your tests inside the assembly you wish to test or in a separate test assembly? I have written an application with the tests in classes in a separate assembly for ease of deloyment as I can just exclude the assembly. Does anyone write there tests within the assembly you wish to test and if so what is the justification for it?
I have a single solution with an interface project, a tests project, a domain project and a data project. When i release i just publish the interface, which doesnt reference Tests so it doesnt get compiled in.
Edit: The bottom line is really that you dont want it to be part of your final release. You can achieve this automatically in VS by using a separate project/assembly. However you can have it in the same assembly, but then just not compile that code if you're using nant or msbuild. Bit messy though, keep things tidy, use a separate assembly :)
This is widely debated all over the net.
We use separate assemblies and the InternalsVisibleTo attribute to help the tester assemblies see all the innards of the tested assemblies. We tend to make one test assembly for each assembly we're testing.
I'd say that the only time you want to incorporate any test code in your final deployment is when the test code is there for monitoring and control of the application.
Leaving the test code in the deployment:
bloats the code to no real advantage at all - all that extra code tagging along that isn't going to be used at all in situ.
affects releases - do you release a new version of the complete app. when what you've improved is only in the test code, e.g. a test suite is extended as a result of a bug fix.
you can't easily reuse the test framework on another project - if you are always releasing app's that are clogged with test code, any subsequent projects have to reuse the same test code. Otherwise, you finish up with the situation where app A is using v1.2 of the test platform for those aspects of the app. that are common across all your app's, e.g. a presentation layer, or a business logic framework, etc. And app. B is using v1.1 of the test framework, app. C is using v1.2.1, etc.
Decoupling on the otherhand allows you to:
easily upgrade and extend the test suite as required.
easily reuse the test suite across multiple projects.
use a common test framework across multiple projects.
HTH
cheers,
Rob
In a different assembly. Otherwise your assembly will reference the test framework (eg Nunit.Framework.dll) and you'll need to install it on the customer machine,
Even if what you ship is a library, and you want your customer to see the unit tests as an example or a specificiation on how to use the objects you provide, there is very little advantage in including in the production assembly.
You can keep them in separate assemblies in the solution, and ILMerge them later for debugging, and don't ILMerge them for release.