Should unit testing be run in debug or release mode?
I am using Visual Studio Standard Edition 2005 which does not come with any unit testing framework. Since I also do not want to make use of any other 3rd party unit testing framework, I used Debug.Assert to perform the actual test inside all unit test methods. However, Debug.Assert only works in debug mode.
Is there an equivalent in release mode or is there any other alternative (without using 3rd party tools)?
Do you know that you can define the DEBUG constant in any project configuration? (in Visual Studio, Project properties - Build - Define DEBUG symbol). Or you could define a custom TEST constant (Project properties - Build - Conditional compilation symbols), and create test methods that only run when this constant is defined, by using the Conditional attribute.
Anyway I would say that using a 3rd party testing framework such as NUnit is the most appropriate way for this task, but maybe you have solid reasons for not willing to use such tools.
Don't abuse or butcher Trace.Assert or Debug.Assert for unit testing purposes.
Use a third-party framework.
http://blogs.msdn.com/billbar/pages/features-and-behavior-of-load-tests-containing-unit-tests-in-vsts-2008.aspx
As for most of your question, it depends somewhat on what unit testing tool your using. However, in general what you want are preprocessor directives
//C#
ifndef DEBUG
//Unit test
end if
Perhaps for your situation
//C# - for NUnit
if !DEBUG
[Ignore("Only valid for release")]
end if
You can actually use Debug.Assert (or Trace.Assert) in a Release config. Check out this article on MSDN for more information.
Personally I don't see a problem with running unit tests on a Debug config. Unit tests are usually aimed at things like logic, not items that are affected by release vs debug, like performance.
Using 3rd Party testing frameworks allow you to make sure that can get better code coverage of your code. By being able to write tests to test a method instead of just checking info along the way means that you can test edge cases before they make it out into production.
Related
I have a C# .NET assembly, in visual studio 2008. If required I could upgrade it to VS 2010.
How can I find which code in the assembly is not called? Ideally if something could analyse the whole solution in one go. It is too much for me to analyse it all myself manually.
The assembly is shared between a client and server project so I need to understand which code is used where.
In VS 2010 you can use code coverage together with unit tests for that - so if you can refactor your client code into actual unit tests you can use that.
To quote from MSDN:
To determine what proportion of your
project's code is actually being
tested using unit tests, you can use
the code coverage feature of Visual
Studio Application Lifecycle
Management. To do this, first edit the
run configuration to indicate the
assembly that contains the code whose
coverage you want to measure. Then run
tests on that code. Detailed code
coverage statistics appear in a
window, and you can also see,
line-by-line, which code has been
tested.
Try getting resharper. That does that for you. You can get a free trial version from their website. http://www.jetbrains.com/resharper/download/
http://www.jetbrains.com/resharper/
Providing you don't mind buying something to do it, ReSharper will go off and find unused code for you.
FxCOP is a free tool from Microsoft that includes dead code detection.
For best resulsts, don't mark classes/methods public unless they really need to be public. It assumes that any public method is used by external assemblies.
Check Code Analysis in Visual Studio, help here.
You could try the safe delete function of ReSharper, keeping in mind it will say it's safe to delete code which is only used via reflection (such as Fluent automapping conventions).
Couldn't hurt to download a 30 day trial to see if it does what you need anyhow.
Summary: I can run unit tests and code-coverage, but the report only includes NUnit classes, not my application classes.
I have successfully used PartCover in the past. Not so this time. I tried the latest PartCover (4.0), downgraded to the next latest (2.0), both with NUnit 2.5.6.
I created a simple .NET 4.0 class library (also tried this with a web application that has a project that's a class library) with a single class in some namespace, and two test methods in another library in another class.
NUnit/PartCover installed correctly; I can run the NUnit tests both in NUnit, and through PartCover (I can see them running and saying "2 passed"), but the report only shows me NUnit namespaces. (Yes, I'm using +[] as my coverage rule.)
Any ideas? As much as I like NUnit, I'd like to see coverage for my own classes :o)
And I also tried aligning the test-DLL and code-DLL namespaces, to no avail.
Edit: I tried re-running my previously working code-covered sample from a year ago; all the tests run, but the actual project namespaces don't appear. There's a hint here, which seems to imply that it depends on the NUnit version you use: http://sourceforge.net/projects/partcover/forums/forum/605222/topic/3308367 (and yes, I already tried the appdomain-reporting checkbox)
I've tried NUnit 2.5.5.x and 2.5.6.x and both give me the same results.
Edit: It seems this fork of the official 4.0 version seems to work, albeit sporadically (google for PartCover fork, I can't add more hyperlinks)
Madness. Apparently, pressing Pause/Break on your keyboard after NUnit prints out the summary of total passed/failed, and waiting approximately one second for the second "CoreProfiler is turned off" message, makes it all work.
Surely, this cannot be the real solution. Sure, I can rig up a batch file that'll sleep ~1 second after executing NUnit, but this seems like a major hack.
The correct way to handle this is to add the required runtime to nunit's configuration. You will notice that NUnit is running in CLR 2.0 instead of 4.0. There are numerous answers to this question on SO, but I found this one first. Doing this alone fixed it for me. Note, your version of the runtime may be slightly different. You may need to confirm.
In .NET(C#) is there any advantage/disadvantage to go with debug/release build for unit testing?
Which target configuration do you usually use for unit testing on a build server? Does it matter?
What about code coverage (for this one I'm guessing debug versions are needed).
I'd recommend running the release code. For a couple of reasons.
1) It is the code that the customers will be using.
2) Some code has special debug conditionals that will produce differences between the debug and release builds.
You must test the code the way it will ultimately run on the client's machine. In most sane deployment scenarios that will be code compiled in the Release configuration.
I would use release build when possible, to get everything as close to the final product as possible.
There are small differences between debug mode and release mode that normally only make a difference for performance, but not result. However, if there is some timing problems with the code they may only show in release mode, so you could take the opportunity to possibly catch those.
Despite most people obviously favour to unit test the release code, I wonder whether the debug build might uncover more errors, though. (I might be wrong)
E.g. afaik in VS debug code, uninitialized variables are forced to some awful value instead of being 0 "by accident". Maybe in .NET it does not make a big difference, but for me, doing mainly algorithmic core code in C++, this can be crucial.
I look forward to any enlightening comments ;).
It is easier (for me) to get to the root of an exception when testing debug code.
Just adding another reason to test in release mode. Some CI services (Appveyor) will fail the build if it comes across a Debug.WriteLine() call, even though the test itself is green.
We are running both, Debug + Release.
We output a separate tests results xml for each build.
Sometimes there are errors in Debug only, and sometimes in Release only, you want to catch them all ASAP.
Good luck!
I try to use NUnit with SnippetCompiler http://www.sliver.com/dotnet/SnippetCompiler/
I have added references to nunit.framework.dll in snippetcompiler (Menu Tools, References) and compiled nunit sample http://www.nunit.org/index.php?p=quickStart&r=2.5.2 to bank.dll
but when I open bank.dll in NUnit GUI it fails saying it cannot load NUnit assembly or one of its dependencies.
Is it possible to fix this ?
I couldn't even get v2.0.8.3 of SnippetCompiler to include the reference. It let me do it, but it wouldn't compile.
In any case writing unit tests isn't the purpose of SnippetCompiler. It's designed to do quick spikes - try something and see if it works. In other words, it's throwaway code.
In addition, the version for .NET 3.5 (the one I'm using) is an alpha release; the developer does not seem to be maintaining this. (Not to put down the author - this was an awesome tool that saved me lots of time!)
For writing spikes against modern versions of .NET, I've switched to LINQPad. Change Edit/Preferences/Query to C# Program and it's very similar to SnippetCompiler. The basic version is free; for a small fee, the registered version provides IntelliSense.
Even if you're writing learning tests with NUnit, you'll want to preserve those tests. Use Visual Studio (or another IDE) and create a separate Class Library project for your tests.
Maybe this is the solution
http://weblogs.asp.net/rosherove/archive/2008/02/21/ad-hoc-unit-tests-with-snippet-compiler.aspx
Ad hoc Unit Tests with Snippet
Compiler
If you're a fan of snippet compiler
(if you're not you should seriously
check it out)Travis Illig published a
little template for writing Typemock
Isolator test inside this handy little
tool.
the reasons you'd need a specialized
template in the first place to write
these kinds of tests in snippet
compiler:
1) Typemock Isolator uses the .NET
profiling APIs to work its magic, so
the .net process running your tests
need to have a couple of environment
variables enabled to work
2) His code template actually creates
and runs a new process that triggers
nunit-console.exe with the path of the
current code you just wrote in snippet
compiler allowing you to effecively
write and run unit tests in snippet
compiler!
3) the nunit-console process will
already have the env. vars as
mentioned in the first item set to it.
Travis' template will work for
anything nunit can run, not just
typemock isolator tests, which is
pretty cool.
When writing unit tests, do you place your tests inside the assembly you wish to test or in a separate test assembly? I have written an application with the tests in classes in a separate assembly for ease of deloyment as I can just exclude the assembly. Does anyone write there tests within the assembly you wish to test and if so what is the justification for it?
I have a single solution with an interface project, a tests project, a domain project and a data project. When i release i just publish the interface, which doesnt reference Tests so it doesnt get compiled in.
Edit: The bottom line is really that you dont want it to be part of your final release. You can achieve this automatically in VS by using a separate project/assembly. However you can have it in the same assembly, but then just not compile that code if you're using nant or msbuild. Bit messy though, keep things tidy, use a separate assembly :)
This is widely debated all over the net.
We use separate assemblies and the InternalsVisibleTo attribute to help the tester assemblies see all the innards of the tested assemblies. We tend to make one test assembly for each assembly we're testing.
I'd say that the only time you want to incorporate any test code in your final deployment is when the test code is there for monitoring and control of the application.
Leaving the test code in the deployment:
bloats the code to no real advantage at all - all that extra code tagging along that isn't going to be used at all in situ.
affects releases - do you release a new version of the complete app. when what you've improved is only in the test code, e.g. a test suite is extended as a result of a bug fix.
you can't easily reuse the test framework on another project - if you are always releasing app's that are clogged with test code, any subsequent projects have to reuse the same test code. Otherwise, you finish up with the situation where app A is using v1.2 of the test platform for those aspects of the app. that are common across all your app's, e.g. a presentation layer, or a business logic framework, etc. And app. B is using v1.1 of the test framework, app. C is using v1.2.1, etc.
Decoupling on the otherhand allows you to:
easily upgrade and extend the test suite as required.
easily reuse the test suite across multiple projects.
use a common test framework across multiple projects.
HTH
cheers,
Rob
In a different assembly. Otherwise your assembly will reference the test framework (eg Nunit.Framework.dll) and you'll need to install it on the customer machine,
Even if what you ship is a library, and you want your customer to see the unit tests as an example or a specificiation on how to use the objects you provide, there is very little advantage in including in the production assembly.
You can keep them in separate assemblies in the solution, and ILMerge them later for debugging, and don't ILMerge them for release.