We Keep Coding

Unit Tests failing when I Run All Tests but pass when I Debug

I'm using NUnit3 in Visual Studio 2017 and doing TDD. Something really strange is happening since I updated my code to make my latest test pass.
Now, 3 of my other tests are failing when I click Run All Tests, as below:
It is telling me that the actual and expected values in my Assert method are not equal.
However, when I put a breakpoint at the line where the Assert method is and start debugging, the stacktrace is showing that expected and actual are the same value and then the test passes, as below:
Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?
This ever happen to anyone else?
[Edit: I should probably add that I have written each test as a separate class]

The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects.
You should also look into static fields or properties in the subjects. They tends to cause issues if not used properly when designing your classes.

Some subtle differences might occur. For instance if a first test change a state which affects the behavior of a second test, then the outcome of this 2nd test may not be the same if I run it alone.
An idea to help understand a test failure when a breakpoint can't be used, could be to add logging.
Anyway, to answer your questions:
This ever happen to anyone else?
Yes
Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?
I bet that it's neither: just a case a bit more subtle

I experienced a similar issue in Visual Studio 2017 using MSTest as the testing framework. Assertions in unit tests were failing when the tests were run but would pass when the unit tests were debugged. This was occurring for a handful of unit tests but not all of them. In addition to the Assertion failures, many of the units tests were also failing due to a System.TypeLoadException (Could not load type from assembly error). I ultimately did the following which solved the problem:
Open the Local.testsettings file in the solution
Go to the "Unit Test" settings
Uncheck the "Use the Load Context for assemblies in the test directory." checkbox
After taking these steps all unit tests started passing when run.

I encountered this phenomenon myself, but found the cause quite easily. More concretely, I tested some matrix calculations, and in my test class I defined data to calculate with as a class variable and performed my calculations with it. My matrix routines, however, modified the original data, so when I used "run tests" on the test class, the first test corrupted the data, and the next test could not succeed.
The sample code below is an attempt to show what I mean.
[TestFixture]
public void MyTestClass()
{
[Test]
public void TestMethod1()
{
MyMatrix m = new MyMatrix();
// Method1() modifies the data...
m.Method1(_data);
}
[Test]
public void TestMethod2()
{
MyMatrix m = new MyMatrix();
// here you test with modified data and, in general, cannot expect success
m.Method2(_data);
}
// the data to test with
private double[] _data = new double[1, 2, 3, 4]{};
}

Code Contract or if statement?

I just tried to use Code Contracts, and I see no real advantages over an if statement.
Consider the following.
private static void BindClassesToInterfacesByConvention(string classesEndingWith
, string interfacesEndingwith) {
Contract.Requires<ArgumentNullexception>(
string.IsNullOrWhiteSpace(classesEndingWith)
, "classesEndingWith");
Contract.Requires<ArgumentNullException>(
string.IsNullOrWhitespace(interfacesEndingWith)
, "interfacesendingWith");
...
}
I find it way more confusing than simply using an if statement
private static void BindClassesToInterfacesByConvention(string classesEndingWith
, string interfacesEndingwith) {
if (string.IsNullOrWhiteSpace(classesEndingWith))
throw new ArgumentNullException("classesEndingWith");
if (string.IsNullOrWhitespace(interfacesEndingWith))
throw new ArgumentNullException("interfacesendingWith");
...
}
Code Contracts are supposed to warn me at compile time that a a contract is violated. So, I was expecting to get an error or a warning when I have written the following.
BindClassesToInterfacesByConvention(null, null);
And nothing happened, everything compiled just fine, and neither an error nor a warning message appeared.
In this scenario, I believe it is best to continue with the it statement. Or perhaps was it an unfair use of Code Contracts?

Code Contracts are a great idea let down by tooling that isn't quite there.
Firstly, in order to make the exceptions actually throw, you have to install the right extensions in Visual Studio and/or configure the correct settings on your project. Great fun if you have unit tests relying on the code contracts throwing exceptions at runtime and run them on a build server.
However, it is important to understand that the real purpose of Code Contracts isn't just to throw exceptions. It enables static code analysis (if you switch it on), which when enabled may be able to give you an error at compile time - but it does require you to do a lot of work to apply it pretty much everywhere in order for the static code analysis to really work. I believe that is the scenario you are trying to test? In which case I suggest you look at the code contracts setting for your project to make sure you have enabled all the static code checking (it will make your build rather long).
Furthermore and importantly, the code contracts allows you to communicate intent to the callers of your methods; Intellisense will pick up on the conditions you have specified (provided you install the correct extensions). The informatation about the code contracts can also automatically be added to the XML file that can accompany assemblies, which will then enable 3rd party users of your assembly to know about your requirements when they write their code, as well as allowing you to include this information in helpfiles built with Sandcastle etc.
It's a great idea, just not really fully implemented in the tools yet so you get some funny behavior once in a while. Personally I have pretty much stopped using them for the time being.

You are specifying a method that requires its arguments to be null or whitespace, then passing null. The contract is satisfied. That's why you got no contract violation. (Requires() requires throws the exception when the condition evaluates to false, not to true.)
Furthermore, even if you correct the contract, you shouldn't be throwing ArgumentNullException if the parameter value is a non-null string containing no characters or only whitespace characters. In that case you should throw an ArgumentException.
I would do this:
private static void BindClassesToInterfacesByConvention(string classesEndingWith
, string interfacesEndingwith) {
Contract.Requires<ArgumentNullException>(classesEndingWith != null
, "classesEndingWith");
Contract.Requires<ArgumentException>(!string.IsNullOrWhiteSpace(classesEndingWith)
, "classesEndingWith");
Contract.Requires<ArgumentNullException>(interfacesEndingWith != null
, "interfacesEndingWith");
Contract.Requires<ArgumentException>(!string.IsNullOrWhiteSpace(interfacesEndingWith)
, "interfacesEndingWith");
...
}
To download the Code Contracts analysis tools, including Visual Studio integration, visit http://visualstudiogallery.msdn.microsoft.com/1ec7db13-3363-46c9-851f-1ce455f66970

This is a subjective answer, but I would say the real statement here is: "I just tried to use Code Contracts, and I see no real advantages over a Unit Test"
For Example:
private void Foo(string something)
{
Contract.Requires<ArgumentNullException>(something != null, "something");
}
is equivalent as (NUnit testing):
void Foo(string something)
{
if (something == null)
throw new ArgumentNullException();
}
[Test]
[ExpectedException( typeof( ArgumentNullException ) )]
void foo_throws_exception_with_null_param()
{
Foo(null);
}
Which is better? Well from my (limited) experience the static analysis addon for VS is quite slow. If you have done a call to foo with an explicit null'ed variable then it'll pick it up. But it won't pick up nulls loaded dynamically and send to foo during user iteraction.
On the other hand, if you have an if-statement and a unit test to ensure it WILL throw an ArgumentNullException then you know that the exception is going to be thrown; and you can deal with it in the run-time environment... you can test anything that uses Foo to make sure it handles the exception.
Ensuring that explicit check is very fast with NUnit. The downside of unit testing is setting up the tests to begin with. So my opinion is that over time you'll save more time by being explicit, making unit tests and ensuring your application can handle if those exceptions are thrown... it'll just cost you more to begin with to set it up.

Yes, Code Contracts will warn you at compile time that a a contract is violated if you enable the static checking.It proved the Code Contract add-ons(Code Contracts for .NET)only works on Visual Studio 2013 or lower versions, but not Visual Studio 2015 or 2017.
On Visual Studio 2013:
On Visual Studio 2015, we can see the error information from Output window, but not in Error List window. This bug was already logged and fixed, but it can still be repro.Warnings and messages from static contract checking don't appear in the VS2015 error list
Latest update:
Install the latest Contracts.devlab9ts.msi(Currently, it's just RC version) will resolve the issue in Visual Studio 2015.
DotNet CodeContracts v.1.10.20606.1-rc2

Using Moles to mock static constructor - user-defined breakpoint encountered

I'm using Moles in a legacy project where I need to mock out a class with a static constructor which does something I don't want it to in a testing environment. No problem - MolesEraseStaticConstructor attribute to the rescue, right? Well, not quite...
When I try to run my tests, I get a pop-up saying: "Microsoft.Moles.VsHost has encountered a user defined breakpoint." with the description: "A breakpoint in an application indicates a program error. After this dialog is dismissed, the application will continue running, but it may be in an unstable state."
The last part of the message is true: If I choose "Close", sometimes the test fails and sometimes it doesn't - and yet other times it gets aborted.
If I choose Debug, I go to some assembly code I can't figure out where originates.
How can I figure out more about what' going wrong here?
(A little aside-question: I tried to mock out a class inside the static constructor to try and get around it that way, but it doesn't seem to work. Am I right is assuming that you can't mock something inside a static constructor with Moles?)

I ran into this problem as well.
I had an #ifdef DEBUG, and inside of it, a call to System.Diagnostics.Debugger.Break()
So... there really was a user-defined breakpoint. How silly of me!

Assert.Fail does not fail test?

I am using VS 2008 for unit testing. Even if my code calls
Assert.Fail("some message");
Test passess ??
and even
Assert.IsTrue(false); this also passes test.
How is this possible. I am expecting test to fail . i want to forcefully fail it .
Please help !

Probably you are catching exceptions somewhere in the test code. Assert class uses exceptions to fail the test, so if you are catching it the test passes.
Please paste the code for more accurate answer.

Are you using the Assert class built into .NET? If so, I suspect TRACE was not defined.

Sometimes, when I click "run tests", I've seen the test engine build in Release configuration but then proceed to run tests against an older Debug configuration build. I would call that a bug in the VS2008 test engine. Anyway, try cleaning your Debug AND Release folders and running it again.

I can't be sure without knowing more about what you're doing, but it sounds like you're expecting the .NET classes to work like the NUnit classes -- can't be sure -- but if you are, that will never work.
The .NET classes only work if Trace is defined (view the Project Properties, the click the Build tab, then click the "Define TRACE constant"). -- But for Unit Tests this is not helpful -- those classes do not throw exceptions or otherwise fail unit tests... they pop up message boxes for those subscribing to the "debug later" mantra.
If you want Assert.Fail to fail a unit test you'll need a unit testing framework that provides a class which does this (again, like NUnit). Please, let us know what testing framework you're using (or not using?) and we'll be able to help you a bit more if you're still having trouble with this.

Unit Testing a Customer's Installation (with NUnit and MSTest)

I've developed a large base of unit tests for my company's application, and dev would like to hand my unit tests over to our support department to help them debug customer installation problems. I wrote my unit tests using mstest, so support would have to install Visual Studio on a customer's machine if they wanted to use my tests out of the box, which is the wrong thing to do, obviously. I have looked into using mstest without VS at the command prompt, but hacking the registry on a customer's system to make it think VS is installed isn't workable either.
To get around this I planned on compiling my mstests for nunit using the information in this post . However, after compiling with NUnit enabled, and adding my test assembly dll to the NUnit runner, I get the error message "This assembly was not built with any known framework."
Has anyone done this and has tips/tricks to get this running? Or is this the entirely wrong way to go about solving this problem? Thanks.

I'm going to go with your second thought on this "Or is this the entirely wrong way to go about solving this problem".
To solve this problem easily and not confuse your support department I would recommend creating a little command-line wrapper around the test class. You can write the command-line tool yourself, or if you prefer you can do the following:
using CSharpTest.Net.Commands;
static void Main(string[] args)
{
MyTest testClass = new MyTest();
// optional: testClass.MySetupMethod();
new CommandInterpreter(testClass).Run(args);
}
Just build the above code as a command-line exe in a new project referencing your test assembly and CSharpTest.Net.Libary.dll. The namespace CSharpTest.Net.Commands is defined in the CSharpTest.Net.Libary.dll assemby from this download.
Essentially the above code will crawl your test class (named MyTest in the example above) and expose all the public methods as commands that can be executed via the command line. By default it provides help output and sets the Environment.ExitCode on failure. If you want to get fancy you can decorate your tests with any of the following:
public class MyTest
{
[System.ComponentModel.DisplayName("rename-this-function")]
[System.ComponentModel.Description("Some description for tech-support")]
[System.ComponentModel.Browsable(true | false)]
public void TestSomeFunction()
{ ... }
}
(And yes I acknowledge I am shamelessly plugging my own code a wee-bit :)

My concern would be someone adding a unit test (or integration test disguised as a unit test) which you inadvertently run against your customer database. Of course I'm making assumptions such as you having a database, but it seems a risky approach unless you can be sure that your tests are non-destructive, now and in the future.
Do you have logging in your application? That's the traditional way to help support, and there are many tools out there to help you.