Assume I have a test class with a theory method that runs with two inputs, "irrelevant" and "irrelevant2". The test first checks if a static class "IsInitialized", if it does, the test fails.
Then, the test initializes the static class by calling "Initialize".
[Theory]
[InlineData("irrelevant")]
[InlineData("irrelevant2")]
public void Test(string param)
{
if (MyStaticClass.IsInitialized()) { throw new Exception(); }
MyStaticClass.Initialize();
}
public static class MyStaticClass
{
private static bool Initialized = false;
public static void Initialize()
{
Initialized = true;
}
public static bool IsInitialized()
{
return Initialized;
}
}
What I expect is that both tests will pass, as the static class is only initialized after calling "Initialize". However, the result is that the first test pass and the second fails, because the static class remains in memory. I'd expect the static class state to revert to its initial. I can understand why this is happening, because of using a static. However, I'm trying to figure out if there's a way to configure the test to dispose the memory of the static class as if I would run a new test case.
This would also happen if I had two "Fact"s with the same code in them. When running each fact separately, both tests would pass. When running the test class, one would pass (the first) and the second would fail.
What you're describing is expected behavior. If you set the value of static field Initialized to "true" in one test, when you run the next test it's going to be "true".
There are a few ways to look at this:
If all you're testing is that setting a field's value actually works, just don't test that. It's not so much a behavior of your code as a feature of the language. Do we need to write tests to verify that setting a property really sets it? Likely not.
If you have to test it, don't write tests that must fail if the language works correctly. Write a test that verifies that the value is whatever you set it to. A test that verifies what the value is not isn't very useful.
Consider not using a static class and property. The reasons for using a static vs. an instance class are specific to whatever you're coding. I don't know your reasons. But often the challenges we encounter when testing reflect the problems we'll have when using the code "for real."
You asked if you could "dispose" it from memory. You can't do that, or at least the process for doing it is so complicated that you shouldn't do it. You could add a method to your static class that "resets" the value. But that's a little messy. If we have to add methods to static classes just so we can test them then maybe we should consider not using a static class.
Related
I have some test-code that is initializing some members which should be done just once at the beginning. This is why I used the constructor for that:
[TestFixture]
public class MyTestClass
{
private readonly IUnitTestGeometryProvider m_GeometryProvider;
public MyTestClass()
{
// do some heavy init-op
}
private IEnumerable<TestCaseData> TestCases()
{
yield return new TestCaseData(this.m_GeometryProvider.GetPolyline())
.Throws(typeof(ArgumentException));
}
[TestCaseSource("TestCases")]
public double Check_GetReducedArea_For_Invalid_Arguments(IGeometry theGeom)
{
return theGeom.GetReducedArea();
}
}
I know of the convention to use the FixtureSetup-attribute for initializing tests, e.g. from this question on SO. However I noticed that the method TestCases is executed before the method marked with that attribute so I run into an NullReferenceException when evaluating the different testcases as the m_GeometryProvider is null at this time.
So I debugged my code and set a breakpoint into the constructor. I noticed, that it is executed twice before any test has even been run. I assumed every testcase having its own instance of the MyTestClass, but as I have three different testcases and the constructor running twice this doesn't explain it.
As the initialization is heavy I´d like to execute it just once. Is there a way to guarantee this? I´d like to avoid a static member as it often attracts colleagues to heavily use other static members just because there already is one. Furthermore I´d consider the test-init to be specific for one instance of my MyTestClass instead of the class itself - assuming there´s just one however.
I'm using NUnit 2.5.10.
Basically NUnit reserves the right to construct your fixture object whenever it needs to, as often as it needs to. Therefore, you should avoid doing heavy initialization in your constructor, especially if some initialization is only needed when the tests are being run.
Generally, initialization for executing the tests should be done in the TestFixtureSetUp method, which is run once and only once each time the fixture is executed. This is more complicated, however, when you are generating test cases using a TestCaseSource attribute.
Because TestCases must be executed in order to create your tests in the first place, eons before they are ever run, an instance of the object must be created to do that. Then when the tests are run, another instance is created for the purpose of running them. There's not enough info in your question to figure out why there are two "extra" calls to your constructor, but it's either due to some other aspect of your code or simply a bug in the somewhat old version of NUnit you are using. You would have to step through those constructor calls and examine the stack trace to see what is calling them.
Is your m_GeometryProvider member used anywhere else besides the TestCases method? If not, you can simplify things by making it a temporary field created in that method. You can then eliminate an extra constructor call by making the TestCases method static. You said you didn't like that, but it's what we recommend. In fact, in NUnit 3.0 and later it is required to be static.
If you have other initialization that's needed to run tests besides the creation of the geometry provider, that should go in a TestFixtureSetUp method.
Im begginer in webdriver and c#. I want to use variable, from first test in another tests, how do I do that? I got to this point with some examples, but it does not work. I see that first test gets the login right, but when I start the second test and try to sendkeys, I get that loginName is null. (code is in short version, only to give you an idea of what Im trying to do)
[TestFixture]
public class TestClass
{
private IWebDriver driver;
private StringBuilder verificationErrors;
private string baseURL;
private bool acceptNextAlert = true;
static public String loginName;
static public String loginPassword;
[SetUp]
public void SetupTest()...
[TearDown]
public void TeardownTest()...
[Test]
public void GetLoginAndPassword()
{
loginName = driver.FindElement(By.XPath("...")).Text;
loginPassword = driver.FindElement(By.XPath("...")).Text;
Console.WriteLine(loginName);
}
[Test]
public void Test1()
{
driver.FindElement(By.Id("UserNameOrEmail")).SendKeys(loginName);
driver.FindElement(By.Id("Password")).SendKeys(loginPassword);
}
You cannot (and should not) send variables between tests. Tests methods are independent from another... and should actually Assert() something.
Your first method GetLoginAndPassword() isn't a test method per se but a utility method. If you use a Selenium PageObject pattern, this is probably a method of your PageObject class that you can run at the begining of your actual Test1() method.
The problem is that the methods marked with TestAttribute do not run sequentially in the same order you implemented them. Thus it might be possible that Test1 runs long before GetLoginAndPassword). You have to either call that method once from within the constructor or during test-initialization, or before every test-run.
[Test]
public void Test1()
{
GetLoginAndPassword();
driver.FindElement(By.Id("UserNameOrEmail")).SendKeys(loginName);
driver.FindElement(By.Id("Password")).SendKeys(loginPassword);
}
Probably your GetLoginAndPassword isn´t even a method to test but a method used from your tests (unless you actually have a method within your system under test called GetLoginAndPassword. However since there are no asserts at all your tests are somewhat weird.
The purpose of unit testing is to test if a specific unit (meaning a group of closely related classes) works as specified. It is not meant to test if your complete elaborate program works as specified.
Test driven design has the advantage that you are more aware what each function should do. Each function is supposes to transform a pre-condition into a specified post-condition, regardless of what you did before or after calling the function.
If your tests assume that other tests are run before your test is run, then you won't test the use case that the other functions are not called, or only some of these other required functions are called.
This leads to the conclusion that each test method should be able to be run independently. Each test should set up the precondition, call the function and check if the postcondition is met.
But what if my function A only works correctly if other function B is called?
In that case the specification of A ought to describe what happens if B was called before A was called, as well as what would happen if A was called without calling B first.
If your unit test would first test B and then A with the prerequisite that B was called, you would not test whether A would react according to specification without calling B.
Example.
Suppose we have a divider class, that will divide any number by a given denominator that can be set using a property.
public class Divider
{
public double Denominator {get; set;}
public double Divide(double numerator)
{
return numerator / this.Denominator;
}
}
It is obvious that in normal usage one ought to set property Denominator before calling Divide:
Divider divider = new divider(){Denominator = 3.14};
Console.WriteLine(divider.Divide(10);
Your specification ought to describe what happens if Divide is called without setting Denominator to a non-zero value. This description would be like:
If method Divide is called with a parameter value X and the value of Denominator is a non-zero Y, then the return value is X/Y. If the value of Denominator is zero, then the System.DivideByZeroException is thrown.
You should create at least two tests. One for the use case that Denominator was set at a proper non-zero value, and one for the use case that the Denominator is not set at all. And if you are very thorough: a test for the use case that the Denominator is first set to a non-zero value and then to a zero value.
I've got a codebase that makes use of static variables in a number of cases where that makes sense, for example flagging that something's already run once since launch, etc etc.
Of course, this can lead to issues with unit testing whereupon suddenly order matters and the outcome of a test on a method of such a class may depend on whether other code has been hit before, etc. My understand of TestTools.UnitTesting is that whenever I run a set of unit tests, any within the same project run within the same process, thus any static state is maintained from test to test, whereas a Unit Test project boundary also implies a process boundary and thus, if I run 3 tests from project A then a fourth from project B, state is maintained from 1>2>3 (in whatever order they run) but then 4 is virgin and any static state is default.
So now my questions are two:
1) is my assessment correct that a unit test projects have a 1:1 relationship with processes when tests are run in a group (run all or run selected), or is there more nuance there that I'm missing?
2) Regardless, if I have a test that definitely needs fresh, default static state for the custom objects it uses and tests, do I have a more elegant option for creating it than giving it its own test project?
Statics are not actually per process, but per application domain, represented by the AppDomain class. A single process can have several AppDomains. AppDomains have their own statics, can provide sandboxing to partially trusted code, and can be unloaded allowing newer versions of the same assembly to be hot swapped without restarting the application.
Your test runner is likely creating a new AppDomain per test assembly so each assembly gets its own static variables. You can create an AppDomain to do the same on the fly. This is not typically great for pure unit tests, but I've had to work with "rude" libraries that do all kinds of static initialization and caching that cannot be cleaned out or reset. In those sorts of integration scenarios it is very useful.
You can use this helper to run a simple delegate:
public static class AppDomainHelper
{
public static void Run(Action action)
{
var domain = AppDomain.CreateDomain("test domain");
try
{
domain.DoCallBack(new CrossAppDomainDelegate(action));
}
finally
{
AppDomain.Unload(domain);
}
}
}
One caution is that the delegate action passed to Run cannot have any captured variables (as in from a lambda). That doesn't work because the compiler will generate a hidden class that is not serializable and so it cannot pass through an AppDomain boundary.
Your assessment is correct as far as I know -- the assembly is loaded at the start of the test process, and any static state is maintained throughout the tests.
You should always start with a "fresh" state. Unit tests should be able to be run in any order, with no dependencies whatsoever. The reason is because your tests need to be reliable -- a test should only ever fail for one reason: The code it's testing changed. If you have tests that depend on other tests, then you can easily end up with one test failing and "breaking the chain" such that a dozen other tests fail.
You can use the TestInitialize attribute to define a method that will run before every test that will reset the state to the baseline.
Another way to enable this is to wrap your static state into a singleton, then put a "back door" into the singleton so that you can inject an instance of the singleton class, allowing you to configure the state of the application as part of arranging your test.
If you're not looking to test global state, but rather what you do with values you get from global state, you can work around it.
Consider this simple class definition that uses some static property.
public class Foo
{
public int Bar(int baz)
{
return baz + GlobalState.StaticValue;
}
}
You can refactor it like this.
public class Foo
{
public virtual int GetGlobalStaticValue
{
return GlobalState.StaticValue;
}
public virtual int Bar(int baz)
{
return baz + this.GetGlobalStaticValue();
}
}
I added virtuals to the method definitions because that's particular to Rhino Mocks, but you get the idea - while running live, your class will pull global state as it does now, but it gives you the hooks to mock out the values that will be returned in your test scenario.
Let's say I write a private method which has a string as an argument and returns a string.
private string SomeMethod(string a)
{ ... }
After writing it, I want to make sure this method works correctly and input a sample string to see what the return value is. Now I test it the following way. I either:
comment out everything I have in main() function and call just the method SomeMethod() or
I call the method where it should be called in the code, set the breakpoint at that place and debug to see if the return value was correct.
I don't like either method. Is there any better way to simply check and test the method while developing code? I can't (shouldn't) write a unit test for this method as it's private.
If you just want to test it once after creating it, I would recommend to keep another instance of your Development Environment (e.g. Visual Studio) open, with a simple console application where you can just copy-paste the function and test it there.
If you want it regularly tested to ensure it doesn't go wrong with later changes, then I fear Unit testing is exactly what you need. You might use precompiler flags to disable the private scope for testing purposes:
#define TESTING
(...)
#if TESTING
public void foo()
#else
private void foo()
#endif
This is the academia approach for all first year classes at the University of Waterloo, I imagine a similar approach is taken in the real world (my first co-op is 2014 so I don't know just yet). I find it very useful but it takes some time.
If you create a class (henceforth mentioned as a module) then you need you create an associative driver module for each class. This driver file will consist of all your tests: testing edge cases and expected input to ensure your function(s) from your module behave as expected. Each driver module will have a main method and for your testing you will need to change the scope of your function from private to public (or call them all in the constructor).
Keep in mind we are programming in C and driver modules are much easier to make without changing scope (the occasional static for modular scope).
Put the logic in a public method, which you unit test while your are developing. When you are satisfied you have got the logic correct, you can copy and paste the logic in to your private method.
I'm using MSTEST inside Visual Studio 2008. How can I have each unit test method in a certain test class act as if it were the first test to run so that all global state is reset before running each test? I do not want to explicitly clean up the world using TestInitialize, ClassInitialize, AssemblyInitialize, etc. For example:
[TestClass]
public class MyClassTests
{
[TestMethod]
public void Test1()
{
// The "Instance" property creates a new instance of "SomeSingleton"
// if it hasn't been created before.
var i1 = SomeSingleton.Instance;
...
}
[TestMethod]
public void Test2()
{
// When I select "Test1" and "Test2" to run, I'd like Test2
// to have a new AppDomain feel so that the static variable inside
// of "SomeSingleton" is reset (it was previously set in Test1) on
// the call to ".Instance"
var i2 = SomeSingleton.Instance;
// some code
}
Although a similar question appeared on this topic, it only clarified that tests do not run in parallel. I realize that tests run serially, but there doesn't seem to be a way to explicitly force a new AppDomain for each method (or something equivalent to clear all state).
Ideally, I'd like to specify this behavior for only a small subset of my unit tests so that I don't have to pay the penalty of a new AppDomain creation for tests that don't care about global state (the vast majority of my tests).
In the end, I wrote a helper that used AppDomain.CreateDomain and then used reflection to call the unit test under a different AppDomain. It provides the isolation I needed.
This post on MSDN's forums shows how to handle the situation if you only have a few statics that need to be reset. It does mention some options (e.g. using Reflection and PrivateType ).
I continue to welcome any further ideas, especially if I'm missing something obvious about MSTEST.
Add a helper in your tests that uses reflection to delete the singleton instance (you can add a reset method to the singleton as well, but I would be concerned about its use). Something like:
public static class SingletonHelper {
public static void CleanDALFactory()
{
typeof(DalFactory)
.GetField("_instance",BindingFlags.Static | BindingFlags.NonPublic)
.SetValue(null, null);
}
}
Call this in your TestInitialize method. [ I know this is "cleaning up the world", but you only have to write the method once in a helper per singleton, its very trivial and gives you explicit control ]
I think you are looking for the TestIntialize attribute and the TestCleanUp attribute. Here is an MSDN blog showing the execution orderlink text
We had a similar issue arise with our MSTests. We handled it by calling a function at the beginning and end of the specific tests that needed it.
We are storing a test expiration date in our app configuration. Three tests needed this date to fall into a specific range to determine the appropriate values. The way our application is set up, the configuration values would only be reset if there was not a value assigned in session. So, we created two new private static functions - one to explicitly set the configuration value to a specified date and one to clear that date from session after the test runs. In our three tests, we called these two functions. When the next test runs, the application sees an empty value for the date and refetches it from the configuration file.
I'm not sure if that's helpful, but that was how we worked around our similar issue.