We Keep Coding

how to organize test data in integration tests

I am working on a large test project consisting of thousands of integration tests.
It is a bit messy with lots of code duplication. Most tests are composed of several steps, and a lot of "dummy" objects are created. With dummy I mean something like this:
new Address {
Name = "fake address",
Email = "some#email.com",
... and so on
}
where it often really doesn't matter what the data is. This kind of code is spread out and duplicated in tests, test base classes, and helpers.
What I want is to have a single "test data builder", having a single responsibility, generate test data which is consumed by the tests.
One approach is to have a class with a bunch of static methods like following:
Something CreateSomething(){
return new Something {
// set default dummy values
}
and an overload:
Something CreateSomething(params){
return new Something {
// create the Something from the params
}
Another approach is to have xml files containing the data but i am afraid then it would be too far away from the tests.
The goal is to move this kind of code out of the tests because right now the tests are big and not readable. In a typical case, of 50 lines of test code, 20-30 is of this kind of code.
Are there any patterns for accomplishing this? Or any example of big open source codebase with something similar that I can have a look at?

For code, use a simple method chaining builder pattern:
public class TestOrderBuilder
{
private order = new Order();
// These methods represent sentances / grammar that describe the sort
// of test data you are building
public AnObjectBuilder AddHighQuantityOrderLine()
{
//... code to add a high quantity order line
return this; // for method chaining
}
// These methods represent sentances / grammar that describe the sort
// of test data you are building
public AnObjectBuilder MakeDescriptionInvalid()
{
//... code to set descriptions etc...
return this; // for method chaining
}
public Order Order
{
get { return this.order; }
}
}
// then using it:
var order = new TestOrderBuilder()
.AddHighQuantityOrderLine()
.MakeDescriptionInvalid()
.Order

I would shy away from xml files that specify test dependencies.
My thought process stems from a lack of refactoring tools that these xml files cannot take advantage of within the Visual Studio ecosystem.
Instead, I would create a TestAPI.
This API will be responsible for serving dependency data to test clients.
Note that the dependency data that is being requested will already be initialized with general data and ready to go for the clients that are requesting the dependencies.
Any values that serve as required test inputs for a particular test would be assigned within the test itself. This is because I want the test to self document its intent or objective and abstract away the dependencies that are not being tested.
XUnit Test Patterns provided me a lot of insight for writing tests.

Unit Testing Methods With File IO

I'm trying to get into the habit of writing unit tests, I've written a few before but they've usually been quite basic...I'd like to start making a move to TDD as i want to improve the quality of my code (design and structure) - reducing coupling, while at the same time hopefully reduce number of regressions which slip through to a testable build.
I have taken a relatively simple project i work on to begin with. The resultant program watches a folder and then acts on files within this folder.
Here is a typical example of some code extracted from the project:
private string RestoreExtension(String file)
{
var unknownFile = Path.GetFileName(file);
var ignoreDir = Path.GetDirectoryName(file) + "\\Unknown";
string newFile;
//We need this library for determining mime
if (CheckLibrary("urlmon.dll"))
{
AddLogLine("Attempting to restore file extension");
var mime = GetMimeType(BufferFile(file, 256));
var extension = FileExtensions.FindExtension(mime);
if (!String.IsNullOrEmpty(extension))
{
AddLogLine("Found extension: " + extension);
newFile = file + "." + extension;
File.Move(file, newFile);
return newFile;
}
}
else
{
AddLogLine("Unable to load urlmon.dll");
}
AddLogLine("Unable to restore file extension");
if (!Directory.Exists(ignoreDir))
{
Directory.CreateDirectory(ignoreDir);
}
var time = DateTime.Now;
const string format = "dd-MM-yyyy HH-mm-ss";
newFile = ignoreDir + "\\" + unknownFile + "-" + time.ToString(format);
File.Move(file, newFile);
return String.Empty;
}
Questions :
How can i test a method using IO?
I don't really want to be using a real file as this would be slow (and more of an integration test?), but i cant really see another way. I could add a switchable IO abstraction layer, but this sounds to me like it may unnecessarily complicate code...
Is a project like this worth unit testing?
By that i mean, is it too simple. Once you strip out .Net and 3rd party library calls there is not much left...So is it a case that the amount of work to make it testable means its not a good candidate to test?
A lot of the methods in this project are private, as this project happens to be a windows service. I've read you should only really test methods which are externally visible (public or exposed through an interface), it is unlikely in this case however that i want to expose any of my own methods. So is this project worth testing (as id probably need to either change method access modifiers to public or add some attributes so that NUnit can see them)?

Regarding how to test File I/O:
A common way of doing this is encapsulating File I/O in a wrapper class. For example:
public class FileHandler : IFileHandler
{
public string GetFilename(string name)
{
return System.IO.GetFileName(name);
}
public string GetDirectoryName(string directory)
{
return System.IO.GetDirectoryName(directory);
}
}
public interface IFileHandler
{
string GetFilename(string name);
string GetDirectoryName(string directory);
}
In your method under test, you only program against the interface. When you run your unit test, you would use a mock object which returns predefined values. For this, you could use Moq.
This is a bit work to do but then you do not need to test File I/O as well.
Best
Jan

How can i test a method using IO?
By introducing abstraction over the .NET's IO libraries and then using real implementation (BCL) in application, and fake (mocked) one in unit test. Such abstraction is fairly simple to write on your own, but those projects already exist (eg. System.IO.Abstraction) so reinventing wheel is pointless, especialy in such trivial matter.
I don't really want to be using a real file as this would be slow
This is correct, in test you want to use fake (mock) objects.
Is a project like this worth unit testing?
Every project is worth testing and you will test it in one way or another (in most extreme case, by manually executing application and doing good, old click-and-wait testing).
Note that almost always it is too time consuming to manually test 10 edge cases of complex parser method which everytime requires you to load large file and wait for results. This is where automated tests in isolated environment help greatly. It is the same with your code - file system, web service, they all introduce large overhead before you can get to actual business logic. Must isolate.
A lot of the methods in this project are private (...) I've read you should only really test methods which are externally visible
Correct again, you should be testing public contract. However, if private stuff gets big enough to test it, most of the times it's a good indicator it is also big enough to put it in its own class. Note that changing members access modifiers only for the unit test usage is not the best idea. Almost always it's better to extract-class/method refactor.

You can do that using Microsoft Fakes. First generate a fake assembly for System.dll - or any other package and then mock expected returns as in:
using Microsoft.QualityTools.Testing.Fakes;
...
using (ShimsContext.Create())
{
System.IO.Fakes.ShimDirectory.ExistsString = (p) => true;
System.IO.Fakes.ShimFile.MoveStringString = (p,n) => {};
// .. and other methods you'd like to test
var result = RestoreExtension("C:\myfile.ext");
// then you can assert with this result
}

I'm pretty new with TDD too, but I think I can get some suggestion to you:
About IO testing. You can use mock objects. Create a mock for the method you want to test and compare it with real passed in value. In Moq Framework it looks like that:
var mock = new Mock<IRepository<PersonalCabinetAccount>>();
mock.Setup(m => m.RemoveByID(expectedID))
.Callback((Int32 a) => Assert.AreEqual(expectedID, a));
var account = new PersonalCabinetAccount {ID = myID};
var repository = mock.Object;
repository.RemoveByID(account.myId);
Absolutely, yes. If you want to get a habit, you need to test everything, i suppose. That way knowledge comes ;)
You can use InternalsVisibleTo attribute to avoid visibility issues.
test methods which are externally visible
I think this one about enterprise programming. just a spare of time with your own trusted code.
Hope my point can help you.

Finding where a method is called for a convention test

I'm trying to write a convention test that specifies that a method should only be called in some contexts - specifically I have a static Empty getter that I only want to allow used in test methods, vis methods decorated with TestAttribute.
I know that I should also mark the getter as obsolete, use another method etc, but I also want a convention test around this so it doesn't break in the future.
I am guessing I want to use static analysis through reflection in my convention test. How would I go about performing this kind of analysis?

Yes, Roslyn can help with this sort of thing. An example of what this might look like as a standalone analysis would be something like:
var solution = Solution.Load(pathToSolution);
foreach (var project in solution.Projects)
{
var type = project.GetCompilation().GetTypeByMetadataName(typeNameContainingMethod);
var method = type.GetMembers("Empty").Single();
var references = method.FindAllReferences(solution);
foreach (var referencedSymbol in references)
{
foreach (var referenceLocation in references)
{
CheckIfCallIsAllowed(referenceLocation);
}
}
}
You might also look at the Creating a Code Issue walkthrough and the Code Issue template that comes with the Roslyn CTP for another approach to doing this at edit time, instead of in a test.

Code Contracts and failure on private static readonly fields

I have a private static readonly field in my class:
public class MyClass
{
// ISSUE #1 -- requires unproven: path != null
private static readonly DirectoryInfo MyDirectory =
new DirectoryInfo(Settings.Default.MyDirectoryPath);
protected virtual void SomeMethod()
{
if (MyDirectory.Exists)
{
// ISSUE #2 -- requires unproven: !string.IsNullOrEmpty(path)
var catalog = new DirectoryCatalog(MyDirectory.FullName);
}
}
}
For issue #1 I used a null coalescing operator to default to some magic string and that fixed it, but I don't really like that solution. I was hoping there was a better solution.
For issue #2 the only thing I can think of is using a Contract.Assumes because if I attempt to use Contract.Requires(MyDirectory.Exists || !String.IsNullOrEmpty(MyDirectory.FullName)); it complains about visibility (private field used in a requires on a protected method).

Issue #1 is a result of Settings.Default.MyDirectoryPath being code generated by Visual Studio without any contracts on the property. This issue is not limited to null strings. Many API's now have contracts that require say a TimeSpan to be non-negative but using a setting directly in the API will generate a Code Contracts warning.
A way to solve this issue is to wrap the setting in a method that has a contract. E.g.:
String GetMyDirectoryPath() {
Contract.Ensures(Contract.Result<String>() != null);
var myDirectoryPath = Settings.Default.MyDirectoryPath;
Contract.Assume(myDirectoryPath != null);
return myDirectoryPath;
}
Notice how the Contract.Assume really performs validation of your setting (which can't be verified by Code Contracts because it is controlled by an external configuration file). Had it been a TimeSpan that is expected to be non-negative you can either use Contract.Assume to do the validation resulting in a ContractException or some other method using your own exception instead.
Adding this extra layer is somewhat tedious but because the setting is defined outside the application it needs to be run-time validated at some point just as you have to validate interactive user input.
Issue #2 is probably because DirectoryInfo doesn't have any contracts defined. The easist way is to use Contract.Assume. This will make a statement about what you believe is the expected behavior of DirectoryInfo but a run-time check will still be in place to ensure that your belief is correct (provided that you keep the checks in your code).
var path = MyDirectory.FullName;
Contract.Assume(!string.IsNullOrEmpty(path));
var catalog = new DirectoryCatalog(path);

After having used Code Contracts in a current project for a while now I have found that it does force you to rewrite your code at times to correct for issues. You really have two options here.
You can add the setting to your project settings to output what the correct attributes to apply are to ignore certain warnings. This is done by adding the "-outputwarnmasks" flag to the "Extra Static Checker Options" under the Advanced section in the Code Contracts tab of the Project file settings. This will add information to the Build Output window giving you the correct attributes to add to ignore the individual cases. (very useful when dealing with Entity Framework).
You can rewrite your code to add the proper Requires and Ensures to your code so that the warnings don't appear.
If you want to rewrite the code:
To solve Issue #1 you would have to wrap the Settings class and expose a new MyDirectoryPath as a property that isn't code generated so that you can add a check in it and return an empty string and add the Contract.Ensures(Contract.Result<string>() != null) at the top of the Getter for the property.
To solve Issue #2 you would have to wrap you access to the class field inside a private static property that adds the proper Ensures and Requires.
I have usually gone with rewriting the code wherever possible except with Entity Framework/LINQ where you need to add the attributes, especially with complex queries.
** Disclaimer ** These are just the ways I have found to solve the issues as there isn't a great deal of information on other ways of working around these types of items.

Well, for Issue#2, I think you might want to use && not ||. But beyond that, perhaps for Issue#1 you can put those checks in the static constructor? Another option for Issue#2 is to have the method to take the directory as a parameter:
private static readonly DirectoryInfo MyDirectory;
static MyClass()
{
Contract.Requires(Settings.Default.MyDirectoryPath != null);
MyDirectory = new DirectoryInfo(Settings.Default.MyDirectoryPath);
}
protected void SomeMethod()
{
SomeOtherMethod(MyDirectory);
}
protected virtual void SomeOtherMethod(DirectoryInfo directory)
{
Contract.Requires(directory.Exists && !String.IsNullOrEmpty(directory.FullName));
var catalog = new DirectoryCatalog(directory.FullName);
}
I don't have much experience working with the Contract API, so I might be off my rocker with all this. :)

Contract.Requires(MyDirectory.Exists || !String.IsNullOrEmpty(MyDirectory.FullName));
Don't do this! MyDirectory.Exists can change at any time and the caller cannot guarantee it. Just throw an exception if the directory doesn't exist - this is what exceptions are for.

Are those unit tests fine?

I'm trying to grasp test driven development, and I'm wondering if those unit tests is fine. I have a interface which looks like this:
public interface IEntryRepository
{
IEnumerable<Entry> FetchAll();
Entry Fetch(int id);
void Add(Entry entry);
void Delete(Entry entry);
}
And then this class which implements that interface:
public class EntryRepository : IEntryRepository
{
public List<Entry> Entries {get; set; }
public EntryRepository()
{
Entries = new List<Entry>();
}
public IEnumerable<Entry> FetchAll()
{
throw new NotImplementedException();
}
public Entry Fetch(int id)
{
return Entries.SingleOrDefault(e => e.ID == id);
}
public void Add(Entry entry)
{
Entries.Add(entry);
}
public void Delete(Entry entry)
{
Entries.Remove(entry);
}
}
Theese are the unit tests I have written so far, are they fine or should I do something different? Should i be mocking the EntryRepository?
[TestClass]
public class EntryRepositoryTests
{
private EntryRepository rep;
public EntryRepositoryTests()
{
rep = new EntryRepository();
}
[TestMethod]
public void TestAddEntry()
{
Entry e = new Entry { ID = 1, Date = DateTime.Now, Task = "Testing" };
rep.Add(e);
Assert.AreEqual(1, rep.Entries.Count, "Add entry failed");
}
[TestMethod]
public void TestRemoveEntry()
{
Entry e = new Entry { ID = 1, Date = DateTime.Now, Task = "Testing" };
rep.Add(e);
rep.Delete(e);
Assert.AreEqual(null, rep.Entries.SingleOrDefault(i => i.ID == 1), "Delete entry failed");
}
[TestMethod]
public void TestFetchEntry()
{
Entry e = new Entry { ID = 2, Date = DateTime.Now, Task = "Testing" };
rep.Add(e);
Assert.AreEqual(2, rep.Fetch(2).ID, "Fetch entry failed");
}
}
Thanks!

Just off the top of my head...
Although your testing of add really only tests the framework:
You've got adding 1 item, that's good
what about adding LOTS of items
(I mean, ridiculous amounts - for what value of n entries does the container add fail?)
what about adding no items? (null entry)
if you add items to the list, are they in a particular order?
should they be?
likewise with your fetch:
what happens in your fetch(x) if x > rep.Count ?
what happens if x < 0?
what happens if the rep is empty?
does x match performance requirements (what's it's algorithmic
complexity? is it within range when there's just one entry and when
there's a ridiculously large amount of entries?
There's a good checklist in the book Pragmatic Unit Testing (good book, highly recommended)
Are the results right?
Are all the boundary conditions CORRECT
Conform to an expected format
Ordered correctly
In a reasonable range
Does it Reference any external dependencies
Is the Cardinality correct? (right number of values)
does it complete in the correct amount of Time (real or relative)
Can you check inverse relationships
Can you cross check the results with another proven method
Can you force error conditions
Are performance characteristics within bounds

Here's some thoughts:
Positive
You're Unit Testing!
You're following the convention Arrange, Act, Assert
Negative
Where's the test to remove an entry when there's no entry?
Where's the test to fetch an entry when there's no entry?
What is supposed to happen when you add two entries and remove one? Which one should be left?
Should Entries be public. The fact that one of your asserts calls rep.Entries.SingleOrDefault suggests to me you're not constructing the class correctly.
Your test naming is a bit vague; typically a good pattern to follow is: {MethodName}_{Context}_{Expected Behavior} that remove the redundancy "test" redundancy.
As a beginner to TDD I found the book Test-Driven Development By Example to be a huge help. Second, Roy Osherove has some good Test Review video tutorials, check those out.

Before I answer, let me state that I am fairly new to unit testing and by no means an expert, so take everything I state with a grain of salt.
But I feel that your unit tests are largely redundant. Many of your methods are simple pass through, like your AddEntry method is simply a call to the underlying List method Add. Your not testing your code, your testing the Java library.
I would recommend only unit testing methods that contain logic that you write. Avoid testing obvious methods like getters and setters, because they operate at the most basic level. That's my philosophy, but I know some people do believe in testing obvious methods, I just happen to think it is pointless.

Seems fine like this. I personally like to give my tests a bit more descriptive names but that's more of a personal preference.
You can use mocking for dependencies of the class you're testing, EntryRepository is the class under test so no need to mock that, else you would end up testing the mock implementation instead of the class.
Just to give a quick example. If your EntryRepository would use a backend database to store the Entries instead of a List you could inject a mock-implementation for the data-access stuff instead of calling a real database.

This looks like a fine start, but you should try to test the 'borderline' cases as much as possible. Think about what might cause your methods to fail - Would passing a null Entry to Add or Delete be valid? Try to write tests that exercise every possible code path. Writing tests in this manner will make your test suite more useful in the future, should you make any changes to the code.
Also, it is useful for each test method to leave the test object in the same state as when it was invoked. I noticed your TestFetchEntry method adds an element to the EntryRepository, but never removes it. Having each method not affect test object state makes it easier to run series of tests.

You should not be mocking the IEntryRepository since the implementing class is the class under test. You might want to mock the List<Entry> and inject it, then just test that the methods that you invoke via your public interface are called appropriately. This would just be an alternative to the way you have it implemented and isn't necessarily better -- unless you want the class to have it's backing class injected in which case writing the tests that way would force that behavior.
You might want some more tests to make sure that when you insert an entry, the right entry is inserted. Likewise with delete -- insert a couple of entries, then delete one and make sure that the proper one has been deleted. Once you come up with tests to get the code to do what you want it to, keep thinking of ways that you could mess up writing the code and write tests to make sure those don't happen. Granted this class is pretty simple and you may be able to convince yourself that the tests you have that drive behavior is sufficient. It doesn't take much complexity, though, to make it worth testing edge cases and unexpected behavior.

For a TDD beginner and for this particular class, your tests are fine. +1 for making the effort.
Post another question once you get to more complex scenarios involving dependency injection and mocking. This is where things get really interesting ;).

Looks good overall. You should use transactions (or create new instance of repository in TestInitialize) to make sure the tests are trully isolated.
Also use more descriptive test methods, like When_a_new_entity_is_added_to_a_EntryRepository_the_total_count_of_objects_should_get_incremented