I have a number of unit tests which rely on the presence of a csv file. They will throw an exception if this file doesn't exist obviously.
Are there any Gallio/MbUnit methods which can conditionally skip a test from running? I'm running Gallio 3.1 and using the CsvData attribute
[Test]
[Timeout(1800)]
[CsvData(FilePath = TestDataFolderPath + "TestData.csv", HasHeader = true)]
public static void CalculateShortfallSingleLifeTest()
{
.
.
.
Thanks
According to the answer in this question, you'll need to make a new TestDecoratorAttribute that calls Assert.Inconclusive if the file is missing.
Assert.Inconclusive is very appropriate for your situation because you aren't saying that the test passed or failed; you're just saying that it couldn't be executed in the current state.
What you have here is not a unit test. A unit test tests a single unit of code (it may be large though), and does not depend on external environmental factors, like files or network connections.
Since you are depending on a file here, what you have is an integration test. You're testing whether your code safely integrates with something outside of the control of the code, in this case, the file system.
If this is indeed an integration test, you should change the test so that you're testing the thing that you actually want tested.
If you're still considering this as a unit test, for instance you're attempting to test CSV parsing, then I would refactor the code so that you can mock/stub/fake out the actual reading of the CSV file contents. This way you can more easily provide test data to the CSV parser, and not depend on any external files.
For instance, have you considered that:
An AntiVirus package might not give you immediate access to the file
A typical programmer tool, like TortoiseSvn, integrates shell overlays into Explorer that sometimes hold on to files for too long and doesn't always give access to a file to a program (you deleted the file, and try to overwrite it with a new one? sure, just let me get through the deletion first, but there is a program holding on to the file so it might take a while...)
The file might not actually be there (why is that?)
You might not have read-access to the path
You might have the wrong file contents (leftover from an earlier debugging session?)
Once you start involving external systems like file systems, network connections, etc. there's so many things that can go wrong that what you have is basically a brittle test.
My advice: Figure out what you're trying to test (file system? CSV parser?), and remove dependencies that are conflicting with that goal.
An easy way would be to include an if condition right at the start of the test that would just execute any code in the test if the CSV file can be found.
Of course this has the big drawback that tests would be green although they haven't actually run and asserted anything.
I agree with Grzenio though, if you have unit tests that rely heavily on external conditions, they're not really helping you. In this scenario you will never really know whether the unit test ran successfully or was just skipped, which contradicts what unit tests are actually for.
In my personal opinion, I would just write the test so that they correctly fail when the file is not there. If they fail this is an indicator that the file in question should be available on the machine where the unit tests run. This might need some manual adjustments at times (getting the file to the computer or server in question), but at least you have reliable unit tests.
In Gallio/MbUnit v3.2 the abstract ContentAttribute and its concrete derived types (such as [CsvData] have a new optional parameter that allows to change the default outcome of a test in case of an error occured while opening or reading the file data source (ref. issue 681). The syntax is the following:
[Test]
[CsvData(..., OutcomeOnFileError = OutcomeOnFileError.Inconclusive)]
public void MyTestMethod()
{
// ...
}
Related
In Visual Studio IDE, I can create a unit test file with a unit test class for the code in a source file, by right clicking inside the code to be tested and selecting the option to create unit test.
The testing code and the code to be tested are not only in the same file, but also not in the same namespace.
Is it possible to write the code to be tested and the testing code in the same source file?
If yes, how? Should I put them in the same or different namespaces?
Can you give some examples?
Thanks.
It is possible, but that also means that you deploy your tests with your code, as well as any mocks, dummy data, etc. All of this is unnecessary and may confuse anyone trying to use the library.
However, to answer the question, just use different namespace blocks to separate the test classes in a separate namespace.
namespace MyCompany.MyLibrary
{
// classes
}
namespace MyCompany.MyLibrary.Test
{
// tests, mocks, etc.
}
Yes, there is no restrictions where "code under test" is coming from.
While it is somewhat strange you can have just UnitTest project and put code you trying next to your tests. If you want - even in the same files using same or different namespaces of your choice (C# is not Java and there is no connection bewteen file name/location and namespace)
Yes, but
If you put them in the same code base as the system under test, then you will be unable to deploy the system without also deploying the tests. Do you want the tests sitting on your production servers?
Also, the same app.config (or web.config, depending on your solution) will apply to both your tests and the system under test. That means you can't set up alternate configurations for things like AutoFac, which normally is handy for unit/isolation testing.
I have some classes that implements some logic related to file system and files. For example, I am performing following tasks as part of this logic:
checking if certain folder has certain structure (eg. it contains subfolders with specific names etc...)
loading some files from those folders and checking their structure (eg. these are some configuration files, located at certain place within certain folder)
load additional files for testing/validation from the configuration file (eg. this config file contains information about other files in the same folder, that should have other internal structure etc...)
Now all this logic has some workflow and exceptions are thrown, if something is not right (eg. configuration file is not found at the specific folder location). In addition, there is Managed Extensibility Framework (MEF) involved in this logic, because some of these files I am checking are managed DLLs that I am manually loading to MEF aggregates etc...
Now I'd like to test all this in some way. I was thinking of creating several physical test folders on HDD, that cover various test cases and then run my code against them. I could create for example:
folder with correct structure and all files being valid
folder with correct structure but with invalid configuration file
folder with correct structure but missing configuration file
etc...
Would this be the right approach? I am not sure though how exactly to run my code in this scenario... I certainly don't want to run the whole application and point it to check these mocked folders. Should I use some unit testing framework to write kind of "unit tests", that executes my code against these file system objects?
In general, is all this a correct approach for this kind of testing scenarios? Are there other better approaches?
First of all, I think, it is better to write unit tests to test your logic without touching any external resources. Here you have two options:
you need to use abstraction layer to isolate your logic from external dependencies such as the file system. You can easily stub or mock (by hand or with help of constrained isolation framework such as NSubstitute, FakeItEasy or Moq) this abstractions in unit tests. I prefer this option, because in this case tests push you to a better design.
if you have to deal with legacy code (only in this case), you can use one of the unconstrained isolation frameworks (such as TypeMock Isolator, JustMock or Microsoft Fakes) that can stub/mock pretty much everything (for instance, sealed and static classes, non-virtual methods). But they costs money. The only "free" option is Microsoft Fakes unless you are the happy owner of Visual Studio 2012/2013 Premium/Ultimate.
In unit tests you don't need to test the logic of external libraries such as MEF.
Secondly, if you want to write integration tests, then you need to write "happy path" test (when everything is OK) and some tests that testing your logic in boundary cases (file or directory not found). Unlike #Sergey Berezovskiy, I recommend creating separate folders for each test case. The main advantages is:
you can give your folder meaningful names that more clearly express your
intentions;
you don't need to write complex (i.e. fragile) setup/teardown logic.
even if you decide later to use another folder structure, then you can change it more easily, because you will already have working code and tests (refactoring under test harness is much easier).
For both, unit and integration tests, you can use ordinary unit testing frameworks (like NUnit or xUnit.NET). With this frameworks is pretty easy to launch your tests in Continuous integration scenarios on your Build server.
If you decide to write both kinds of tests, then you need to separate unit tests from integration tests (you can create separate projects for every kind of tests). Reasons for it:
unit tests is a safety net for developers. They must provide quick feedback about expected behavior of system units after last code changes (bug fixes, new features). If they are run frequently, then developer can quickly and easily identify piece of code, that broke the system. Nobody wants to run slow unit tests.
integration tests are generally slower than unit tests. But they have different purpose. They check that units works as expected with real dependencies.
You should test as much logic as possible with unit tests, by abstracting calls to the file system behind interfaces. Using dependency injection and a testing-framework such as FakeItEasy will allow you to test that your interfaces are actually being used/called to operate on the files and folders.
At some point however, you will have to test the implementations working on the file-system too, and this is where you will need integration tests.
The things you need to test seem to be relatively isolated since all you want to test is your own files and directories, on your own file system. If you wanted to test a database, or some other external system with multiple users, etc, things might be more complicated.
I don't think you'll find any "official rules" for how best to do integration tests of this type, but I believe you are on the right track. Some ideas you should strive towards:
Clear standards: Make the rules and purpose of each test absolutely clear.
Automation: The ability to re-run tests quickly and without too much manual tweaking.
Repeatability: A test-situation that you can "reset", so you can re-run tests quickly, with only slight variations.
Create a repeatable test-scenario
In your situation, I would set up two main folders: One in which everything is as it is supposed to be (i.e. working correctly), and one in which all the rules are broken.
I would create these folders and any files in them, then zip each of the folders, and write logic in a test-class for unzipping each of them.
These are not really tests; think of them instead as "scripts" for setting up your test-scenario, enabling you to delete and recreate your folders and files easily and quickly, even if your main integration tests should change or mess them up during testing. The reason for putting them in a test-class, is simply to make then easy to run from the same interface as you will be working with during testing.
Testing
Create two sets of test-classes, one set for each situation (correctly set up folder vs. folder with broken rules). Place these tests in a hierarchy of folders that feels meaningful to you (depending on the complexity of your situation).
It's not clear how familiar you are with unit-/integration-testing. In any case, I would recommend NUnit. I like to use the extensions in Should as well. You can get both of these from Nuget:
install-package Nunit
install-package Should
The should-package will let you write the test-code in a manner like the following:
someCalculatedIntValue.ShouldEqual(3);
someFoundBoolValue.ShouldBeTrue();
Note that there are several test-runners available, to run your tests with. I've personally only had any real experience with the runner built into Resharper, but I'm quite satisfied with it and I have no problems recommending it.
Below is an example of a simple test-class with two tests. Note that in the first, we check for an expected value using an extension method from Should, while we don't explicitly test anything in the second. That is because it is tagged with [ExpectedException], meaning it will fail if an Exception of the specified type is not thrown when the test is run. You can use this to verify that an appropriate exception is thrown whenever one of your rules is broken.
[TestFixture]
public class When_calculating_sums
{
private MyCalculator _calc;
private int _result;
[SetUp] // Runs before each test
public void SetUp()
{
// Create an instance of the class to test:
_calc = new MyCalculator();
// Logic to test the result of:
_result = _calc.Add(1, 1);
}
[Test] // First test
public void Should_return_correct_sum()
{
_result.ShouldEqual(2);
}
[Test] // Second test
[ExpectedException(typeof (DivideByZeroException))]
public void Should_throw_exception_for_invalid_values()
{
// Divide by 0 should throw a DivideByZeroException:
var otherResult = _calc.Divide(5, 0);
}
[TearDown] // Runs after each test (seldom needed in practice)
public void TearDown()
{
_calc.Dispose();
}
}
With all of this in place, you should be able to create and recreate test-scenarios, and run tests on them in a easy and repeatable way.
Edit: As pointed out in a comment, Assert.Throws() is another option for ensuring exceptions are thrown as required. Personally, I like the tag-variant though, and with parameters, you can check things like the error message there too. Another example (assuming a custom error message is being thrown from your calculator):
[ExpectedException(typeof(DivideByZeroException),
ExpectedMessage="Attempted to divide by zero" )]
public void When_attempting_something_silly(){
...
}
I'd go with single test folder. For various test cases you can put different valid/invalid files into that folder as part of context setup. In test teardown just remove those files from folder.
E.g. with Specflow:
Given configuration file not exist
When something
Then foo
Given configuration file exists
And some dll not exists
When something
Then bar
Define each context setup step as copying/not copying appropriate file to your folder. You also can use table for defining which file should be copied to folder:
Given some scenario
| FileName |
| a.config |
| b.invalid.config |
When something
Then foobar
I don't know your program's architecture to give a good advice, but I will try
I believe that you don't need to test real file structure. File access services are defined by system/framework, and they're don't need to be tested. You need to mock this services in related tests.
Also you don't need to test MEF. It is already tested.
Use SOLID principles to make unit tests. Especially take look at Single Responsibility Principle this will allow you to to create unit tests, which won't be related to each others. Just don't forget about mocking to avoid dependencies.
To make integration tests, you can create a set of helper classes, which will emulate scenarios of file structures, which you want to test. This will allow you to stay not attached to machine on which you will run this tests. Such approach maybe more complicated than creating real file structure, but I like it.
I would build framework logic and test concurrency issues and file system exceptions to ensure a well defined test environment.
Try to list all the boundaries of the problem domain. If there are too many, then consider the possibility that your problem is too broadly defined and needs to be broken down. What is the full set of necessary and sufficient conditions required to make your system pass all tests? Then look at every condition and treat it as an individual attack point. And list all the ways you can think of, of breaching that. Try to prove to yourself that you have found them all. Then write a test for each.
I would go through the above process first for the environment, build and test that first to a satisfactory standard and then for the more detailed logic within the workflow. Some iteration may be required if dependencies between the environment and the detailed logic occur to you during testing.
I have a method which deletes a file after processing. I'm writing tests for it, using the same file in the test project. But as expected, this file will be deleted after each execution. How can I make it... come back each time?^^ Sorry if it is very stupid from me
If the code you're testing deletes it then you'll obviously need to recreate it before each test. You can make use of the 'set-up' facility in your unit-testing library to do it.
If you're using Visual Studio Unit-Testing Framework, then this method is created by decorating it with a TestInitialize attribute:
[TestInitialize]
public void Setup()
{
// recreate file here
}
If you're using NUnit then use the SetUp attribute:
[SetUp]
public void Setup()
{
// recreate file here
}
I had a kind of similar situation where I needed to load in a sqlite db and perform actions on it, then dispose of it.
I ended up including the file as an embedded resource, streaming the resource out and creating a file in the test setup method and then removing it afterwards in the cleanup.
In your test setup, you would create a test version of the file, then execute your method, then check that the file is gone.
It sounds like perhaps your test is using the 'production' version of the file - this is a bad idea and to make your method more testable, it should be passed the name of the file to process and delete, so your can use a test file for testing.
Well it depends on the nature of the file and the test. Typically you would want to have your unit test generate the data that it is going to operate on, to ensure a consistent data set each time the unit test is run.
If that is not possible or feasible, then what I would probably do is make a copy of the file in the unit test, maybe with the same name plus a .test extension, and pass that file to the method you are testing.
If the method you are testing gets the filename on it's own somehow, then you can do the opposite; make a copy of the file before the method is called, then after the method executes, rename the copied file back to the original filename.
I'm new to testing and I need an advice for the best testing strategy (and its application).
Here's my problem:
I have a program that reads a file and automatically extracts its structure. I want to test the method that makes this "intelligent" extraction. Initially I can use a few files to check if the method is doing the correct extraction. Then I want to use these files and the (correct) extraction results for testing purposes. As the extraction results have been verified they should (and must) be used for further tests.
So, I have something like: for "this particular file" I expect "this result".
The problems:
It's easy to get the input files for the test. I will store them in a specific directory. What about the results? They affect the contents of the object that stores the file structure. In this case I may need to save this object in a file as well. With serialization I'm afraid that with object's structure changes, it will be difficult to reuse the previous saved objects.
With more and more results I may have hundreds of files and results and the test will take a lot of time. I expect that testing time will not be a big issue.
I need testing because my "extraction algorithm" used in the method will change often. I can't cope with all possibilities in order to have a perfect extraction algorithm. So, my solution is to build an initial algorithm that works in a dozen files and each time that I find a failure in the algorithm for a particular file I change the algorithm in order to solve the problem with that file. This change should be tested in order the previous files and results are still valid.
Any suggestion for the testing strategy?
For tests you need somewhere a place where you can inject input test data and some place where you can observe some behavior or output.
On the input side: Are files really the only possibility to inject input test data? If yes, the application does not have a good testable design. Tests with files are hard to maintain. On the output side: The application does not seem to offer a possibility to observe behavior or output. This is points to a design which is not testable.
Even if you find a way to observe behavior our output, there will be only end-to-end tests for all the extraction algorithms. Such end-to-end tests are brittle and a maintenance nightmare. The cause is a not good testable design.
You will be not able to implement a good test strategy without a good testable design. You will need to change the design of your application. On the other hand you may argue that you do not want to change the design without any tests in place. It seems to be a chicken-and-egg problem.
How to get out of such a situation? A combination of test and refactoring strategies might help. On a high-level this might work like this:
Build a few representative end-to-end tests. Therefore even use the
serialization trick. This is just to verify that your program works
like before you start refactoring. They act as migration tests.
Refactor you program. Give it places to inject and observe. Such
places are known as seams.
As a result you will have testable chunks, which you can put into a
test harness.
You refactor and put new seams into the code, to test smaller
chunks, until you get to the point that you have unit tests in
place. Ideally you will have all your algorithms encapsulated into a family of classes which are all unit tested.
Sounds like a hard job? No, in reality it is even harder than it sounds. Refactoring an application to a testable design needs a lot of experience. Luckily there is a guy who wrote a book about this: Michael Feather’s 'Working Effectively with Legacy Code'.
If you really, really want to implement a good test strategy for the existing application, then read that book. If you want to know what you can do better next time, read that book. If you think that unit testing might be the key to avoid not testable design then start learning about unit tests now. There are a lot of resources in the internet and books about unit testing.
If I understand the problem, you need to persist the results of a test for a later inspection or additional testing.
It has often been reluctant to invest too much time in writing test code, but in this case I see no alternative immediately available.
My advice is to decouple as much as possible the parts involved: the algorithm, the persistence layer (serialization/deserialization), its product and the verification code.
It is possible also that later algorithms implementations can share the same interface, e.g.:
interface IMyAlgorithm {
AbstractOutput DoSomething (InputData);
}
class ConcreteOutput : AbstractOutput {
// Output for version XXX of your algorithm
}
class XXXAlgorithm {
ConcreteOutput DoSomething (InputData inputData)
// Version XXX of you alogorithm
}
}
interface IPersistenceManager {
Serialize(AbstractOutput output, string filename);
AbstractOutput Deserialize(string filename)
}
class XXXPersistenceManager : IPersistenceManager {
// Handle persistence for XXX hierarchy
}
class XXXTestFixture {
void BuildObjectWithXXXAlgorithm() {
IMyAlgorithm XXX = new XXXAlgorithm();
// run XXX here
AbstractOutput objXXX = XXX.DoSomething(new InputData());
IPersistenceManager pmXXX = new XXXPersistenceManager();
pmXXX.Serialize(objXXX);
}
void VerifyThatXXXWorkAsExpected() {
IPersistenceManager pmXXX = new XXXPersistenceManager();
AbstractOutput objXXX = pmXXX.Deserialize(path);
// check object here
}
}
So when you need to create a new algorithm, say YYY, you create the corresponding hierarchy. Anyway I don't know details and this is just a draft of pseudocode, placed here only to put emphasis on loosely coupling application components.
You can try to use approvaltests to verify that
a given input file always generates the same in memory-object-graph.
To do this you need code that converts the in-memory-object-graph into a string representation of it (i.e overwriting ToString() or having an xml-serializer)
approvaltests verifies that the generated string is always the same.
if the string representation chages you will get a diff-viewer and you are asked to verify if the changes are still ok or not. If the changes are ok you will take this result for later verifications.
Background: In our project, we have a bunch of xml files that define tests that are loaded into a report generator. Since we want to make sure our tests stay accurate, we have data driven unit tests set up to test that the xml input produces the same output. This is currently held in a large excel spreadsheet, and works great with Visual Studio's data driven unit tests. But, this leads to problems because often more than one developer will be working on tests, and they will need to get the spreadsheet to add the unit tests. If another developer has the spreadsheet, then they are stalled waiting on it to get checked back in since VS can't merge the excel file. In addition, we are wanting to switch to git, and this approach won't work well there either due to the lack of locking on checkouts and inability to merge binary files.
So, we are trying to switch to having a separate file for each individual test's set of unit tests in a folder. Then, when the unit tests are run, we load up all the files in the folder and run the test data in each of the files. This way, when a developer makes a new test, they just have to add a new file for the unit tests, instead of having to add an entry to a central file.
The data format is going to be the same across all the files, so really, even concatenating them together should work. But, I'm trying to do this and retain the data driven behavior where each row in the test data is equivalent to one test in the output; they all pass or fail/produce output messages separately.
Is it possible to run a data driven unit test off of multiple files? If so, can we still retain the 'one row == one test' style output in the results?
I wound up having a ClassInitialize method for the unit test class and made the class include the folder of files I needed as a deployment item. Then, the ClassInitialize method took all of the data files in the folder and concatenated them together. The other tests in the class then referenced this file in their DataSource attribute. This allowed me to use the multiple file layout and still retain the output style we wanted.
Main lesson out of this: The data source file for a data driven unit test does not have to exist at compile time, apparently.