I have some classes that implements some logic related to file system and files. For example, I am performing following tasks as part of this logic:
checking if certain folder has certain structure (eg. it contains subfolders with specific names etc...)
loading some files from those folders and checking their structure (eg. these are some configuration files, located at certain place within certain folder)
load additional files for testing/validation from the configuration file (eg. this config file contains information about other files in the same folder, that should have other internal structure etc...)
Now all this logic has some workflow and exceptions are thrown, if something is not right (eg. configuration file is not found at the specific folder location). In addition, there is Managed Extensibility Framework (MEF) involved in this logic, because some of these files I am checking are managed DLLs that I am manually loading to MEF aggregates etc...
Now I'd like to test all this in some way. I was thinking of creating several physical test folders on HDD, that cover various test cases and then run my code against them. I could create for example:
folder with correct structure and all files being valid
folder with correct structure but with invalid configuration file
folder with correct structure but missing configuration file
etc...
Would this be the right approach? I am not sure though how exactly to run my code in this scenario... I certainly don't want to run the whole application and point it to check these mocked folders. Should I use some unit testing framework to write kind of "unit tests", that executes my code against these file system objects?
In general, is all this a correct approach for this kind of testing scenarios? Are there other better approaches?
First of all, I think, it is better to write unit tests to test your logic without touching any external resources. Here you have two options:
you need to use abstraction layer to isolate your logic from external dependencies such as the file system. You can easily stub or mock (by hand or with help of constrained isolation framework such as NSubstitute, FakeItEasy or Moq) this abstractions in unit tests. I prefer this option, because in this case tests push you to a better design.
if you have to deal with legacy code (only in this case), you can use one of the unconstrained isolation frameworks (such as TypeMock Isolator, JustMock or Microsoft Fakes) that can stub/mock pretty much everything (for instance, sealed and static classes, non-virtual methods). But they costs money. The only "free" option is Microsoft Fakes unless you are the happy owner of Visual Studio 2012/2013 Premium/Ultimate.
In unit tests you don't need to test the logic of external libraries such as MEF.
Secondly, if you want to write integration tests, then you need to write "happy path" test (when everything is OK) and some tests that testing your logic in boundary cases (file or directory not found). Unlike #Sergey Berezovskiy, I recommend creating separate folders for each test case. The main advantages is:
you can give your folder meaningful names that more clearly express your
intentions;
you don't need to write complex (i.e. fragile) setup/teardown logic.
even if you decide later to use another folder structure, then you can change it more easily, because you will already have working code and tests (refactoring under test harness is much easier).
For both, unit and integration tests, you can use ordinary unit testing frameworks (like NUnit or xUnit.NET). With this frameworks is pretty easy to launch your tests in Continuous integration scenarios on your Build server.
If you decide to write both kinds of tests, then you need to separate unit tests from integration tests (you can create separate projects for every kind of tests). Reasons for it:
unit tests is a safety net for developers. They must provide quick feedback about expected behavior of system units after last code changes (bug fixes, new features). If they are run frequently, then developer can quickly and easily identify piece of code, that broke the system. Nobody wants to run slow unit tests.
integration tests are generally slower than unit tests. But they have different purpose. They check that units works as expected with real dependencies.
You should test as much logic as possible with unit tests, by abstracting calls to the file system behind interfaces. Using dependency injection and a testing-framework such as FakeItEasy will allow you to test that your interfaces are actually being used/called to operate on the files and folders.
At some point however, you will have to test the implementations working on the file-system too, and this is where you will need integration tests.
The things you need to test seem to be relatively isolated since all you want to test is your own files and directories, on your own file system. If you wanted to test a database, or some other external system with multiple users, etc, things might be more complicated.
I don't think you'll find any "official rules" for how best to do integration tests of this type, but I believe you are on the right track. Some ideas you should strive towards:
Clear standards: Make the rules and purpose of each test absolutely clear.
Automation: The ability to re-run tests quickly and without too much manual tweaking.
Repeatability: A test-situation that you can "reset", so you can re-run tests quickly, with only slight variations.
Create a repeatable test-scenario
In your situation, I would set up two main folders: One in which everything is as it is supposed to be (i.e. working correctly), and one in which all the rules are broken.
I would create these folders and any files in them, then zip each of the folders, and write logic in a test-class for unzipping each of them.
These are not really tests; think of them instead as "scripts" for setting up your test-scenario, enabling you to delete and recreate your folders and files easily and quickly, even if your main integration tests should change or mess them up during testing. The reason for putting them in a test-class, is simply to make then easy to run from the same interface as you will be working with during testing.
Testing
Create two sets of test-classes, one set for each situation (correctly set up folder vs. folder with broken rules). Place these tests in a hierarchy of folders that feels meaningful to you (depending on the complexity of your situation).
It's not clear how familiar you are with unit-/integration-testing. In any case, I would recommend NUnit. I like to use the extensions in Should as well. You can get both of these from Nuget:
install-package Nunit
install-package Should
The should-package will let you write the test-code in a manner like the following:
someCalculatedIntValue.ShouldEqual(3);
someFoundBoolValue.ShouldBeTrue();
Note that there are several test-runners available, to run your tests with. I've personally only had any real experience with the runner built into Resharper, but I'm quite satisfied with it and I have no problems recommending it.
Below is an example of a simple test-class with two tests. Note that in the first, we check for an expected value using an extension method from Should, while we don't explicitly test anything in the second. That is because it is tagged with [ExpectedException], meaning it will fail if an Exception of the specified type is not thrown when the test is run. You can use this to verify that an appropriate exception is thrown whenever one of your rules is broken.
[TestFixture]
public class When_calculating_sums
{
private MyCalculator _calc;
private int _result;
[SetUp] // Runs before each test
public void SetUp()
{
// Create an instance of the class to test:
_calc = new MyCalculator();
// Logic to test the result of:
_result = _calc.Add(1, 1);
}
[Test] // First test
public void Should_return_correct_sum()
{
_result.ShouldEqual(2);
}
[Test] // Second test
[ExpectedException(typeof (DivideByZeroException))]
public void Should_throw_exception_for_invalid_values()
{
// Divide by 0 should throw a DivideByZeroException:
var otherResult = _calc.Divide(5, 0);
}
[TearDown] // Runs after each test (seldom needed in practice)
public void TearDown()
{
_calc.Dispose();
}
}
With all of this in place, you should be able to create and recreate test-scenarios, and run tests on them in a easy and repeatable way.
Edit: As pointed out in a comment, Assert.Throws() is another option for ensuring exceptions are thrown as required. Personally, I like the tag-variant though, and with parameters, you can check things like the error message there too. Another example (assuming a custom error message is being thrown from your calculator):
[ExpectedException(typeof(DivideByZeroException),
ExpectedMessage="Attempted to divide by zero" )]
public void When_attempting_something_silly(){
...
}
I'd go with single test folder. For various test cases you can put different valid/invalid files into that folder as part of context setup. In test teardown just remove those files from folder.
E.g. with Specflow:
Given configuration file not exist
When something
Then foo
Given configuration file exists
And some dll not exists
When something
Then bar
Define each context setup step as copying/not copying appropriate file to your folder. You also can use table for defining which file should be copied to folder:
Given some scenario
| FileName |
| a.config |
| b.invalid.config |
When something
Then foobar
I don't know your program's architecture to give a good advice, but I will try
I believe that you don't need to test real file structure. File access services are defined by system/framework, and they're don't need to be tested. You need to mock this services in related tests.
Also you don't need to test MEF. It is already tested.
Use SOLID principles to make unit tests. Especially take look at Single Responsibility Principle this will allow you to to create unit tests, which won't be related to each others. Just don't forget about mocking to avoid dependencies.
To make integration tests, you can create a set of helper classes, which will emulate scenarios of file structures, which you want to test. This will allow you to stay not attached to machine on which you will run this tests. Such approach maybe more complicated than creating real file structure, but I like it.
I would build framework logic and test concurrency issues and file system exceptions to ensure a well defined test environment.
Try to list all the boundaries of the problem domain. If there are too many, then consider the possibility that your problem is too broadly defined and needs to be broken down. What is the full set of necessary and sufficient conditions required to make your system pass all tests? Then look at every condition and treat it as an individual attack point. And list all the ways you can think of, of breaching that. Try to prove to yourself that you have found them all. Then write a test for each.
I would go through the above process first for the environment, build and test that first to a satisfactory standard and then for the more detailed logic within the workflow. Some iteration may be required if dependencies between the environment and the detailed logic occur to you during testing.
Related
We have created P2P sharing in c# and I want to have some integration tests which tests for example if a file downloaded correctly from sender, and check size, speed etc.
I have tried to send a file and then checked but I can not check this file without running the program.
Any idea how I can create some tests to check data?
Let's start off by the difference from an integration to a unit test. Whereas in a unit test you'll test individual parts in isolation (usually achieved via mocking the dependencies) the integration tests works against the broader (or full) system.
I'm assuming you want to run the integration test in automated fashion. Basically you can just run a test with the unit test framework you're using already (for example NUnit) and when creating the necessary services you don't use any mocks but you inject the actual dependencies.
How you exactly setup the "frame" of the integration test up depends on your project, for example if you are using an IoC library to inject dependencies you might be able to use this one also within the integration tests rather than having to set your services up by hand.
What you have to be careful is also the fact that when you run integration tests it might affect the system you're running it on. So if you're doing something on the file system it's a good practice to make sure to clean up after the test.
I would recommend to create some basic "framework" that fits your project to setup integration tests, that would include the generic code you need to setup the tests to run with your system and maybe creates dedicated folders in a temp directory that will be removed after every test run.
Now to your more concrete question: If I get it right you would need to create a "sender" that will provide the test file you want to download. As part of the test you could deploy this test file to the above mentioned temp folder and configure the sender to provide that file. Then you could create the client that would interact with this sender and download the file to somewhere on your system.
Before you initiate the download you could track the time and figure out how long it took. Additionally you could after the download has finished check it's properties, or compare it to the original file, as I assume it should be identical.
Following some pseudo-code that could show the general concept:
[TestMethod]
[TestCategory("Integration")
public void DownloadFileFromSender_ConnectionDoesNotGetInterrupted_SuccessfullyDownloadsFile()
{
// Arrange - do the setup of files and temp folders, make create sender and receiver
SetupTestFile(#"Tempfolder\Testfile.txt");
var sender = new Sender(...);
var receiver = new Receiver(...);
sender.ProvideFile(#"Tempfolder\Testfile.txt");
// Act - Put your actual test here
var timeBeforeDownload = DateTime.Now;
receiver.DownloadFile(sender, #"Tempfolder\Testfile.txt", #"Tempfolder\DownloadedFile.txt");
var totalDownloadTime = DateTime.Now - timeBeforeDownload;
// Assert - Verify here your assumptions, e.g. download time or file properties
Assert.IsTrue(totalDownloadTime.TotalMilliseconds < 10000);
Assert.IsTrue(File.Exists(#"Tempfolder\DownloadedFile.txt"));
}
Be aware that running integration tests might have a longer time to setup and run depending on the size/complexity of the parts that you are testing. They do not replace unit tests but rather complement them. Due to their difference it's also a good idea to tag them to be able to just run either unit or integration tests.
Again the specifics on how to setup the test environment is up to you and depends heavily on your project you want to test.
Every time, when I deal with writing Tests in .NET (C#), I read, you can test DLLs only. But my app hasn't any DLLs and I want test some controllers and behaviours too. How it is possible, without creating an DLL project? Is it possible?
Hints are welcome.
Your unit test project(s) can still reference application projects like any other class library. It may be considered less ideal, but there's nothing preventing it from working. As the logic of your system grows, even a little bit, you'll certainly want to consider moving the business logic into Class Library projects and keeping the application layer as thin as possible.
I read, you can test DLLs only.
This was an oversimplification. What they probably meant was that you can test discrete functionality only. Regardless of what kind of project is hosting that functionality, the functionality itself has to be well defined, with separated concerns, and individually testable. If that's causing a problem in your design, then you have more work to do in order to unit test your code.
But to get started, simply create your unit test project and reference your other project. Then start writing tests for your individual units of functionality. Each test should be testing only one discrete thing, and should consist of the simple steps of:
Arrange
Act
Assert
Unit tests shouldn't require any additional setup, nor should they produce any side effects. This is where the "discrete" part comes in, each one should be individual and not depend on others.
Question
For a large number of Unit Tests, Is there an easy way to generate a "OrderedTest" file for each test class within my project, allowing me to run each test method in the order that they appear within their respective classes?
Background
I have a large number(1000+) of Selenium Functional Tests contained within a Unit Test Project. In my project each class represents a page and each "Unit Tests" represents one of the my functional tests. Typically the tests are run in the following manner:
Create - Complex object within the page(10ish tests)
Manipulate\Edit - The already created complex object(100ish tests)
Tear-down\Delete - Remove the complex object piece by piece until the test page is resorted to its original state(10ish tests)
Due to the many complexities and load times of each page, each one of these tests(really just the groups) must be run in a specific order within their given class. I understand that this is "not optimal" to structure my tests in the manner, but unfortunately I have not found an alternative design for my tests to run in any reasonable amount of time.
I previously use ReSharpers test tool to run these tests, with this tool I'm able to run each test in the order that is appears in each class. Now I'm attempting(for various irrelevant reasons) to us MSTest to run my tests. MSTest runs each test by default in a "non-deterministic" order.
I would like to use "Ordered Tests" to enforce the order of each test. However, since I follow this convention my tests are currently not named in the order they are to be run. The order that I need to run the tests is currently defined by their order within their class.
So here's my problem, when I create a new "Ordered Test" file, the interface does not allow me to sort the "Avaliable tests" by their "natural order"(the order in which they appear in their class), it also does not allow me move the order of each of the "Selected Tests" more than one space(once per click). In a small scale project this would be just annoying, with my 1000+ test project(with many more 1000's on the way) it's very difficult to generate a ordered test for each one of my classes due to the overhead of having to order each item by hand.
Follow Up
The simpilist way I can think of to solve this is to write a script to generate "Orderedtest" files exactly as I've stated in my question, but it really strikes me as excessive, maybe I'm not following a standard(or recommended) path in coding my selenium test structure. I would think if many people had already followed this path, that there would be more documentation on the subject, but the little I find in relation to this subject does provide me with a clear path to follow.
I wonder if there is an alternate way that I can accomplish the same functionality with MSTest?
I am in the process of retrofitting unit tests for a asp.net solution written in VB.Net and c#.
The unit tests need to verify the current functionality and act as a check for future breaking changes.
The solution comprises of:
1 MVC web project
written in vb.net (don't ask, it's a legacy thing)
10 other supporting projects each containing logically grouped functionality
written in C#, each project contains repositories and DAL
All the classes are tightly coupled as there is no inversion of control (IOC) implemented anywhere, yet.
currently to test a controller there is the following stack:
controller
repository
dal
logging
First question, to unit test this correctly would I setup 1 test project and run all tests from it, or should I setup 1 test project for each project to test the functionality of that DLL only?
Second question, do I need to implement IOC to be able to use MOQ?
Third question, is it even possible to refactor IOC into a huge solution like this?
Forth question, what other options are available to get this done asap?
I am in the process of retrofitting unit tests for a asp.net solution written in VB.Net and c#. The unit tests need to verify the current functionality and act as a check for future breaking changes.
When working with a large code base that doesn't have unit tests and hasn't been written with testing in mind, there is a good chance that in order to write a useful set of unit tests you will have to modify the code, hence you're going to be triggering the event that you're planning on writing the unit tests to support. This is obviously risky, but may not be any riskier than what you're already doing on a day to day basis.
There are a number of approaches that you could take (and there's a good chance that this question will be closed as too broad). One approach is to create a good set of integration tests ensure that the core functionality is working. These tests won't be as fast to run as unit tests, but they will be further decoupled from the legacy code base. This will give you a good safety net for any changes that you need to make as part of introducing unit testing.
If you have an appropriate version of visual studio, then you may also be able to use shims (or if you have funds, typemock may be an option) to isolate elements of your application when writing your initial tests. So, you could for example create shims of your dal to isolate the rest of your code from the db.
First question, to unit test this correctly would i setup 1 test project and run all tests from it, or should i setup 1 test project for each project to test the functionality of that dll only?
Personally, I prefer think of each assembly as a testable unit, so I tend to create at least one test project for each assembly containing production code. Whether or not that makes sense though, depends a bit on what's contained in each of the assemblies... I'd also tend to have at least one test project for integration tests of the top level project.
Second question, do i need to implement IOC to be able to use MOQ?
The short answer is no, but it depends what your classes do. If you want to test using Moq, then it's certainly easier to do so if your classes support dependency injection, although you don't need to use an IOC container to achieve this. Hand rolled injection either through constructors like below, or through properties can form a bridge to allow testing stubs to be injected.
public SomeConstructor(ISomeDependency someDependency = null) {
if(null == someDependency) {
someDependency = new SomeDependency();
}
_someDependency = someDependency;
}
Third question, is it even possible refactor IOC into a huge solution like this?
Yes it's possible. The bigger question is it worth it? You appear to be suggesting a big bang approach to the migration. If you have a team of developers that don't have much experience in this area, this seems awfully risky. A safer approach might be to target a specific area of the application and migrate that section. If your assemblies are discrete then they should form fairly easy split points in your application. Learn what works and what doesn't, along with what benefits and unexpected pain you're feeling. Use that to inform your decision about how and when to migrate the rest of the code.
Forth question, what other options are available to get this done asap?
As I've said above, I'm not sure that ASAP is really the right approach to take. Working towards unit-testing can be done as a slow migration, adding tests as you actually change the code due to business requirements. This helps to ensure that testers are also allocated to catch any errors that you introduce as part of the refactoring that might need to take place to support the testing.
I have a number of unit tests which rely on the presence of a csv file. They will throw an exception if this file doesn't exist obviously.
Are there any Gallio/MbUnit methods which can conditionally skip a test from running? I'm running Gallio 3.1 and using the CsvData attribute
[Test]
[Timeout(1800)]
[CsvData(FilePath = TestDataFolderPath + "TestData.csv", HasHeader = true)]
public static void CalculateShortfallSingleLifeTest()
{
.
.
.
Thanks
According to the answer in this question, you'll need to make a new TestDecoratorAttribute that calls Assert.Inconclusive if the file is missing.
Assert.Inconclusive is very appropriate for your situation because you aren't saying that the test passed or failed; you're just saying that it couldn't be executed in the current state.
What you have here is not a unit test. A unit test tests a single unit of code (it may be large though), and does not depend on external environmental factors, like files or network connections.
Since you are depending on a file here, what you have is an integration test. You're testing whether your code safely integrates with something outside of the control of the code, in this case, the file system.
If this is indeed an integration test, you should change the test so that you're testing the thing that you actually want tested.
If you're still considering this as a unit test, for instance you're attempting to test CSV parsing, then I would refactor the code so that you can mock/stub/fake out the actual reading of the CSV file contents. This way you can more easily provide test data to the CSV parser, and not depend on any external files.
For instance, have you considered that:
An AntiVirus package might not give you immediate access to the file
A typical programmer tool, like TortoiseSvn, integrates shell overlays into Explorer that sometimes hold on to files for too long and doesn't always give access to a file to a program (you deleted the file, and try to overwrite it with a new one? sure, just let me get through the deletion first, but there is a program holding on to the file so it might take a while...)
The file might not actually be there (why is that?)
You might not have read-access to the path
You might have the wrong file contents (leftover from an earlier debugging session?)
Once you start involving external systems like file systems, network connections, etc. there's so many things that can go wrong that what you have is basically a brittle test.
My advice: Figure out what you're trying to test (file system? CSV parser?), and remove dependencies that are conflicting with that goal.
An easy way would be to include an if condition right at the start of the test that would just execute any code in the test if the CSV file can be found.
Of course this has the big drawback that tests would be green although they haven't actually run and asserted anything.
I agree with Grzenio though, if you have unit tests that rely heavily on external conditions, they're not really helping you. In this scenario you will never really know whether the unit test ran successfully or was just skipped, which contradicts what unit tests are actually for.
In my personal opinion, I would just write the test so that they correctly fail when the file is not there. If they fail this is an indicator that the file in question should be available on the machine where the unit tests run. This might need some manual adjustments at times (getting the file to the computer or server in question), but at least you have reliable unit tests.
In Gallio/MbUnit v3.2 the abstract ContentAttribute and its concrete derived types (such as [CsvData] have a new optional parameter that allows to change the default outcome of a test in case of an error occured while opening or reading the file data source (ref. issue 681). The syntax is the following:
[Test]
[CsvData(..., OutcomeOnFileError = OutcomeOnFileError.Inconclusive)]
public void MyTestMethod()
{
// ...
}