Here's an integration test I wrote for a class that interacts with a database:
[Test]
public void SaveUser()
{
// Arrange
var user = new User();
// Set a bunch of properties of the above User object
// Act
var usersCountPreSave = repository.SearchSubscribersByUsername(user.Username).Count();
repository.Save(user);
var usersCountPostSave = repository.SearchSubscribersByUsername(user.Username).Count();
// Assert
Assert.AreEqual(userCountPreSave + 1, userCountPostSave);
}
It seems to me that I can't test the Save function without involving the SearchSubscriberByUsername function to find out if the user was successfully saved. I realize that integration tests aren't meant to be unit tests which are supposed to test one unit of code at a time. But ideally, it would be nice if I could test one function in my repository class per test but I don't know how I can accomplish that.
Is it fine how I've written the code so far or is there a better way?
You have a problem with your test. When you're testing that data is saved into the database, you should be testing that it's in the database, not that the repository says that it's in the database.
If you're testing the functionality of repository, then you can't verify that functionality by asking if it has done it correctly. It's the equivalent of saying to someone 'Did you do this correctly?' They are going to say yes.
Imagine that repository never commits. Your test will pass fine, but the data won't be in the database.
So, what I would do is to to open a connection (pure SQL) to the database and check that the data has been saved correctly. You only need to a select count(*) before and after to ensure that the user has been saved. If you do this, you can avoid using the SearchSubscribersByUsername as well.
If you're testing the functionality of repository, you can't trust repository, by definition.
To unit test something like a "Save" function, you will definitely need some trustworthy channel to check the result of the operation. If you trust SearchSubscribersByUsername (because you have already made some unit tests for that function on its own), you can use it here.
If you don't trust SearchSubscribersByUsername and you think your unit test could also break because there is an error in that function (and not in Save), you should think about a different channel (perhaps you have a possibility to make a bypassing SQL access to your DB to check the Save result, which may be simpler than the implementation of SearchSubscribersByUsername)? However, do not reimplement SearchSubscribersByUsername again, that would be getting pointless. Either way, you will need at least some other function you can trust.
Unless the method you are testing returns definitive information about what you have done I don't see any way to avoid calling other methods. I think you are correct in your assumption that Integration testing needs a different way of thinking from Unit testing.
I would still build tests that focus on individual methods. So in testing Save() I may well use the capabilities of Search(), but my focus is on the edge cases of Save(). I build tests that deal with duplicate insertions or invalid input data. Then later I build a whole raft of Search() tests that deal with the edge cases of Search().
Now one possible way of thinking is that Save and Search have some commonality, a bug in Search might mask a bug in Save. Imagine, for example, if you had a caching layer down there. So possibly an alternative approach is to use some other verification mechanism. For example a direct JDBC call to the database, or alteratively introducing mocking layers at some point in your infrastructure. When building complex Integrated Systems this kind of "Back Door" verification may be essential.
Personally, I've written countless tests very similar to this and think its fine. The alternative is to stub out the database so that searchSubscribers never actually does anything but thats great deal of work for what I'd say is little gain.
Related
How would I use NUnit and a test database to verify my code? I would in theory use mocks (moq) but my code is more in maintenance shape and fix it mode and I don't have the to setup all the mocks.
Do I just create a test project, then write tests that actually connect to my test database and execute the code as I wwould in the app? Then I check the code with asserts and make sure what I'm requesting is what I'm getting back correctly?
How would I use NUnit and a test database to verify my code? I would
in theory use mocks (moq) but my code is more in maintenance shape and
fix it mode and I don't have the to setup all the mocks.
Using mocks is only useful if you want to test the exact implementation behavior of a class. That means you are literally asserting that one class calls a specific method on another class. For example: I want to assert that Ninja.Attack() calls Sword.Unsheath().
Do I just create a test project, then write tests that actually
connect to my test database and execute the code as I wwould in the
app? Then I check the code with asserts and make sure what I'm
requesting is what I'm getting back correctly?
This is just a plain old unit test. If there are no obstacles to achieving this, that's a good indicator that this is going to be your most effective method of testing. It's practical and highly effective.
There's one other testing tool you didn't mention, which is called a stub. I highly recommend you read this classic article for more info:
http://martinfowler.com/articles/mocksArentStubs.html
Since we are not talking about theoretical case, this is what I would do - From my understanding what you want to test is that whether your app is properly connecting to the DB and fetching the desired data or not.
Create a test DB with the same schema
Add some dummy data in that
Open a connection to the DB from the code, request desired data
Write assertions to test what you got from the DB against what you expected
Also, I don't think these tests should be called unit tests because they are not self contained and are dependent on other factors like whether your database is up and running or not. I would say they fall close to integration tests that will test if different components of your applications are working as expectation when used together.
(Dan's answer ^^ pretty much sums what I wanted to say)
I'm in the midst of rewriting a database application and using Entity Framework to access the DB. Currently I am using MSTest and a copy of the underlying database as part of these tests. My MSTest involves the following code as part of each test:
[TestInitialize()]
public void MyTestInitialize()
{
transScope = new TransactionScope(TransactionScopeOption.RequiresNew, new TransactionOptions { Timeout = new TimeSpan(0, 10, 0) });
}
[TestCleanup()]
public void MyTestCleanup()
{
Transaction.Current.Rollback();
transScope.Dispose();
}
Now, this seems to work pretty well for testing and resets the DB between tests. My tests use the DB context to do CRUD operations against the test DB and then roll them back afterward.
I've read a bit about isolating the C# library from the underlying DB for testing but I'm wondering what this actually buys me. As part of this rewrite, most (but not all) of the code that was in stored procedures has been moved in the C# layer but a few remain which are called via triggers on tables. What do I gain from going through the exercise of mocking out that database layer? Frankly, when I look at doing this, it seems like a lot of additional work without any obvious value, but perhaps I'm missing the point here.
Thoughts?
It depends on the the kinds of tests you are writing. When writing unit tests where you want to test one unit of code--typically a class--then usually it's good for two things to hold true:
The tests should run as fast as possible. Meaning 100's of tests per second.
The code you are testing should be isolated from other code such that you can control the way dependencies work and test different kinds of inputs and outputs easily.
Both of these things are difficult to do if you use a real database for all tests. The first because a round-trip to the database usually takes a lot more time than just running some code. The second because you need to have your database setup with lots of different kinds of data, including negative cases and corner cases, and sometimes even make the database fail. It's often easier to instead mock the dependencies of your class and pass in any inputs needed.
That being said, I have written many tests that use a similar pattern to the one you describe and they work well and run relatively fast. Personally, I would use a combination of real unit tests without a database, and tests like the ones you have that are more like functional testing of a component.
In the end, do what works for you.
In my example, I've got two methods for Enabling and Disabling and account and I'm gonna to write a test method for each one.
The problem is that I've got to consider the original status for the data and restore it after testing it, even in a sample database, to keep the data consisted for the next test.
public void DisableAndEnableAccount()
{
var client = new GwIntegrationServiceSoapClient();
string userName = "admin";
Account account = client.GetAccountByUsername(userName);
int accountID = account.Id;
bool isActiveOrginalValue = account.IsActive;
if (isActiveOrginalValue)
{
client.DisableAccount(accountID);
account = client.GetAccountByUsername(userName);
Assert.IsFalse(account.IsActive);
client.EnableAccount(accountID);
account = client.GetAccountByUsername(userName);
Assert.IsTrue(account.IsActive);
}
else
{
client.EnableAccount(accountID);
account = client.GetAccountByUsername(userName);
Assert.IsTrue(account.IsActive);
client.DisableAccount(accountID);
account = client.GetAccountByUsername(userName);
Assert.IsFalse(account.IsActive);
}
}
I think that my test method isn't written in a good way, Any idea how to deal with such case?
You should use test data (test user accounts) in your tests, not real ones. (In fact, it is strongly recommended to use a separate test DB for your tests, never the real live production DB.) Then you are free to set it up any way you need prior to the test. Btw it is recommended to do the setup in the separate setUp() method (or one annotated with #Before in JUnit 4).
Note however, that "classic" unit tests should not depend on external entities like a DB or the file system: they should focus on testing one unit (class, method), isolated from the rest of the world. This is usually achieved by dependency injection and mocking, i.e. hiding them behind interfaces, so that in unit tests, you can supply a dummy implementation which e.g. doesn't connect to the DB, just verifies the calls made to it and the parameters passed.
Testing the whole integrated system is still useful, just it is not unit, rather integration testing. Unit tests can be much finer grained, easier to understand and maintain, and faster, so whenever you can, it is best to start with unit tests, then once you are sure the smaller parts are working fine, put together some integration tests to verify that the system works end to end.
You're always going to have trouble restoring the data to its original state using this method. The solution, in general, is to avoid trying to do this, as #Péter Török points out (I also happen to agree almost completely with his post).
Many things could go wrong between the various steps in your test: what if the internet connection is dropped, and you lose connection to the web service? What happens if you make a mistake in your test code and break the test account's live data?
If absolutely have to have some kind of automated testing with an external web service, I would suggest coming to an arrangement with the owner of the service to be able to reset your test entities automatically, or something like that. This way, you can focus on having your tests actually test functionality, not worry about how you will trick the data back to the way it was to start with.
I need to develop a fairly simple algorithm, but am kindof confused as how to best write a test for it.
General description: User needs to be able to delete a Plan. Plan has Tasks associated with it, these need to be deleted as well (as long as they're not already done).
Pseudo-code as how the algorithm should behave:
PlanController.DeletePlan(plan)
=>
PlanDbRepository.DeletePlan()
ForEach Task t in plan.Tasks
If t.Status = Status.Open Then
TaskDbRepository.DeleteTask(t)
End If
End ForEach
Now as far as I understand it, unit tests are not supposed to touch the Database or generally require access to any outside systems, so I'm guessing I have two options here:
1) Mock out the Repository calls, and check whether they have been called the appropriate number of times as Asserts
2) Create stubs for both repository classes, setting their delete flag manually and then verify that the appropriate objects have been marked for deletion.
In both approaches, the big question is: What exactly am I testing here? What is the EXTRA value that such tests would give me?
Any insight in this would be highly appreciated. This is technically not linked to any specific unit testing framework, although we have RhinoMocks to be used. But I'd prefer a general explanation, so that I can properly wrap my head around this.
You should mock the repository and then construct a dummy plan in your unit test containing both Open and Closed tasks. Then call the actual method passing this plan and at the end verify that the DeleteTask method was called with correct arguments (tasks with only status = Open). This way you would ensure that only open tasks associated to this plan have been deleted by your method. Also don't forget (probably in a separate unit test) to verify that the plan itself has been deleted by asserting that the DeletePlan method has been called on the object your are passing.
To add to Darin's answer I'd like to tell you what you are actually testing. There's a bit of business logic in there, for example the check on the status.
This unit test might seem a bit dumb right now, but what about future changes to your code and model? This test is necessary to make sure this seemingly simple functionality will always keep working.
As you noted, you are testing that the logic in the algorithm behaves as expected. Your approach is correct, but consider the future - Months down the road, this algorithm may need to be changed, a different developer chops it up and redoes it, missing a critical piece of logic. Your unit tests will now fail, and the developer will be alerted to their mistake. Unit testing is useful at the start, and weeks/months/years down the road as well.
If you want to add more, consider how failure is handled. Have your DB mock throw an exception on the delete command, test that your algorithm handles this correctly.
The extra value provided by your tests is to check that your code does the right things (in this case, delete the plan, delete any open tasks associated with the plan and leave any closed tasks associated with the plan).
Assuming that you have tests in place for your Repository classes (i.e. that they do the right things when delete is called on them), then all you need to do is check that the delete methods are called appropriately.
Some tests you could write are:
Does deleting an empty plan only call DeletePlan?
Does deleting a plan with two open tasks call DeleteTask for both tasks?
Does deleting a plan with two closed tasks not call DeleteTask at all?
Does deleting a plan with one open and one closed task call DeleteTask once on the right task?
Edit: I'd use Darin's answer as the way to go about it though.
Interesting, I find unit testing helps to focus the mind on the specifications.
To that end let me ask this question...
If I have a plan with 3 tasks:
Plan1 {
Task1: completed
Task2: todo
Task3: todo
}
and I call delete on them, what should the happen to the Plan?
Plan1 : ?
Task1: not deleted
Task2: deleted
Task3: deleted
Is plan1 deleted, orphaning task1? or is it otherwise marked deleted?.
This is a big part of the Value I see in unit tests (Although it is only 1 of the 4 values:
1) Spec
2) Feedback
3) Regression
4) granularity
As for how to test, I wouldn't suggest mocks at all. I would consider a 2 part method
The first would look like
public void DeletePlan(Plan p)
{
var objectsToDelete = GetDeletedPlanObjects(p);
DeleteObjects(objectsToDelete);
}
And I wouldn't test this method.
I would test the method GetDeletedPlanObjects, which wouldn't touch the database anyways, and would allow you to send in scenarios like the above situation.... which I would then assert with www.approvaltests.com , but that's another story :-)
Happy Testing,
Llewellyn
I would not write unit tests for this because to me this is not testing behaviour but rather implementation. If at some point you want to chance the behaviour to not delete the tasks but rather set them to a state of 'disabled' or 'ignored', your unit tests will fail. If you test all controllers this way your unit tests are very brittle and will need to be changed often.
Refactor out the business logic to a 'TaskRemovalStrategy' if you want to test the business logic for this and leave the implementation details of the removal up to the class itself.
IMO you can write your unit tests around the abstract PlanRepository and the same tests should be useful in testing the data integrity in the database also.
For example you could write a test -
void DeletePlanTest()
{
PlanRepository repo = new PlanDbRepository("connection string");
repo.CreateNewPlan(); // create plan and populate with tasks
AssertIsTrue(repo.Plan.OpenTasks.Count == 2); // check tasks are in open state
repo.DeletePlan();
AssertIsTrue(repo.Plan.OpenTasks.Count == 0);
}
This test will work even if your repository deletes the plan and your database deletes the related tasks via a cascaded delete trigger.
The value of such test is whether the test is run for PlanDbRepository or a MockRepository it will still check that the behavior is correct. So when you change any repository code or even your database schema, you can run the tests to check nothing is broken.
You can create such tests which cover all the possible behaviors of your repository and then use them to make sure that any of your changes do not break the implementation.
You can also parameterize this test with a concrete repository instance and reuse them the test any future implementations of repositories.
I want to write unit tests with NUnit that hit the database. I'd like to have the database in a consistent state for each test. I thought transactions would allow me to "undo" each test so I searched around and found several articles from 2004-05 on the topic:
http://weblogs.asp.net/rosherove/archive/2004/07/12/180189.aspx
http://weblogs.asp.net/rosherove/archive/2004/10/05/238201.aspx
http://davidhayden.com/blog/dave/archive/2004/07/12/365.aspx
http://haacked.com/archive/2005/12/28/11377.aspx
These seem to resolve around implementing a custom attribute for NUnit which builds in the ability to rollback DB operations after each test executes.
That's great but...
Does this functionality exists somewhere in NUnit natively?
Has this technique been improved upon in the last 4 years?
Is this still the best way to test database-related code?
Edit: it's not that I want to test my DAL specifically, it's more that I want to test pieces of my code that interact with the database. For these tests to be "no-touch" and repeatable, it'd be awesome if I could reset the database after each one.
Further, I want to ease this into an existing project that has no testing place at the moment. For that reason, I can't practically script up a database and data from scratch for each test.
NUnit now has a [Rollback] attribute, but I prefer to do it a different way. I use the TransactionScope class. There are a couple of ways to use it.
[Test]
public void YourTest()
{
using (TransactionScope scope = new TransactionScope())
{
// your test code here
}
}
Since you didn't tell the TransactionScope to commit it will rollback automatically. It works even if an assertion fails or some other exception is thrown.
The other way is to use the [SetUp] to create the TransactionScope and [TearDown] to call Dispose on it. It cuts out some code duplication, but accomplishes the same thing.
[TestFixture]
public class YourFixture
{
private TransactionScope scope;
[SetUp]
public void SetUp()
{
scope = new TransactionScope();
}
[TearDown]
public void TearDown()
{
scope.Dispose();
}
[Test]
public void YourTest()
{
// your test code here
}
}
This is as safe as the using statement in an individual test because NUnit will guarantee that TearDown is called.
Having said all that I do think that tests that hit the database are not really unit tests. I still write them, but I think of them as integration tests. I still see them as providing value. One place I use them often is in testing LINQ to SQL code. I don't use the designer. I hand write the DTO's and attributes. I've been known to get it wrong. The integration tests help catch my mistake.
I just went to a .NET user group and the presenter said he used SQLlite in test setup and teardown and used the in memory option. He had to fudge the connection a little and explicit destroy the connection, but it would give a clean DB every time.
http://houseofbilz.com/archive/2008/11/14/update-for-the-activerecord-quotmockquot-framework.aspx
I would call these integration tests, but no matter. What I have done for such tests is have my setup methods in the test class clear all the tables of interest before each test. I generally hand write the SQL to do this so that I'm not using the classes under test.
Generally, I rely on an ORM for my datalayer and thus I don't write unit tests for much there. I don't feel a need to unit test code that I don't write. For code that I add in the layer, I generally use dependency injection to abstract out the actual connection to the database so that when I test my code, it doesn't touch the actual database. Do this in conjunction with a mocking framework for best results.
For this sort of testing, I experimented with NDbUnit (working in concert with NUnit). If memory serves, it was a port of DbUnit from the Java platform. It had a lot of slick commands for just the sort of thing you're trying to do. The project appears to have moved here:
http://code.google.com/p/ndbunit/
(it used to be at http://ndbunit.org).
The source appears to be available via this link:
http://ndbunit.googlecode.com/svn/trunk/
Consider creating a database script so that you can run it automatically from NUnit as well as manually for other types of testing. For example, if using Oracle then kick off SqlPlus from within NUnit and run the scripts. These scripts are usually faster to write and easier to read. Also, very importantly, running SQL from Toad or equivalent is more illuminating than running SQL from code or going through an ORM from code. Generally I'll create both a setup and teardown script and put them in setup and teardown methods.
Whether you should be going through the DB at all from unit tests is another discussion. I believe it often does make sense to do so. For many apps the database is the absolute center of action, the logic is highly set based, and all the other technologies and languages and techniques are passing ghosts. And with the rise of functional languages we are starting to realize that SQL, like JavaScript, is actually a great language that was right there under our noses all these years.
Just as an aside, Linq to SQL (which I like in concept though have never used) almost seems to me like a way to do raw SQL from within code without admitting what we are doing. Some people like SQL and know they like it, others like it and don't know they like it. :)
For anyone coming to this thread these days like me, I'd like to recommend trying the Reseed library I'm developing currently for this specific case.
Neither in-memory db replacement (lack of features) nor transaction rollback (transactions can't be nested) were a suitable option for me, so I ended up with a simple delete/insert cycle for the data restore purpose. Ended up with a library to generate those, while trying to optimize my tests speed and simplicity of setup. Would be happy if it helps anyone else.
Another alternative I'd recommend is using database snapshots to restore data, which is of comparable performance and usability.
Workflow is as follows:
delete existing snapshots;
create db;
insert data;
create snapshot ;
execute test;
restore from snapshot;
go to "execute test" until none left;
drop snapshot.
It's suitable if you could have the only data script for all the tests and allows you to execute the insertion (which is supposed to be the slowest) the only time, moreover you don't need data cleanup script at all.
For further performance improvement, as such tests could take a lot of time, consider using a pool of databases and tests parallelization.