There are some other variations of this question here at SO, but please read the entire question.
By just using fakes, we look at the constructor to see what kind of dependencies that a class have and then create fakes for them accordingly.
Then we write a test for a method by just looking at it's contract (method signature). If we can't figure out how to test the method by doing so, shouldn't we rather try to refactor the method (most likely break it up in smaller pieces) than to look inside it to figure our how we should test it? In other words, it also gives us a quality control by doing so.
Isn't mocks a bad thing since they require us to look inside the method that we are going to test? And therefore skip the whole "look at the signature as a critic".
Update to answer the comment
Say a stub then (just a dummy class providing the requested objects).
A framework like Moq makes sure that Method A gets called with the arguments X and Y. And to be able to setup those checks, one needs to look inside the tested method.
Isn't the important thing (the method contract) forgotten when setting up all those checks, as the focus is shifted from the method signature/contract to look inside the method and create the checks.
Isn't it better to try to test the method by just looking at the contract? After all, when we use the method we'll just look at the contract when using it. So it's quite important the it's contract is easy to follow and understand.
This is a bit of a grey area and I think that there is some overlap. On the whole I would say using mock objects is preferred by me.
I guess some of it depends on how you go about testing code - test or code first?
If you follow a test driven design plan with objects implementing interfaces then you effectively produce a mock object as you go.
Each test treats the tested object / method as a black box.
It focuses you onto writing simpler method code in that you know what answer you want.
But above all else it allows you to have runtime code that uses mock objects for unwritten areas of the code.
On the macro level it also allows for major areas of the code to be switched at runtime to use mock objects e.g. a mock data access layer rather than one with actual database access.
Fakes are just stupid dummy objects. Mocks enable you to verify that the controlflow of the unit is correct (e.g. that it calls the correct functions with the expected arguments). Doing so is very often a good way to test things. An example is that a saveProject()-function probably want's to call something like saveToProject() on the objects to be saved. I consider doing this a lot better than saving the project to a temporary buffer, then loading it to verify that everything was fine (this tests more than it should - it also verifies that the saveToProject() implementation(s) are correct).
As of mocks vs stubs, I usually (not always) find that mocks provide clearer tests and (optionally) more fine-grained control over the expectations. Mocks can be too powerful though, allowing you to test an implementation to the level that changing implementation under test leaving the result unchanged, but the test failing.
By just looking on method/function signature you can test only the output, providing some input (stubs that are only able to feed you with needed data). While this is ok in some cases, sometimes you do need to test what's happening inside that method, you need to test whether it behaves correctly.
string readDoc(name, fileManager) { return fileManager.Read(name).ToString() }
You can directly test returned value here, so stub works just fine.
void saveDoc(doc, fileManager) { fileManager.Save(doc) }
here you would much rather like to test, whether method Save got called with proper arguments (doc). The doc content is not changing, the fileManager is not outputting anything. This is because the method that is tested depends on some other functionality provided by the interface. And, the interface is the contract, so you not only want to test whether your method gives correct results. You also test whether it uses provided contract in correct way.
I see it a little different. Let me explain my view:
I use a mocking framework. When I try to test a class, to ensure it will work as intended, I have to test all the situations may happening. When my class under test uses other classes, I have to ensure in certain test situation that a special exceptions is raised by a used class or a certain return value, and so on... This is hardly to simulate with the real implementations of those classes, so I have to write fakes of them. But I think that in the case I use fakes, tests are not so easy to understand. In my tests I use MOQ-Framework and have the setup for the mocks in my test method. In case I have to analyse my testmethod, I can easy see how the mocks are configured and have not to switch to the coding of the fakes to understand the test.
Hope that helps you finding your answer ...
Related
I'm writing unit tests for the implementation of an API I wrote myself in my company's application. Still new to this whole thing. When looking for answeres on how to unit test certain things I come across a certain pattern. It goes something like this:
Question:
I have this private method I need to unit test.
Top voted answer:
Don't.
I also came across this article arguing against unit testing private methods as well.
Basically how I'm implementing an API I'm given is I write the code first, then I write unit tests to "break it the worst way possible" (as my superior puts it). Once I notice something broke I fix it in the code. To me this seems like a mash-up of OOD and TDD. Is that a legit approach?
The reason I got so many private methods in the first place is that I'm required to break up larger chunks of code into methods. Since these methods are only supposed to be used within the scope of this API implementation I set them to private. Since the file structure established by my team requires me to write all the code into a single file corresponding to an API I can't separate these private methods into a new class and set them to public.
My superior expects me to test these private methods as well. But I'm beginning to doubt if this is even really necessary if the Asserts on the public methods all run successfully?
From my point of view, if my tests on the public methods return the values I expected, I infer that my private methods also work like I intended.
Or am I missing something?
The core point is: unit tests exist to guarantee that your class under tests behaves as expected.
The behavior of your classes manifests itself via those methods that can be called from "outside" of your classes.
Therefore there is neither need nor sense in trying to directly test private methods.
Of course, it is fair to measure coverage while running unit tests; in order to understand which paths in your code are taken. This information can be used to either enhance test cases (to gain more coverage); or to delete production code (which isn't required).
And to align with your question: you do not use TDD to implement private methods.
You use TDD to create a special form of your "contract" that can be executed automatically. You verify what needs to be done; not how it is actually done in detail. That is especially true since the TDD methodology includes continuous refactoring. You write your tests, you turn them green (by writing production code); and then, at some point, you look into improving the quality of your code. Meaning: you start reworking internal aspects of your class under test. Like: creating more private methods, moving content around; maybe even creating internal-only helper classes and so on. But you keep running your existing tests ... which should still all work; because as said: you write them to check the externally observable behavior (as far as possible).
And beyond that: you should rather looking into "fuzzying" the test data that your unit tests drive into your code instead of worrying about private methods.
What I mean: instead of trying to manually find that test data that makes your production code break, look into concepts like QuickCheck that try to do exactly that automatically.
Final words: and if your management keeps hammering on "test private methods"; then it is your responsibility as engineer to convince them that they are wrong about this. And there is plenty of material out there to back that up.
The way you are splitting your code at the moment is out of necessity. You are delegating some work in a private method, because, well, other public methods need to re-use this, and you don't want to copy-paste that code. Of course, since these methods don't make sense being used as standalone methods, you keep them private.
Good, at least you're true to the DRY (Don't Repeat Yourself) principle.
Now, another way to look it is that you want to separate your private methods from the rest of the code, because you want to have a Separation of Concerns. If you do this, you will see that these private methods, although they can't be used on their own, don't really belong to the class containing your public methods, because they don't solve the same concern : This is the Single Responsibility principle: the S in SOLID.
Instead of having your private method within your class, what you can do is move it to another class (a service as I call them), inject it in the class in which they were before, and call these methods instead of the call to the private ones.
Why should you do this ?
Because it will be so much easier to test: you delegate a big part of the code, that you will not have to test under a big combination of scenarios.
Because you can then inject an alternative implementation (think maintainability: it's easier to replace a brick, than a part of a brick)
Because you can delegate the implementation (and the testing) of this service to someone else (you can have 2 developers in parallel working on a very small area of the code)
Sometimes, it makes even more sense, because these service classes will then be re-used by other completely different classes that will have the same needs, if they really take care of one single concern.
This last point doesn't always happen, but quite often, it does. I found it is easier to re-use existing data services when they are self-documented: properly-named services and properly-named methods. (your co-workers will discover them more easily)
Now, you don't need to test a private method... because it's public.
You may think it's cheating, because you just made it public, but this comes from a very legitimate approach: Separation of Concerns.
Final notes:
I am convinced your superior is right about asking you to test this code. One thing he could have added was to do that separation into different classes. Also, make sure that you inject these classes using Dependency Injection and Inversion of Control containers. don't instantiate them using the new statement, otherwise, you will not be able to assert that the right method was called with the right arguments !
I've been reading up on Mocks and Stubs, their differences and uses. I'm still a bit confused, but I think I've got the jist of it.
Now I'm wondering about applications. I can see the use in creating "fake" objects in testing scenarios where the actual objects are too complicated to haul around to test one aspect.
But let's consider my application: I'm working on a computational geometry library. Our library defines points, lines, linesegments, vectors, polygons, and polyhedra, along with a bunch of other objects and all the usual geometric operations. Any given object is stored as a list of points or directions, or lower level objects. But none of these objects takes more than a few milliseconds to generate.
When I'm testing this library, does it make sense to use Mocks/Stubs anywhere?
Right now we just use particular test cases. We're calling them stubs, but I don't think they meet the technical definition of a stub. What do you think better vocab for that would be? "TestCases"? "Examples"?
SourceCode: https://bitbucket.org/Clearspan/geometry-class-library/src
Edit: Note that we're striving for immutability in all our geometry objects, so it only makes sense to test the results of operations, but not state changes to the initial objects.
The fundamental difference between mock and stub is that mock can make your test fail. Stub can't. Stub is used to guarantee correct program flow. It is never part of assert.
Note that mock can also be used to guarantee flow. In other words, every mock is also a stub, and stub is never a mock. Because of such overlapping responsibilities nowadays you don't see much distinction between mock and stub and framework designers go for more general terms (like fake, substitute or catch-all mock).
This realization (mock - assert, stub - flow) helps us narrow down some usage scenarios. To start with the easier one...
Mock
As I mentioned mocks are used in asserts. When the expected behavior of your component is that it should talk to this other component - use mock. All those
emailSender.SendEmail(email);
endOfDayRunner.Run();
jobScheduler.ScheduleJob(jobDetails);
can be only tested by asking "Did it call ScheduleJob with such and such parameters?" This is where you go for mock. Usually this will be mock's only usage scenario.
Stub
With stub it's a bit different. Whether to use stub or not is a design question. Once you follow regular loosely coupled, dependency injection-based design, eventually you will end up with a lot of interfaces.
Now, when in test, how do return value from interface? You either stub it or use real implementation. Each approach has its pros and cons:
with library-generated stubs, your tests will be less brittle but might require more up-front work (setting up stub and such)
with real implementations, setup work is already done but when Angle class changes CoordinateSystem might fail... Is such behavior desirable or not?
Is it? Which one to use? Both! It all depends on...
Unit of work
We arrived at final and the actual part of the problem. What is the scope of your unit test? What is the unit? Can CoordinateSystem be detached from its inner workings and dependencies (Angle, Point, Line) and can they be stubbed? Or more importantly, should they be?
You always need to identify what your unit is. Is it CoordinateSystem alone or perhaps Angle, Line and Point play important part of it? In many, many cases, the unit will be formed by both method and its surrounding ecosystem, including domain objects, helper classes, extensions or sometimes even other methods and other classes.
Naturally, you can separate them and stub all the way around but then... is it really your unit?
As a rule of thumb, use Mocks when you need to simulate behavior, and stubs when the only thing that matters in your test is the state of the object you're communicating with.
Taking into consideration the edit you made to your post, when you need to receive an immutable object use a stub, but when you need to call operations that object exposes then go for a mock, this way you are not prone to failing tests due to errors in another class implementation.
I am working on unit testing stuffs for my controller and service layers(C#,MVC). And I am using Moq dll for mocking the real/dependency objects in unit testing.
But I am little bit confuse regarding mocking the dependencies or real objects. Lets take a example of below unit test method :-
[TestMethod]
public void ShouldReturnDtosWhenCustomersFound_GetCustomers ()
{
// Arrrange
var name = "ricky";
var description = "this is the test";
// setup mocked dal to return list of customers
// when name and description passed to GetCustomers method
_customerDalMock.Setup(d => d.GetCustomers(name, description)).Returns(_customerList);
// Act
List<CustomerDto> actual = _CustomerService.GetCustomers(name, description);
// Assert
Assert.IsNotNull(actual);
Assert.IsTrue(actual.Any());
// verify all setups of mocked dal were called by service
_customerDalMock.VerifyAll();
}
In the above unit test method I am mocking the GetCustomers method and returning a customer list. Which is already defined. And looks like below:
List<Customer> _customerList = new List<Customer>
{
new Customer { CustomerID = 1, Name="Mariya",Description="description"},
new Customer { CustomerID = 2, Name="Soniya",Description="des"},
new Customer { CustomerID = 3, Name="Bill",Description="my desc"},
new Customer { CustomerID = 4, Name="jay",Description="test"},
};
And lets have a look on the Assertion of Customer mocked object and actual object Assertion :-
Assert.AreEqual(_customer.CustomerID, actual.CustomerID);
Assert.AreEqual(_customer.Name, actual.Name);
Assert.AreEqual(_customer.Description, actual.Description);
But here I am not understanding that it(above unit test) always work fine. Means we are just testing(in Assertion) which we passed or which we are returning(in mocking object). And we know that the real/actual object will always return which list or object that we passed.
So what is the meaning of doing unit testing or mocking here?
The true purpose of mocking is to achieve true isolation.
Say you have a CustomerService class, that depends on a CustomerRepository. You write a few unit tests covering the features provided by CustomerService. They all pass.
A month later, a few changes were made, and suddenly your CustomerServices unit tests start failing - and you need to find where the problem is.
So you assume:
Because a unit test that tests CustomerServices is failing, the problem must be in that class!!
Right? Wrong! The problem could be either in CustomerServices or in any of its depencies, i.e., CustomerRepository. If any of its dependencies fail, chances are the class under test will fail too.
Now picture a huge chain of dependencies: A depends on B, B depends on C, ... Y depends on Z. If a fault is introduced in Z, all your unit tests will fail.
And that's why you need to isolate the class under test from its dependencies (may it be a domain object, a database connection, file resources, etc). You want to test a unit.
Your example is too simplistic to show off the real benefit of mocking. That's because your logic under test isn't really doing much beyond returning some data.
But imagine as an example that your logic did something based on wall clock time, say scheduled some process every hour. In a situation like that, mocking the time source lets you actually unit test such logic so that your test doesn't have to run for hours, waiting for the time to pass.
In addition to what already been said:
We can have classes without dependencies. And the only thing we have is unit testing without mocks and stubs.
When we have dependencies there are several kinds of them:
Service that our class uses mostly in a 'fire and forget' way, i.e. services that do not affect control flow of the consuming code.
We can mock these (and all other kinds) services to test they were called correctly (integration testing) or simply for injecting as they could be required by our code.
Two Way Services that provide result but do not have an internal
state and do not affect the state of the system. They can be dubbed complex data transformations.
By mocking these services you can test you expectations about code behavior for different variants of service implementation without need to heave all of them.
Services which affect the state of the system or depend on real world
phenomena or something out of your control. '#500 - Internal Server Error' gave a good example of the time service.
With mocking you can let the time flow at the speed (and direction) whatever is needed. Another example is working with DB. When unit testing it is usually desirable not to change DB state what is not true about functional test. For such kind of services 'isolation' is the main (but not the only) motivation for mocking.
Services with internal state your code depends on.
Consider Entity Framework:
When SaveChanges() is called, many things happen behind the scene. EF detects changes and fixups navigation properties. Also EF won't allow you to add several entities with the same key.
Evidently, it can be very difficult to mock the behavior and the complexity of such dependencies...but usually you have not if they are designed well. If you heavily rely on the functionality some component provides you hardly will be able to substitute this dependency. What is probably needed is isolation again. You don't want to leave traces when testing, thus butter approach is to tell EF not to use real DB. Yes, dependency means more than a mere interface. More often it is not the methods signatures but the contract for expected behavior. For instance IDbConnection has Open() and Close() methods what implies certain sequence of calls.
Sure, it is not strict classification. Better to treat it as extremes.
#dcastro writes: You want to test a unit. Yet the statement doesn't answer the question whether you should.
Lets not discount integration tests. Sometimes knowing that some composite part of the system has a failure is ok.
As to example with the chain of dependencies given by #dcastro we can try to find the place where the bag is likely to by:
Assume, Z is a final dependency. We create unit tests without mocks for it. All boundary conditions are known. 100% coverage is a must here. After that we say that Z works correctly. And if Z fails our unit tests must indicate it.
The analogue comes from engineering. Nobody tests each screw and bolt when building a plane.Statistic methods are used to prove with some certainty that factory producing the details works fine.
On the other hand, for very critical parts of your system it is reasonable to spend time and mock complex behavior of the dependency. Yes, the more complex it is the less maintainable tests are going to be. And here I'd rather call them as the specification checks.
Yes your api and tests both can be wrong but code review and other forms of testing can assure the correctness of the code to some degree. And as soon as these tests fail after some changes are made you either need to change specs and corresponding tests or find the bug and cover the case with the test.
I highly recommend you watching Roy's videos: http://youtube.com/watch?v=fAb_OnooCsQ
In this very case mocking allowed you to fake a database connection, so that you can run a test in place and in-memory, without relying on any additional resource, i.e. the database. This tests asserts that, when a service is called, a corresponded method of DAL is called.
However the later asserts of the list and the values in list aren't necessary. As you correctly noticed you just asserting that the values you "mocked" are returned. This would be useful within the mocking framework itself, to assert that the mocking methods behave as expected. But in your code is is just excess.
In general case, mocking allow one to:
Test behaviour (when something happens, then a particular method is executed)
Fake resources (for example, email servers, web servers, HTTP API request/response, database)
In contrast, unit-tests without mocking usually allow you to test the state. That is, you can detect a change in a state of an object, when a particular method was called.
All previous answers assume that mocking has some value, and then they proceed to explain what that value supposedly is.
For the sake of future generations that might arrive at this question looking to satisfy their philosophical objections on the issue, here is a dissenting opinion:
Mocking, despite being a nifty trick, should be avoided at (almost) all costs.
When you mock a dependency of your code-under-test, you are by definition making two kinds of assumptions:
Assumptions about the behavior of the dependency
Assumptions about the inner workings of your code-under-test
It can be argued that the assumptions about the behavior of the dependency are innocent because they are simply a stipulation of how the real dependency should behave according to some requirements or specification document. I would be willing to accept this, with the footnote that they are still assumptions, and whenever you make assumptions you are living your life dangerously.
Now, what cannot be argued is that the assumptions you are making about the inner workings of your code-under-test are essentially turning your test into a white-box test: the mock expects the code-under-test to issue specific calls to its dependencies, with specific parameters, and as the mock returns specific results, the code-under-test is expected to behave in specific ways.
White-box testing might be suitable if you are building high criticality (aerospace grade) software, where the goal is to leave absolutely nothing to chance, and cost is not a concern. It is orders of magnitude more labor intensive than black-box testing, so it is immensely expensive, and it is a complete overkill for commercial software, where the goal is simply to meet the requirements, not to ensure that every single bit in memory has some exact expected value at any given moment.
White-box testing is labor intensive because it renders tests extremely fragile: every single time you modify the code-under-test, even if the modification is not in response to a change in requirements, you will have to go modify every single mock you have written to test that code. That is an insanely high maintenance level.
How to avoid mocks and black-box testing
Use fakes instead of mocks
For an explanation of what the difference is, you can read this article by Martin Fowler: https://martinfowler.com/bliki/TestDouble.html but to give you an example, an in-memory database can be used as fake in place of a full-blown RDBMS. (Note how fakes are a lot less fake than mocks.)
Fakes will give you the same amount of isolation as mocks would, but without all the risky and costly assumptions, and most importantly, without all the fragility.
Do integration testing instead of unit testing
Using the fakes whenever possible, of course.
For a longer article with my thoughts on the subject, see https://blog.michael.gr/2021/12/white-box-vs-black-box-testing.html
I'm creating a TestClass for unit testing an application, and one thing I'd like to do is run a test method, check method has run correctly, store a value in class property within test class base on outcome, and then use that value in a later method.
I've tried doing this and found that as soon as the compiler moves from one method to another, all the properties I have set are wiped clean. I have checked with breakpoints and at the end of the first method the value is in the property, and then at the beginning of the second method that same property is null.
Looked this up and no one else seems to be attempting the same thing, so is it possible to share a value between methods or am I taking the wrong approach?
Thanks in advance.
You are taking the wrong approach.
Unit tests, by definition, should be completely self-contained and deterministic. They should not depend on one another.
You should be able to refactor out the repetitive portion of your first unit test into a helper method which can be invoked by your other unit test. The work will be done twice, but Unit Tests should be really fast, so the overhead should be really minimal.
It's not the compiler - it's the test runner which will (potentially) create a new instance for each test.
Tests should generally be independent - even if you could find a way of getting this to work, I would avoid doing so. Design your way around it as best you can.
This smells like bad practice to me, no matter what testing framework you're using. All automated tests (let alone formal unit tests) should be independent of one another. A static field/property might work, but I would recommend refactoring your tests first.
I recently joined a company where they use TDD, but it's still not clear for me, for example, we have a test:
[TestMethod]
public void ShouldReturnGameIsRunning()
{
var game = new Mock<IGame>();
game.Setup(g => g.IsRunning).Returns(true);
Assert.IsTrue(game.Object.IsRunning);
}
What's the purpose of it? As far as I understand, this is testing nothing! I say that because it's mocking an interface, and saying, return true for IsRunning, it will never return a different value...
I'm probably thinking wrong, as everybody says it's a good practice etc etc... or is this test wrong?
This test is excerising an interface, and verifying that the interface exposes a Boolean getter named IsRunning.
This test was probably written before the interface existed, and probably well before any concrete classes of IGame existed either. I would imagine, if the project is of any maturity, there are other tests that exercise the actual implementions and verify behavior for that level, and probably use mocking in the more traditional isolationist context rather than stubbing theoretical shapes.
The value of these kinds of tests is that it forces one to think about how a shape or object is going to be used. Something along the lines of "I don't know exactly what a game is going to do yet, but I do know that I'm going to want to check if it's running...". So this style of testing is more to drive design decisions, as well as validate behavior.
Not the most valuable test on the planet, but serves its purpose in my opinion.
Edit: I was on the train last night when I was typing this, but I wanted to address Andrew Whitaker comment directly in the answer.
One could argue that the signature in the interface itself is enough of an enforcement of the desired contract. However, this really depends on how you treat your interfaces and ultimately how you validate the system that you are trying to test.
There may be tangible value in explicitly stating this functionality as a test. You are explicitly verifying that this is a desired functionality for an IGame.
Lets take a use case, lets say that as a designer you know that you want to expose a public getter for IsRunning, so you add it to the interface. But lets say that another developer gets this and sees a property, but doesn't see any usages of it anywhere else in the code, one might assume that it is eligible for deletion (or code-rot removal, which is normally a good thing) without harm. The existence of this test, however, explicitly states this properity should exist.
Without this test, a developer could modify the interface, drop the implementation and the project would still compile and run seemingly correctly. And it would be until much later that someone might notice that interface has been changed.
With this test, even if it appears as if no one is using it, your test suite will fail. The self documenting nature of the test tells you that ShouldReturnGameIsRunning(), which at the very least should spawn a conversation if IGame should really expose an IsRunning getter.
The test is invalid. Moq is a framework used to "mock" objects used by the method under test. If you were testing IsRunning, you might use Mock to provide a certain implementation for those methods that IsRunning depends on, for example to cause certain execution paths to execute.
People can tell you all day that testing first is better; you won't realize that it actually is until you start doing it yourself and realize that you have fewer bugs with code you wrote tests for first, which will save you and your co-workers time and probably make the quality of your code higher.
It could be that the test has been written (before) the code which is good practice.
It might equally be that this is a pointless test created by a tool.
Another possibility is that the code was written by somebody who, at the time, didn't understand how to use that particular mocking framework - and wrote one or more simple tests to learn. Kent Beck talked about that sort of "learning test" in his original TDD book.
This test could have had a point in the past. It could have been testing a concrete class. It could have been testing an abstract one -- as it sometimes makes sense to mock an abstract class that you want to test. I can see how a (less informed(like myself)) follower of the TDD way of doing things could have left a small mess like this. Cleanup of dead code and dead test code, does not really fit in the natural workflow of red, green, refactor (since it doesn't break any tests!).
However, history is not justification for a test that does nothing but confuse the reader. So, be a good boyscout and remove it.
In my honest oppinion this is just rubbish, it only proves that moq works as expected