I have a method that takes a JSON object and puts it through several stages of processing, updating values in the database at each stage. We wanted this method to be fault tolerant, and decided that the desired behaviour would be, if any processing stage failed, log an error to the database and carry on with the next stage of processing, rather than aborting.
I've just made several changes to the behaviour of one of the processing steps. I then ran our unit test suite, expecting several of the tests to fail due to the new behaviour and point me at potential problem areas. Instead, the tests all passed.
After investigating, I realised that the mock data the tests run against didn't include certain key values important for the new behaviour. The tests were in fact throwing exceptions when they ran, but the exceptions were being caught and handled - and, because the tests don't run with a logger enabled, they were completely suppressed. So the new code didn't change the data in a way that would cause the tests to fail, because it was silently erroring instead.
This seems like the sort of problem that unit tests are there to catch, and the fact that they showed no trace means they're not serving their purpose. Is there any way that I can use NUnit to assert that no exception was ever thrown, even if it was handled? Or alternatively, is there a sensible way to refactor that would expose this issue better?
(Working in C#, but the question seems fairly language-agnostic)
First and foremost, in the scenario you describe, the presence or absence of exceptions is secondary. If you wrote the code to produce a desired result while catching and handling exceptions, then that result - whether it's a return value or some other effect - is the most important thing to test.
If the exceptions that you didn't see caused that result to be incorrect, then testing for the correct result will always reveal the problems. If you don't know what the expected result will be and are only interested in whether or not exceptions are getting handled, something is wrong. We can never determine whether or not anything works correctly according to whether or not it throws exceptions.
That aside, here's how to test whether or not your code is catching and logging exceptions that you wouldn't otherwise be able to observe:
If you're injecting a logger that looks something like this:
public interface ILogger
{
void LogError(Exception ex);
void LogMessage(string message);
}
...then a simple approach is to create a test double which stores the exceptions so that you can inspect it and see what was logged.
public class ListLoggerDouble : ILogger
{
public List<Exception> Exceptions = new List<Exception>();
public List<string> Messages = new List<string>();
public void LogError(Exception ex)
{
Exceptions.Add(ex);
}
public void LogMessage(string message)
{
Messages.Add(message);
}
}
After you've executed the method you're testing you can assert that a collection contains the exception(s) or message(s) you expect. If you wish you can also verify that there are none, although it seems like that might be redundant if the result you're testing for is correct.
I wouldn't create a logger that throws an exception and then write a test that checks for a thrown exception. That makes it look like the expected behavior of your code is to throw an exception, which is exactly the opposite of what it does. Tests help us to document expected behaviors. Also, what will you do if you want to verify that you caught and logged two exceptions?
Related
Using Selenium in Visual Studio. Using NUnit to sort my testcases.
I'm writing a testcase that compares two serialnumbers with a if statement like this:
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
else
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
//method for stopping test missing!
I want to be able to abort the rest of the testsequence if the serialnumbers don't match.
Is there a "end/stop test" method or something similar I could put in else section?
I think you have a couple of options.
1) simply throw an exception (and fail the test)
Throwing an exception will fail a unit test. There are loads of different types of exceptions but the base is simply Exception. You can check the different types of exceptions available here. Where possible try to pick the exception that most closely represents the error (so bad arguments for example use ArgumentException or some derivative there of).
Your test would then look something like this:
[Test]
public void CompareVariables()
{
if (!string.Equals(serialNumberInfo, serialNumberReport))
throw new Exception($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
2) Use an assertion (and fail the test)
Unit tests are usually supposed to assert that something desirable happened. If that thing didn't happen then an exception should be thrown (which is often handled for you by some assertion framework).
So you could flip the test to do this:
[Test]
public void CompareVariables()
{
serialNumberInfo.ShouldBe(serialNumberReport);
// The rest of your tests (only run if serialNumberInfo and serialNumberReport) are equal.
}
This is done with Shouldly but there are countless assertion frameworks so pick your favourite. (mstest has one built in but I find it less readable but that is a personal preference).
Note, only use an assertion when you want to explicitly make sure that it should have happened. I.e. This needs to be true for my test to pass, rather than if this happened then abort. That's hard to explain so I hope that makes sense?
Exceptions for when something went wrong, Assertions for when something should have gone right.
3) Leave the test (and pass the test)
If the test exits without an exception being thrown (either manually or via an assertion framework) then the test is considered to be a passing test. Therefor if you wanted to treat this as a pass you could simply return from the test.
[Test]
public void CompareVariables()
{
if (string.Equals(serialNumberInfo, serialNumberReport))
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} are a match! Proceed!");
}
else
{
Console.WriteLine($"{serialNumberInfo} and {serialNumberReport} don't match! Cancel test!");
return;
}
// The rest of your tests
}
This will mark the test as passing, but mean the rest of the operations in the test are not run. I would try not to do this however - unless you really understand why you want this because you could start passing tests without knowing why they passed (i.e. without asserting anything)
I hope that helps
If you want to end the test early without failing it, simply use return.
[Test]
public void MyTest() {
//Do some stuff
if(!shouldContinue) {
return;
}
}
I do this reasonably often given certain conditions may warrant additional assertions, and other conditions may not. Throwing an exception will fail the test. This will not fail it.
Edit: I just noticed that the other responder mentioned this at the end of their answer. So ignore me :)
I've read this question about simulating throwing exceptions. Answers suggest I create a mock object that pretends being a real object.
That's not what I want to have. If I replace a real object I lose part of real code. I want to invoke real code with minimal changes and have an exception thrown from inside it at random point of that code.
Is it possible to have an exception thrown at random point of code being called from a Unit test?
don't put randomness into your unit tests. it will only bring you troubles. a unit tests should always have some consistence. especially if you want him to tell you what went wrong when he went red. if you implement a random exception throwing into your unit tests it can happen that he SOMETIMES crashes and returns a red bar. maybe the next run it the error is gone again. this is really not what unit tests are here for and you will have big troubles finding the issue for that once-occurred failed test.
a much better approach is to systematically test critical parts of your code. replace each crucial method with a mocked object, which throws the type of exception you want to test for and cover each test case like this. this will give you much more information about what went wrong when you are looking for that error.
hope this helps.
I can think about this work around.
1- Make a function called RandomException(); that throw exception if a random value is divisible by 3 otherwise it won't throw the exception. make sure this function is encapsulcate in this block of code.
#if DEBUG
void RandomException()
{
// gen random value
// if random value % 3 == 0 throw the exception now
}
#else
void RandomException()
{
}
#endif
that way when you release the code these function calls won't affect the program.
I hope that idea helps you.
It seems like you are trying to test the exception handling, i dont think this is a good approach to throw an exception randomly and check if it gets cought, if you need to test handling in certain conditions you need to simulate the condition in a real object, like an illegal situation which should cause an exception.
You still can produce this behaviour through a local varibale which you can set from outside (in your Unittest) which will cause the code to throw an exception depending on this variable, but as saied i dont think its a good approach.
I'm looking for the best method of handling errors in a c# winforms class that I have. The gist of the application is that it has a data analyzer that analyzes the data for statistics and other such stuff. However, I'm looking for the proper way of handling an ABORT.
For example, I have the class called Analyzer
namespace PHOEBE
{
public class Analyzer
{
public Analyzer(){
DoAnalysis();
DoFurtherAnalysis();
}
public class DoAnalysis(){
try{
Convert.ToInt32("someNumber...."); //obviously fails..
}
catch{
//ERROR OCCURRED, ABORT ALL ANALYSIS
return;
}
}
}
Obviously, when DoAnalysis() is called, there will be an error that occurs. The catch block will catch the exception. However, when this catch occurs, it will return to the constructor and run DoFurtherAnalysis(). This is a problem.
I know that you could do return values from each method where each value indicates a certain outcome (ie. 1 = success, 0 = fail). However, a lot of the methods I call, use return values already. I could also use a boolean that gets flagged when an error occurs and check that value before calling the next method from the constructor, but checking this value each time is annoying and repetitive.
I was really hoping for some sort of like "abort mechanism" that I could use. Is there any other ways of working around this? Any interesting work-arounds for this?
Assume this class is being called from a form.
Just let the exception propagate up - you should only catch the exception if you can actually handle it. Exceptions are the "abort mechanism" in .NET. You're currently swallowing the signal that everything's gone wrong, and returning as if all were well.
Generally I find catching exceptions to be pretty rare - usually it's either at the top level (to stop a whole server from going down just because of one request) or in order to transform an exception of one kind into another in order to maintain appropriate abstractions.
I was really hoping for some sort of like "abort mechanism" that I
could use. Is there any other ways of working around this? Any
interesting work-arounds for this?
Yes, there is. It is called exception handling.
Let's rewrite your code:
namespace PHOEBE
{
public class Analyzer
{
public Analyzer()
{
try
{
DoAnalysis();
DoFurtherAnalysis();
}
catch
{
//ERROR OCCURRED, ABORT ALL ANALYSIS
return;
}
}
public class DoAnalysis()
{
Convert.ToInt32("someNumber...."); //obviously fails..
}
}
Now, the constructor will abort and not run the second method since the exception will "bubble through" and be catched where you want it.
On an unrelated note: Please try to catch as specific exceptions as possible, in this case a FormatException
You are subverting the existing "abort" mechanism by catching an exception that you are not doing anything about and swallowing it.
You should not use a try{}catch{} block in this case and let the exception bubble up and cause the application to abort.
The easiest work-around is don't catch the exception. If that were to happen, it'd go straight past the DoFurtherAnalysis() function and out to the original caller.
Don't see anything anoying in returning and checking bool return value from the function. It's much much better solution then having some tricky internal state management, that you for sure will messed up after a couple of months when you return to your code.
Make code sumple and streghtforward. It's not anoying, it's good.
In your specific case if you want just abort everything, just do not catch exception it will abort your program.
use a try...catch in the constructor?
Well, you've got several issues mixed up here. First, it looks like you do possibly-very expensive processing from your constructor. If that processing can throw, you really don't want to call it from your constructor becuase you don't even have the option of returning an error code.
Second, (and you'll read in many threads here,) how you handlee errors really depends on the application and expectation of your users. Some errors could be corrected by changes to inputs. Others might happen in the middle of the night if analysis takes a long time and you might want to continue with another analysis, logging this error.
So I think we're really going to punt back to you for more information about the above.
You could just move DoFurtherAnalysis(); into the the try block
And I would do this entire process somewhere other than the constructor.
Only thing I ever do in the constructor is initialize properties.
When using try-catch in methods, if you want you application to continue even if errors come along, is it okay to return the default value as return through the catch block, and log the error for later review?
For Example:
public static string GetNameById(int id)
{
string name;
try
{
//Connect to Db and get name - some error occured
}
catch(Exception ex)
{
Log(ex);
name = String.Empty;
}
return name;
}
Example 2:
public static string GetIdByName(string name)
{
int id;
try
{
//Connect to Db and get id- some error occured
}
catch(Exception ex)
{
Log(ex);
id = 0;
}
return id;
}
Is it okay to return any default value (depending on the return type of the method ...???) so that the application logic that required the result from this method do not crash and keeps going ....
Thanks in advance...
Regards.
The advice for exception handling is that mostly you should only catch exceptions that you can do something about (e.g. retry an operation, try a different approach, elevate security etc). If you have logging elsewhere in your application at a global level, this kind of catch-log-return is unnecessary.
Personally - typically - in this situation I'd do this:
public static string GetNameById(int id)
{
string name;
try
{
//Connect to Db and get name - some error occured
}
catch(Exception ex)
{
Log(ex);
throw; // Re-throw the exception, don't throw a new one...
}
return name;
}
So as usual - it depends.
Be aware of other pitfalls though, such as the calling method not being aware that there was a failure, and continuing to perform work on the assumption that the method throwing the exception actually worked. At this point you start the conversation about "return codes vs. throwing exceptions", which you'll find a lot of resources for both on SO.com and the internets in general.
I do not think that is a good solution. In my opinion it would be better to let the caller handle the exception. Alternatively you can catch the exception in the method and throw a custom exception (with the caught exception as the inner exception).
Another way of going about it would be to make a TryGet method, such as:
public static bool TryGetNameById(int id, out string name)
{
try
{
//Connect to Db and get name - some error occured
name = actualName
return true;
}
catch(Exception ex)
{
Log(ex);
name = String.Empty;
return false;
}
}
I think this approach is more intention revealing. The method name itself communicates that it may not always be able to produce a useful result. In my opinion this is better than returning some default value that the caller has to be able to know about to do the correct business logic.
My opinion is I'll never mute errors.
If some exception is thrown, it should be treated as a fatal error.
Maybe the problem is throwing exceptions for things that aren't exceptions. For example, business validation shoudn't be throwing such exceptions.
I'd prefer to validate the business and translate the errors in "broken rules" and transmit them to the presentation or wherever, that would save CPU and memory because there's no stack trace.
Another situation is a data connection loss or another situation that makes the application fall in a wrong state. Then, I'd prefer to log the error and prompt the user to re-open the application or maybe the application may restart itself.
I want to make you some suggestion: have you ever heard about PostSharp? Check it, you can implement that exception logging with an AOP paradigm.
It is advised that you only catch errors that you can handle and recover from. Catching and consuming exceptions that you cannot handle is bad form. However, some environments / applications define strict rules that go against this ideal behaviour.
In those cases, I would say in cases you don't have a choice, you can do what you like with returns - it is up to your application to work with them.
Based on the assumption that your application can handle any sort of failure in trying to get an ID, then returning a default ID is a good enough choice. You can go further and use something like the special case pattern to return empty / missing objects, but for simple data types this seems unwarranted.
this depends very much on the context and on the way your calling method is designed. I have used to return default values in the past as you are doing now and I understood only afterwards that it was deeply wrong...
you should imagine the catch block to throw the exception back, after logging it properly, then the calling method could have another try catch and depending on the severity of the error could inform the user or behave accordingly.
the fact is that if you return an empty string, in some cases, could be that the caller "thinks" there is a user with empty name, while would probably be better to notify the user that the record was not found, for example.
depending on the way your logger works you could log the exception only where you handle it and not in all catches...
That depends on what you want. If it's important for you that you log the exception, but that everything else keeps working, then this is - in my honest opinion- ok.
On the other hand: if an exception occurs, you have to make sure this does not have an impact on the further working of your application.
The point of a try/catch is to allow you to catch and handle errors in a graceful manor, allowing application execution to continue, rather than simply crashing the application and stopping execution.
Therefore it is perfectly acceptable to return a default value. However, be sure that all following code will continue to function if a default value is returned rather than causing further errors.
Eg - in your example, if the DB connection fails, ensure there are no further commands to edit / retrieve values from the database.
It REALLY depends. In most scenarios it is not - problems i that if there is a scenario specific issue you amy never find out. Soemtiems a recovery attempt is good. This normalyl depends on circumstances and actual logic.
The scenarios where "swallow and document" are really valid are rare and far in between in the normal world. They come in more often when writing driver type of thigns (like loaded moduels talkign to an external system). Whether a return value or default(T) equivalent maeks sense also depends.
I would say in 99% of the cases let the exception run up. In 1% of the cases it may well make sense to return some sensible default.
It is better to log the exception but then re-trow it so that it can be properly handled in the upper layers.
Note: the "log the exception part" is of course optional and depends a lot on your handling strategy(will you log it again somewhere else?)
Instead of making the app not crash by swalowing exception, it is better to let them pass and try to find the root cause why they were thrown in the first place.
Depends on what you want your app to do...
For example, display a friendly message, "Cannot Login" if you get a SqlException when trying to connect to the Database is OK.
Handling all errors is sometimes considered bad, although people will disagree...
Your application encountered an exception you did not expect, you can no longer be 100% sure what state your application is in, which lines of codes executed and which didn't, etc. Further reading on this: http://blogs.msdn.com/b/larryosterman/archive/2005/05/31/423507.aspx.
And more here : http://blogs.msdn.com/b/oldnewthing/archive/2011/01/20/10117963.aspx
I think the answer would have to be "it depends".
In some cases it may be acceptable to return an empty string from a function in case of an error. If you are looking up somebody's address to display then an empty string works fine rather than crashing the whole thing.
In other cases it may not work so well. If you are returning a string to use for a security lookup for example (eg getSecurityGroup) then you are going to have very undesired behaviour if you return the wrong thing and you might be better off keeping the error thrown and letting the user know something has gone wrong rather than pretending otherwise.
Even worse might be if you are persisting data provided to you by the user. Something goes wrong when you are getting their data, you return a default and store that without even telling them... That's gotta be bad.
So I think you need to look at each method where you are considering this and decide if it makes sense to throw an error or return a default value.
In fact as a more general rule any time you catch an error you should be thinking hard about whether it is an acceptable error and continuing is permissable or whether it is a show stopping error and you should just give up.
I have this sort of format
asp.net MVC View -> Service Layer -> Repository.
So the view calls the service layer which has business/validation logic in it which in turns calls the Repository.
Now my service layer method usually has a bool return type so that I can return true if the database query has gone through good. Or if it failed. Then a generic message is shown to the user.
I of course will log the error with elmah. However I am not sure how I should get to this point.
Like right now my Repository has void return types for update,create,delete.
So say if an update fails should I have a try/catch in my repository that throws the error, Then my service layer catches it and does elmah signaling and returns false?
Or should I have these repository methods return a "bool", try/catch the error in the repository and then return "true" or "false" to the service layer what in turn returns "true" or "false" to the view?
Exception handling still confuses me how handle the errors and when to throw and when to catch the error.
The rule of thumb I always use is:
At low levels, throw when an operation cannot complete due to exceptional circumstances.
In middle layers, catch multiple exception types and rewrap in a single exception type.
Handle exceptions at the last responsible moment.
DOCUMENT!
Here's an example in pseudocode for a multi-layer ASP.NET MVC app (UI, Controller, Logic, Security, Repository):
User clicks submit button.
Controller action is executed and calls into the Logic (business) layer.
Logic method calls into Security with the current User credentials
User is invalid
Security layer throws SecurityException
Logic layer catches, wraps in LogicException with a more generic error message
Controller catches LogicException, redirects to Error page.
User is valid and Security returns
Logic layer calls into the Repository to complete action
Repository fails
Repository throws RepositoryException
Logic layer catches, wraps in LogicException with a more generic error message
Controller catches LogicException, redirects to Error page.
Repository succeeds
Logic layer returns
Controller redirects to the Success view.
Notice, the Logic layer only throws a single exception type -- LogicException. Any lower-level exceptions that bubble up are caught, wrapped in a new instance of LogicException, which is thrown. This gives us many advantages.
First, the stack trace is accessible. Second, callers only have to deal with a single exception type rather than multiple exceptions. Third, technical exception messages can be massaged for display to users while still retaining the original exception messages. Lastly, only the code responsible for handling user input can truly know what the user's intent was and determine what an appropriate response is when an operation fails. The Repository doesn't know if the UI should display the error page or request the user try again with different values. The controller knows this.
By the way, nothing says you can't do this:
try
{
var result = DoSomethingOhMyWhatIsTheReturnType();
}
catch(LogicException e)
{
if(e.InnerException is SqlException)
{
// handle sql exceptions
}else if(e.InnerException is InvalidCastException)
{
// handle cast exceptions
}
// blah blah blah
}
I like to think of exception handling this way: You define your method signature, as to what you are expecting to do. Now if you are not able to do that, then you must throw an exception. So if you are expecting something to fail based on the input data that you have, (ignoring the ambient state), then your method signature must indicate whether an operation has succeeded or failed. But if your method is not expecting to fail based on the input you have (again, ignoring all the other ambient state), then an exception is in order when the method fails.
Consider these two APIs:
int int.Parse(string integerValue); // In this case, the method will return int
// or it will die! That means your data must be
// valid for this method to function.
bool int.TryParse(string integerValue, out number); // In this case, we expect the data
// we passed in might not be fully
// valid, hence a boolean.
While returning an error (or success) code is often the better way, exceptions have one huge advantage over returning codes or silently suppressing errors: at least you can't just ignore them!
Don't abuse exceptions for simple flow control - that would be the silliest thing to do.
But if a function of yours really runs into an "exceptional" problem, then definitely throw an execption. The caller must then either handle it explicitly and thus know what's going on, or it'll bomb out on him.
Just returning an error code is dangerous since the caller might just not bother inspecting the code and could possibly still go on - even if in your app's logic, there's really something wrong and needs to be dealt with.
So: don't abuse exceptions, but if a real exception happens that requires the caller to do something about it, I would definitely recommend using that mechanism for signalling exceptional conditions.
As for handling exceptions: handle those that you can really deal with. E.g. if you try to save a file and get a security exception, show the user a dialog asking for some other location to save to (since he might not have permissions to save to where he wanted to).
However, exceptions you can't really deal with (what do you want to do about a "OutOfMemory exception", really?) should be left untouched - maybe a caller further up the call stack can handle those - or not.
Marc
First of all there is no one way and there certainly isn't a perfect one so don't overthink it.
In general you want to use exceptions for exceptional cases (exceptions incur a performance overhead so overusing them especially in "loopy" situations can have a perf impact). So let's say the repository cannot connect to the database server for some reason. Then you would use an exception. But if the repository executes a search for some object by id and the object is not found then you would want to return null instead of throwing an exception saying that object with ID x doesn't exist.
Same thing for the validation logic. Since it's validating it is assumed that sometimes the input won't validate so in that case it would be good to return false from the validation service (or perhaps a more complex type including some additional information as to why it didn't validate). But if the validation logic includes checking if a username is taken or not and it can't do this for some reason then you would throw an exception.
So say if an update fails should I
have a try/catch in my repository that
throws the error, Then my service
layer catches it and does elmah
signalling and returns false?
Why would the update fail? Is there a perfectly fine reason for this happening that's part of the normal process? Then don't throw an exception if it happens because of a strange reason (let's say something removed the record being updated before it was updated) then an exception seams logical. There really is no way to recover from this situation.