You can’t guarantee bug-free code. But you can validate your test suite to make sure you’re catching as many bugs as possible.
Let’s be clear: Testing is probably the most inefficient way to eliminate bugs imaginable … but it’s what we’ve got right now. And, while I’m obviously a big fan of automated testing, automated testing is not a panacea. But, telling people to “write good, SOLID code” isn’t sufficient protection because (as we all know) even code written with the best of intentions has bugs.
But why isn’t automated testing the final solution? Because, after all, your test suite consists of two things: inputs and … (wait for it) … code. That test code, like all other code, is subject to bugs. And, when you have a bug in your test suite, then your tests can’t, in fact, protect you from implementing bugs in your production system.
This all just means that you should be checking for bugs in your automated tests as diligently as your automated tests check for bugs in your application code. However, you don’t want to test your test code by writing more code—if you do that, you’re going to get into an infinite regression of writing code that checks code that checks code that… You need different mechanisms to validate your automated tests.
To understand how you can validate your test code, you need to consider the three ways that your automated testing can fail:
The false negative: A test fails when there’s nothing wrong because the test (code or inputs) is badly written. This isn’t actually a problem, though. First: so long as you have a failed test, your code isn’t going to move to production which is where your bugs matter. Second: you’re going to investigate any failing test and fix the problem. It’s too bad about any delay or costs associated with fixing the test but no bugs will be deployed to production—the false negative is a self-correcting problem.
The missing test: You didn’t recognize a potential point of failure, didn’t write a test to check it, and (as a result) don’t catch the inevitable bug that a test would have caught. This, by the way, is fallout from the Fifth Law in my 10 Immutable Laws of Testing: Anything you don’t test has a bug in it.
The false positive: A test that reports success when, in fact, something is wrong. Think of these as “crummy tests” and they come in two varieties:
» A test that doesn’t prove what you think it proves. You are, for example, trying to prove that anything on or before the shipping date will be rejected but the test is only checking for dates before the shipping date.
» A test that can never fail. Mea culpa: I’ve written one of those tests—I was checking a sorting routine and used test data that was already sorted. Even if my sort routine did nothing at all, my test was going to pass (and, as we discovered in production, my sorting code was doing nothing at all).
Given those are all bugs that your automated tests can have, how do you make sure that you don’t have missing or crummy tests?
For the missing test, you should consider every possible way your code can go wrong and provide a test to check for each of those ways. This is stupid advice because, of course, you feel you’re already doing that.
But, I bet, what you’re doing is creating a list of inputs that you consider “dangerous.” That’s not the same thing as creating inputs with every possible value (both valid and invalid) and every possible combination of those values. It might be worthwhile to consider a test data generation tool that will be more objective than you in generating all the potential inputs for your application.
But, in addition to your inputs, you need to consider all the different ways your application can be used. You’re not as good at that as you think you are because you’re looking at the application from a developer’s or testing engineer’s perspective. Instead, start bringing in some end users (who have a very different perspective than you do) and ask them to stress your application. They will generate multiple tests that—I guarantee—you will not have thought of.
And, while I’m not a big fan of coverage statistics, they can be useful here. Let me be clear about coverage statistics: my feeling is that if you’ve passed all of your tests, then you’ve passed all of your tests—your application is as bug-free as you can make it, regardless of what code is or isn’t executed. But my claim does assume you have “all the tests.”
So, if your coverage report shows that you have code that isn’t being executed, then you want to consider if that’s code that can never be accessed (“dead code”) or code that isn’t executed because you’re missing a test case. Based on what you determine, you should then take action:
If you’re concerned about deleting code because, after all, change is our enemy, I’ll just quote my Fifth Law of Testing (again): Anything that you don’t test has a bug in it. That being true, since “dead code” is code you’re not testing, then you’re leaving buggy code in your application. Q.E.D.
To find those tests that aren’t doing what you think they’re doing, you need a special set of inputs that are guaranteed to cause every one of your tests to fail … and to fail in every way possible. Think of this as your “guaranteed failure test suite.”
If you run a test with those inputs and some test doesn’t fail, then you have found a crummy test. Once you’ve found those tests you need, again, to take action:
If you can’t generate a test that will cause some code to fail (for example, the test has to mimic the application being simultaneously offline and still, somehow, accepting inputs), then you really do have a condition that can never happen. You’ll just have to hope that the conditions that will cause that code to run can’t occur in production. But, having said that, I’d make sure that I couldn’t find some ingenious way to create a relevant test.
And it’s always a good idea to have some other person check your tests to make sure that your tests are doing what you think they are doing. To support that, you’ll need to document what each test is supposed to prove (the test’s intent) so that the “other person” can assess the test against the test’s intent.
You can document the test’s intent with a comment, but a better solution is to write the code to indicate the intent of your test (calling the test “ShippingDateIsOnOrBeforeOrderDate,” for example, rather than “ShippingDateBad”). Adding a comment should be your second choice.
It’s also a good idea to keep your tests as simple as possible. Keep the act phase of your tests to one or two lines, written in as clear as fashion as possible so that, if the test is crummy, it will be instantly obvious to anyone who reads it. That will result in you having lots of simple tests, but I think that’s preferable to having a few large, complex tests.
But that last piece of advice isn’t new to you—it is, after all, one of the coping skills we’ve adopted to reduce bugs in any code.
Since we’ve come full circle and are back to “writing good code,” here’s a final tip that is “test suite related”: If, while auditing your test suite, you do find a bug, look for more. If you were in a hotel room and saw a cockroach, you wouldn’t say, “Oh, look there’s a cockroach.” Nope: You’d say, “Oh my gosh, this place is infested.” Like real-world bugs, bugs in code—including test code—typically travel in packs: When you see one bug, look for more.
Will these tools and techniques ensure you’ll never have a bug get into production? No (and don’t be silly). But it will help to validate your test suite (code and inputs) so that when a bug does get into production, you won’t look stupid.
Peter Vogel is a system architect and principal in PH&V Information Services. PH&V provides full-stack consulting from UX design through object modeling to database design. Peter also writes courses and teaches for Learning Tree International.