Asking the Wrong Questions on Test Automation

Whether at conferences or in casual discussions at companies, I talk to a lot of testers and in particular testing managers about the benefits of automation.  Almost to a person, they ask some variation of the following:

“It will take a lot of money and time to buy tools and training, and get up to speed on effectively building automated tests.  When will we see a return on this investment?”

I understand where that question is coming from.  Even a mid-sized testing organization is likely spending tens of thousands of dollars, and giving up hundreds of hours of productive testing time in buying and learning automation tools.

But it’s the wrong question.  Or at least it’s not the most important question.  Certainly teams and organizations want to get the most out of their investment.  But in my mind, saving time and/or money isn’t the primary purpose of automation.

That’s right.  If you’re looking to save time or money with automation, you’re off the mark.

So if not for more comprehensive testing or shortening testing times, why are you looking at automation?  Manual testing has a significant flaw.  Depending on who is doing the testing, there are often variations in how those tests are conducted from test run to test run.  One tester may follow steps deliberately, while another may run through the test fast, using shortcuts and other accelerators.  That type of variation has the potential to generate different results, which may cause delays and misunderstandings among the team.

The primary goal of automation is to reduce tester bias and individual variation between test runs, so that testers can have confidence in the results over time.  If variations among individual testers performing manual testing can be reduced, test results are more reliable, and there are fewer odd behaviors that have to be investigated further.

Ultimately, it is a savings in time or money, but not the one that most testing organizations think they are getting.  Rather than faster testing, it is more consistent testing, reducing confusion and further investigation.  It takes bias out of the equation.  I talk about this more in my upcoming Pacific Northwest Software Quality Conference presentation “How Did I Miss That Bug?”

And speaking of when and what to automate, here’s an interesting post from testing guru Michael Larsen on that topic.

Peter Varhol

About the author

Peter Varhol

Peter Varhol is an Evangelist for Telerik’s TestStudio. He’s been a software developer and software product manager, technology journalist, and university professor among the many roles in his past, and believes that his best talent is explaining concepts and practices to others. He’s on Twitter at @pvarhol.