NOTE: I’m running a series called “31 Days of Testing” on my personal blog at FrazzledDad.com. The series covers a really broad range of testing topics, so I haven’t been cross-posting things here. This specific post; however, hits perfectly on things I’ve spoken about in Test Studio webinars and previous posts, so I’m posting it here, too. I’ll cross-post a few others; however, if you want to read the full series then please join me over at FrazzledDad!
I wanted to write a blog post on “Introduction to Functional Automation Testing” but I realized I firmly believe there are a number of things to know before you ever start down the road of functional automation so that you don’t get bushwhacked a few weeks or months in to your effort.
In my view, functional tests via a browser or desktop automation tool, are by far the slowest, most brittle, most curmudgeonly type of automated test. Maybe that’s why I like them so much…
I’ve been fortunate to have spent a few trips over the last month working with customers to train them up on Telerik’s Test Studio, the great automation tool I’ve joined Telerik to help promote. I think the customers are a little taken aback when I get very frank and animated about the difficulties around functional test automation. I also make it very clear that everything you do around functional test automation is closely tied to how well your tests will be running in four weeks or four months. (NOTE: Test Studio is an awesome tool from an awesome team that I’m proud to be part of, but functional testing isn’t just some point-click-click-click-run magic. We help you get close to that state; however, there’s a lot of things you need to keep in mind for a successful, long-term automation effort!)
I’ve also been spending a LOT of time wandering around to various conferences and user groups giving my talk on Automation Isn’t Shiny Toys. That talk really drives home the point that you’ve got to carefully focus on your automation strategy if you want to succeed and stay sane.
In this post I want to highlight a few of the things I discuss in that talk. I’ll dive in to greater detail on a few of those points in later posts, but the post you’re reading now will give you the broad view.
I’ve seen automation fail, or cause tremendous stress, in a number of situations: Projects I’ve been on, projects I’ve seen colleagues on, projects I’ve talked about with friends and other conference attendees, and projects I’ve seen at customer sites. I’ve come to see three common threads among these conversations: long-running tests, brittle tests which break too frequently, and tests which take too much time to maintain over the life of the project. (I also admit that items two and three are likely the same root cause.)
Long-running test suites can suck the trust out of your project. Functional tests at the UI level will always be much slower than other sorts of tests, but you can’t have a productive, trustworthy automation environment if your tests can only be run once a month over a weekend. You need a moderately short execution time so you can have your UI tests running several times a day.
Keep in mind the scale of time in this context. 800 tests taking 30 seconds per test will take you nearly seven hours to execute. Small changes across many tests will net you huge gains.
Here are a few causes I’ve seen for overly long execution times:
Brittle tests break for odd reasons. Brittle tests work in the developers’ environments, but fail in the QA or staging environments. Brittle tests fail when run in your suite, but pass when run individually.
Here are a few causes of brittle tests I’ve run in to (or shot myself in the foot with!):
If you’re not careful, you can easily end up spending more time maintaining your existing automated tests than you spend creating new ones or doing your exploratory testing. Roy Osherove, in his amazing book The Art of Unit Testing talks about a project that failed because his team spent too much time maintaining tests. Note they didn’t abandon testing on their project—the entire project failed! Ouch.
In my experience, for functional/UI tests, the number one cause of high-maintenance tests is poor element locator management. Side effects and poor test design contribute, but locators is the number one PITA around. Locators are how your testing framework/tool finds elements on the page or in the application. (I’ll have much more on locators in a following post.) If you aren’t careful, you’ll end up spending huge amounts of time fixing up your tests’ locators every time the UI changes even a small amount.
Two things can greatly help you cut down pain around your locator management.
I’ll be doing some follow on posts introducing you to functional test automation; however, past experience has taught me that you’ll see the most success when you keep your automation suite’s maintainability in mind from the start. Gee, isn’t that sort of the same approach great software developers use? Maybe you should treat your testing codebase like production code—because it is production code!
I hope you’re enjoying this series so far and are finding it useful. I’m enjoying writing it!
Subscribe to be the first to get our expert-written articles and tutorials for developers!