Avoiding Brittle Automation

This morning I gave a “Speed Geeking Breakfast Talk” at the Software Testing Professionals Conference in New Orleans.

Speed Geeking talks at STPCon are eight minute lightning format talks. There’s not really any time for Q&A during the talks, but they’re great for laying out important points and starting a conversation after the talk.

My talk was whimsically titled “Brittle Automation or a Sharp Stick in the Eye? Wait, Let Me Think.” My point with the title is that badly done test automation can become such a time-consuming, frustrating task that the pain of it is metaphorically worse than  a sharp stick in the eye.

I thought I’d post up photos of my flip charts along with some quick explanation of the points!

What’s brittle automation?

Brittle automation are tests which intermittently fail for unclear reasons—the epic “Works on my machine” kind of test! Brittle automation tests are those which are high-maintenance and require a lot of time to update when the system under test changes. Note that’s when the system changes, not if! I also lump slow test suites in with brittle tests because it’s an indicator of root cause issues with your testing approach.

What are some of the impacts of brittle automation?

Brittle tests impact your velocity by soaking up time and effort as you run around fixing failing tests. Brittle tests erode your team’s trust in the value of automation, and can even leak over in to eroded confidence in your testers/QA. Worst of all, teams take a huge morale hit playing “Whack a Mole” with the automation suite.

Why do we have brittle automation?

Brittle comes about because we often try to test too much, or make our tests too complex. Duplication is perhaps one of the largest causes of brittle tests, or at least the pain associated with fixing broken tests – you have to touch tens or hundreds of tests when one locator on the UI changes, or one piece of a workflow changes. Manual pauses in functional automation are a huge source of brittleness. Timing delays you craft for your environment will inevitably fail in other environments. Finally, especially in UI-based automation, coupling yourself to hardwired, brittle locators for UI elements (Xpath, for example) is a recipe for pain.

Fixing brittle automation

Save yourself the pain and frustration around brittle automation.The reality of automated testing is that it’s a software development task – you need to use many of the same approaches in your automation work as in the systems we’re building.

Work closely with your developers to learn what parts of the system under test can be leveraged for speeding up tasks like creating baseline or setup data. Collaborate with them on creating great test cases – they’ll be able to help you understand good software design concepts like the “DRY” (Don’t Repeat Yourself) principle.

Above all stay lean and focused on writing only high-value automation. Not everything can or should be automated. Think about what’s the most important piece to automate. Do that, then move on to something else that’s critical.

Wrapping up

Automation can be incredibly frustrating and time-consuming if you don’t step back and carefully consider how you’ll accomplish the work. Avoid brittle automation and instead focus on writing high-value, low-maintenance test suites.

That way you’ll never have to pause when considering a choice between automation and a sharp stick in the eye…

About the author

Jim Holmes

Jim Holmes has around 25 years IT experience. He is co-author of "Windows Developer Power Tools" and Chief Cat Herder of the CodeMash Conference. He's a blogger and evangelist for Telerik’s Test Studio, an awesome set of tools to help teams deliver better software. Find him as @aJimHolmes on Twitter.