Updated: I realized that I didn’t make it clear I was writing this post from a vendor’s viewpoint! My choices of what to automate below are based on the view of a tester working on the team for this component—the creators of this control. You, as a consumer, should rarely, rarely ever automate third party controls or components. You may need some high-level test to ensure custom functionality works correctly, but your value is much better focused on testing your own software. Don’t invest in automating Google, Telerik, or other commercial products.
When you’re working with automation, choosing what NOT to test is every bit as important as choosing what TO test. Viewing your automation test suite with a disciplined eye and a Lean mindset will reap rewards as your project continues.
Automation is a wonderful thing that I love dearly. (This may not come as a surprise to you since I am an evangelist for a set of tools that helps people automate their testing…) That said, I’ve been burned by, and seen many others burned by, overly complex test suites filled with low-value tests that would be better left to human-driven testing.
For example, let’s look at the Kendo UI’s grid control: http://demos.kendoui.com/grid/index.html
This control gives us the ability to group on any table field by dragging that column’s header to the top of the grid. As you’re dragging the header you see some small indicators on the screen. After you’ve dropped the header in place, you see the grid changed to its grouping display.
If I were to automate tests around this I’d write solid automation tests around what the grid looks like after the header’s been dropped in place: Does the correct column header appear at the top? Are entries in the grid actually grouped by that column? Does sorting still work after grouping? There are a number of other high-value tests I could come up with, depending on how far out of the cage I let my evil QA personality.
What I would NOT test in this example are the appearance of the visual cues. Those small stylistic hints are wonderful additions to our system, and are important to our customer, but they’re a hassle to write tests for in the first place, and they’ll likely be brittle and irritating to maintain over the life of the system.
Other examples of things on this page I wouldn’t look to automate:
- Precise layout factors such as the width of the left sidebar column or main content pane
- Validating the color schemes and logo graphics in the main menu bar
- Validating the proper icons display in the right sidebar
These three points are all vital in our overall site, but they’re things I can quickly validate with my eyes through manual testing. The effort of writing, running, and maintaining these tests just doesn’t pay off in my experience.
The Lean mindset that’s taken hold in the software industry the last several years needs to be forefront in our minds as testers, too. Where am I adding the most value to my tests? Where am I wasting time for me, my team, or those who come after us?
Focus your automation on high-value aspects of your system: business logic, behavior, and basic functionality. Use manual testing where it makes the most sense: visual checks, usability, overall UI.
Keep your automation lean. You’ll be thankful over the long run!