How to Succeed at Automating UI Testing

When it comes to picking the right tools to help you succeed at automated UI testing, here’s what you need to understand.

Using automated UI testing effectively isn’t about the tools you use. Don’t misunderstand that, though: You’re going to need the right tools. It’s just that you’re going to need the right set of tools for you.

Why You Can’t Ignore Automated UI Testing Anymore

If you’re one of the people who’ve been skeptical of automated UI testing in the past, you’re neither alone… nor wrong. While automated code-oriented testing tools have become more common, most development shops have ignored automated UI testing.

The primary reason for that has been the cost of maintaining a UI test suite. With most/all UI testing tools, virtually any change to an application’s UI caused UI testing tools to flag the whole application as broken. As a result, much of the course of modern software development practice has been organized around separating the UI from the code precisely so the code could be tested without touching the UI. And there’s the problem.

The reality is that your users don’t interact with your code: Your users interact with your UI. From your users’ point of view, your UI is your application. The current practice of proving that your code works while deliberately ignoring the UI is missing the point. In contrast to current practice, UI testing makes one simple claim: To prove that your application is “ready for production,” you must prove that you UI works and drives your application to do the right thing.

That’s a claim that’s hard to argue with.

Some Tool-Based Options

As DevOps and the demand for user acceptance testing has increased, that claim has become more important. The result has been an evolution in UI test tools that makes getting the right toolset both harder and easier. Harder because there’s more to choose from; easier because there’s more likely to be a tool that makes sense for you. For example, when looking at UI testing, you can choose between codeless and code-based tools.

Codeless tools allow testers to create a UI test by interacting with the application while the tool generates a test script by “observing” both the user’s interactions and the application’s response. These tools leverage the “UI is the application” paradigm and don’t require the tester to know more than the application (and its related business requirements).

Code-based tools, on the other hand, require the tester to write a script that manipulates the UI through code (i.e. finding buttons on the page and then extracting data from UI elements). These tools can, however, check for “side effects” that don’t necessarily show up in any UI (or “any UI that could be accessed as part of the test”) and can handle a wider variety of responses. Code-based tools do require testers to know how to code (no surprise there).

It’s obvious that there’s no one right answer here. Codeless tools take developers out of the critical path for testing and empower users to create tests that have validity for them; code-based tools support deeper, more thorough probes and handle a wider variety of responses, reducing the number of false negatives (failure reports when, in fact, the application is working correctly).

And, at this point, you can see that this article, which started off sounding like a potentially useful discussion, is about to veer off into a bunch of handwaving and “It all depends…” territory.

What Matters

Again, you’re not wrong. Whatever tools you end up using will need to integrate into your processes without getting in the way of you delivering applications… and do that while meeting your organization’s, your users’, and your own goals. That means answering some key questions.

The first one is: Do you need automated UI testing? It’s worth remembering that the goal of testing is to transfer the costs of failure out of the production environment and into the development environment. If your organization is comfortable with the current level of production failures and unwilling to modify development practices, then you may not need automated UI testing. How does automated UI testing fit with your organization’s strategic goals?

That first question overlaps with the second one: How does automated testing fit into your organization’s culture? Does your organization value delivering new functionality as fast and as early as possible to a user community that’s willing to deal with a high rate of change, even if it means some glitches? Or does your organization value highly reliable applications that are stable over time and, as a result, can meet rigorous (perhaps, even regulatory) standards?

And that question, in turn, overlaps with the third one: How will automated UI testing fit into your processes? That answer begins with where and when users get to do acceptance testing (maybe: not at all). A UI testing strategy that leverages your users may not make sense if, for example, there’s a long history of users not being involved in the development process. However, in your organization, if “coder-driven UI tests” is an oxymoron (i.e. only end users get to say if the UI is “right”) then a coder-based approach won’t fit into the way you do things.

Finally, the last question: What sort of skillsets and existing toolsets can you leverage? Codeless testing, for example, only makes sense if you have a pool of users who don’t just “use” an application but are competent to know what counts as a “correct” or “incorrect” response in a test. On the developer side, you want to look at the toolchain used to deliver your application—exploiting your team’s experience with that toolchain and integrating with it has real benefits for you. Interestingly enough, though, the development tools you use to build your applications aren’t especially critical when picking a UI test tool, especially for web applications.

Suite Success

This all means that you’re more likely to be looking at a test suite that can be configured to meet your particular needs than a single “UI testing tool.” You may well end up putting together a “best of breed” suite yourself that meets your unique needs, but it would obviously be more convenient to get a complete solution from a single source.

It’s not surprising, then, that vendors in the automated UI testing field both value flexibility and support for integrating with other tools. Telerik Test Studio, for example, supports codeless testing but also supports converting those codeless tests into coded tests, combining coded steps with codeless tests, and integrating with third-party libraries for special needs.

The ability to create codeless tests means that non-programmers (e.g. the QA team or end users) can create tests that demonstrate that the system does what the users want the system to do. The ability to seamlessly combine those codeless tests with coded tests means that when non-programmers hit roadblocks, developers can extend those tests to handle “hard-to-automate” scenarios. The users create the tests that make sense to them, and programmers are taken out of the critical path, except when they’re required.

But don’t miss the point: It’s still not about the tools—it’s whether those tools support your goals, your processes and your existing skillsets/toolchains. If you’ve got a good understanding of what those are, then you’re in a position to get the tools that will let you succeed at automated UI testing.