Mastering the Essentials of UI Automation—Webinar Q&A Follow Up

Wednesday, March 25, Dave Haeffner, joined me (Jim Holmes) for a webinar targeted at helping teams and organizations start up successful UI test automation projects. This webinar was based on a series of blogposts on the same topic hosted here on the Telerik blogs.

You can find the recording of the webinar hosted online and you can sign up to receive a copy of an eBook assembled from those same blogposts.

We had a very interactive audience for the webinar. Unfortunately we couldn’t answer all questions during the webinar itself. We’ve addressed quite a few of the unanswered questions below.

Q: Companies think that automation will make development results better. Not actually true because you cannot test in quality.

A: Absolutely true. “Quality” is a whole team effort. Automation is only part of the equation.

Q: Do we have a way to measure the quality of automation itself? Is it just the same as measuring manual tests, like using some metrics?

A: Metrics are always a tricky thing because too often they drive bad behavior or measure the wrong things. Some metrics I’ve found useful at the team (not organizational level) are number of bugs found by automation, and number of automation scripts written as a result of input from support or other up-stream organizations.

Q: Hi! Currently we're using UFT. Our devs use Jenkins as well. Can we integrate UFT automation scripts to be integrated with Jenkins?

A: I don’t have experience with HP’s UFT, however, I’m certain it can integrate with Jenkins—nearly every test automation toolset has some form of Jenkins integration. Hit up your UFT support folks, UFT’s documentation, or good old Google/Bing.

Q: How do we evaluate which tool we need to use..??

A: The eBook I mentioned at the top of this post will have some very helpful info. Also, I can’t over-recommend Elisabeth Hendrickson’s wonderful post on choosing the right tool.

Q: If all else fails, bribe with FOOD or DRINK!

A: (This comment was in response to a discussion around developer-tester collaboration.) Absolutely. There’s no better way to bring your team together than over food and beverages, adult beverages or not. Struggling to get your team to gel? Throw a lunch and learn, but buy the team food!

Q: My company currently has only manual testers--no automated tests. I am a technical tester and I plan to own Selenium integration tests on a new software product we are planning. How do you recommend I work with development as they write the application code and I write integration tests? How do you identify business value that needs integration tests on a new product when you don't have past data on what has been broken or what has been used the most?

A: First off, you’re not alone! Secondly, your mantra should be “Business value, business value, business value!” Don’t worry about past metrics—focus on what the stakeholder/customer/sponsor has identified as their primary goals moving forward. Get great tests around that, then have the conversation about whether or not it makes sense to test (or even build!) lower value pieces.

Q: Should any project ever have a goal of zero manual testing?

A: No. Next question.

Seriously, automation has a place in an overall test strategy. It is not an overall test strategy by itself. Automation frees testers from burdensome regression and acceptance testing and lets them use their intuition, domain knowledge and skill to evaluate other areas of the system.

Automation plus great testers makes for awesome value delivered to customers.

Q: UI automation is fragile, what are important factors to be considered?

A: A top priority is to work tightly with developers whenever possible. They’ll help with gradually adjusting the UI into a much more testable component.

I’ve always felt teams, with or without tight developer collaboration, need to focus on these factors for less-brittle automation. (Note I didn’t say anything about eliminating fragility of tests, just making it less fragile!)

• Good locators/find logic
• Clearly understood async actions (AJAX, etc.)
• Solid backing APIs for configuration, setup and oracles/heuristics

Q: What are some essentials for any manual testers to know or learn before jumping in to automation?

A: Learn the concepts in the question above about UI automation fragility. Understand that test code is dramatically different from system code. Learn to work with the rest of your team. Perhaps most importantly, learn to focus on value before look and feel.

Q: What components should an Automation Framework have?

A: “Components” can mean lots of different things to different people! Here’s what I’d start out with:

• Solves the business-level problems you’re trying to fix
• Can be easily learned and supported by your team/organization over the lifetime of the tests
• Support for some form of page objects/page repository for centralized element definition
• Enables your testers and developers to work closely together
• Enables you to write as much or little code as you need to effectively tie into your system
• Easy support for handling asynchronous operations in your application
• Easy way to remotely execute your test suites in a distributed fashion
• Ease of integrating into systems like TFS, Jenkins, Team City, etc.

Q: What's the most common thing that results into a dropped/killed automation framework?

A: In my experience it’s low value and high cost of maintenance. Few issues are discovered or prevented, but the team needs a huge amount of time to build and maintain the tests.

Q: Where do you draw the line between adding code to a product for automation testability?

A: The line moves from project to project! My line is when the work to make a system more testable outweighs the value (and opportunity cost) of time needed to get that particular aspect of the system tested. Testability is wonderful, but not at the cost of continuing to move forward with delivering value!

Q: With companies going to an agile world, where do automation developers fit in? When do they step in? Do they develop tests for screens being built or screens that have been built already?

A: Teams should focus on writing tests for features in development. A well-functioning team should be able to write automation for features that are currently in development. If you’re not able to do that, it’s an indication you’ve got too few testers, a poor toolset (either for automation or your system), or perhaps your team isn’t collaborating effectively.

As your team improves, you should look at writing automation to backfill functionality that was built before you got adept. Do NOT automate everything. Instead, focus on the high-risk, high-value pieces that don’t have testing around them

Q: Won’t automation help reduce testing efforts which will eventually help reduce testing resources?

A: Perhaps, but using automation to reduce headcount of testers is generally the wrong way to approach the effort. Instead, you should focus on solving other business problems: long ship cycles due to long testing efforts, escaped bugs, high support ticket costs, fear of changing a brittle system, etc.

Q: Can you give some examples of good tests vs bad tests?

A: Bad tests are unclear, mix or conflate test cases, are overly complex, contain duplication and use brittle locators/async handling. Good tests are short, concise, focused, clear and aren’t brittle due to duplication, etc.

Q: Can you link us to the books?

A: Dave’s books, Java and Ruby editions, can be found at http://seleniumguidebook.com/. Jim’s book on Leadership can be found at https://leanpub.com/theleadershipjourney

Q: Can you post a link to the newsletter?

A: Dave’s newsletter is at http://elementalselenium.com/

Q: Do you think it's a good idea to keep automated Cucumber acceptance tests separate from automated Selenium integration/acceptance tests?

A: My (Jim’s) opinion is yes. I prefer to keep different test domains separated. Unit, integration, and acceptance are all best kept separate for two reasons: clarity, and ease of tool support.

Q: I am in charge of GUI testing on our agile team using Telerik Testing Framework on a WPF application and I am trying to find a way to efficiently contribute within the sprint.

A: Read the blog series and eBook! You can best contribute by having early conversations and making sure you’re focusing on writing high-value tests. Early conversations means it’s also easier for you to write GUI tests since you can ask devs for help with a testable UI!

Q: Does a UI tester contribute at the start of an iteration when the GUI isn’t built or after the iteration is done?

A: Start at the start! Work as early in the iteration as possible. All good automation tools let you build your tests ahead of the GUI (now including Test Studio, too! See what’s included in the latest release here), so there’s no reason not to work alongside the development of the feature!

Q: Is automation applicable only to “test driven development” for unit tests, or can it be used for functional tests by an outside tester too?

A: Automation is applicable to unit, integration, functional and other types of testing too.

Q: Our company isn’t supportive of tester/developer collaboration. How can we change that?

A: That’s not uncommon, unfortunately. First, step back and make sure you have a problem that really needs solving. (Read back through the blogpost and eBook for some help.) If you feel you do have a problem that needs addressing, try a couple different approaches.

• Assemble some cost data around current impacts of your poor quality issues. (Think cost of time to fix issues, customer support impacts, etc.) Give some estimates on how you might reduce those costs while perhaps increasing revenue (higher sales, faster delivery of releases, etc.).
• Run a short spike to implement a few automated tests around your highest value features, or those which have the most regressions. Collect data around bugs caught by those automated tests.

These approaches help you get hard facts in place to help convince your leadership of the need for and benefits of automated testing.

I can personally attest to the value of the second approach. I once went guerilla and built 50 or so automated tests in secret without my boss’s permission. He was anti-automation until I showed him reports of how many bugs I’d detected while running those tests on a scheduled job over a week’s time.

Q: Is it mandatory for developers to create more test-friendly HTML? I’m often missing good IDs.

A: No. Nor is it mandatory to avoid hitting yourself in the head with a rock; however, both test-friendly HTML and avoiding striking oneself in the head with a rock are much less painful approaches to life.

You can always fall back to combinations of other locator strategies in the situations where developers won’t or can’t get you testable UI. It’s not as fun or easy, but it’s absolutely workable.

Thank you all for your interest! For upcoming webinars, visit this page.