QA Automation Testing—Balancing Time, Cost & Quality

This guide describes the benefits of test automation, and the critical importance of planning for successfully balancing time, cost and software quality.

Automated testing is not new. Tools and coding options have been around for 20+ years. Many organizations have adopted some sort of automated testing strategy for selected testing efforts, others have abandoned it completely, and still others have moved to test automation exclusively. Why such a variation in adoption? Successful test automation requires balancing realistic expectations of time, cost and quality.

Organizations that are quick to adopt automation without first planning, budgeting or strategizing are frequently doomed to failure. Automated testing offers significant benefits. Take the time to design an automated test strategy that fits the quality requirements and the QA testing team.

This guide describes the benefits of test automation, and the critical importance of planning for successfully balancing time, cost and software quality.

Top Benefits of Automated Software Testing

The top benefits of automated software testing include:

• Increasing the depth and breadth of test coverage.
• Enables a QA testing team to create valid performance and load tests.
• Releases QA testers from basic, repetitive test execution so they can focus on complex test scenarios.
• Can speed up test execution time, or be executed as needed.

When well-planned and thoughtfully developed, automated test suites increase the scope of testing coverage. For example, your QA team plans and develops a suite of smoke tests for the application UI. The suite can be run against any browser or operating system. Instantly, you have increased test coverage and save the testing team a great deal of time having to manually test different browser and OS combinations.

Another benefit of well-designed test automation is creating a suite of smoke or critical priority regression tests. If the QA team can automate the most critical workflow functionality for an application, those tests can be quickly executed as unit tests for each build, as smoke tests, or executed daily or for a special release.

Testing flexibility increases with reliable test automation suites that are executable at any time. One important note: When an organization first decides to automate, do not be tempted to skip planning. Automated testing that is unplanned usually starts as a company declaration to automate X tests by the end of the year. You may end up with X number of tests, but they won’t be organized and, if you look closely, they may not even be valid.

For example, I’ve seen test automation suites that met the numbers by repeating a single working test 1,000 times and changing one data point without any added validation. What gets tested is that the tool can run through the test, but the application functionality is not validated. Those types of tests are useless for determining software quality.

Planning for Test Automation

A well-designed test automation plan starts with the following:

• Defined budget
• Prioritized list of areas for test development
• Realistic schedule
• Resource identification
• Tool selection
• Tool training
• Test design definition
• Review of existing tests

Defining a budget includes the cost of resources over time as well as the cost of a tool, training and test maintenance. Don’t forget to budget for maintaining tests throughout the development cycle. Unless your application doesn’t change or update features, you need to budget for test script maintenance to keep the scripts running effectively.

Prioritizing application areas means creating a prioritized list of application functionality. Each application function should be prioritized as critical, high, medium or low. Once you have test areas grouped, then you can decide where test automation development should start.

Next, create a realistic schedule for test development. When creating the schedule, plug in your identified testing and/or developer resources who will be working on test automation development. Make sure to understand each resource’s current workload and assignments and all of those tasks are covered so they can focus on test automation. If you try to squeeze test automation development on top of a tester’s current workload, one thing will happen—there will be no time left for test development.

Now, the easy part. Select a tool that works for your application and your team from literally thousands of quality options. Consider having the testers assigned to test automation participate in free trials of various tools and work together to find the best match. Consider the codeless test automation tool Progress Telerik Test Studio for a quality tool that empowers the QA team with support, intuitive design and AI assistance for maintenance.

You’re not done yet, but you’re close. Develop an automated test design approach. The scripts may differ slightly but should be as standardized as possible for easier maintenance, failure analysis and troubleshooting. Spend time having the QA test automation team create a workable test design template. It’ll save time throughout the script’s lifetime.

Finish up by reviewing the current manual test suites. Figure out which tests should remain manual and which, if any, can be updated to an automated script. Many complex end-to-end workflow scenarios are not suitable for test automation, especially for highly integrated applications. Keep complex tests manual, and make a list of tests that are more direct with one or two verification points for test automation.

Now that you have a well-designed automated test strategy, you’re ready to move forward.

Dividing Forces—What to Test Manually?

When you are starting down the automated test path, don’t try to automate every test. Many tests are best suited for manual test execution.

Manual testing is most effective for:

• Irregular test cases or tests that only need to be executed once
• Complex workflow test scenarios
• End-to-end system tests for highly integrated applications
• Usability tests
• Exploratory tests
• Tests for functionality that is challenging to repeat and rely on the experience of a QA tester

Manual testing is highly effective for testing human interactions or complicated workflow scenarios that require moving between different applications. For example, many healthcare workflows test functionality between a software application and a medical device. These types of tests rely on the experience of an end user interacting with the device. For complex integrated scenarios, manual testing is more cost-effective and efficient.

Dividing Forces—What Tests to Automate?

Get the most value from test automation by automating the following test types:

• Smoke
• Unit
• Load and performance
• Data
• Cross-browser
• Cross-platform

Smoke testing may involve both functional regression and unit tests. Both focus on the automation of modular tests to verify a single function. The intent for these types of tests is they are schedulable and repeatable at will. Both smoke and unit test automation focuses on critical application functionality.

Application load and performance automated scripts are a way to extend test coverage. Many organizations simply skip or skimp on load and performance tests. However, with test automation testers can create suites for both load and application performance. Performance is a vital ingredient of customer satisfaction and should not be ignored. The same holds for testing an application’s load range. Both are critically important to a positive customer experience.

Testing data structure is essential to preventing database corruption or failure. Developing automated tests to prevent corrupt data from being entered or accepted in an application goes a long way to improving application quality. Data testing is typically straightforward and does well in repeatable automated scripts.

Finally, cross-browser and cross-platform testing is made more efficient when tested through automation. Manually testing in different browsers alone is a resource-intensive test that consumes a great deal of time. Most cross-browser tests can be developed once and executed on multiple browsers. The same holds true for cross-platform testing. Make platform compatibility testing simpler and faster with test automation.

Finding the Right Balance is Key

Automated test development offers distinct advantages in the form of reduced testing time and increased repeatability. QA testing teams may use both manual and automated tests based on the type and complexity of the test. Successful test automation does require planning and a well-designed test approach to balance cost, time and software quality.

Many organizations are tempted to simply purchase a tool, assign a resource and start automating everything. Automated test development is more beneficial and successful when it is planned and organized. Having a well-designed and defined automated test strategy reduces the risk of automation failure. If you’re going to invest the time and budget into test automation, do it right and plan first. Find the balance between cost, time and software quality to move the application forward at the highest quality possible.