Releasing Faster (and More Frequently) Without Defects: The Automation Game

Increasing release frequency helps companies go to market faster, but it comes with its fair share of challenges. This article investigates what’s changed in the industry and how automation can speed up release cycles and ensure a level playing field for everyone on the team, including testing.

Cutting time to market is a business decision that has its logic lying with competition, customer demand, economic situation, compliance or security requirements—or a combination of all of these. The market dictates the rules, which has a two-fold impact on the software world. There is an opportunity for start-ups to emerge and answer the demand while the pressure on existing market players grows exponentially, by the minute.

When we at Test Studio talk to customers and ask them about the reasons behind the decision to introduce test automation, we get the same answer again and again, almost in every interview: We need to release faster, sooner, and more frequently and meet customer, stakeholder or market expectations.

Increased release cycle frequency, however, increases the potential for defects in production, which is not ideal. With the extremely fast go-to-market cycles and the growing demand for quality, the greatest challenge of software testing is keeping up with the software development pace.

In this article, we will investigate what’s changed in the industry and how automation helps create a level playing field for everyone on the team, including testing.

The 3 Pitfalls of Software Testing

There are several seemingly basic and yet important non-technical aspects of software testing where people fail. Acknowledging them and taking appropriate action is the first step of the journey.

Lack of Testing Strategy and Prioritization

In reality, you never have time to test everything and cover all possible scenarios. With the increased release frequency and shorter release cycles, most teams can barely test the critical if not basic functionality for regressions.

If you want to minimize the number of critical bugs slipping into production, you need to plan very carefully what to test, when to test it, who to task with testing it and what tooling you need to maximize testing and release successfully, as often as you need to.

Failing To Select Appropriate Testing Tools

Many less senior testers and engineers go astray believing that automation can and will solve all their problems. Test automation is a must for most testing projects, but after all it is just one tool. To be successful at testing, teams need to evaluate and pick the right set of tooling, minding their skills and experience along with the specifics of the project or application under test.

Often, they will end up automating only a part of the test scenarios and executing them within their CI/CD pipelines in combination with exploratory tests being done manually. Yes, there is and will always be a need for a healthy proportion of manual testing installed as an additional quality gate, and this is the right way to go forward.

Poor Communication

Whether to understand feature requirements, user scenarios, application specifics, the morphology of bugs, or to simply get help from a developer colleague over some automation issues, communication is a key element for the modern test engineer to succeed.

It seems clear, but we see so many cases of team fragmentation where a simple five-minute conversation could have saved hours of work. Not to mention having in place appropriate processes for monitoring results, sharing feedback, reporting and fixing bugs, which also falls back to communication.

Another interesting aspect here is understanding the importance of testing and allocating enough time to the testing operations. Surprising or not, test engineers we talk to complain that testing is still undervalued and treated as a bottleneck by product owners or higher-level managers. Therefore, properly communicating and showcasing the role of testing is vital for any testing project’s success.

Building Quality Gates in Your Pipelines

Provided that we have ruled out the above discussed pitfalls, it all comes down to the question of how to speed up release cycles and increase release frequency without sacrificing quality (and security).

Focus on Small Releases

Our experience shows that there is no magic formula applicable to all organizations. But if there is one that has proven to work, it is to keep the focus on small releases, supported by frequent, if not continuous builds and integrations.

Splitting the work into bits and shipping these bits in short and even periods of time has its advantages: the work load is manageable, resources are distributed based on priorities, it is much easier to avoid resource density and capacity bursts, while teams get timely feedback as to prevent issues before they occur.

Add Automated Checks to Your Pipelines

The CI (continuous integration) practice of frequently pulling source code from the repository to compile the link steps needed to fully build software has gained a lot of popularity in the industry for a reason—resulting in the evolution of CD (continuous deployment/delivery).

Pushing these frequent builds from CI to appropriate deployable environments where further automated checks can be performed makes sure that code which doesn’t pass any of the various manual or automated quality gates (unit tests, build & integration tests, automated end-to-end or API tests, static code analysis, regression or exploratory tests, etc.) does not move further down the delivery pipeline.

These concepts of CI/CD were created and promoted in the first place to mitigate risks in environments where teams traditionally wouldn’t merge or integrate bits of work for longer periods of time, up to months sometimes.

The idea of small, frequent releases is not new, but the way organizations manage it in practice is evolving. What’s been missing for too long, but slowly gaining traction, is the integration of UI or end-to-end testing into the CI/CD pipeline. And that is the only way to go when it comes to quality and security.

Everyone Has a Seat at the DevOps Table

Integrating builds several times a day on its own is not enough to speed up releases and ensure software quality. Including additional quality gates as part of the automation process is what takes quality assurance to the next level—continuous integration testing—or, for the sake of simplicity, we will call it continuous testing. Receiving feedback earlier in the process helps everyone understand better what’s being built and ensure the team has the appropriate instruments and information to do the job and meet their goals.

Shifting Left To Make Enough Room for Testing

Adopting a “shift left” mindset to bring testing earlier into the delivery cycle helps address problems before they even occur, which is one of the main principles behind DevOps and DevSecOps.

Shifting left implies the continuous effort to replace the traditional practice of testing after completing all development by running a set of tests after each build is integrated. Here we don’t want to limit ourselves to unit or integration tests, which traditionally are part of the development process and a very important component of ensuring the code does what it should.

In end-user-facing applications, though—and not only them—it is extremely important to make sure that critical functionality of the website or application is covered, and regressions don’t occur after a new piece of code is deployed. By employing a build server, users can run static and dynamic tests around the clock to monitor the quality of the build, which often is providing insight into overall release health, not only prior to release but whenever that information is needed and could help improve things.

Testing Tools: The Right Cocktail of Vitamins

Testing tools nowadays offer some sort of integration with popular CI/CD platforms such as Azure DevOps, Jira, Jenkins, Bamboo, TeamCity and others. When it comes to integrating UI test automation into CI/CD, it is important to be able to schedule tests and run them remotely, often as part of nightly builds.

That is the only way to make room for testing as part of the continuous delivery concept and to prevent unnecessary bottlenecks. Here the key is test automation stability. Having flaky tests interrupt the nightly build, when speed is critical, is the last thing a development team would wish for. However, test automation is advancing quite fast in offering a solution—in fact, more than one.

Headless testing is one of DevOps’ best friends. It ensures that rendering a browser’s UI won’t cause the test suite to fail by simply eliminating the need for running the test in a traditional browser. If speed is as critical as it seems, plus one for headless testing, which can speed up test execution three times and beyond.

The Testing Pyramid

If there is one thing this model continues to be absolutely right about, it’s that the closer to the top of the pyramid the delivery process gets, the more critical it is to make sure business transactions happening through the application’s UI work seamlessly.

User interface has a huge impact on how and if your customers will achieve their goals by interacting with your application. Higher in the pyramid means bigger value of the performed testing, but it also implies rising costs, which organizations need to take into consideration and integrate into resource planning, budgeting, etc. Yes, quality is expensive indeed, but allowing defects to slip into production because of cost savings is, for the most part, risky business.

The Role of Automation in Security

When it comes to security, the complexity increases. Security requires having a clear and easily traceable quality assurance process, which for some organizations that operate under strict governmental laws might even imply full compliance and auditability readiness. All CI/CD platforms offer some sort of audit trail features with best-practice security controls of what tasks were executed by whom at what time.

Including information on automated tests that have been integrated into the CI/CD pipeline is usually also supported by some of the proprietary software automation tools, allowing organizations to control and document their process, including security/compliance issues, and have decision-critical information on the health of their products and processes.

Automation offers audit trails, compliance coverage and risk elimination for all tasks and stages involved in building, testing and deploying software.

Summary

It takes more than just good reasoning to integrate UI testing into CI/CD and DevOps. It takes time and dedication. Fortunately, most companies are way beyond the phase of thinking about it. The fact that it has gotten its own discipline called QAOps speaks for itself. With the right set of tools, it is not only possible but practical that QAOps can account for tremendous results, affecting both release speed and software quality, and—ultimately—cut time to market.