An Overview of Automated Testing for Developers

This article provides a high-level overview of automated testing; what it is, how it works, and some things to consider when looking into incorporating it into your development process.

Automated testing is a vital part of the development process for building robust software. Unfortunately, it is sometimes viewed as unnecessary or too cumbersome for developers. Much like flossing and filing your taxes on time, automated testing is an area of a developer's life that should not be avoided - especially given the evidence of its positive affects and outcomes.

What is Automated Testing?

Automated testing (AKA, test automation) is a development process that involves tools to execute predefined tests against software based on an event. These tests typically involve operations that are repetitive in both their nature and outcome.

There are many different ways for developers to test the code they write. Most of the time, these tests are conducted manually. Automated testing differs by having a process, system, or service that conducts these tests in response to developers committing changes to source code. In practice, the impact on the developer's workflow is minimal since the tests are conducted in an asynchronous and autonomous fashion.

The nature of automated testing for developers is clear: we're performing a predefined series of tasks (i.e. tests) against a set of preconditions and postconditions (i.e. assertions) based upon a triggering event (i.e. source code change).

While this description seems simple, the practicality of integrating automated testing into a developer's workflow can be far more nuanced. There are many benefits associated with automated testing, just as there are many associated costs as well.

Practical Automated Testing for Developers

Typically, developers wish to test particular aspects of an application such as its underlying framework and APIs. This is done through unit tests. Unit tests are snippets of code that assert claims about how parts of an application—its libraries, frameworks, APIs, and so—should behave under certain conditions. Unit tests inform the developer when a change in one unit interferes with the functionality of another. Modern unit testing frameworks are typically implemented using the same code used by the system under test.

There is a cost associated with the authoring and ongoing maintenance of unit tests. They can take time to write and require discipline to keep them up-to-date with ongoing changes to the code. However, that cost should be considered an investment since unit tests instill confidence; the code will operate as expected.

Now, I mention all of this because automated testing can leverage unit tests as part of the development process. For example, an IDE can run these tests after a successful build is completed. Or, in a more elaborate scenario, infrastructure can be set up to run unit tests on a schedule (e.g. every night at 11 PM). Whatever configuration is used, there are tangible wins to be gained as a developer through automated testing of this kind. But, like I stated earlier, the "cost" comes in the authoring and maintaining of the unit tests themselves.

Note: This article does not cover related methodologies to unit testing such test-driven development (TDD) and behavior-driven development (BDD).

Another aspect of testing is one that targets the UI. This is similar in some respects to unit tests in the sense that UI testing can be small and focused on particular aspects of an application. However, it differs in that it's running these tests against parts of the UI that are present at runtime. This requires infrastructure to support. It also requires a framework for authoring these tests. These frameworks provide the necessary building blocks for testing UI. However, they are often limited to a similar environment as unit tests.

Some solutions provide a means for recording a user's interaction with the application as a means for authoring these tests. They act as a code generator of tests by watching all of the interactions that a user has with an application and uses that as a basis of tests. Once authored, these tests can be modified to accommodate other requirements.

At the end of the day, developers have a choice when deciding on which testing framework to use. Although it is possible to author and conduct unit tests without such a framework, the process can be difficult, complicated and very manual. There are a lot of unit testing frameworks available. Each of the frameworks has its own merits and selecting one depends on what features are needed and the level of expertise of the development team. These frameworks provide the infrastructure and functions necessary for testing code.

The Benefits and Costs of Automated Testing

There's a strong case to be made for incorporating automated testing into your workflow. The Wikipedia entry on Test Automation elaborates on some of its benefits:

Some software testing tasks, such as extensive low-level interface regression testing, can be laborious and time-consuming to do manually. In addition, a manual approach might not always be effective in finding certain classes of defects. Test automation offers a possibility to perform these types of testing effectively. Once automated tests have been developed, they can be run quickly and repeatedly. Many times, this can be a cost-effective method for regression testing of software products that have a long maintenance life.

The key phrase in this description (above) is "testing effectively." There's nothing worse than a tool that's a burden in its usage or application. Good automated testing solutions are ones that complement workflows; not hinder them. They should improve a developer's productivity; not impair it.

Ultimately, testing is about feedback; its purpose is to provide insight into the robustness of the software solutions we're building. Automated testing provides a different approach to manual testing by ensuring that code is continually measured and validated against our expectations. Unfortunately, it's not a perfect. There are real costs associated with it. Consider the infrastructure that could be necessary to support automated testing. Servers or services may be required to run these tests. These will require maintenance over time. A database may be needed to store test results and report against them.

This is just part of the story when it comes to the costs associated with automated testing. There's the entire matter of managing and maintaining the tests themselves. If you decide to change your software in a significant way, you'll have to update your tests to reflect any new assumptions or expectations as to how it should operate and behave. A good automated testing solution will consider this as a real possibility and provide a way to mitigate against it.

Some Criteria for Automated Testing Solutions

There are many solutions and products in the marketplace that support automated testing. Some integrate directly into development environments with little-to-no supporting infrastructure, while others may require a more elaborate configuration. Some are built to support individual developers, while others are built for scale to support teams. Whatever automated testing solution you choose, it is important to consider the following criteria:

• What kind of application or system are you testing against? For example, is it a desktop application or a website with backend integrated systems? Can your automated testing solution target them?
• Do you have the infrastructure that's required to support automated testing? Some automated testing solutions have significant requirements. (Note: Hosted solutions are available.)
• How are tests authored and integrated into your automated testing solution? Test management and maintenance represents a significant cost associated with automated testing solutions.
• Does your automated testing solution integrate with your existing tools and development environments like Visual Studio?
• What reporting capabilities are available from your automated testing solution? How do you know if your tests have passed or failed? Can you get a historical view of your test runs to observe your progress over time?
• How are tests run through your automated testing solution? Are they triggered based on certain events? Can they be scheduled to run at certain times during the day? Some test runs can take a long time to execute.

Take the time to consider these and other questions. Their answers will help guide you towards a suitable automated testing solution.

Automated Testing: Yes or No?

The evidence for developers to use and support automated testing is clear; it has a dramatic impact on developer productivity when integrated properly into a workflow. While it's true to claim that there are costs associated with automated testing, they are amortized over the lifetime of the software project and can even save you time and money in the long-term.

We've seen the impact of automated testing ourselves. We have built products that have been used by millions of developers for many years. Over that time, we used automated testing for popular products like Kendo UI and Sitefinity with great success. We've also helped developers improve their productivity through automated testing solutions we've built like Telerik Test Studio.

For example, with products like Telerik Test Studio, companies like Insperity have achieved a 30% time savings by eliminating manual testing, reducing the time it takes to do repetitive scenarios, and enabling performance testing using the same tool.

Ultimately, the decision to incorporate automated testing will rest on a detailed analysis of your requirements and development processes that you employ to build software. Take a measured approach when evaluating your options. Deploy them in isolation, use them for a while, and consider the positive (and negative) impacts they have for you and your team.

You can also check out this article covering some of the most popular automated testing tools on the market.