Data-Driven Testing

Data-driven testing will help improve the maintainability of your tests by reusing logic and reducing repeated tests—all while providing the same test coverage. But it’s important to make sure we don’t overcomplicate them or over-test.

What Is Data-Driven Testing

Let’s take a test case—“a set of preconditions, inputs, actions, expected results and postconditions, developed based on test conditions” (ISTQB Glossary). Typically a test case will take the form of a list of instructions containing the information required to check that an application, or part of an application, works as expected. A test case can be run manually or it can be automated. To ensure adequate test coverage of the application, there usually need to be several test cases.

Data-driven testing is a method that allows tests to be run using a table of conditions. Instead of having a large number of individual test cases, we can run individual test cases multiple times using different input and expected output parameters.

Setting up a data-driven test is actually quite simple. Let’s take the following test case:

GIVEN I have a calculator
WHEN I press “1”
AND I press “+”
AND I press “2”
THEN I should see the answer “3”

Instead of having the inputs and outputs hardcoded, you can set them up to allow alternative parameters to be used in the test instead. This test case could be changed to this:

GIVEN I have a calculator
WHEN I press <input1>
AND I press <input2>
AND I press <input3>
THEN I should see the answer <output1>

The alternative parameter values are stored in a data file containing a table of values, each line being an individual test iteration. The table might look something like this:

Common Mistakes with Data-Driven Testing

There are many benefits of data-driven testing. It reduces the number of repeated steps and allows existing logic to be reused. This reduces the amount of maintenance required as it reduces the number of test cases that need updating. It also allows the tester to increase test coverage by simply adding a new row to the data table.

However, it can be easy to get a little carried away when setting up the data-driven tests.

Too Few Test Cases, Too Many Parameters

It may be tempting to reduce the number of test cases and have just a few data-driven tests instead. We’d only have to focus on a single data table, and additional iterations can be created by adding just another line. However, too many parameters can make keeping track of the data challenging. Also, if executing the test manually, it can actually be a bit of a headache to run.

Let’s say we are testing a page with four checkboxes and four text boxes which require an integer value between 1 and 100. A test case might look like this:

GIVEN I am on the application form page
WHEN I set the value of checkbox1 to <checkbox1value>
AND I set the value of checkbox2 to <checkbox2value>
AND I set the value of checkbox3 to <checkbox3value>
AND I set the value of checkbox4 to <checkbox4value>
AND I set the value of textbox1 to <textbox1value>
AND I set the value of textbox1 to <textbox2value>
AND I set the value of textbox1 to <textbox3value>
AND I set the value of textbox1 to <textbox4value>
AND I click submit
THEN the form should be successfully submitted

A data file for the above test might look something like this:

This is a lot of parameters to keep track of. It is often advised to keep the tests as simple as possible to reduce the risk of mistakes being made when setting up the data. Simple tests are easier to maintain than more complex ones. Breaking the data-driven test down to two separate test cases with four parameters, or even four test cases with two parameters, will reduce the complexity and improve the maintainability of the data file.

Over-Testing

Just as it’s tempting to add too many parameters to a test case, it can also be tempting to add too many iterations. It takes time to run a test, and each additional iteration adds to that time. To run a test, a test environment is needed. Only one test can be run in that environment at a time, which could be used to run alternative tests. When increasing the number of iterations, have a think: Is this many iterations really necessary?

Let’s say we have a text box that should only allow a number between 1 and 60 to be entered. Since it’s so easy to set up a data-driven test, it might be tempting to create a data table that covers every single number between 1 and 60. Is this really necessary? If each iteration takes 30 seconds, the entire test would take 30 minutes total to run. That is 30 minutes where the test environment would be unavailable for alternative tests to be run.

A better way would be to use boundary value analysis and only cover the boundary values and maybe a few invalid and mid-range values (0, 1, 60, 61, 24, 47, -3, “string”, 3.234). If there was a concern about this particular feature, then you might increase the number of iterations just to be safe—all you have to do is add a few more rows in.

Summary

Data-driven tests are great for improving maintainability by reducing the number of repeated steps and reusing logic. They allow us to reduce the number of test cases without impacting the overall test coverage. However, it is easy to get carried away. Care must be taken to design the tests so we don’t overcomplicate them by introducing too many parameters in the data table, and we don’t take up valuable time and resources by increasing the number of iterations too much.

Editor’s Note: Progress Telerik offers an end-to-end testing tool in Test Studio. It comes with a 30-day free trial so you can explore all Test Studio has to offer.