Automated Tests With Feature Toggles

Feature toggles are helpful when developing new features and releasing them to production—you can test out new features in production but hide them from end users. How do they fit into your automated testing plan?

Feature toggles (also known as switches or flags) are a useful method that allows features to be activated or deactivated without having to change the code. To enable or disable the feature, there will usually be a setting in a config file which allows the feature to be switched on or off at a moment’s notice. It allows the development team to test the feature before it is formally released.

An alternative to feature toggles would be to have two separate code branches. This reduces complexity, but can cause issues when merging the two branches together. By including both sets of code in a single branch, we significantly reduce the risk of major merge conflicts.

Another advantage of feature toggles is that it allows the feature to only be made available for certain users. For example, the feature may only be switched on for users who belong to a particular channel or user group. This allows the business to opt for a phased release where they can assess the user behavior before deciding if the feature can be released to more users.

How Do Feature Toggles Work?

The “switch” itself may exist within a config file or a database. In some cases, an additional UI application exists with a list of features and a “switch” next to them. If the switch is in the “on” position, then the feature will be available within the target application.

The switch can be used to hide or show a new feature, or replace an existing feature with a new version.

Feature Toggles That Enable a New Feature

When a feature toggle is enabling a new feature, the code might look something like this:

if (enableNewFeature)  
{  
  newFeatureCode()  
}

If the switch for the new feature is “on,” then enableNewFeature will be set to true. The function newFeatureCode() will be executed, and the new feature will be available to the user.

If the feature switch is “off,” then enableNewFeature is false and newFeatureCode() is not executed. The feature is not made available to the user.

Feature Toggles That Replace a Feature With a New Feature

When a feature toggle is replacing an existing feature with a new feature, then the code might look something like this:

if (enableNewFeature)  
{  
  newFeatureCode()  
}else  
{  
  oldFeatureCode()  
}

If the switch for the new feature is “on,” then the function newFeatureCode() is executed and the new feature is made available to the user.

If the switch for the new feature is “off,” then the function oldFeatureCode() is executed and the original feature is made available to the user instead.

What This Means for Automated Tests

Unfortunately, the inclusion of feature toggles can cause problems for automated tests.

Developing tests for a feature that could be switched on or off at a moment’s notice is challenging, as the tests will fail if the feature toggle is not in the correct state.

Tests for New Features Controlled by Feature Toggles

A new feature will require new tests—however, these tests will fail if the feature is currently disabled. A simple way around this is to remove these tests from the main test run until the feature has been formally released. Until then, only run the tests when required and if the feature has been enabled. Once the feature has been formally released, the tests can be added to the main test run.

If desired, you could include a test that makes sure the feature is not available when disabled. However, this could be seen as excessive when it’s known that the test will need deleting once the feature is released.

Tests for Features Replacing Existing Features

Automated tests for these features will be a little more complicated and dependent on several factors, including if there are existing tests for the original feature.

If no automated tests exist (which is common for older applications with features implemented before an automation strategy was developed), then you would treat the feature as if it were a completely new one. Simply create new tests that cover the new version of the feature, only run them when the feature is enabled, and only include them in the main test run once the feature has been fully released. There is no point in creating tests for the original version of the feature as this now has a limited shelf life.

If automated tests already exist for the original version of the feature, then these tests will need updating as they will fail once the new feature is released and the original feature is disabled. There are two options for handling this situation:

1. Create a set of new tests, run the old tests when the feature is disabled, and run the new tests when the feature is enabled.
2. Introduce another feature toggle in a config file which sets if the new checkout is enabled or not, and introduce if statements to the sections of code where new code is required to cover the new version of the checkout.

Here are the pros and cons of these options:

Developing two sets of tests for old and new features:

+ Less code complexity as new and old tests will be separate.
+ Easier to delete old tests once the new feature is released and feature toggle is no longer required.
- Test maintenance will increase as both the new and old tests will require maintenance.
- If there are lots of tests to run, picking the correct tests to run might be confusing.

Introducing a test feature toggle so existing tests contain steps for both the new and old feature:

+ Only one set of tests will need to be maintained. This will be especially beneficial if there are some shared steps for both the old and new features.
+ Since the test steps are controlled by a single setting in the config file, there is no need to make any changes to the actual test run. All the tester needs to do is make sure the test config setting matches the feature toggle setting.
- Introductions of if statements into the test code will increase code complexity as each test will cover two different features.
- Once the new feature has been released and the feature toggle is no longer needed, all the code covering the old feature will need to be removed. This will need to be done so that the new test code is not accidentally impacted by the removal of the code.

Avoid Toggle Debt

One common mistake made with feature toggles is that development teams forget to remove them once the new feature has been launched. The issue is that businesses like to introduce a cooling-off period where they make sure the new feature is working and fits its purpose. Having the option to quickly remove a feature if it is not performing the way it should can be reassuring.

Unfortunately, by the time it is decided that a feature toggle is no longer needed, the development team is likely to have moved on to a new project. As a result, removing the old feature toggles becomes less of a priority. This is assuming a decision to remove the toggle is made. Sometimes, the toggle is forgotten about and remains within the application until the application is no longer required. The issue here is that this can result in a large amount of redundant code being present within the software.

If more feature toggles are introduced to the application, then keeping track of which toggles exist and which ones are enabled can become a real challenge. This also makes testing more complicated as testers may find themselves having to check not only if a feature toggle works, but also how multiple feature toggle settings work together (e.g., Feature A is enabled, B is disabled, C is disabled, etc.).

Keeping track of which feature toggles currently exist, and being very strict when making sure that the feature toggles and all related code are removed once the toggle is no longer needed are essential. Removing feature toggles can create additional problems, and it is important to not only remove them but also check that removing the toggle has not impacted any existing features.

Summary

Feature toggles are a useful way of developing new features and releasing them to production, while providing the option to switch them off as needed. This provides the opportunity to test the new feature in production but keeping it hidden from the end users.

Automated tests that cover the new feature should only be run when the new feature has been enabled. Existing tests that cover an older version of the feature can be either replaced with new tests or adapted so they can be run against the new feature.

Once the feature toggle is no longer required, then the feature toggle and any code related to the feature toggle should be permanently removed. To avoid a situation where there are a large number of feature toggles that are difficult to keep track of, it is essential that feature toggles are removed once the feature has been tested, released and confirmed as being fit for purpose.

Further Reading

An eye-opening article that highlights the problems feature toggles can bring in more detail:

• https://dzone.com/articles/feature-toggles-are-one-worst

A couple of articles that provide details on variations of feature toggles, circuit breakers and the science experiment model:

• https://martinfowler.com/bliki/CircuitBreaker.html
• https://www.ministryoftesting.com/dojo/lessons/testing-in-production-the-mad-science-way?s_id=27125