A Deployment Is Not a Release: Control Your Launches With Feature Flags

In the world of software development, the excitement of shipping code to users is often tempered by the inherent risks that come with each release. To mitigate these risks, one approach to choose is to decouple the deployment of code from when the feature goes live. This is achieved through a technique known as feature flagging.

Feature flagging is a powerful tool that allows developers to safely push out new features to customers in a measured and gradual way. These features can be hidden behind if/then expressions in the code and can be made specific to some users or all of them.  In this blog, you’ll learn about what feature flags are and how to use them, and discover the benefits they offer.

Why Decouple Deployment From Release?

The way developers release updates or add features to their web applications has changed over the past decade. It used to be that once, or maybe twice a year, organizations would roll out a new version of their application to all their users at the same time, and hope that all the new features and updates work in the way intended. This is known as a "push-and-pray" deployment because it’s impossible to know how the code will behave once it’s in the production environment.

The problem is that there are usually issues for at least some users. Maybe the updated code clashes with other apps, or with certain user configurations. Perhaps the feature updates aren’t being used as expected and need redesigning. It’s also possible that users in different countries or regions have different expectations about the way a feature is supposed to be—it could be something as simple as the size, color, or positioning of a button.

The way to tackle this issue is to decouple the deployment process from the release process, which means that the new app version is deployed but the new features are deployed iteratively to small numbers of users at a time. This reduces the "blast radius," so if there are any critical issues with the new code, only a select group of users will be affected. The initial users might be part of a beta testing group, or they could be chosen at random, or be selected from a specific region.

Separating Deployment From Release Using Feature Flags

Feature flags, also known as feature toggles, are used to selectively enable or disable features in your application without changing the underlying codebase. This allows you to release new code to your production environment but only reveal specific features to a selected number of users, while the rest of the users continue to use the old functionality.

For example, developers can use feature flags to release a new language in a voice assistant application, but limit access to it to a small subset of users. The developers can then collect feedback, work on calibrating the language model, and gradually release the new feature to the larger user base.

There are two approaches to implementing feature flags: by using a configuration file or through API calls to a centralized feature flag service that provides developers an interface to manage the feature flags.

Configuration Feature Flags

Configuration feature flags facilitate faster propagation of control updates due to low latency, are easier to implement, and permit dynamic updates.

When the code on the server changes, the client downloads the new configuration and uses it to determine which features to show each user. When the server-side configuration is modified, the client-side configuration is loaded and then acted upon accordingly.

API Calls Driven Feature Flags

If feature flags are driven by API calls, this provides greater flexibility and the ability to centrally monitor feature flag usage. When a page containing a new feature is viewed, the API is invoked to determine whether or not the user will see the new feature. This is ideal for testing and experimenting with different features over an extended period of time, or for testing different versions of features to determine which is the most popular.

Something to note here is an increased latency due to multiple API calls and less resilience if the feature flag control plane is not designed with redundancy.

Feature Flag Use Cases

There are numerous applications for feature flags. For example, they may be utilized for experiments in which two variations of the same feature are deployed into production. A typical use case is a retail website where different colors of the "buy" button or the size of the button are compared to determine which version generates the most revenue. 

Similar to release, feature flags may also be used for launch control. Consider the scenario where your team has built a new service and modified existing services for compatibility, but since you are still awaiting the marketing announcement publicly you cannot allow users to interact with the new functionality. At the same time, all the new services you encounter at events such as AWS re:Invent have been deployed weeks or even months in advance. They become accessible in the console on the day they are announced. 

People may increase the rate of logging, or security personnel may alter permissions using a feature flag, in the case of operational changes. 

Canary testing is yet another good illustration. Canary testing entails releasing a new feature to a very small subset of your population and observing how it behaves or how your customers respond to it. The term "canary testing" originates from the use of canaries in coal mines to detect dangerous gases. If the bird perished due to excessive gas levels, it was a signal for the miners to leave the mine.

Getting Started With Feature Flagging

When considering a native service in AWS to adopt a configuration feature flagging strategy, refer to AWS AppConfig, which is ideal for safely distributing your updated features to your chosen users. It is designed to safely distribute and deploy configurations for your applications. While initially launched to support applications running on a virtual machine, it now supports the distribution of configurations to serverless AWS Lambda functions or web applications.

Similarly, for API call-driven feature flagging refer to Amazon CloudWatch Evidently, which is ideal for experimenting with new features and A/B testing. It fits more on the API side and is used when you want to release a new feature to your user population in a very safe and controlled way. AppConfig can progressively distribute the configuration file to your customers and can roll back automatically if something goes wrong during the deployment.

AWS recently launched a hybrid version called Evidently Client-Side Evaluation, which uses the Amazon CloudWatch Evidently API, but calls AWS AppConfig in the background and downloads the latest configuration into cache. This is refreshed when the code changes on the server, which provides developers with the best of both worlds.

Feature Flagging Best Practices

Implementing feature flagging can be a powerful tool for managing the release of new features in your application. However, it's important to follow some best practices to ensure that your use of feature flags is effective and doesn't introduce unnecessary complexity or risk. Here are some best practices to consider:

1. Use feature flags for short-lived experiments: Feature flags are great for testing new features with a subset of users, but they should not be used as a long-term solution for maintaining multiple versions of a feature. Over time, maintaining multiple code paths can lead to code complexity and technical debt.
2. Clean up old flags: Once a feature has been fully rolled out and is no longer being tested, the associated feature flag should be removed. This helps to keep your codebase clean and reduces the risk of accumulating "dead" code.
3. Use a consistent naming convention: It's important to use a consistent naming convention for your feature flags to make it easier to understand what each flag controls. This can also make it easier to find and remove old flags.
4. Limit the scope of flags: Each feature flag should control a single feature or behavior. Avoid using a single flag to control multiple unrelated features.
5. Monitor your flags: Use monitoring and analytics tools to track the usage and performance of your feature flags. This can help you understand how your features are being used and can provide valuable insights for future development.
6. Use a feature flag management system: While it's possible to build your own feature flag system, using a dedicated feature flag management system can provide additional features like flag rollouts, user targeting, and flag lifecycle management.
7. Test your flags: Just like any other part of your code, your feature flags should be tested to ensure they work as expected. This includes testing both the "on" and "off" states of each flag.
8. Document your flags: Keep a record of why each flag was introduced, what it controls, and who has the authority to change it. This can be invaluable for maintaining control over your feature flags as your team and codebase grow.
9. Secure your flags: Feature flags can be powerful tools, but they can also be dangerous if they fall into the wrong hands. Make sure your flags are secure and can't be manipulated by unauthorized users.
10. Use flags for operational switches: Feature flags can also be used as operational switches, allowing you to turn off certain features in response to operational issues or incidents. This can be a powerful tool for maintaining system stability and performance.

Summary

In summary, feature flags are a powerful technique for decoupling functionality from deployment in software development. By enabling developers to release new features incrementally, feature flags can help organizations to be more agile and responsive to customer feedback.  If you want to safely deploy new features and roll out using a configuration file for feature flags, AWS AppConfig is the way to do it. If you want to experiment in the long term with new features, A/B testing, and different variants of the same feature, AWS CloudWatch Evidently is the best choice. It all depends on your use case.

If interested to watch the on-demand session on this topic presented at AWS re:Invent 2022, then click here.