Dependent Feature Flags

Recently I was given the task of refactoring and reimplementing the service, which determines the users' authorizations within our SaaS application.

The major challenge was to do the work without anyone realizing it: the charade is up once errors start popping and paying customers have difficulties doing their job. Worst possible end-case: users are completely blocked from accessing the app, and customers flee to your competitors. No pressure, right?

I quickly concluded that this effort required generous use of feature flags and then leaped to dependent feature flags to make the safe, granular release of the new implementation. (and potentially save my job).

Have You Never Heard of Feature Flags?

Using Feature Flags software provides a way to deploy code changes without releasing those changes to users: features added, functionality modified or extended, internal tooling changed, existing functionality deprecated or removed, data handling changed, etc.

Feature flags can be implemented in the front-end or back-end tiers of your application.

Example

Assume that we've deployed a new process for creating application users - what has changed is immaterial. A feature flag determines which implementation of user creation is used.

fun createUser (userInfo: CreateUserRequest) : UserDTO {
    if (checkFeatureFlag("my-new-user-creation")) {
        return createUserV2(userInfo);
    } else {
        return createUserV1(userInfo);
    }
}

The new implementation is hidden behind the feature flag, and the V2 method is not called until the feature flag is enabled. Once enabled, the V2 method is called, and if everything goes well, the V1-related code is removed in future pull requests.

Why Feature Flags?

While conceptually similar to boolean configuration properties, Feature Flag applications provide capabilities beyond flipping a flag, such as:

• Centralized Management: single-pane-of-glass for flags in all environments, tags for grouping flags, etc.;
• Context-Specific: the flag returns true or false based on criteria, such as a current user or application-specific values;
• A/B Testing: slow rollout by declaring the percentage of flag checks which return true to ensure application behavior is as expected before always returning true;
• Immediate: no need to restart the app, reload the configuration, or revert to the previous version;
• Reporting: what flags are used, breakdown of true/false, etc.

To anyone who will listen: Feature Flag the f*ck out of everything! More than once, my team has benefited when the unexpected occurs, disabling the feature flag to revert the change and then RCA without leadership panicking.

So, What Are Dependent Feature Flags?

Essentially a multipart conditional, where the true/false value for a feature flag also depends on another feature flag value (true or false, depends on your requirements).

Let's expand the example above by adding the requirement that the new authentication flow must be enabled before we can create users differently.   

fun createUser (userInfo: CreateUserRequest) : UserDTO {
    if (checkFeatureFlag("my-new-authentication") &&
       (checkFeatureFlag("my-new-user-creation")) {
        return createUserV2(userInfo);
    } else {
        return createUserV1(userInfo);
    }
}

Code maintainability is the problem: any change in dependencies requires code changes.  Originally flag-A is dependent on flag-B, which later is made dependent flag-C.  See the problem?

Feature flag software allows dependencies between flags to be defined in the UI, taking immediate effect without code changes: the code checks for its singular feature flag, while in the background, all flags are evaluated.

Using Launch Darkly, I have created two feature flags: scott-test-flag-1 and scott-test-flag-2, whose prerequisite (dependency) is the first flag validating true.

This screengrab shows the prerequisite defined on scott-test-flag-2.

This screengrab shows how the user is notified that there is a prerequisite defined for scott-test-flag-1.

Back to the Original Problem Statement

I identified the non-functional requirements desired to minimize the chance of breaking application authorization during development:

• Unchanged and usable original implementation available throughout;
• Small, manageable PRs of new code that are continually deployed but not released;
• Gradual and incremental releases of new implementation;
• Comparison testing in non-prod environments without impacting overall user experience.

For my work, the authorizations are generated for four separate contexts, each context a specific implementation.  There are additional subcategories within a context that are not important for this discussion.

Design Feature Flags

The non-functional requirements helped to design a series of feature flags whose dependencies are represented in the following directed graph.

The purpose of each feature flag:

• auth-new-global: the top-level master feature flag, which serves as a precondition for all other flags, turning off (returning false) immediately disables all other flags and reverts to the original implementation.
• auth-new-service: determines whether the service is ready to run new implementation code. Precondition: auth-new-global.
• auth-new-factory: determines whether the new factory is enabled, which allows for both original and new authorization generation to occur. Precondition: auth-new-service.
• auth-new-context[1..4]: determines which implementation for context authorizations is enabled.  Precondition: auth-new-factory.
• auth-new-test: determines whether the runtime comparison of original vs. new authorization generation is enabled. Precondition: auth-new-global.
• auth-new-test-context[1..4]: determines whether a runtime comparison for a specific context is enabled. Preconditions: auth-new-test and NOT auth-new-context[1..4].

For example, generating authorizations for context3 using the new implementation requires the flags auth-new-global, auth-new-service, auth-new-factory, and auth-new-context3 are enabled.

Create Feature Flags

I next created each feature flag using the UI for the feature flag software my organization uses.

It's important to double-check for consistency across environments.  I discovered that my feature flag software does not copy dependencies (prerequisites) across environments, requiring dependencies to be defined individually across the environments.

Implement Feature Flag Stubs

I identified approximately where each feature flag needed to be checked and made code changes, which usually require simple restructuring. The stub for the new implementation throws a 501 Not Implemented exception to make it blatantly obvious is a flag is prematurely enabled.

Original Implementation

fun genContext1Auth (request: AuthRequest) : AuthResponse {
    <generate auth for context 1>
    return response
}

Stubbed-Out Implementation

fun genCtxt1Auth (request: AuthRequest) : AuthResponse {
    if (checkFeatureFlag("auth-new-context1")) {
        return genCtxt1AuthNew(request);
    } else {
        return genCtxt1AuthOrig(request);
    }
}

private fun genCtxt1AuthOrig (request: AuthRequest) : AuthResponse {
    <generate auth for context 1>
    return response
}

private fun genCtxt1AuthNew (request: AuthRequest) : AuthResponse {
    throw NotImplementedException()
}

Implement New Functionality

The meat of the work. Pick a context, implement, test, repeat.

I used Postman to define a catalog of tests that represented the different scenarios and compared results from running against original and new implementations.

Changing which feature flags were enabled at any time was more difficult than I expected, and I often resorted to brute-force code changes to test a path.

Test

The wide variability of requests raised the possibility of developers testing missing scenarios that could identify bugs in the new implementation. The existing automated tests are fairly similar in design and scope.

I implemented a background process for generating authorizations by the new implementation, which are compared with authorizations generated with the original implementation. Differences were logged, analyzed, and, if necessary, changes made.

Fortunately, most differences were minor — i.e., null vs. empty string or different ordering of authorizations — but it did add to the overall comfort level.

Enable Feature Flags

We've coded, we've tested, we've peer-reviewed, we've released in non-prod.  Next up: production.

The first three flags — auth-new-global, auth-new-service and auth-new-factory — are parent dependencies non-events which hopefully won't change the implementation used; nevertheless, each flag was enabled individually to prevent surprises.

The rubber hits the road with the auth-new-context flags. The contexts were enabled from least-impactful to most-impactful, each context enabled individually with a week pause between each to monitor execution, resources, bugs, etc.

Remove Original Implementation

After a few weeks, I was confident that the new implementation was solid and that retaining the original implementation just in case was no longer necessary. The original implementation and the feature-flag checks are removed.

Note: These changes must be promoted to production before the next step; otherwise, you will inadvertently revert to the original implementation!

Archive Feature Flags

I archived the feature flags once they were no longer required (previous step), which also shows kindness towards your Feature Flag administrator and fellow engineers by reducing the number of flags seen when doing their own feature flag work.

Note: Your feature flag software may require additional steps before archiving the flags, such as removing prerequisites or dependencies. For example, Launch Darkly cannot archive until prerequisites are removed in each environment.

Outcome

I kept my job! The functional goals achieved, the non-functional goals of no one noticing achieved, and — surprisingly — no issues logged. A success any way you look at it.

The dependent feature flags because useful for the following:

• Disabling the global flag to use the original implementation while researching a permissions problem;
• Preventing incomplete work from being enabled when someone inadvertently enabled the wrong feature flag.
• Turning off testing when the new implementation was enabled.

The dependent feature flags most provided me a level of comfort, that in a pinch, I could disable the global feature flag and immediately revert to the original implementation without much effort.

Conclusion

Feature flag software is a lot more than simple true/false flagging. It's worth the time and effort to understand your product and take advantage of whatever is offered to make your engineering life even better.