DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • The Role of AI and Programming in the Gaming Industry: A Look Beyond the Tables
  • VPN Architecture for Internal Networks
  • Micro Frontends on Monorepo With Remote State Management
  • Five Java Books Beginners and Professionals Should Read

Trending

  • The Role of AI and Programming in the Gaming Industry: A Look Beyond the Tables
  • VPN Architecture for Internal Networks
  • Micro Frontends on Monorepo With Remote State Management
  • Five Java Books Beginners and Professionals Should Read
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Deployment
  4. No Need to Support Experiments While You Sleep

No Need to Support Experiments While You Sleep

Fredric Paul user avatar by
Fredric Paul
·
Apr. 16, 14 · Interview
Like (0)
Save
Tweet
Share
1.35K Views

Join the DZone community and get the full member experience.

Join For Free

This post was written by David Celis for the New Relic blog.


At New Relic, we didn’t want experimental features waking us up in the middle of the night. Going straight from testing in a staging environment to a production environment is a bigger step than your feature might be ready for, especially if you’.d rather not be getting paged in the middle of the night. To that end, we created a feature flag to make sure we were only testing in production when we could reasonably support our experimental features. Here’s how we got there, and why it will save your precious Z’s.

You need to test in production

So you have an awesome, new — albeit very experimental — feature that you’d like to test out. You’ve written automated tests, so you’re fairly confident, but you need to test your feature in a production setting to build real confidence. You create a feature toggle that’s read from the database per-request. You can just flip a switch, turn it on, and watch your traffic. See any errors? Flip it right back off, fix the problem, and re-enable it. Repeat ad infinitum.

However, going from automated testing to production (even with testing in a staging environment) can be a big leap with experimental features. What if, for whatever reason, errors don’t start coming in until a while after you’ve left the feature on? Nobody wants to get woken up by alerts at three in the morning to disable the feature if it does break, but, of course, this is simply the nature of running a SaaS application. Still … we wanted to attempt to narrow the gap between our staging environment and our production environment and reduce the probability that we would get called off hours for experiments.

Take authentication, for instance; it’s one of the most critical parts of a web application. If users can’t log into your application, it may as well be down. Last year at New Relic, we began work on our own single sign-on (SSO) solution to provide a more unified login experience for our users who will be on both RPM and our upcoming Software Analytics project. We were confident in launching our SSO feature, but we wanted to be available to react quickly if any issues occurred, and we believed that it might take a while for potential issues to surface.

Just turn it off at night

Rather than get notified of issues in the middle of the night when alertness is questionable, we thought, why don’t we just disable the service when we go home for the night? This ended up being a great idea for us; if something with the new SSO solution were to break, we would already be at work, awake and alert.

We could react very quickly and with our heads in the game.

We could disable the feature flag ourselves before going home for the day and flip it back on in the morning, but we’re programmers, and we automate things. Instead, we created a simple feature flag that would restrict access to the feature to between the hours of 10:00AM to 4:00PM Pacific Time on weekdays. Here’s a basic example in Ruby (plus ActiveSupport):

require 'active_support/core_ext/time'
 
class BusinessHoursFeature
# This feature is only available between the hours of 10am and 4pm
def enabled?
Time.use_zone('Pacific Time (US & Canada)') do
now = Time.zone.now
am, pm = Time.zone.parse('10:00'), Time.zone.parse('16:00')
weekday = !(now.saturday? || now.sunday?)
 
now.between?(am, pm) && weekday
end
end
end
 
module Features
class NewRelicSingleSignOn < BusinessHoursFeature; end
end
 
Features::NewRelicSingleSignOn.new.enabled?

We decided to set the feature to enable itself at 10:00AM to give ourselves time to settle in at the office or arrive a bit late. Disabling the feature at 4:00PM would give us time to address any possible issues that could arise from falling back to our old authentication method. This ended up being an additional benefit of this feature flag: gaining confidence in deploying and rolling back the feature. In our case, we discovered added complexity due to the change of state resulting from the aforementioned fallback. If your feature involves complicated state changes, make sure you test turning your feature off and on again!

How it paid off

Thanks to this business hours feature flag, we ended up greatly improving our supportability of the new SSO solution. Whereas we would have typically gone directly from a staging environment to our production environment, adding an interim step of production during the daytime gave us ample time to safely learn how to better support our new authentication method. We were able to determine that we had adequate reporting, and we became well practiced in quickly triaging and fixing issues. When problems arose (and they did), we were able to react quickly and not need to be on-call for an experimental feature. And we’ve never slept better.

And you?

Have you come up with other sorts of cool feature flags? Let me know over here in our Community Forum.

feature flag Production (computer science)

Published at DZone with permission of Fredric Paul, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • The Role of AI and Programming in the Gaming Industry: A Look Beyond the Tables
  • VPN Architecture for Internal Networks
  • Micro Frontends on Monorepo With Remote State Management
  • Five Java Books Beginners and Professionals Should Read

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: