Reflections on A Major Feature Launch

In November 2018, we launched CircleCI orbs. I recently wrote about the design decisions we made while building a new package manager. Here, I want to share some of the lessons we learned (or re-learned) launching a major new feature. You likely won’t be surprised that we think a lot about how to ship working software, and our retro process helps us put in words what we observe and learn along the way.

Committing Dedicated but Contained Time to An Internal Proof-Of-Concept Was Invaluable

An internal deadline is a great motivator to stay focused on having a working prototype. In our case, it also gave us a clear but fairly low bar of “good enough” to test the concept. Coming out of the first major migration wave after the launch of CircleCI 2.0, we had a renewed interest in addressing the difficulties of bootstrapping and managing our new configuration format. In March 2018 (8 months prior to the marketing launch of orbs), we started the preliminary design work on orbs.

Around that time we were also putting together the agenda for a major get-together of our entire engineering organization (50+ engineers coming from no fewer than 25 different locations across 11 states and 10 countries). That meeting set up the perfect opportunity: not just many different kinds of engineers solving myriad problems as a captive, invested audience, but all of them are users of our platform in the projects they have to ship. Having a short deadline for a test like that is easier than a short deadline to deliver production software because we didn’t make design compromises, opting instead to compromise some of the end-to-end features and polish in order to test our riskiest assumptions. We invested a month's worth of effort from our (then newly-formed) developer experience team, creating enough working software to test the core design concepts of orbs. After a month of investment from 4 engineers, it took a single day to get various feedback on the initial design decisions quickly, allowing us to both validate the strength of the concept but also to set the objectives and priorities to get orbs production-ready.

Having Time to Step Away from Backlog Delivery and Do Open-Ended Design at The Start of The Initiative Was Invaluable

We spent non-trivial cycles doing open-ended design discussion, including many concrete sketches, theoretical explorations, and big-picture concept development. In addition to the feature details, we had to invent an idiom around orb development, including formalizing our vernacular to have productive conversations about DSL design. We had meandering conversations without clear-cut agendas, normally anathema in a business setting. This could easily have seemed like a smell at the time because the team’s velocity fell off precipitously, but those kinds of abstractions and frameworks were essential to doing the engineering work to turn orbs into production-ready software.

“Docs-First Design” Worked Well

You’ve likely heard the stories about Amazon using a model in their early days of writing the press release before rather than after a product initiative is started, as a way to demonstrate the end state. In our case, we did something similar with documentation in our preview docs. Our preview SDK evolved out of at least two other repos, designed as a place to do sketching of concrete code and pseudo-docs that we used internally. We stood it up to be read by potential customers as if it were the docs to a working product before we’d gotten it live. This meant we could get a steady trickle of feedback from a few early adopters about whether we were on the right track. When we went to preview release, our docs were already sufficiently rich that the transition from development to production was relatively “cheap.”

Incremental, Quiet Releases Ahead of A Marketing Launch Were Critical

Handfuls of users were able to try out various iterations of orbs prior to their official launch. We were feature complete and quietly inviting our partners and customers to use them out a month before our marketing launch. Leaving time before launch where the team could do several waves of review, bug fixing, tech debt, and coordination with other teams proved to be essential to a fairly stable launch.

The value of having a small yet active and vocal user base can really not be overstated. We were fortunate to be able to engage with some of our customers and not have to keep things secret. We were able to limit the scope of risk we took at each stage, from validating the concept to specific syntax decisions. We were able to wring out a few critical operational and logic bugs before going to wide release. We were able to instrument our monitoring systems and set up our operations runbook under controlled but real-world usage patterns before experiencing a spike in traffic after our marketing launch.

Some Ideas Take a Long Time to Become a Reality

The idea for orbs took a long time to percolate before ultimately getting the green light. The original conception of reusable config components dates back years and can be traced in conceptual lineage to design work our founder, Paul Biggar, did in 2015 (over 3 years before orbs launched). The concept evolved in fits and starts before it received earnest attention from a product engineering team. Even after it was first proposed, it took several months to get everyone comfortable with the concept. In the end, this nearly-forgotten idea became our “next up” initiative because the stars aligned around the timing of our software maturity, our medium-term business needs, and various customer requests.

Cross-Org Coordination Work Should Start Early

There are many pieces of a successful launch that come from departments outside engineering all of which require non-trivial lead time. Our general practice is to have a basic briefing with our SRE, security, documentation, marketing, and solutions departments at the beginning rather than the end of an initiative. For this launch specifically, it also included legal help with our new Code Sharing Terms of Service, security review of the orb publishing/importing model, documentation, wiring up analytics, coordinating with marketing, and establishing a production-ready operating posture were all pieces of the launch plan that had to be considered. It tooks many elapsed weeks for each of these cross-team processes to play out, so it was essential to plan well in advance if we had any hope of hitting our dates.

At CircleCI we get to think a lot about the practices of software teams, and at the same time we’re constantly managing our own growing team and ambitions. While the retro document the above was drawn from was much longer, I’ve shared the parts containing general lessons I hope can be valuable to your team.

We love to hear about practices and techniques that teams rely on to continuously ship quality software products, and we’d love to hear from you! Tweet us @circleci with your stories and lessons.