Pushing the envelope, and thinking outside of the box by navigating uncharted waters is how the human species innovates. Undoubtedly some attempts to innovate have less than stellar results. Some attempts fail in the most spectacular ways (think the Hindenburg). Others simply sputter out (but still fail)... But every once in a while an attempt succeeds and innovation inches forward.
One of the more recent examples is the notorious SpaceX rocket landing attempts. Conducted in June of 2015, the first attempts at landing a SpaceX rocket failed spectacularly. With a determination to succeed and willingness to innovate the SpaceX team would eventually pave the way for a monumental success. In December of 2015 SpaceX safely landed a rocket on a landing pad, thus creating the first reusable rocket in history. As the old saying goes “without disappointment we cannot appreciate victory.”
The victory of success continues to encourage us to try. Failure for humanity doesn’t seem to deter us; instead it inspires us. Inspires us to try harder, to push further and never give up. Humanities insatiable instinct to explore and innovate fuels the will to continue, despite our propensity for failure and the alluring ease to give up. Etsy’s Ian Malpass outlined a blueprint for failure quite adequately in his 2015 Orielly Velocity conference speech.
"While failure is inevitable, costly failure is not. How can we fail privately instead of publicly, how can we fail strategically instead of clumsily? How we embrace failure is everything."
Software engineering organizations and businesses each approach innovation differently. One commonality within the technology ecosphere is that innovation involves originating a good idea and iterating on it. When an organization observes a fault, some create barriers, instill fear and add process; while others remove boundaries alleviate fear and embrace failure as a learning experience. As the software landscape becomes increasingly competitive, engineering teams are under immense pressure to develop and deliver high-quality implementations on time and under budget. So how then can we innovate and learn from failure or experiment and conduct research under such tight constraints? Enter A/B split testing, or A/B testing for short.
A/B testing is a software approach in which two viable options are provided and the feedback metrics determine the path to continue on. In a practical sense it means creating features and software implementations as hypothesis and allowing the business and its customers to determine where to apply further engineering resources. The definitive approach is based on metrics and data points collected and not intuition and whim. This means by nature that at least one of the choices provided in an A/B testing scenario will inherently fail. When this occurs the non-performing option is deprecated and the failure is hidden. In a sense we are failing strategically and not publicly.
To better understand this novel concept let’s take a step back and rethink how we innovate in the software industry. In an A/B split testing world we implement ideas and changes as experiments. Metrics are then collected to determine if an experiment worked as expected (provided value) or does not. Users are presented with either option A or option B and they decide, which option is better. The winning experiments are further iterated on and improved, and the failing one is deprecated. A/B testing embraces failure, mitigates risk and provides insights into customer demand or performance of a proposed change. In effect A/B testing helps businesses determine where to steer product development in the future and where not to.
Getting started with A/B testing isn’t actually all that hard. The most basic approach to A/B testing can be accomplished with a simple IF THEN ELSE statement as the diagram illustrated above. However, there are plenty of mature frameworks, which have been developed and supported by the open source software community.
To date numerous A/B split testing frameworks are freely available. Some of the more popular ones are listed below.
Six pack - http://sixpack.seatgeek.com
Ruby on Rails:
PHPABTest - http://phpabtest.com
easyAB - http://romainstrock.com/easyAB/
Scenario.js - http://makerstudios.github.io/Scenario.js/
The idea of A/B split testing provides numerous benefits and allows engineering organizations to experiment without an immediate worry of failure. When a feature or proposed enhancement does not meet the criteria needed for success it can be simply disabled. This provides room to increase delivery velocity and reduce the size of releases. The possibilities with this type of solution are virtually endless.