Over a million developers have joined DZone.

The Anatomy of an Experimental Organization

DZone's Guide to

The Anatomy of an Experimental Organization

Experimentation is a key facet of achieving not only continuous delivery, but tangible business results and new ways to operate effectively.

· DevOps Zone
Free Resource

Download “The DevOps Journey - From Waterfall to Continuous Delivery” to learn learn about the importance of integrating automated testing into the DevOps workflow, brought to you in partnership with Sauce Labs.

I am a software developer. I see the world from that perspective. In reality though that is only one viewpoint. While it is important that we are effective at delivering software, what really matters is that we are effective at delivering business value.

When I describe Continuous Delivery to people I generally spend a fair amount of time impressing on them that it is not about tools and technicalities. It is not even about the relationship between developers and operations or product owners and testers. Continuous Delivery is about minimizing the gap between having an idea and getting that idea, in the form of working software, into the hands of users and seeing what they make of it. This vital feedback loop is at the core of not just good development, but of good business too.

I have been lucky to work in, or with, several companies that I would describe as Agile companies, not just companies practicing Agile development. These organizations are fundamentally different in approach in almost everything that they do. The principal characteristic of this kind of organization is that they are experimental in their approach to everything. For some more traditional organizations this sounds scary. “Experimenting, that sounds like you don’t know what you are doing”. Well, yes, we don’t know what we are doing, none of us really know!


To quote one of my heroes, Richard Feynman:

It doesn’t matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence – then it is still a guess!

Research into the performance of successful organizations says that 2/3rd of their ideas that are implemented in software produce zero or negative value:

“Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve”

“Avinash Kaushik wrote in his Experimentation and Testing primer that “80% of the time you/we are wrong about what a customer wants”

“Mike Moran wrote that Netflix considers 90% of what they try to be wrong”

(Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, Nils Pohlmann 2013).


Two Thirds, or More, of the Ideas of GOOD Companies Are Waste

Further, the experience of these companies, and the data, says that nobody can tell which third of ideas are the good ones before they are delivered. Clearly there is a place for guessing and predicting customer demand for innovative ideas, but it is important to remember that for every iPhone that you invent, you will have to go through three or four Newtons first.

If we are generally so bad at guessing, then the only sane strategy is to embrace the uncertainty. Optimize to have lots of ideas and to evaluate them quickly so that you can discard the poor ones as fast, and cheaply, as possible. This is what really effective organizations do. Watch Henrik Kniberg’s wonderful descriptions of the Spotify Culture. (See how Netflix works).

These companies know that their ability to guess is poor, because they are becoming more scientific in their approach. Instead of guessing what their customers want, and then assuming that their customers like what they are getting, they are measuring. These companies are designing experiments. They define metrics, which will identify what their customers think, and then carry out the experiment and reflect on the results in order to learn and improve.

So how do these experimental companies differ from others?

Value Innovation Over Prediction

Innovation is the thing that differentiates the great companies from the rest. Great companies create new markets, provide new products or services that change how people do things.

More traditional companies value predictability. The trouble is that you can’t be both predictable and innovative at the same time. They are different ends of a spectrum. The only really predictable thing is status-quo. So if you want your company to be great, you need to value innovation and discard your reliance on predictability – at least for some of your products. One aspect of this is launch-dates. Apple don’t pre-announce their products, they are very secretive. Only when the products are ready do they announce. This allows them to strive for “Insanely great” as Steve Jobs so memorably put it.

State Your Hypothesis

This shared vision is kind of fractal. It operates at all levels in effective organisztions. Being hypothesis driven helps to establish this clarity, this shared purpose. Useful hypotheses can range from, stating what you expect the outcome of a test to be before you execute it, to stating how you think your global business strategy will work out. Some people even talk about “Hypothesis Driven Development.”

Seek the Right Experiment

As soon as you have a new idea. Whatever its nature, the next question should be “How can we test this idea?”. If it is an idea about how to improve your process, figure out an experiment to see if it works: “Ok, let’s try doing without the estimation meeting for a while and see if we save time”. If it is an idea about improving your product figure out an experiment to test that too: “Let’s release this feature and A/B test it against the existing service to see which one makes more money”.

Establish Feedback Loops

Experimentation means nothing without the closure of the feedback loop. After each experiment there should be a way to evaluate the results and figure out what to do next. The outcome from each experiment should be a real action of some kind. Depending on the results of the experiment you should either: Drop the change; Adopt the change; Run through the cycle again with a new experiment to learn more.

The establishment of effective feedback loops is a vital attribute of experimental organizations. Continuous Integration, Continuous Delivery, Experiments in Production, Test Driven Development, Retrospective meetings and Incident Reviews are all mechanisms for establishing effective feedback.

Assume the Fallibility of Experts

All of the really high-performance organizations that I have seen are notable in their attempts to minimize hierarchy. Decisions based on the highest paid person in the room are always only guesses! Seek evidence, allow individuals and teams the freedom to experiment and make decisions based on what they find. Treat people’s opinions with respect, but recognize that whomever it is that holds them, they are still only opinions. Test the hypothesis!

Question Everything

It is not just important to remember that experts are fallible, but there should also be no sacred cows. Everything should be up for evaluation, and if it is found wanting should be amenable to change. Technology, practice, process, office-space, team organization, even personnel.

If it is not working, try something else.

Eliminate Waste

Successful organizations avoid doing unnecessary work. The Lean mantra of eliminating waste is a powerful one. We should be rational and objective in assessing everything that we do in the light of our real goals and finding how to make our work more efficient.

The most common response that I get when coaching teams how to do better is: “We don’t have time to improve”. Investing in making your team and their working practices more efficient is almost never a cost!

Leadership is Different to Management

Spotify talk of having “Loosely-coupled, tightly aligned teams” as being one of their goals. They aim for high autonomy and high alignment. Dan Pink talks of the importance of “Autonomy, Mastery and Purpose” in motivating people towards excellence. These ideas require leadership, not management. The goal of leadership should be to establish a common vision, a shared purpose for the organization without telling people how to achieve it. Ideally leadership is about inspiring people.

My experience has been that experimental organizations are generally much more successful than their more traditional counterparts. They also tend to appeal to the most talented people, because of the creative freedom that they offer.

People sometimes ask me “how do you know when your organization is working well?”. I think that you know things are on the right track when your first response to any challenge or question is: “How could we try that out?” rather than “I think this is the answer”.

Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure, brought to you in partnership with Sauce Labs

devops ,management ,leadership

Published at DZone with permission of Dave Farley, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.


Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.


{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}