Failure has seldom been sexier, with advocates believing that if you're not failing regularly, you're not pushing the boundaries far enough. Such cheerleaders often evoke the spirit of Edison, who famously remarked that his thousands of failed experiments were a necessary precursor to the invention of the lightbulb.

Edison's notorious example merely serves to illustrate the importance of learning from each dead end so that you can be more successful next time. To take such constructive feedback from failure, it's vital that we understand the essence of what our failures represent.

How Failure and Experimentation Lead to Success

Researchers from Northwestern University have attempted to shed some light on just what failure often entails in a recently published paper where three distinct datasets were examined.

The researchers began by analyzing data on nearly 800,000 research proposals submitted to the National Institutes of Health between 1985 and 2015. The proposals were submitted by nearly 140,000 researchers, and each proposal contains both the nature of the proposal and whether it was successfully backed or not.

The researchers then completed a second analysis of the VentureXpert database, which contains records of investments made in startup companies. They analyzed over 58,000 companies that had been backed by venture capitalists between 1970 and 2017. Success in this instance was when the startup secured an IPO or acquisition within five years of starting.

The third and final dataset was the Global Terrorism Database, which logs instances of terrorist incidents around the world. The team analyzed over 170,000 such incidents from over 3,000 organizations between 1970 and 2017. In this example, success was regarded as an attack that claimed at least a single life.

The road to success

This very diverse array of examples provides an interesting insight into the very nature of success and failure as researchers, entrepreneurs, and terrorists attempt multiple strategies to achieve success in their chosen endeavor.

The researchers hone in on two factors they believe are key to both success and failure. The first of these is chance, as luck inevitably plays a vital role, whilst the second is our ability to learn from each event.

When exploring the role luck plays, they construct a model that allows them to test whether success is largely a random occurrence that will inevitably come your way if you persist long enough. With this hypothesis, the more attempts one has, the more likely they are to eventually succeed.

Perhaps not that surprisingly, this didn't turn out to be the case, and there wasn't any clear indication that luck was the sole determinant of success. In large part, this was because when each attempt was analyzed, later efforts were significantly better than earlier ones.

The ability to learn

This highlights the crucial role learning plays in our ability to successfully digest the lessons that failures can provide so that we improve with each attempt. Indeed, if learning is a key determinant, the number of attempts we need before we succeed should result in a narrower distribution of failure streaks than proposed in the chance model.

Interestingly however, this didn't appear to be the case either, with an even fatter-tailed distribution emerging. It seems that neither chance or learning are enough to explain the patterns that underpin failure rates.

To further explore this peculiar finding, the researchers attempted to model just how people appeared to learn from each failure. Were people digesting all previous failures or merely some of them? This data was fed into a model to illustrate all ends of the spectrum, from those who digest all past experiences to those who digest none of them.

When the model was run, it did appear to accurately reflect what the data was showing, with the ability to learn from past experiences crucial to subsequent success. Indeed, when teams were incapable of learning from the past, their subsequent efforts actually got worse. When learning was above a certain threshold however, each future iteration was better than the last until success was eventually achieved.

So it seems clear. Unless you're able to learn from each failure and ensure that subsequent attempts are better than the last, you're very unlikely to succeed in future.

Further reading: 

If You're Not Experimenting in Production, You're Falling Behind

Failures Are Really Mini Successes