Over a million developers have joined DZone.

The Lindy Effect and Technology Survival Patterns

DZone's Guide to

The Lindy Effect and Technology Survival Patterns

· Big Data Zone
Free Resource

Learn best practices according to DataOps. Download the free O'Reilly eBook on building a modern Big Data platform.

The longer a technology has been around, the longer it’s likely to stay around. This is a consequence of the Lindy effect. Nassim Taleb describes this effect in Antifragile but doesn’t provide much mathematical detail. Here I’ll fill in some detail.

Taleb, following Mandelbrot, says that the lifetimes of intellectual artifacts follow a power law distribution. So assume the survival time of a particular technology is a random variable X with a Pareto distribution. That is, X has a probability density of the form

f(t) = c/tc+1

for t ≥ 1 and for some c > 0. This is called a power law because the density is proportional to a power of t.

If c > 1, the expected value of X exists and equals c/(c-1). The conditional expectation of Xgiven that X has survived for at least time k is ck/(c-1). This says that the expected additional life X is ck/(c-1) – k = k/(c-1), and so the expected additional life of X is proportional to the amount of life seen so far. The proportionality constant 1/(c-1) depends on the power c that controls the thickness of the tails. The closer c is to 1, the longer the tail and the larger the proportionality constant. If c = 2, the proportionality constant is 1. That is, the expected additional life equals the life seen so far.

Note that this derivation computed E( X | X > k ), i.e. it only conditions on knowing that Xk. If you have additional information, such as evidence that a technology is in decline, then you need to condition on that information. But if all you know is that a technology has survived a certain amount of time, you can estimate that it will survive about that much longer.

This says that technologies have different survival patterns than people or atoms. The older a person is, the fewer expected years he has left. That is because human lifetimes follow thin-tailed distributions. Atomic decay follows a medium-tailed exponential distribution. The expected additional time to decay is independent of how long an atom has been around. But for technologies follow a thick-tailed distribution.

Another way to look at this is to say that human survival times have an increasing hazard function and atoms have a constant hazard function. The hazard function for a Pareto distribution is c/and so decreases with time.

Find the perfect platform for a scalable self-service model to manage Big Data workloads in the Cloud. Download the free O'Reilly eBook to learn more.


Published at DZone with permission of John Cook, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}