Survivorship Bias and Lessons From History

Following successful role models

Have you been following the Olympics? Watching an exciting variety of different sports and rooting for your national athletes isn’t just entertaining. Media tends to highlight those who’ve been successful. Successful athletes and sports personalities often act as role models and motivate people to follow their example. Olympics can have the added effect of inspiring “ordinary” people to get off the couch and start moving.

Becoming a successful athlete is an admirable aspiration, but it’s important to keep a sense of perspective. Only 180 of 1.5 million youth footballers in the UK are likely to reach their goal and make it as professional players in the Premier League ¹. I’ve done the math for you and here’s some bad news: 99.99 percent of all young football hopefuls will end up disappointed. Similarly, male basketball players in the US only have a 0.03 percent chance of turning their sporty hobby into a professional career ².

For many centuries, blood-letting was considered a tried and true remedy for certain conditions. It was recommended for fevers, inflammations, a variety of disease conditions and, ironically, for hemorrhage. Although it fell in and out of favor, it persisted into the 20th century and was even recommended by Sir William Osler in the 1923 edition his Principles and Practice of Medicine.³. This practice, which was based on the belief that withdrawing blood could help the body maintain balance and health, had been performed over 2000 years until the late 19th century. Why for so long? In some cases, bloodletting might have resulted in a positive outcome, and without a throughout scientific approach, only the survivors were taken into account.

The Missing Bullet Holes

One of the most popular stories of survivorship bias played out during World War II. Suppose that for American planes, the bullet holes on returning planes were distributed as in figure bellow, where should the mechanics reinforce planes so that more of them come back safely?

These spots were the wings, tail, and cockpit. Due to weight constraints, it couldn’t be everything. Initially military engineers wanted to reinforce those places where surviving planes had bullet holes, figuring that these were commonly hit spots: wings and tail. The American military called on statistician Abraham Wald to help them determine what parts of the planes they should reinforce. Here’s an excerpt of Wald’s analysis:

“What you should do is reinforce the area around the motors and the cockpit. You should remember that the worst-hit planes never come back. All the data we have come from planes that make it to the bases. You don’t see that the spots with no damage are the worst places to be hit because these planes never come back.”

Wald’s insight and reasoning was based on understanding what we now call survivorship bias. Bias is any factor in the research process which skews the results. Survivorship bias describes the error of looking only at subjects who’ve reached a certain point without considering the (often invisible) subjects who haven’t. In the case of the US military they were only studying the planes which had returned to base following conflict i.e. the survivors. In other words what their diagram of bullet holes actually showed was the areas their planes could sustain damage and still be able to fly and bring their pilots home.

Survivorship bias is a type of selection bias where the results, or survivors, of a particular outcome are disproportionately evaluated. Those who “failed”, or did not survive, might even be ignored. Focusing on the survivors can result in a false, or incorrect, estimate of probability.

Survivorship Bias in Financial Markets

A common phrase in any investment prospectus is “past performance is not indicative of future returns”. Yet many investors still rely on a fund or market’s historical performance or fail to factor survivorship bias into their judgment. Survivorship bias risk is the chance of an investor making a misguided investment decision based on published investment fund return data.

For instance, the S&P 500 is one of the leading stock indexes in the world, indexing to 500 stocks that its governors feel best represent the overall market. However, an individual investor would not necessarily generate the same returns as the S&P 500 simply by investing directly into the currently indexed 500 companies. This is because the S&P cuts and replaces companies overtime. It selects only those it feels most fit, biasing towards those companies that are succeeded and survive. Numerous studies have been done discussing survivorship bias and its effects ⁴.

Reverse survivorship bias describes a far less common situation where low-performers remain in the game, while high performers are inadvertently dropped from the running. An example of reverse survivorship can be observed in the Russell 2000 index that is a subset of the 2000 smallest securities from the Russell 3000. The loser stocks stay small and stay in the small-cap index while the winners leave the index once they become too big and successful. ⁵

Why this is important for business

Survivorship bias is the logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that didn’t because of their lack of visibility. This can lead to false conclusions in several different ways. Survivorship bias can lead to overly optimistic beliefs because failures are ignored, such as the tendency for failed companies to be excluded from performance studies because they no longer exist.

It often causes the results of studies to skew higher because only companies which were successful enough to survive until the end of the period are included. Wald’s story has important implications for all sorts of problems we face as individuals and as managers: whether it involves making good technical decisions, hiring decisions, and even diagnosing health problems.

“In a nutshell, the survivorship bias implies that the highest performing realization will be the most visible. Why? Because the losers do not show up… The mistake of ignoring the survivorship bias is chronic, even (or perhaps especially) among professionals. How? Because we are trained to take advantage of the information that is lying in front of our eyes, ignoring the information that we do not see.” - Nassim Nicholas Taleb in Fooled By Randomness

In his book The Black Swan, financial writer Nassim Taleb called the data obscured by survivorship bias “silent evidence.” We might miss problems, opportunities, and threats that remain hidden until their impact becomes clear.

Think also about the testimonials you hear from people who have signed up to be a part of a business and have made thousands of dollars in a short amount of time. By focusing on these success stories, internet businesses and multi-level marketing companies (MLM) are using survivorship bias to recruit more people to join and pay into the company.

These stories from “ordinary” people are convincing and can make one assume that if that person was successful, then they could be, too. However, statistically, 95% of people who join MLM companies fail ⁶; but this failure rate is often dismissed and only the top 2% of salespeople are highlighted. But you see these stories enough to believe they might be the norm.

Whether it be movie stars, or athletes, or musicians, or CEOs of multibillion-dollar corporations who dropped out of school, popular media often tells the story of the determined individual who pursues their dreams and beats the odds ⁷. There is much less focus on the many people that may be similarly skilled and determined but fail to ever find success because of factors beyond their control or other (seemingly) random events.

How To Combat Survivorship Bias

All around you, you only see winners. When you’re watching movies or YouTube or Twitch, you’re watching the actors who got the part, the creators who were boosted by the algorithm, and the gamers who made it big. Admiring celebrities and success stories is great, but there is much to be learned from those who tried and failed.

To get a clear look at the odds of something happening without the allure of a really great story, seek out stories of failure. Survivorship bias isn’t only limited to celebrity idolization, is one of the issues discussed in the provocative 2005 paper “Why Most Published Research Findings Are False” ⁸.

“New or rapidly developing industries, whether glamorous or not, very often provide more opportunities to get rich than established sectors. The three reasons for this are availability of risk capital, ignorance and the power of a rising tide." - Felix Dennis

Having successful role models is a key source of motivation for many people. The problem with the overwhelming focus on success is that valuable insights from failure are drowned out. By keeping a sense of perspective, the motivational benefits of role models can outweigh the dangers of survivorship bias.

The advice business is a monopoly run by survivors. When something becomes a non-survivor, it is either completely eliminated, or whatever voice it has is muted to zero," - David McRaney ⁹

What makes survivorship bias so insidious is that winners may not really know why they came out on top. In fact, luck and timing probably played larger roles than most realize or will admit, especially those busy trumpeting their incisive strategy and bold leadership.

“A stupid decision that works out well becomes a brilliant decision in hindsight." — Daniel Kahneman in Thinking Fast and Slow

Try and find out how many people started on the same path you’re considering, but didn’t make it. Then learn as much as you can from their failures.

External References