We examine the cause and effects of survivorship bias.
If you’ve ever called the NDW analyst line for help constructing a custom model, you may have heard us mention survivorship bias and how it can skew the results of a backtest and make the historical performance look better than it would have been in practice. If you understood that survivorship bias was something you wanted to avoid, but didn’t fully understand what it was, you’re far from alone. That’s why today, we thought it would be helpful to give a more in-depth explanation.
What is survivorship bias?
Survivorship bias is a statistical error which arises from focusing only on entities that passed a selection process and ignoring those that did not, which can lead to overly optimistic assumptions and suboptimal decisions. In finance, survivorship bias is often talked about in the context of stocks or mutual funds/ETFs. But it exists in other fields as well.
During WWII, a statistician proposed a method for estimating the vulnerability of various parts of airplanes based on the damage to aircraft returning from combat. However, this methodology was flawed because it ignored aircraft that didn’t survive combat. If the data collected in such a study was used to try to reinforce parts of the plane that appeared most susceptible to damage, it would potentially leave out more critical components because any aircraft that was damaged in those critical areas did not survive combat and was thus excluded from the data.
How does survivorship bias impact finance and investing?
One of the easiest places to see survivorship bias in investing is in the aggregation of fund data. Consider a hypothetical company—ABC Investment Co—that has 30 mutual funds but 10 years ago had 50 mutual funds. ABC advertises that its 30 existing funds have returned an average of 10% over the last 10 years. We’ll assume that ABC’s advertisement is accurate, but it still may not accurately reflect the experience of ABC’s investors because it omits the 20 funds that are now defunct. And, as we know, in this business, high performing funds tend to attract more money and stay in business while weak performers tend to see assets dwindle and eventually disappear. So, it’s likely that an investor who started out in one of the 20 defunct funds did not achieve the advertised 10% return, as survivorship bias has skewed the average.
When it comes to backtesting models and matrices when we talk about survivorship bias, we are typically talking about it as it relates to stocks, which is the same idea but can be a bit more difficult to conceptualize. The basic idea is that the stocks in a given universe, like the S&P 500 (SPX), are not static. Just as in the mutual fund example, if we look only at the current constituents (survivors), it tends to make performance look better than it would if we included the stocks that didn’t survive.
It can be easy to think of an index like the S&P 500 as being stable, which can cause us to ignore survivorship bias or underestimate its effects. After all, the S&P this year looks pretty much the same as it did last year. But, over time, the effects of additions and deletions and the resulting survivorship bias can be substantial. Consider this—of the 500 stocks in the S&P in 1996, only 135 or about 27% remain in the index.
So, what happened to the other 365 stocks? Some of them merged with other companies or were acquired and may survive today under other names. Some likely still exist but were removed from the S&P due to declines in market cap, while others, like Enron and Circuit City, ended in bankruptcy. If we look at the average performance of only the current S&P 500 constituents, we ignore the negative effects of those that have been removed due to declines in market value or bankruptcy. To further illustrate this point, below is a table of the average performance of all the stocks currently in the S&P 500 and the performance of the S&P 500 Equal Weight Index (SPXEWI) over various time periods. Theoretically, if the index were static, the average performance of the constituents and the performance of SPXEWI should be equal. But as you can see over longer periods, there are significant differences with the average for all current S&P 500 stocks significantly outperforming the equal weight index. This is not to say that the entire difference is due to survivorship bias, but it is a significant factor.
We have used the S&P 500 as an example, but survivorship bias is not unique to the S&P or even indexes in general. If, theoretically, you had a universe of every stock that currently trades on any market in the world, it would still not be free of survivorship bias. You might lessen the bias, as you would overcome the problem of excluding stocks that have been removed due to market cap requirements, but you would still exclude all stocks that have ceased trading.
Mitigating Survivorship Bias
So, how do you overcome survivorship bias? Conceptually, this is easy. You simply need to include the non-survivors in your data set. In practice, it can be much more difficult. In our aircraft example, it would mean retrieving planes shot down in combat. For backtesting a stock strategy, it means including all the stocks that were in your universe at an any point during the backtest period, which often means obtaining & including performance history for stocks that no longer trade, which can be a complex task.
Luckily, on the NDW platform, this task has been made easier. Virtually all the pre-made matrices in the system have been constructed to correct for survivorship bias. So, when you’re backtesting a strategy, if you use one of the pre-made matrices, it will include all stocks that have been in that universe over the backtest period, even those that may no longer trade. To illustrate this point, below is the top 19 stocks from the pre-made Top 500 Large Cap matrix as of September 30, 1999. All the stocks with an asterisk after their name no longer trade but are still present in the matrix because it has been adjusted to remove survivorship bias.
