Why is this interesting? - The Survivorship Bias Edition

Noah here. The NBA has returned, and, with it, I’ve been spending more time watching, thinking about, and arguing basketball. This season is unique for a whole bunch of reasons, not the least of which is they’re playing games without fans on a campus at Walt Disney World in Orlando. This setup, of course, substantially changes the dynamics of the playoffs. Usually, the prize for winning lots of games during the regular season is home-court advantage, which involves getting to play four of a round’s seven games on your home floor, critically including the first two. As teams battle in the postseason, narratives naturally emerge around the location of games. Away team down 2-0? Now they get to return home and feed off the advantage of their home crowd. The lower-ranked team took game two? They stole one on the road.

Of course, in bubble basketball, there’s no home-court advantage because there are no home-courts. But, even if there were, the evidence suggests that the benefit of playing in front of your fans has been eroding in the NBA over the last two decades. What once represented a nearly 4-point edge, home court is now worth around one extra basket. What’s more, home teams who used to enjoy a 65% winning percentage are now much closer to 50%. Despite the data, home-court narratives have persisted.

Why is this interesting?

Beyond the athleticism, sports are fueled by stories. For those of us not on the floor, the theories are all we have. Will my team finally win this season? Who is the greatest of all time? How would this player from a previous generation survive in today’s game? The only game fans get to play are ones that involve debate (and betting, I guess). The greatest players in history are the ones who won the championships. And while that is certainly partly true—particularly in a strong link sport like basketball where a single individual can have a massive impact on the outcome—it also brazenly fails basic controls around things like teammates and competition that can massively sway the way a career looks. If Michael Jordan and Lebron James had lived in the same era, whose team would have won? What if Scottie Pippen had left the Bulls all those years ago? We obviously can’t ever know the answers, but factors like those alter outcomes and, in turn, alter our view of history.

If all of this sounds a bit unscientific, that’s because it is. Many of the arguments surrounding the NBA and other sports boil down to survivorship bias—a standard error in logic that speaks to our tendency to over-interpret non-random data. One of the more famous examples of survivorship bias comes from World War II, when the United States Air Force was trying to figure out how to better protect planes from being shot down. Their initial approach was to examine the aircraft that had returned from combat and bulk up the areas that had been hit. That was until the statistician Abraham Wald came along: 

Wald assumed, on good evidence, that hits in combat were uniformly distributed over the planes. It follows that hits on the more vulnerable parts were less likely to be found on returning planes than hits on the less vulnerable parts, since planes receiving hits on the more vulnerable parts were less likely to return to provide data. From these premises, he devised methods for estimating the vulnerability of various parts.

(As an aside, that comes from a Rejoinder published in the Journal of the American Statistical Association in 1980 by the economist W. Allen Wallis and seems to be the only real reference to this story that exists.)

The image above, from Wikipedia, has nothing to do with Wald’s work, but it does a reasonably good job of visualizing the point.

In its essence, survivorship bias, and its parent selection bias, are cases of misplaced causation. Take the marshmallow test: the famous psychological study that told children they could have two marshmallows if they could control their urge for a short bit of time. Those better able to do so performed better on standardized tests and a host of other academic measures—a finding that was attributed to their self-control. All of this held until researchers recently went back and realized the study participants were a completely non-representative sample of professors’ children. When the experiment was readministered with a group of kids that look more like regular preschoolers, the results fell away. The selection, in other words, biased the outcome. Survivorship takes things a step further by creating a situation where there are big barriers in front of whatever you’re measuring. What kind of person makes a great Senator or CEO? We can’t ever really know the answer because the pool we have to look at is so small and non-representative of the broader public, but it’s safe to assume it goes far beyond the homogeneity that presently represents the norm.

Back to the NBA, there’s a kind of NBA meme called “ringz” that refers to the argument of many fans and ex-players that the only measure that matters is the number of championships a player wins. In this version of the universe, the German-born Dirk Nowitzki is “soft” until he finally wins a ring in 2011, and then it’s all just erased and he can enter the pearly gates of greatness. Kevin Garnett is thankfully remembered as an all-time NBA player (because he was), but it’s hard to know how he’d be viewed if he hadn’t left Minnesota for Boston where he was able to finally win a title. Part of the story here is survivorship bias—allowing ourselves to be swayed by the data from after a massively unlikely outcome—and part of it is our addiction to narratives. NBA writer extraordinaire Zach Lowe pulled on this string for Grantland in 2014:

The word “narrative” pops up in sportswriting more often than a non-fan might expect, since we’re writing about a bunch of grown men wearing funny uniforms running around throwing balls of various shapes. The need to turn games into something grander is natural. It can feel silly to invest so much anguish into a game that doesn’t hold larger metaphysical importance. … 

Sports absolutely can reveal something deeper about a player’s core nature. Some players thrive under pressure, demanding the ball in the hothouse of a Game 7, as if shouldering the burden of a team’s season is no biggie. Some players shrink from those moments, prove themselves to be selfish, or morph into injury-riddled tragic figures. There is grist for the literary in basketball.

But it can seem at times as if “narratives,” and anxiety over them, are taking over sportswriting.

I was recently having a conversation with a friend who said he believes grand narratives poison people’s brains, and, I have to say, the comment has certainly stuck in mine. Our attempts to reduce a hugely complex set of interactions, whether on a basketball court, in geopolitics, or culture, is not only often unhelpful, it’s likely counterproductive to us actually seeking a deeper and more nuanced understanding of what’s actually happening beneath the surface. While it’s only natural to seek out those stories that seemingly explain everything, we should also look much more skeptically at both them and ourselves for believing things could be so simple. Or, as Benford’s law of controversy states, “Passion is inversely proportional to the amount of real information available.” (NRB)

Longread of the Day:

We’ve long been fans of The California Sunday Magazine, and thought this was a powerful portrait of what is required to keep a bar running during the pandemic. As the magazine describes, it is “A tale of difficult decisions, impossible bureaucracy, sleepless nights, and no guarantees that any of it will work.” (CJN)

Quick Links:

• Really enjoyed this episode of the Mindscape podcast with physicist Neil Johnson on Complexity, Conflict, and Infodemiology (NRB)

• While we are talking about basketball, this ESPN piece trying to find a comp for second-year player Luka Doncic is a fun read (NRB)

• How to Add More Play to Your Grown-Up Life, Even Now (NRB)

Thanks for reading,

Noah (NRB) & Colin (CJN)

PS — Noah here. Variance, my new company, is just getting going with our Alpha. If you work in sales, services, marketing, or engineering and want to try out/give feedback on a tool to help your team work more effectively with their apps, please request an invite on the site. Thanks.

—

Why is this interesting? is a daily email from Noah Brier & Colin Nagy (and friends!) about interesting things. If you’ve enjoyed this edition, please consider forwarding it to a friend. If you’re reading it for the first time, consider subscribing (it’s free!).