“They don’t make them like they used to.” I’m sure most of us have heard that a lot. From cars to refrigerators or even our homes, it seems that everything lacks the durability or quality that existed years ago. While the mechanical engineer in me likes to see how things are made, the forensic investigator in me loves to find out how things break.
Understanding how things break is a key part of improving them and designing future gadgets, structures, or widgets that will last for a long time. As clever as we are, people are often subject to fallacious reasoning, and the fallacy I encounter most in my field is survivorship bias.
Survivorship Bias and Operation Sea Lion
What is survivorship bias? It’s when you focus too much on the survivors of a given population. This selection bias pops up when we talk about manufacturing, finance, or even cats. However, my favorite example comes from Operation Sea Lion.
Sea Lion was the code name for Germany’s planned invasion of the U.K. during the Battle of Britain, just after knocking France out of the fight. As the Nazis amassed their navy, armor, and infantry in preparation for crossing the English Channel, both sides knew that victory would be determined by one thing: air superiority.
To that end, the Royal Air Force had to figure out a way to increase the survivability of bombing runs. While they had designed effective armor that could be added to their planes, it was too heavy to place everywhere.
So where should you armor a plane to increase the chances of it (and the crew) returning home? Researchers studied the damage done to aircraft that had returned from previous missions, and they recommended adding armor to the areas that had taken the most damage.
With the damage of surviving planes carefully tracked and illustrated, an overlay like the one above makes it clear that the damage is focused in certain areas. Doesn’t it make sense, then, to put the armor where those planes were taking the most damage?
“Not So Fast,” Says Abraham Wald
Without the intervention of Abraham Wald, a statistician with the U.S. military’s Statistical Research Group, the British might have put that armor in all the wrong places, forever changing world history and making The Man in the High Castle a documentary series.
Wald noted that the study and its recommendations suffered from selection bias. Bombers that were shot down never returned and were therefore not assessed for damage. This means that the researchers had their conclusion backwards.
The damaged areas of returning aircraft didn’t need to be reinforced, as evidenced by the fact that the planes shot in those places still made it back. Instead, the blank areas in the diagram above represent the areas sorely in need of armor, as planes shot in those areas didn’t return to tell their tale.
Forensic Investigators Don’t Let Failures Go to Waste
Survivorship bias means that you aren’t accounting for the failures. Yes, your mother’s sewing machine has lasted for 50 years, but that doesn’t mean that “they don’t make them like they used to.” You might not be aware of all the lower-quality sewing machines manufactured around the same time that went to the junk heap decades ago.
As a forensic investigator, I take a special interest in learning why structures or materials fail. When you know the causes of failure, you can prevent it.
What are your favorite examples of survivorship bias? Leave a comment below!
Marc Meadows