CHAPTER 20
The Illusion of Validity
Read It
Kahneman kept seeing the same thing in the army: the method for evaluating officer candidates was no better than random, yet every assessor remained fully confident in their next judgment. This isn’t stubbornness. It’s a deeper illusion—confidence comes from how smoothly the brain assembles limited information into a coherent story, not from how much evidence there is. Knowing a method fails overall doesn’t automatically make you cautious about a specific case.
Open full image ↗Draw It
Prose tends to treat two levels as the same thing: the overall assessment of a method, and the immediate judgment of an individual case. The first is a statistical question; the second is a feeling. Visually, the two layers separate: the upper layer says “this method’s long-term accuracy is near chance,” the lower layer says “I’m very sure about this person.” There’s no automatic channel between them. That structure explains why people stay confident even when they know the method is invalid—the fluency of a case judgment and the overall accuracy rate live on different cognitive floors.
Rethink It
In interviews, one interviewer may feel extremely confident about a candidate’s technical judgment while the interview process itself has low predictive validity. If the team only discusses “does this interviewer have good instincts” and never tests “does this process reliably separate good from bad,” the illusion of validity persists. More commonly, after every hiring mistake the team tweaks specific questions but never verifies whether the revised process actually predicts better.
Take It With You
The only reliable test for real skill in a domain is whether individual performance repeats consistently over time. How certain a judgment sounds is not evidence.