CHAPTER 10
The Law of Small Numbers
Read It
The phrase “law of small numbers” invites misunderstanding. It isn’t a law about small samples; it’s a pseudo-law the human mind imposes on them. We assume a small sample represents the whole, so we mistake sampling variation for real difference.
The counties with the lowest kidney cancer rates are mostly the counties with the fewest people. That’s not because of clean living; it’s because small samples produce extreme values more often. The mind doesn’t like “this is noise.” It wants a story. So we invent one: better diet, cleaner air, slower pace. The smoother the story, the less we want to ask: how big was the sample?
Open full image ↗Draw It
The chapter rests on a contrast: statistics says small samples produce extremes; intuition says extremes must have causes. These aren’t competing explanations on the same level. One is math, the other is narrative. A diagram puts causal stories and sampling artifacts on separate paths, so the same observation can be seen as coming from two different generating mechanisms.
Rethink It
In A/B testing, the classic error is treating a significant difference in a small traffic slice as proof of product improvement. A button color lifts conversion by 20%—impressive, until you notice the experiment group had a few hundred users. The right question isn’t “why did it lift?” but “how common is a 20% swing at this sample size?”
Take It With You
Before accepting any statistical conclusion, ask about sample size. If the answer is smaller than your intuition expected, suspend the causal story until a larger sample catches it.