Cognitive Biases · CB-24
People treat small samples as if they're just as reliable as large ones — expecting a handful of observations to look just like the true underlying distribution.
The mistaken intuitive belief that a small sample of observations should closely resemble the properties of the larger population it's drawn from — leading people to draw overly confident conclusions from limited data and to be surprised by perfectly normal statistical variability in small samples.
Named and analyzed by Amos Tversky and Daniel Kahneman in their 1971 paper 'Belief in the Law of Small Numbers,' which found that even trained research psychologists routinely underestimated how much random variability to expect in small experimental samples.
The Mechanism
Small samples are much noisier than intuition expects
The red line's wide swings at small sample sizes are genuine statistical noise, not evidence of anything meaningful happening — Tversky and Kahneman found that even trained scientists routinely misjudged how much of this normal small-sample variability to expect, drawing overconfident conclusions from too little data.
01 · IT'S DISTINCT FROM, BUT UNDERLIES, BOTH THE GAMBLER'S AND HOT-HAND FALLACIES
A shared root cause of several related statistical misjudgments
The core error — expecting small samples to closely mirror the true underlying probability — is the same underlying mechanism behind expecting a coin to 'even out' quickly (gambler's fallacy) or reading meaning into a short streak (hot hand): all stem from underestimating how much randomness a small sample can display even when nothing unusual is happening.
02 · IT AFFECTS TRAINED RESEARCHERS, NOT JUST LAY INTUITION
Tversky and Kahneman's original finding specifically targeted scientific practice
The 1971 study specifically surveyed professional research psychologists and found they routinely designed studies with sample sizes too small to reliably detect the effects they were looking for, because their intuitive sense of how much random variation to expect in small samples was itself systematically miscalibrated.
03 · IT LEADS TO PREMATURE PATTERN-DETECTION FROM LIMITED DATA
Small samples get treated as if they were conclusive
Because people underestimate small-sample noise, a handful of data points showing an apparent trend or difference is often treated as far more conclusive than it statistically warrants, leading to premature conclusions, false pattern recognition, and business or policy decisions made on insufficient evidence.
Where It Fails / Inversion
Where it fails / inversion
Not every conclusion drawn from a small sample is wrong — sometimes a small sample provides genuinely strong evidence, particularly when the effect size is large and the measurement is precise; the error specifically lies in failing to account for the wider uncertainty small samples inherently carry, not in ever using small samples at all.
How To Use It
Worked example · evaluating an A/B test with too little traffic
A team observing an apparent 15% improvement in a metric after an A/B test with only a few dozen users per group should treat that as weak, highly uncertain evidence, not a confirmed win — calculating a proper confidence interval (which will likely be very wide with so little data) rather than trusting the point estimate alone prevents premature rollout decisions based on what may be pure sampling noise.
How to use it
Before drawing a firm conclusion from a small number of observations, explicitly estimate how much random variation you should expect at that sample size, and treat any apparent pattern as tentative until it either persists at a larger sample size or comes with a properly calculated confidence interval.
See Also