Cognitive Biases · CB-27
An unusually extreme result is naturally followed by a more average one — not because of any real change, but because extremes are, statistically, partly luck that doesn't repeat.
The failure to account for regression to the mean — the statistical tendency for an unusually extreme measurement to be followed by a measurement closer to the average, purely because part of what made the first measurement extreme was random variation that isn't expected to recur — leading people to invent causal explanations for what is actually a purely statistical phenomenon.
The underlying statistical phenomenon was identified by Francis Galton in the 1880s studying the heights of parents and children; its systematic neglect in everyday causal reasoning was highlighted prominently by Daniel Kahneman, including his well-known account of Israeli flight instructors who mistakenly believed criticism improved performance more than praise did.
The Mechanism
Praise and punishment both 'work' due to regression alone
Kahneman's flight instructors saw exactly this pattern — pilots praised after an unusually smooth landing tended to perform worse next time, and pilots criticized after a rough landing tended to perform better next time, purely because extreme performances regress toward the average regardless of any feedback given — yet the instructors concluded criticism worked and praise backfired.
01 · EXTREME OUTCOMES ARE PARTLY SIGNAL, PARTLY NOISE — AND THE NOISE DOESN'T REPEAT
Any measurement combining real skill and randomness will regress
Any real-world outcome that includes some random variation on top of a stable underlying ability will show extreme results occasionally, driven partly by unusually good or bad luck — and because luck doesn't carry forward, the next measurement naturally moves back toward the person's true average level, independent of any intervention.
02 · IT SYSTEMATICALLY DISTORTS EVALUATIONS OF INTERVENTIONS AND FEEDBACK
This is precisely how ineffective (or actively harmful) interventions get credited with success
Any intervention applied specifically after an extreme result (praise after a great performance, punishment after a poor one, a new program launched after an unusually bad year) will appear to 'work' simply because regression to the mean would have produced improvement anyway — a major source of false confidence in interventions that have no real effect.
03 · IT REQUIRES A COMPARISON GROUP OR A KNOWN EFFECT SIZE TO CORRECT FOR
You can't diagnose the neglect from a single before/after comparison alone
Distinguishing a genuine intervention effect from ordinary regression to the mean requires either a proper control group that didn't receive the intervention, or a well-established statistical estimate of how much regression to expect given the measurement's known reliability — a single before/after comparison, however dramatic, can't distinguish the two.
Where It Fails / Inversion
Where it fails / inversion
Real interventions genuinely can and do work — the corrective isn't to dismiss every improvement following an intervention as pure regression, but to specifically check whether the improvement exceeds what regression to the mean alone would predict, typically by comparing against an untreated control group experiencing the same regression effect.
How To Use It
Worked example · evaluating a performance-improvement program at work
Before crediting a coaching or performance-improvement program with an employee's improved results, check whether that employee was selected for the program specifically because of an unusually poor prior performance — if so, some improvement was statistically likely regardless of the program, and a fair evaluation requires comparing against similarly low-performing employees who didn't receive the intervention.
How to use it
Before crediting any intervention that followed an unusually extreme result — good or bad — with causing the subsequent change, ask whether that change is larger than what regression to the mean alone would predict. Without a proper comparison group, an apparent 'effect' following an extreme outcome is often just statistics.
See Also