Calibration Is a Skill

Probability estimates can be graded for accuracy. Assign a 70% probability to 100 statements — if roughly 70 turn out true, you are well-calibrated. If 95 turn out true, you were too cautious. If only 40 are true, you were overconfident. Weather forecasters are famously well-calibrated; experts in most other fields are not.

Philip Tetlock's Good Judgment Project tested thousands of forecasters and found that the best — "superforecasters" — share specific habits: quantify beliefs in percentages, update on new evidence, track their own hit rates. Calibration can be trained. Once you start scoring your predictions, you get better at probability the way a musician gets better at pitch.