A model that outputs 0.9 for a prediction should be right about 90% of the time it says that, not 60%. Model calibration in machine learning is the gap between a confidence score looking meaningful and it actually being meaningful, and the two aren’t the same thing by default.
What an uncalibrated model actually does wrong
Most classifiers output a probability, but that number isn’t automatically calibrated just because it’s between 0 and 1. A model can be systematically overconfident, clustering most of its outputs near 0 or 1 regardless of how uncertain the case actually is. Accuracy alone won’t reveal this. A model can classify correctly most of the time while still being badly calibrated, since calibration is about the confidence values themselves, not just whether the final label was right.
How to check calibration
A reliability diagram bins predictions by their stated confidence and plots that against actual observed accuracy within each bin. A perfectly calibrated model traces a straight diagonal line. Anything bowing away from that line shows over- or under-confidence at specific probability ranges, which a single accuracy number would never surface.
Why this matters more than it sounds like it should
Calibration matters most whenever a downstream decision depends on the probability itself, not just the predicted label. A fraud system that flags anything above a 0.7 threshold is trusting that 0.7 actually means roughly 70% likely, not just “probably fraud, probably not, who knows.” If the underlying probabilities are miscalibrated, that threshold stops meaning what it’s supposed to mean.
A real example: where calibration would change the decision
The support ticket triage platform predicts both category and priority for incoming tickets. If priority routing relies on a confidence threshold to decide when a ticket needs immediate escalation versus normal queueing, that threshold is only meaningful if the underlying probability is calibrated. An overconfident model would escalate too aggressively. An underconfident one would let urgent tickets sit.
Fixing calibration
- Platt scaling fits a simple logistic regression on top of the model’s raw outputs to rescale them into better-calibrated probabilities.
- Isotonic regression is a more flexible, non-parametric alternative, useful when the miscalibration doesn’t follow a simple sigmoid-shaped pattern.
- Both require a held-out calibration set, separate from the data used to train the original model, so the rescaling itself doesn’t leak information from training.
A quick checklist
- Does anything downstream actually use the raw probability, or only the final predicted label?
- Have you plotted a reliability diagram, or only checked accuracy?
- If a threshold decision depends on the probability meaning something specific, has calibration been verified?
- Was any calibration fix fit on a separate held-out set, not the original training data?
FAQ
Do all models need calibration?
Only when the probability itself is used downstream, for thresholding, ranking by confidence, or risk-based decisions. If only the final predicted label matters, calibration is less critical.
Are tree-based models usually well calibrated by default?
Not reliably. Random forests in particular tend to push probabilities toward the extremes, which often benefits from a calibration step.
Is model calibration the same as model accuracy?
No. A model can be accurate on its final predictions while still being poorly calibrated in its stated confidence, since these measure different things.
