Source
Guo, Pleiss, Sun and Weinberger (2017), LIT-616.
When this applies
Your classifier’s output probabilities are used for something other than taking the argmax — a threshold, a ranking, an abstention rule, a downstream expected-value calculation, a hand-off to a person — and you have a held-out set you can spend. Modern architectures are the case that needs it: typical ECE is 4–10%, and depth, width, BatchNorm and reduced weight decay each make it worse while improving accuracy.
Do this
Fit one scalar. After training, divide the logits by a temperature T
and optimize T for NLL on the validation set with the network frozen:
q̂ᵢ = max_k softmax(zᵢ / T)⁽ᵏ⁾
T > 1 softens the distribution. It is a one-dimensional convex problem —
about ten conjugate-gradient iterations, a fraction of a second — and in a
framework it is a constant multiplier inserted between the logits and the
softmax, set to 1 during training.
The method and the recommendation are Guo et al.’s, in LIT-616. They found typical ECE of 4–10% in modern convolutional, recurrent and deep-averaging networks, and temperature scaling beat vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ, though the first two strictly contain it.
Note what it cannot do, because that is the reason to prefer it. A single positive scalar does not move the argmax, so accuracy is unchanged by construction. Nothing has to be re-validated after applying it. The binning methods do not have this property: histogram binning, isotonic regression and BBQ all change class predictions, and cost accuracy for it.
Do not reach for the more general version. Vector scaling (a diagonal matrix) and matrix scaling (a full one) strictly contain temperature scaling and both do worse. The vector solution comes out with nearly constant entries — the paper’s reading is that miscalibration is intrinsically low dimensional. Matrix scaling is worse still above a few hundred classes, because its parameter count grows quadratically, and it fails to converge at all on ImageNet’s 1000.
Why the problem exists at all
Networks overfit to NLL without overfitting to 0/1 loss. On a CIFAR-100 ResNet the test error falls from 29% to 27% inside the region where NLL is already rising — the model buys accuracy with confidence it has not earned. That is why the four things that worsen calibration are all things adopted for accuracy, and why the fix is post hoc rather than a change to training. The paper is explicit that why those trends do this is open.
Limitations
- It does not fix a model, it fixes a report. Refinement — how well the predictions separate the classes — is untouched, by the same argument that makes accuracy untouched. THEORY-060 is about that other term.
- It assumes train, validation and test come from one distribution, which is stated in the paper and is the first thing to go in deployment.
- One dataset in the study was not improved by any method. Reuters, at ECE below 1% before calibration; the authors say post-processing may not have been necessary there, and allow that their measurements may be affected by the dataset split or the binning scheme.
- ECE itself is a binned estimate, which the paper computes at M=15. The measure is Naeini et al. (2015), not this paper — this is where it became the default.