Ethics, Safety & Alignment

How Sure Is Sure?

A robot guesses if fruit is ripe, and says how sure it is. Is "95% sure" really right 95% of the time? Let's check, then teach it to say "I don't know".

If I say "90% sure" I should be right 9 times in 10. Look at the chart: the dots should sit on the dashed line. They don't yet!

Is its confidence honest?

Bottom: how sure it said it was. Side: how often it was really right. Bars on the dashed line mean honest. Bars below it mean overconfident.

Challenges

Catch it bragging: with "As trained", find 40 or more wrong answers where it was 90% sure or more.
Make it honest: pick a method that brings the honesty error to 0.05 or less.
Know when you don't know: get 85% right when it answers, while still answering 30% or more.
Almost no confident mistakes: show 15 or fewer wrong answers, while still answering 15% or more.

What just happened?

For grown-ups: what's really going on

A logistic regression with 14 features (3 informative, 11 noise) is trained on 50 examples with no regularisation for many steps, so it fits noise and produces extreme logits. This is overconfidence, common in modern neural networks too.

  • The reliability diagram bins predictions by confidence (the probability of the chosen answer) and plots the bin's real accuracy. Expected calibration error is the weighted average gap: ECE = Σ (nb/N) · |accb − confb|.
  • Temperature scaling divides the logit by one number T fitted on a separate validation set by minimising log loss: p = σ(z / T). It never changes which answer is picked, only how sure it sounds.
  • Platt scaling fits p = σ(a·z + c) on the validation set, which can also shift the model if it is biased towards one answer.
  • Selective prediction (abstention) answers only above a confidence threshold, producing a risk-coverage trade-off. It works only if confidence ranks correct above incorrect answers and is well calibrated.
  • Language models can state false claims fluently (hallucination). Calibrated uncertainty, retrieval, and abstention are active ways to reduce harm, but token probability is not truth.

Further reading: Guo et al., On Calibration of Modern Neural Networks (2017) · Platt, Probabilistic Outputs for Support Vector Machines (1999) · Geifman and El-Yaniv, Selective Classification for Deep Neural Networks (2017).