Neural Network Lexicon

Temperature Scaling

Short Definition

Temperature scaling is a post-training method for calibrating model confidence.

Definition

Temperature scaling is a calibration technique that adjusts a model’s output probabilities by dividing logits by a scalar temperature parameter before applying the softmax function. The temperature parameter is learned on a validation set and does not change model accuracy or ranking.

Temperature scaling improves probability calibration without retraining the model.

Why It Matters

Neural networks are often overconfident. Temperature scaling provides a simple, effective way to reduce overconfidence while preserving predictive performance.

It is widely used in modern deep learning systems due to its simplicity and effectiveness.

How It Works (Conceptually)

  • The model produces logits
  • Logits are divided by a temperature value T
  • Softmax is applied to scaled logits
  • T is optimized to minimize calibration error

Higher temperatures soften probabilities; lower temperatures sharpen them.

Mathematical Formulation

softmax(z_i / T)

Where:

  • z_i are the logits
  • T > 0 is the temperature parameter

Minimal Python Example

Python
scaled_logits = logits / temperature
probabilities = softmax(scaled_logits)

Common Pitfalls

  • Optimizing temperature on test data
  • Assuming temperature scaling fixes all calibration issues
  • Applying temperature scaling to already calibrated models
  • Using temperature scaling for regression tasks

Related Concepts

  • Calibration
  • Model Confidence
  • Expected Calibration Error (ECE)
  • Reliability Diagrams
  • Uncertainty Estimation