Short Definition
Convergence occurs when training reaches a stable solution.
Definition
Convergence refers to the point at which a neural network’s parameters and loss values stabilize and further training produces little or no improvement. Convergence does not necessarily imply optimality or good generalization.
A model can converge to a poor solution.
Why It Matters
Understanding convergence helps decide:
- when to stop training
- whether optimization is effective
- if hyperparameters are appropriate
Convergence behavior reveals much about training dynamics.
How It Works (Conceptually)
- Gradients decrease in magnitude
- Loss plateaus or decreases slowly
- Parameter updates become small
- Validation performance stabilizes
Convergence is influenced by optimizer choice, learning rate, and batch size.
Minimal Python Example
Python
if abs(loss_t - loss_t_minus_1) < tolerance: converged = True
Common Pitfalls
- Confusing convergence with generalization
- Stopping too early
- Training too long after convergence
- Ignoring validation behavior
Related Concepts
- Optimization
- Training Dynamics
- Early Stopping
- Learning Rate Schedules
- Generalization
Neural Network Lexicon 