Short Definition
Training dynamics describe how a neural network’s behavior evolves during training.
Definition
Training dynamics refer to the patterns and changes observed in a model’s parameters, losses, gradients, and predictions as training progresses. Rather than focusing only on final performance, training dynamics emphasize how learning unfolds over time.
Understanding training dynamics helps explain why models converge, diverge, overfit, underfit, or become unstable during training.
Why It Matters
Two models with the same final accuracy may have very different training behaviors. Training dynamics reveal:
- whether learning is stable or chaotic
- how quickly a model converges
- when overfitting begins
- whether optimization is effective
Ignoring training dynamics often leads to inefficient training and brittle models.
How It Works (Conceptually)
- Gradients update parameters step by step
- Loss values change over epochs
- Training and validation metrics evolve differently
- Learning rate, batch size, and initialization shape trajectories
Training dynamics emerge from the interaction between:
- data
- model architecture
- optimization algorithm
- hyperparameters
Minimal Python Example
Python
for epoch in range(epochs): print(epoch, train_loss, val_loss)
Common Pitfalls
- Focusing only on final metrics
- Misinterpreting short-term fluctuations
- Using learning rates that cause instability
- Ignoring divergence between training and validation curves
- Treating training as a black box
Related Concepts
- Learning Dynamics
- Training Monitoring
- Gradient Descent
- Optimization
- Overfitting
- Underfitting
- Generalization
Neural Network Lexicon 