Short Definition
The learning rate controls the size of parameter updates.
Definition
The learning rate determines how much model parameters change in response to gradients. It directly affects training speed and stability.
Choosing an appropriate learning rate is critical: too small leads to slow learning, too large leads to divergence.
Why It Matters
A poor learning rate can prevent learning entirely.
How It Works (Conceptually)
- Scales gradient updates
- Controls convergence speed
- Influences stability
Minimal Python Example
learning_rate = 0.01
Common Pitfalls
- Using one value forever
- Copying defaults blindly
- Ignoring schedules
Related Concepts
- Gradient Descent
- Optimization
- Training Stability
Neural Network Lexicon 