Short Definition
Validation curves show how model performance varies with a hyperparameter.
Definition
Validation curves plot training and validation performance as a function of a specific hyperparameter, such as model capacity, regularization strength, or learning rate. They help identify ranges where the model underfits, overfits, or performs optimally.
Validation curves are used primarily for model and hyperparameter selection.
Why It Matters
Validation curves help answer:
- Is the model too simple or too complex?
- Is regularization too weak or too strong?
- How sensitive is performance to this hyperparameter?
They provide structured guidance for tuning decisions.
How It Works (Conceptually)
- Fix all but one hyperparameter
- Train models across a range of values
- Plot training vs validation performance
- Identify regions of underfitting and overfitting
Minimal Python Example
Python
plot(hyperparam_value, val_score)Common Pitfalls
- Tuning multiple hyperparameters simultaneously
- Using test data instead of validation data
- Assuming smooth curves in noisy regimes
- Overfitting to validation results
Related Concepts
- Hyperparameters
- Hyperparameter Optimization
- Bias–Variance Tradeoff
- Model Capacity
- Generalization
Neural Network Lexicon 