Neural Network Lexicon

Curriculum Learning

Short Definition

Curriculum learning trains models by presenting examples in a meaningful order, typically from easy to hard.

Definition

Curriculum learning is a training strategy in which a model is exposed to training data according to a predefined progression, rather than sampling uniformly at random. The curriculum organizes examples by difficulty, complexity, or relevance, allowing the model to gradually build more sophisticated representations.

Curriculum learning mirrors structured learning in humans and animals.

Why It Matters

Training on all examples uniformly can overwhelm models early in learning, slow convergence, or trap optimization in poor local minima. By controlling the order of examples, curriculum learning can stabilize training and accelerate learning.

It reshapes when data is seen, not what data exists.

How Curriculum Learning Works

A typical curriculum learning setup:

  1. Define a notion of example difficulty or complexity
  2. Start training on easier examples
  3. Gradually introduce harder or more complex samples
  4. Eventually train on the full data distribution

The curriculum can be static or adaptive.

Common Curriculum Strategies

Frequently used strategies include:

  • Difficulty-based curricula: ordered by loss, margin, or heuristic difficulty
  • Complexity-based curricula: simple patterns before complex ones
  • Length-based curricula: shorter sequences before longer ones
  • Noise-based curricula: clean data before noisy data
  • Self-paced learning: model determines difficulty dynamically

Curriculum design depends heavily on the task.

Minimal Conceptual Example

# conceptual curriculum schedule
for stage in curriculum:
  train(model, data_at_difficulty(stage))

Curriculum Learning vs Random Sampling

  • Random sampling: unbiased but unstructured
  • Curriculum learning: structured but biased

Curricula bias training intentionally to improve optimization.

Benefits and Trade-offs

Benefits include:

  • faster convergence
  • improved training stability
  • better representations in early learning
  • reduced sensitivity to initialization

Trade-offs include:

  • potential bias in learned representations
  • sensitivity to poorly designed curricula
  • unclear transfer benefits in some tasks

Curriculum design is both an art and a science.

Relationship to Optimization

Curriculum learning influences optimization dynamics by smoothing the loss landscape early in training. It can reduce gradient noise and help models escape unstable regimes.

However, it does not change the final objective.

Relationship to Generalization

Curriculum learning may improve generalization indirectly by encouraging robust representations, but poorly aligned curricula can harm calibration or performance on hard cases.

Curriculum success depends on alignment with deployment difficulty.

Relationship to Active and Importance Sampling

Curriculum learning controls training order, while active and importance sampling control sample selection or weighting. These strategies can be combined but serve different purposes.

Common Pitfalls

  • defining difficulty using leaky or post-outcome information
  • never exposing the model to full data complexity
  • assuming curriculum learning always improves performance
  • overfitting to the curriculum schedule

The curriculum should eventually disappear.

Related Concepts