Neural Network Lexicon

Short Definition

Architecture scaling laws describe how model performance changes predictably as architecture size, data, and compute are scaled.

Definition

Architecture scaling laws are empirical relationships showing that neural network performance follows smooth, often power-law trends as key dimensions—such as parameter count, depth, width, data volume, and training compute—are increased. These laws reveal that architecture design and scaling choices jointly determine achievable performance.

Scale reveals regularity.

Why It Matters

Scaling laws guide:

• model sizing decisions
• compute budgeting
• data collection strategies
• architecture selection
• performance forecasting

They replace guesswork with prediction.

Core Scaling Dimensions

Scaling laws typically involve:

• Model size (parameters, depth, width)
• Dataset size
• Training compute
• Architecture efficiency

No dimension scales in isolation.

Minimal Conceptual Illustration

Performance ↑
│
│ ●
│ ●
│ ●
│●
└──────────────────→ Scale (parameters / data / compute)

Power-Law Behavior

Empirically, many models exhibit:

Loss ≈ a × (Scale)^−b + c

where improvements diminish smoothly as scale increases.

Returns diminish—but remain predictable.

Architecture Matters in Scaling

Not all architectures scale equally well.

• residual connections enable deeper scaling
• normalization stabilizes large models
• attention scales differently than convolution
• dense connectivity affects memory efficiency

Good scaling requires good architecture.

Depth vs Width Scaling

• Depth scaling increases representational hierarchy
• Width scaling increases parallel capacity
• optimal trade-offs depend on task and architecture

Balance beats extremes.

Compute-Optimal Scaling

Given fixed compute, scaling laws suggest:

• how to allocate compute between model size and data
• when larger models underperform due to insufficient data
• when data becomes the bottleneck

More compute does not mean better models.

Relationship to Feature Learning

As models scale:

• learned features become more abstract
• representations stabilize
• transferability often improves

Scale shapes representation quality.

Scaling Laws and Generalization

Scaling laws describe average-case behavior but do not guarantee:

• robustness
• calibration
• fairness
• real-world performance

Scale improves benchmarks—not necessarily outcomes.

Failure Modes

Blind scaling can cause:

• overfitting when data is insufficient
• brittle behavior under distribution shift
• inflated confidence
• unsustainable compute costs

Scale amplifies assumptions.

Architectural Efficiency

Efficient architectures shift scaling curves:

• better performance at smaller scale
• lower compute cost for similar accuracy
• improved deployment feasibility

Efficiency bends the curve.

Practical Implications

Scaling laws inform:

• architecture selection
• budget planning
• experiment prioritization
• model roadmap decisions

Scaling is a strategy, not an accident.

Common Pitfalls

• extrapolating scaling laws beyond observed regimes
• ignoring data quality
• assuming scaling fixes misalignment
• conflating benchmark gains with business value
• optimizing scale without governance

Prediction is not permission.

Summary Characteristics

Related Concepts

• Architecture & Representation
• Model Capacity
• Feature Learning
• Generalization
• Compute–Data Trade-offs
• Residual Connections
• Attention vs Convolution
• Benchmark Performance vs Real-World Performance