Data and distribution define what a neural network sees, learns from, and ultimately assumes about the world.
They shape model behavior long before optimization begins and long after training ends.
This section of the Neural Network Lexicon focuses on how data is generated, how distributions change, and why many model failures are caused not by architecture or optimization—but by mismatches between data and assumptions.
Understanding data and distribution is essential for building models that remain reliable outside controlled training environments.
Data as the Foundation of Learning
Neural networks do not discover truth in isolation; they learn patterns present in data. The quality, structure, and labeling of that data determine what the model can and cannot learn.
The following entries explain how data influences learning:
- Training Data
- Validation Data
- Test Data
- Label Noise
- Class Imbalance
These concepts clarify why more data is not always better, and why data quality often matters more than model complexity.
Distributional Assumptions
Most machine learning methods rely—explicitly or implicitly—on assumptions about how data is distributed.
This group explains what those assumptions are and why they matter:
- Data Distribution
- Independent and Identically Distributed (IID) Assumption
Violations of these assumptions are a major source of unexpected model behavior.
Distribution Shift and Change Over Time
Real-world data rarely stays static. When data changes, model performance can degrade silently.
These entries focus on how and why distributions shift:
- Distribution Shift
- Concept Drift
They explain the difference between changes in input data and changes in the underlying relationship between inputs and targets.
Leakage and Dataset Contamination
Some of the most damaging failures in machine learning occur when information flows where it should not.
This group covers critical evaluation and data-handling pitfalls:
- Data Leakage
- Target Leakage
- Train/Test Contamination
These concepts explain why models can appear highly accurate while learning nothing useful.
Sampling and Representation
How data is sampled and represented affects what the model considers “normal.”
This section focuses on representational and sampling effects:
- Sampling Bias
- Out-of-Distribution Data (OOD)
These entries explain why models may fail on rare, unseen, or systematically excluded cases.
How to Use This Section
If you are diagnosing unexpected performance drops, start with Distribution Shift and Concept Drift.
If your model performs unrealistically well during development, review Data Leakage and Target Leakage.
If your model struggles with rare or critical cases, explore Class Imbalance, Sampling Bias, and Out-of-Distribution Data.
Data and distribution form the hidden substrate of learning.
Many failures attributed to models or algorithms are, in reality, failures of data understanding.
Neural Network Lexicon 