Label Latency

Short Definition

Label latency is the delay between an event occurring and its ground-truth label becoming available.

Definition

Label latency refers to the time gap between when a prediction target event occurs (event time) and when the corresponding label is observed, verified, or recorded in the dataset. This delay can range from seconds to months depending on the domain and labeling process.

Label latency constrains what information is legitimately available at training and prediction time.

Why It Matters

Many real-world ML systems operate under delayed supervision. If models are trained or evaluated assuming immediate label availability, they may inadvertently use future information, leading to data leakage and unrealistic performance estimates.

Label latency is a primary source of temporal leakage in production systems.

Common Causes of Label Latency

Label latency commonly arises from:

manual review or human annotation
delayed outcomes (e.g., chargebacks, churn, medical diagnoses)
aggregation windows or reporting cycles
verification and quality-control processes
legal or compliance delays

Latency is often unavoidable.

How Label Latency Affects Modeling

Label latency impacts:

feature eligibility at prediction time
training set construction
evaluation split design
retraining cadence
online vs offline consistency

Ignoring latency breaks causal validity.

Minimal Conceptual Example

			
# conceptual illustration
event_time = t
label_available_time = t + latency

Label Latency vs Missing Labels

Label latency: labels exist but arrive later
Missing labels: labels may never arrive

Latency requires waiting; missingness requires inference or exclusion.

Interaction with Time-Aware Evaluation

Time-aware sampling and evaluation must ensure that:

training data includes only labels available by the cutoff time
test data excludes labels not yet observed
features do not depend on future labels

Without this, evaluation becomes optimistic.

Relationship to Temporal Leakage

Label latency is a root cause of:

temporal feature leakage
processing-time leakage
validation leakage in time-series tasks

Many leakage bugs stem from mishandled latency.

Relationship to Rolling Retraining

Rolling retraining schedules must account for label latency. Retraining too frequently can incorporate incomplete or biased labels, degrading performance.

Latency-aware retraining often uses delayed cutoffs.

Common Pitfalls

training on labels that would not exist at prediction time
evaluating models before labels stabilize
backfilling labels without adjusting feature timelines
mixing real-time and delayed labels inconsistently

Latency must be modeled explicitly.

Relationship to Generalization

Ignoring label latency inflates apparent generalization by letting models learn from future outcomes. Correct handling produces more conservative but realistic performance estimates.

Neural Network Lexicon