Neural Network Lexicon

Short Definition

Feedback loops occur when a model’s predictions or decisions influence future data, which in turn affects subsequent model behavior and evaluation.

Definition

A feedback loop in machine learning arises when a deployed model alters the environment or user behavior in ways that change the data it later observes. This creates a recursive dependency between model outputs and inputs, breaking the assumption that data is independently generated from a fixed distribution.

Models can shape the world they learn from.

Why It Matters

Feedback loops can silently bias data, inflate metrics, distort evaluation, and reinforce undesirable behaviors. They are a major reason why offline evaluation often fails to predict real-world performance.

Once deployed, models stop being passive observers.

Types of Feedback Loops

Positive Feedback Loops

Model decisions reinforce existing patterns.

• example: recommender systems amplifying popular content
• effect: concentration, bias amplification

Negative Feedback Loops

Model decisions suppress certain outcomes.

• example: fraud prevention blocking risky transactions
• effect: reduced label availability, censored data

Self-Fulfilling Feedback

Predictions cause the predicted outcome.

• example: credit denial increasing default risk

Self-Negating Feedback

Predictions prevent the outcome from occurring.

• example: early intervention averting failure

Feedback direction matters.

Minimal Conceptual Illustration

Model → Decision → Environment → Data → Model

Relationship to Data Distribution

Feedback loops violate the IID assumption by making future data conditional on past model behavior. This induces non-stationarity even if the external environment is stable.

The distribution moves because the model moves it.

Relationship to Causal Evaluation

Feedback loops require causal evaluation because correlations observed after deployment may be artifacts of the model’s own influence rather than evidence of effectiveness.

Correlation after intervention is suspect.

Impact on Metrics and Evaluation

Feedback loops can:

• inflate offline metrics
• hide false negatives or true outcomes
• create misleading performance stability
• accelerate metric drift
• enable metric gaming unintentionally

Metrics become endogenous.

Relationship to Proxy Metrics

Proxy metrics are especially vulnerable to feedback loops, as models may optimize the proxy by shaping the data-generating process rather than improving true outcomes.

The proxy becomes the process.

Long-Term Effects

Over time, feedback loops may cause:

• data homogenization
• reduced exploration
• blind spots in model behavior
• brittleness under distribution shift
• unfair or unsafe outcomes

Short-term gains can produce long-term fragility.

Mitigation Strategies

Common mitigation approaches include:

• explicit exploration policies
• randomized decision assignment
• delayed or partial automation
• shadow deployment and audits
• counterfactual logging
• periodic model resets or retraining on external data

Feedback must be designed, not ignored.

Role in Evaluation Governance

Evaluation governance should:

• identify feedback-prone decisions
• require causal or online evaluation
• mandate long-term audits
• prevent over-reliance on post-feedback metrics

Governance keeps loops visible.

Common Pitfalls

• assuming deployment data is unbiased
• interpreting post-deployment accuracy at face value
• retraining on feedback-contaminated labels
• ignoring suppressed outcomes
• failing to document intervention effects

Feedback loops are subtle but powerful.

Summary Characteristics

Aspect	Feedback Loops
Trigger	Model affects data
Assumption broken	IID
Risk	Silent bias and metric inflation
Detection	Difficult
Mitigation	Causal and governance-based

Related Concepts

• Generalization & Evaluation
• Causal Evaluation
• Outcome-Aware Evaluation
• Metric Drift
• Proxy Metrics
• Data Distribution
• Delayed Feedback Loops
• Model Update Policies