Neural Network Lexicon

Short Definition

Soft and hard halting are two approaches to deciding when a neural network should stop computation, differing in whether the stopping decision is continuous (soft) or discrete (hard).

Definition

Soft halting represents stopping decisions using continuous, differentiable signals that allow gradients to flow through the halting mechanism. Hard halting makes discrete stop-or-continue decisions that terminate computation explicitly. The choice affects training stability, optimization complexity, and inference behavior.

Stopping can be smooth—or abrupt.

Why It Matters

Halting decisions directly control computation cost and accuracy. The choice between soft and hard halting determines:

• whether halting can be trained end-to-end
• how stable optimization is
• how predictable inference latency becomes
• how closely training matches deployment

Halting design shapes learning dynamics.

Core Distinction

• Soft halting: differentiable, probabilistic, gradual
• Hard halting: discrete, deterministic or stochastic, final

Differentiability is the dividing line.

Minimal Conceptual Illustration

Soft halting:
Layer 1 → 0.6 continue
Layer 2 → 0.3 continue
Layer 3 → 0.1 continue → Output (weighted)

Hard halting:
Layer 1 → continue
Layer 2 → continue
Layer 3 → stop → Output

Soft Halting

Characteristics

• continuous halting signals
• gradients flow through stopping logic
• expected compute is optimized
• often uses weighted layer outputs

Advantages

• stable end-to-end training
• easier optimization
• smooth trade-off between accuracy and compute

Limitations

• less interpretable stopping decisions
• mismatch between training and inference
• execution of unused computation at inference

Soft halting optimizes expectation, not execution.

Hard Halting

Characteristics

• discrete stop decisions
• computation terminates immediately
• non-differentiable control flow
• exact inference behavior

Advantages

• clear stopping semantics
• predictable compute savings
• faithful deployment behavior

Limitations

• difficult or unstable training
• requires surrogate gradients or RL methods
• sensitive to threshold choices

Hard halting optimizes reality, not gradients.

Training Implications

• Soft halting enables standard backpropagation
• Hard halting often requires:
  • straight-through estimators
  • reinforcement learning
  • policy gradients
  • delayed activation during training

Training difficulty increases with discreteness.

Inference Behavior

• Soft halting may still compute deeper layers
• Hard halting guarantees early termination
• Latency predictability favors hard halting

Inference exposes true cost.

Robustness Considerations

Under distribution shift:

• soft halting may degrade gracefully
• hard halting may fail abruptly
• both rely on calibrated difficulty signals

Stopping errors amplify under shift.

Evaluation Metrics

Halting strategies should be evaluated with:

• accuracy vs compute curves
• expected vs actual compute
• halt-depth distributions
• tail-latency metrics
• OOD performance

Compute-aware evaluation is essential.

Hybrid Approaches

Many systems combine both:

• soft halting during training
• hard halting during inference
• gradual transition from soft to hard

Hybrid designs balance stability and realism.

Failure Modes

Common failures include:

• soft halting never truly stopping
• hard halting collapsing to shallow exits
• unstable threshold tuning
• training–inference mismatch

Stopping must be aligned.

Practical Design Guidelines

• start with soft halting for stability
• delay hard halting until representations mature
• calibrate stopping signals explicitly
• monitor halt distributions over time
• test under deployment-like conditions

Halting evolves over training.

Common Pitfalls

• assuming soft halting saves real compute
• using hard halting without exploration
• ignoring tail latency
• freezing thresholds prematurely
• evaluating only average accuracy

Stopping decisions deserve scrutiny.

Summary Characteristics

Aspect	Soft Halting	Hard Halting
Differentiable	Yes	No
Training stability	High	Lower
Inference efficiency	Approximate	Exact
Latency predictability	Lower	Higher
Deployment fidelity	Lower	Higher

Related Concepts

• Architecture & Representation
• Halting Functions
• Adaptive Computation Depth
• Early Exit Networks
• Conditional Computation
• Sparse Inference Optimization
• Compute-Aware Evaluation