Neural Network Lexicon

Short Definition

Admission control is the practice of deciding whether to accept, delay, or reject incoming inference requests in order to maintain system stability and SLA compliance.

Definition

Admission control regulates the flow of requests entering an ML inference system by enforcing limits based on current load, resource availability, and service-level objectives. Rather than allowing unlimited requests to queue and degrade performance, admission control proactively protects latency, reliability, and fairness by controlling system utilization.

Stability is preserved by saying “no” early.

Why It Matters

In production ML systems:

• queues grow faster than they drain under overload
• tail latency explodes near capacity
• adaptive models increase service-time variance
• failures cascade when overload is unchecked

Admission control prevents overload from becoming outage.

Core Principle

Rejecting early is better than failing late.

Backpressure protects reliability.

Minimal Conceptual Illustration

Incoming Requests
        ↓
 Admission Control
    ┌── Accept → Inference
    │
    ├── Delay / Queue
    │
    └── Reject / Shed Load

What Admission Control Regulates

Admission control may limit:

• request arrival rate
• concurrent in-flight requests
• queue length
• compute or memory usage
• priority class saturation

Control targets system pressure points.

Relationship to Queueing Effects

Queueing effects cause latency to grow nonlinearly near capacity. Admission control caps utilization to prevent entering unstable queueing regimes.

Queues are symptoms; admission is prevention.

Relationship to SLA-Aware Inference Policies

SLA-aware policies define acceptable behavior; admission control enforces those limits by blocking or deferring requests before SLAs are violated.

Admission is an enforcement mechanism.

Admission Control Strategies

Rate Limiting

Caps request rates per client, endpoint, or priority class.

Concurrency Limits

Restricts the number of simultaneous inferences.

Queue Length Thresholds

Rejects or sheds load when queues exceed safe bounds.

Priority-Based Admission

Allows critical traffic while rejecting best-effort requests.

Adaptive Admission

Adjusts thresholds dynamically based on latency or load.

Policies encode priorities.

Interaction with Graceful Degradation

Admission control is often combined with graceful degradation:

• degrade first (fallback models, early exits)
• reject only when degradation is insufficient

Rejection is the last resort.

Evaluation Considerations

Admission control should be evaluated on:

• SLA violation reduction
• rejection rates under load
• fairness across request classes
• recovery behavior after overload

Effectiveness is situational.

Monitoring in Production

Key signals include:

• rejection and throttling rates
• queue length trends
• correlation with latency spikes
• per-priority admission patterns

Admission events reveal system stress.

Failure Modes

Poor admission control can cause:

• unnecessary rejection of valid traffic
• unfair starvation of certain users
• oscillatory behavior
• masking of capacity planning issues

Admission must be tuned carefully.

Practical Design Guidelines

• define hard capacity limits explicitly
• prioritize tail latency protection
• separate critical from non-critical traffic
• log and audit admission decisions
• combine with fallback and degradation strategies

Admission is a policy decision, not just infrastructure.

Common Pitfalls

• allowing unbounded queues
• rejecting traffic too late
• treating admission as an infra-only concern
• ignoring priority and fairness
• failing to test overload scenarios

Overload must be designed for.

Summary Characteristics

Related Concepts

• Generalization & Evaluation
• Queueing Effects in ML Systems
• SLA-Aware Inference Policies
• Graceful Degradation
• Budget-Constrained Inference
• Tail Latency Metrics
• Throughput vs Latency
• Efficiency Governance