Neural Network Lexicon

Short Definition

The attention mechanism is a neural computation strategy that allows a model to dynamically focus on the most relevant parts of its input when producing an output.

Definition

The attention mechanism enables neural networks to assign different importance weights to different input elements when generating predictions. Instead of compressing an entire input sequence into a fixed representation, attention computes weighted combinations of input states, allowing the model to selectively emphasize relevant information.

Representation becomes selective rather than compressed.

Why It Matters

Early Seq2Seq models relied on a single fixed-size context vector, creating a bottleneck for long sequences. Attention:

• removes fixed-length compression constraints
• improves long-range dependency modeling
• increases interpretability through attention weights
• enables parallelizable architectures (Transformers)

Attention removed the information bottleneck.

Core Idea

Given:

• Query (Q)
• Keys (K)
• Values (V)

Attention computes:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

Where:

• Q determines what we are looking for
• K determines where to look
• V contains the information to retrieve

Attention is weighted retrieval.

Minimal Conceptual Illustration

Input States:  h₁   h₂   h₃   h₄
                    ↑
              Higher weight
Output at step t = weighted sum of h₁...h₄

The model decides where to focus.

Types of Attention

Additive (Bahdanau) Attention

Uses a learned feedforward network to compute alignment scores.

Dot-Product Attention

Uses similarity between query and key vectors.

Scaled Dot-Product Attention

Introduces scaling by √d to stabilize gradients (used in Transformers).

Scaling improves numerical stability.

Relationship to Seq2Seq

In classic attention-based Seq2Seq:

• The encoder produces hidden states for all input tokens.
• The decoder attends to these states at each output step.
• Different parts of the input are emphasized dynamically.

Attention replaces fixed compression.

Attention vs Recurrence

Attention bypasses temporal chains.

Computational Properties

• Complexity grows with sequence length (O(n²) in full attention).
• Enables parallel training across positions.
• Removes strict sequential dependency in computation.

Parallelism is transformative.

Interpretability

Attention weights provide:

• soft alignment between inputs and outputs
• insight into model focus
• limited but useful interpretability signals

Weights indicate relevance, not causality.

Limitations

• Quadratic complexity in long sequences
• Attention weights may not reflect causal importance
• Can overfit to spurious correlations

Attention is powerful but not magical.

From Attention to Transformers

Transformers generalize attention by:

• removing recurrence entirely
• using self-attention across all tokens
• stacking multi-head attention layers

Attention became the primary computation mechanism.

Practical Considerations

When using attention:

• monitor memory scaling with sequence length
• combine with positional encoding
• consider sparse or linear attention variants for long sequences

Efficiency becomes critical at scale.

Common Pitfalls

• assuming attention equals explainability
• ignoring computational cost
• misunderstanding Q–K–V semantics
• overlooking positional encoding requirements

Attention requires careful design.

Summary Characteristics

Related Concepts

• Architecture & Representation
• Sequence-to-Sequence Models (Seq2Seq)
• Bidirectional RNNs
• Self-Attention
• Transformers
• Positional Encoding
• Multi-Head Attention