Short Definition
The Attention Mechanism is a neural network technique that allows models to dynamically focus on the most relevant parts of an input when producing an output.
It enables models to selectively weight information instead of processing all inputs equally.
Definition
In many machine learning tasks, not all parts of an input are equally important.
For example, when translating a sentence, a model generating a word in the target language should focus on the corresponding word in the source sentence.
The attention mechanism computes a weighted combination of input representations:
[
c = \sum_{i=1}^{n} \alpha_i h_i
]
Where:
- (h_i) = representation of input element (i)
- (\alpha_i) = attention weight
- (c) = context vector used for prediction
The weights (\alpha_i) determine how strongly each input contributes to the final representation.
Core Idea
Instead of compressing an entire input sequence into a single vector, attention allows the model to look back at all relevant inputs whenever needed.
Conceptually:
Input sequence → attention weights → weighted combination
This allows the model to dynamically retrieve useful information from the input.
Minimal Conceptual Illustration
Example sentence:
The cat sat on the mat
When predicting the word sat in another language, the model may assign attention like:
The cat sat on the mat
0.05 0.40 0.35 0.10 0.05 0.05
The model focuses primarily on cat and sat.
Computing Attention
The most common formulation computes attention scores using a compatibility function between a query and keys.
[
score(q, k_i)
]
These scores are normalized using softmax:
[
\alpha_i = \frac{e^{score(q,k_i)}}{\sum_j e^{score(q,k_j)}}
]
The final context vector is:
[
c = \sum_i \alpha_i v_i
]
Where:
- (q) = query
- (k_i) = key
- (v_i) = value
This Query–Key–Value framework is central to modern attention models.
Historical Context
Attention was first introduced in neural machine translation (Bahdanau et al., 2014).
Early sequence-to-sequence models used a single fixed representation for the entire input sentence, which limited performance.
Attention solved this bottleneck by allowing the decoder to access all encoder states.
Self-Attention vs Cross-Attention
Different types of attention mechanisms exist.
Self-Attention
A token attends to other tokens in the same sequence.
Used in Transformers.
Cross-Attention
One sequence attends to another sequence.
Used in encoder–decoder models.
Importance in Modern AI
Attention mechanisms have become central to modern machine learning architectures.
They are used in:
- Transformers
- Large language models
- Vision transformers
- Multimodal models
- speech recognition systems
The Transformer architecture is built entirely around attention mechanisms.
Advantages
Attention provides several benefits:
- dynamic context selection
- improved long-range dependency modeling
- interpretability through attention weights
- improved performance in sequence tasks
It removes the need for strict sequential processing.
Limitations
Attention mechanisms can be computationally expensive.
For sequence length (n):
[
O(n^2)
]
This quadratic scaling makes attention costly for very long sequences.
Many research efforts focus on improving attention efficiency.
Summary
The attention mechanism allows neural networks to dynamically focus on relevant parts of the input when generating outputs.
By computing weighted combinations of input representations, attention enables flexible context modeling and forms the foundation of modern architectures such as Transformers.
Related Concepts
- Self-Attention
- Transformer Architecture
- Multi-Head Attention
- Encoder–Decoder Models
- Positional Encoding
- Sequence-to-Sequence Models
- Cross-Attention
Neural Network Lexicon 