Neural Network Lexicon

Transformer Architecture

Short Definition

The Transformer Architecture is a neural network design for sequence modeling that replaces recurrence and convolution with self-attention mechanisms, enabling parallel computation and efficient modeling of long-range dependencies.

Transformers form the foundation of modern large language models.

Definition

The Transformer architecture processes sequences by allowing each token to attend to all other tokens in the sequence.

Unlike RNNs or CNNs, Transformers do not rely on sequential recurrence or sliding filters. Instead, they compute relationships between tokens using self-attention.

A typical Transformer layer consists of:

  1. Multi-Head Self-Attention
  2. Feedforward Neural Network
  3. Residual Connections
  4. Layer Normalization

These components are stacked to form deep models capable of learning complex patterns in sequential data.

Core Idea

The key innovation of the Transformer is self-attention.

Instead of processing tokens sequentially, the model computes relationships between all tokens simultaneously.

Given an input sequence:

x₁, x₂, x₃, …, xₙ

each token computes attention weights relative to every other token.

This allows the model to capture dependencies regardless of distance.

Minimal Conceptual Illustration

Traditional RNN processing:

x1 → x2 → x3 → x4 → x5

Sequential dependency.

Transformer processing:

x1 ↔ x2 ↔ x3 ↔ x4 ↔ x5

Every token can attend to every other token.

Self-Attention Mechanism

Self-attention uses three learned projections:

  • Query (Q)
  • Key (K)
  • Value (V)

The attention computation is:

[
Attention(Q, K, V) =
softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V
]

Where:

  • (QK^T) computes similarity between tokens
  • (d_k) scales the dot product
  • softmax produces attention weights

The result is a weighted combination of values.

Multi-Head Attention

Instead of computing one attention function, Transformers use multiple attention heads.

Each head learns different relational patterns.

[
MultiHead(Q,K,V) = Concat(head_1, …, head_h)W^O
]

Multiple heads allow the model to capture different contextual relationships simultaneously

Feedforward Network

After attention, each token passes through a position-wise feedforward network:

[
FFN(x) = max(0, xW_1 + b_1)W_2 + b_2
]

This component introduces nonlinearity and expands representational capacity.

Positional Encoding

Because Transformers process tokens in parallel, they require a way to represent sequence order.

Positional encodings are added to token embeddings:

[
x_i = embedding_i + positional_encoding_i
]

Common methods include:

  • sinusoidal encodings
  • learned position embeddings
  • rotary positional embeddings

This enables the model to reason about order.

Transformer Layer Structure

Each Transformer block typically follows this structure:

Input
↓
Self-Attention
↓
Residual Connection
↓
Layer Normalization
↓
Feedforward Network
↓
Residual Connection
↓
Layer Normalization

Stacking many layers enables deep representation learning.

Encoder vs Decoder Transformers

Two major variants exist.

Encoder

Used for understanding tasks.

Examples:

  • BERT
  • Vision Transformers

Encoders attend bidirectionallyDecoder

Used for generative tasks.

Examples:

  • GPT models

Decoders use causal masking to prevent access to future tokens.

Encoder–Decoder

Used in sequence-to-sequence tasks such as translation.

Examples:

  • original Transformer model
  • T5

The decoder attends both to previous outputs and encoder representations.

Advantages

Transformers provide several benefits:

  • Parallel computation across tokens
  • Strong long-range dependency modeling
  • High scalability
  • Efficient GPU utilization
  • Flexible architecture

These properties enabled scaling to billions and trillions of parameters.

Limitations

Transformers also have limitations.

Self-attention complexity scales as:

[
O(n^2)
]

with sequence length (n).

This makes long sequences computationally expensive.

Research explores alternatives such as

Historical Context

The Transformer architecture was introduced in the paper:

“Attention Is All You Need” (Vaswani et al., 2017).

It replaced RNN-based models in many NLP tasks and eventually became the dominant architecture for large-scale AI systems.

Role in Modern AI

Transformers power many modern systems including:

  • Large Language Models
  • Multimodal models
  • Vision Transformers
  • Code generation models
  • Retrieval systems

Their scalability has enabled major advances in AI capabilities.

Summary

The Transformer Architecture replaces sequential recurrence with self-attention, enabling parallel processing and powerful long-range reasoning.

Its modular structure of attention, feedforward networks, residual connections, and normalization has made it the dominant architecture for modern AI systems.

Related Concepts