Neural Network Lexicon

Short Definition

Sequence-to-Sequence (Seq2Seq) models are neural architectures that map an input sequence to an output sequence, potentially of different length.

Definition

Sequence-to-Sequence (Seq2Seq) models are encoder–decoder architectures designed to transform one sequence into another. They were originally implemented using recurrent neural networks (RNNs), LSTMs, or GRUs, and later extended with attention mechanisms and transformers.

Input sequence → representation → output sequence.

Why It Matters

Many real-world tasks require sequence transformation:

• machine translation
• speech-to-text
• text summarization
• dialogue generation
• time-series forecasting

Seq2Seq made end-to-end sequence learning practical.

Core Architecture

A classic Seq2Seq model consists of:

1. Encoder
   Processes the input sequence and compresses it into a context representation.
2. Decoder
   Generates the output sequence step-by-step from the encoded representation.

Input Sequence → Encoder → Context Vector → Decoder → Output Sequence

The model learns to translate sequences between domains.

Minimal Conceptual Illustration

Encoder:
x₁ → h₁
x₂ → h₂
x₃ → h₃ → Context
Decoder:
Context → y₁
y₁ → y₂
y₂ → y₃

Encoding compresses; decoding generates.

The Context Vector Problem

Early Seq2Seq models compressed the entire input into a single fixed-length vector.
For long sequences, this caused information bottlenecks and degraded performance.

Compression limits capacity.

Attention Mechanism Extension

Attention mechanisms addressed this limitation by allowing the decoder to:

• access all encoder states
• dynamically focus on relevant parts of the input
• avoid fixed-size bottlenecks

Attention removed the compression constraint.

Training Strategy

Seq2Seq models are typically trained using:

• Backpropagation Through Time (BPTT)
• Teacher Forcing
• Cross-entropy loss over predicted tokens

Training remains autoregressive.

Exposure Bias

Because teacher forcing is commonly used:

• training conditions differ from inference
• small inference errors can accumulate
• exposure bias may degrade long outputs

Inference realism must be evaluated explicitly.

Variants

Common Seq2Seq variants include:

• RNN-based encoder–decoder
• LSTM-based Seq2Seq
• GRU-based Seq2Seq
• Attention-based Seq2Seq
• Transformer-based Seq2Seq

Modern large language models are transformer-based Seq2Seq systems.

Seq2Seq vs Transformer

Transformers generalized Seq2Seq.

Applications

Seq2Seq models have powered:

• Google Neural Machine Translation (GNMT)
• Early chatbot systems
• Speech recognition pipelines
• Neural text summarizers

They marked the transition to neural language systems.

Practical Considerations

When building Seq2Seq models:

• handle variable-length sequences carefully
• apply attention for long inputs
• manage teacher forcing vs inference gap
• monitor long-sequence degradation

Sequence length remains a challenge.

Common Pitfalls

• relying on a fixed context vector for long sequences
• ignoring exposure bias
• evaluating only with teacher forcing
• not testing long output stability

Sequence generation magnifies small errors.

Summary Characteristics

Related Concepts

• Architecture & Representation
• Recurrent Neural Network (RNN)
• Long Short-Term Memory (LSTM)
• Gated Recurrent Unit (GRU)
• Backpropagation Through Time (BPTT)
• Teacher Forcing
• Exposure Bias
• Attention Mechanism
• Transformers