Neural Network Lexicon

Short Definition

Residual Networks (ResNets) are deep neural network architectures that use skip (residual) connections to enable stable training of very deep models.

Definition

Residual Networks introduce residual connections—direct pathways that bypass one or more layers—allowing layers to learn residual functions relative to their inputs. Instead of learning a full transformation, layers learn how to adjust an identity mapping.

Depth becomes trainable.

Why It Matters

As networks grow deeper, optimization becomes difficult due to vanishing gradients and degradation of training accuracy. ResNets address this by preserving gradient flow and enabling effective learning in networks with dozens or hundreds of layers.

ResNets made extreme depth practical.

Core Idea: Residual Learning

A residual block computes:

y = F(x) + x

where:

• x is the input
• F(x) is the residual function learned by stacked layers

Learning deviations is easier than learning from scratch.

Minimal Conceptual Illustration

Input ───────────┐
      ┌─ Layers ─┤→ Add → Output
      └──────────┘

Residual Connections

Residual connections:

• preserve information across layers
• improve gradient flow
• reduce optimization difficulty
• stabilize very deep networks

Identity shortcuts act as highways.

Why Residuals Help Optimization

Residual connections:

• mitigate vanishing gradients
• reduce sensitivity to initialization
• make layers optional (can learn near-zero)
• prevent degradation with depth

Depth no longer guarantees failure.

Basic ResNet Block Variants

Basic Block

Used in shallow ResNets (e.g., ResNet-18, ResNet-34).

• two 3×3 convolutions

Bottleneck Block

Used in deeper ResNets (e.g., ResNet-50+).

• 1×1 → 3×3 → 1×1 convolutions
• reduces computation while increasing depth

Structure scales depth efficiently.

Residual Networks vs Plain CNNs

Skip connections change the game.

Relationship to Receptive Fields

Deeper ResNets increase effective receptive fields without severe optimization penalties, enabling richer hierarchical representations.

Depth with stability expands context.

Relationship to Normalization

ResNets are often combined with normalization layers (e.g., BatchNorm) to further stabilize training. Pre-norm and post-norm variants affect gradient flow and robustness.

Ordering matters.

Impact on Modern Architectures

Residual connections are now ubiquitous:

• CNNs
• Transformers
• diffusion models
• graph neural networks

Residual learning is a general principle.

Limitations of ResNets

ResNets may:

• encourage unnecessary depth
• obscure interpretability
• increase inference cost
• still struggle with global reasoning

Depth is not a substitute for design.

Common Pitfalls

• stacking residual blocks without task need
• ignoring normalization placement
• assuming deeper is always better
• overlooking calibration and robustness
• misinterpreting skip connections as ensembling

Architecture amplifies assumptions.

Summary Characteristics

Related Concepts

• Architecture & Representation
• Residual Connections
• Convolutional Neural Network (CNN)
• Optimization Stability
• Vanishing Gradients
• Normalization Layers
• Pre-Norm vs Post-Norm Architectures
• Deep Learning Architectures