Neural Network Lexicon

Short Definition

The convolution operation applies a small, learnable filter across an input to extract local patterns using shared weights.

Definition

The convolution operation is a mathematical procedure in which a kernel (or filter) is slid across an input tensor to produce a feature map. At each spatial location, the kernel computes a weighted sum of local input values, capturing structured patterns such as edges, textures, or temporal motifs.

Convolution is localized pattern matching with shared parameters.

Why It Matters

Convolution enables neural networks to process high-dimensional structured data efficiently by exploiting locality and parameter sharing. It drastically reduces the number of parameters compared to fully connected layers while preserving spatial structure.

Convolution makes deep vision feasible.

Core Elements of Convolution

A convolution operation is defined by:

• Kernel (filter): small learnable weight matrix
• Stride: step size when sliding the kernel
• Padding: added border values to control output size
• Input channels: depth of the input tensor
• Output channels: number of learned feature maps

Each choice affects representation and computation.

Mathematical Intuition

At each position:

Output(x, y) = Σ Kernel(i, j) × Input(x+i, y+j)

The same kernel is reused across all spatial locations.

Weight sharing encodes translation structure.

Minimal Conceptual Illustration

Input → Sliding Kernel → Feature Map

Local Receptive Fields

Convolution restricts each output unit to a local receptive field in the input. Deeper layers increase the effective receptive field, allowing the network to capture more global structure hierarchically.

Local first, global later.

Translation Equivariance

Convolution is translation equivariant: shifting the input results in a corresponding shift in the output feature map. This property underlies robustness to object location in images.

Structure is preserved under movement.

Parameter Sharing

Unlike fully connected layers, convolution reuses the same kernel weights across all spatial locations, dramatically reducing parameter count and improving generalization.

Fewer parameters, stronger bias.

Variants of Convolution

Common convolution variants include:

• 1D convolution: sequences and time series
• 2D convolution: images and spatial grids
• 3D convolution: video and volumetric data
• Dilated convolution: expanded receptive fields
• Depthwise separable convolution: efficiency-focused
• Grouped convolution: channel partitioning

Convolution adapts to data structure.

Convolution vs Fully Connected Operation

Bias enables efficiency.

Role in Feature Learning

Convolutional layers learn increasingly abstract features:

• early layers: edges, corners
• middle layers: textures, motifs
• deep layers: objects or concepts

Features emerge hierarchically.

Limitations

Convolution may struggle when:

• long-range dependencies dominate
• global context is critical
• spatial invariance is harmful
• data lacks grid structure

Bias can misalign with reality.

Relationship to Modern Architectures

Convolution remains foundational but is often combined with:

• residual connections
• attention mechanisms
• normalization layers
• hybrid CNN–Transformer models

Convolution is no longer alone.

Common Pitfalls

• assuming convolution implies invariance (it is equivariant)
• excessive kernel sizes without benefit
• ignoring padding effects
• misinterpreting learned filters
• assuming convolution generalizes under shift

Understanding the operation matters.

Summary Characteristics

Related Concepts

• Architecture & Representation
• Convolutional Neural Network (CNN)
• Receptive Fields
• Feature Maps
• Inductive Bias
• Pooling Layers
• Residual Connections
• Vision Models