Architecture and representation describe how neural networks are structured and how information is encoded internally.
They determine what kinds of patterns a model can express, what assumptions it makes about data, and how efficiently it can learn.
This section of the Neural Network Lexicon focuses on model structure, layer composition, and learned representations. It explains why different architectures exist, how representations emerge during training, and how design choices shape learning behavior.
Understanding architecture and representation is essential for choosing the right model for a problem—and for interpreting what a trained model has actually learned.
Model Architecture Fundamentals
A neural network’s architecture defines how computation flows and how parameters are organized.
The following entries explain the core building blocks of neural networks:
- Neurons
- Layers
- Hidden Layers
- Output Layers
- Activation Functions
These concepts describe how signals are transformed and combined as they pass through a network.
Architectural Design Choices
Different tasks require different structural assumptions. Architecture encodes inductive bias—the assumptions a model makes before seeing data.
This group focuses on design-level concepts:
- Model Architecture
- Model Capacity
- Inductive Bias
- Depth vs Width
These entries explain why some architectures generalize well while others overfit or underfit.
Representation Learning
Neural networks learn internal representations that transform raw inputs into increasingly abstract features.
The following entries focus on how representations are formed and used:
- Representation Learning
- Embeddings
- Feature Hierarchies
These concepts explain why neural networks are effective at handling complex, high-dimensional data such as images, text, and audio.
Specialized Architectures
Certain architectural patterns are designed for specific data types or tasks.
This section includes entries such as:
- Convolutional Neural Networks (CNNs)
- Recurrent Neural Networks (RNNs)
- Transformers
These architectures embed domain-specific assumptions that shape how information is processed.
Architectural Regularization and Control
Architecture also interacts with regularization and control mechanisms that influence learning behavior.
This group focuses on structural techniques that affect training and generalization:
- Dropout
- Weight Sharing
- Parameter Tying
These entries explain how architecture can constrain learning in beneficial ways.
How to Use This Section
If you are new to neural networks, start with Neurons, Layers, and Activation Functions to understand basic computation.
If you are choosing or designing a model, explore Model Capacity, Inductive Bias, and Depth vs Width.
If you want to understand what a trained model has learned, focus on Representation Learning and Embeddings.
Architecture and representation define the expressive limits of learning.
They shape not only what a model can learn, but also how it learns and why it behaves the way it does.
Neural Network Lexicon 