Short Definition
Sparse neural networks use fewer active connections.
Definition
Sparse neural networks deliberately reduce the number of non-zero connections between neurons. By enforcing sparsity, models become more efficient, interpretable, and sometimes more robust.
Sparsity can be applied during or after training.
Why It Matters
Sparse models reduce memory usage and computational cost.
How It Works (Conceptually)
- Remove or zero unimportant weights
- Retain only essential connections
- Optionally retrain after pruning
Minimal Python Example
if abs(weight) < threshold: weight = 0.0Common Pitfalls
- Over-pruning
- Assuming sparsity always improves accuracy
- Ignoring hardware constraints
Related Concepts
- Pruning
- Efficiency
- Interpretability
Neural Network Lexicon 