Neural Network Pruning

Model Pruning

Remove unnecessary weights.

Types

Weight pruning: remove individual weights. Structured pruning: remove neurons/filters.

Magnitude Pruning

Remove weights with smallest magnitude. Often done iteratively. Can achieve 90%+ sparsity.

Lottery Ticket Hypothesis

Dense networks contain sparse sub-networks that train to same performance. Find winning tickets.

Key Takeaways

1. Remove unnecessary weights
2. Magnitude-based common
3. Lottery tickets: sparse can match dense