Advanced Optimization

Advanced Optimizers

Better optimization algorithms.

AdamW

Adam with weight decay. L2 regularization separate from adaptive learning rate.

LAMB

Layer-wise Adaptive Moments for Batch. Large batch training. Different LR per layer.

Sharpness-Aware Minimization

SAM: seeks flat minima. Adversarial perturbation improves generalization.

Key Takeaways

1. AdamW: Adam + proper weight decay
2. LAMB: for large batch training
3. SAM: improves generalization