Quantifying ML Uncertainty
Know what you don't know.
Aleatoric Uncertainty
Inherent data noise. Cannot be reduced. Epistemic uncertainty: model uncertainty. Can be reduced with more data.
Methods
Ensemble methods: variance across models. Monte Carlo dropout: variance across forward passes. Deep ensembles: multiple models with different seeds.
Calibration
Platt scaling, temperature scaling. Reliability diagrams show calibration.
Key Takeaways
- Aleatoric vs epistemic uncertainty
- Ensembles capture model uncertainty
- Calibration improves probability estimates