Self-Supervised Learning

Learning from Unlabeled Data

Use data itself as supervision.

Contrastive Learning

Positive pairs: augmented views. Negative pairs: different samples. InfoNCE loss maximizes mutual information.

SimCLR, MoCo, BYOL: contrastive frameworks.

Masked Prediction

BERT: mask tokens, predict. MAE: mask patches, reconstruct. DALL-E: masked pixel prediction.

Downstream Tasks

Pre-train on large unlabeled data. Fine-tune on labeled data.

Key Takeaways

1. Use data as supervision
2. Contrastive and masked prediction
3. Pre-train then fine-tune