A/B Testing Fundamentals
A/B testing compares variants in experiments.
Experimental Design
Random assignment: users randomly assigned to control/treatment.
Sample size: power analysis determines required users. Minimum detectable effect matters.
Statistical Analysis
Two-sample t-test compares means. Chi-square for proportions.
Confidence intervals show uncertainty. P-values indicate significance.
Pitfalls
Novelty effect, change aversion. Seasonality. Selection bias in analysis.
Key Takeaways
- Randomization ensures valid comparison
- Sample size determined by power analysis
- Monitor for validity threats