Fundamentals of Regression Analysis
Regression analysis represents the cornerstone of statistical modeling, providing methods for understanding relationships between variables and making predictions. From simple models relating one predictor to an outcome to complex systems incorporating numerous variables, regression provides a versatile framework applicable across scientific and business domains.
The origins of regression trace to Francis Galton's work in the 19th century studying relationships between parent and offspring characteristics. The term "regression" originally described the tendency for extreme values to move toward the average in successive generations. Modern regression encompasses a broad class of techniques far beyond Galton's original formulation.
Regression serves two primary purposes: explaining relationships among variables and predicting future outcomes. Both purposes are valuable in data science applications. Understanding what drives outcomes enables intervention and optimization. Accurate prediction enables forecasting and decision-making under uncertainty.
Simple Linear Regression
Simple linear regression examines the relationship between one predictor variable and one outcome variable. Despite its simplicity, it provides fundamental concepts that extend to more complex models.
Model Specification
The simple linear regression model assumes: Y = β₀ + β₁X + ε, where Y is the outcome, X is the predictor, β₀ is the intercept, β₁ is the slope, and ε is the error term representing unexplained variation.
The model assumes errors have mean zero, constant variance (homoscedasticity), and are uncorrelated. These assumptions enable reliable inference about coefficients and predictions.
The slope coefficient (β₁) indicates the expected change in Y for a one-unit change in X. The intercept (β₀) indicates the expected Y value when X equals zero. Both parameters have meaningful interpretations in context.
Parameter Estimation
Ordinary Least Squares (OLS) estimates regression coefficients by minimizing the sum of squared residuals. This criterion produces estimates with desirable properties including unbiasedness and minimum variance among linear estimators under certain conditions.
The OLS slope estimate equals the covariance of X and Y divided by the variance of X. This formula reveals that the slope depends on the strength of linear association. Zero covariance implies zero slope.
Estimation produces point estimates for coefficients. These estimates reveal estimated relationships but do not convey uncertainty. Standard errors enable inference about underlying population coefficients.
Inference and Interpretation
Hypothesis tests evaluate whether predictors significantly relate to outcomes. The t-test for slope tests H₀: β₁ = 0 (no linear relationship) against H₁: β₁ ≠ 0. Significant results indicate the relationship is likely not due to chance.
Confidence intervals for coefficients provide ranges capturing true parameter values with specified confidence. The interval width reflects estimation uncertainty: smaller samples and more variable data produce wider intervals.
The coefficient of determination (R²) measures the proportion of outcome variance explained by the predictor. Values range from 0 to 1, with higher values indicating better fit. R² = 0.17 means X explains 17% of Y variance.
Prediction and Prediction Intervals
Regression enables predicting outcomes for new predictor values. Point predictions use the fitted model with new X values. The predicted value equals Ŷ = b₀ + b₁X_new.
Prediction intervals quantify uncertainty for individual predictions. They account for both estimation uncertainty and outcome variability. Wider intervals for X values far from the sample mean reflect greater uncertainty.
Confidence intervals for mean responses (expected Y at given X) are narrower than prediction intervals for individual outcomes. The difference reflects reduced uncertainty when averaging over individuals.
Multiple Linear Regression
Multiple linear regression extends simple regression to include multiple predictors. This extension enables controlling for confounders, examining unique contributions of predictors, and building more complete models.
Model Specification and Estimation
The multiple regression model is: Y = β₀ + β₁X₁ + β₂X₂ + ... + βₚXₚ + ε, where p is the number of predictors. Matrix notation efficiently represents this model for computation.
Estimation via OLS minimizes sum of squared residuals. The solution involves matrix operations. Software handles computation, but understanding the underlying logic remains important.
Each coefficient represents the expected change in Y for a one-unit change in that predictor, holding other predictors constant. This ceteris paribus interpretation distinguishes multiple regression from bivariate relationships.
Multicollinearity
Multicollinearity occurs when predictors are highly correlated. This creates estimation difficulties including inflated standard errors, unstable coefficient estimates, and sensitivity to small data changes.
Detection uses variance inflation factors (VIFs). VIF > 5 or 10 indicates problematic multicollinearity requiring attention. Correlation matrices reveal bivariate correlations that might cause problems.
Remediation strategies include removing redundant predictors, combining correlated variables, or using regularization methods that handle multicollinearity more robustly.
Model Building
Model building involves selecting which predictors to include. Tradeoffs exist between model complexity (more predictors) and parsimony (fewer predictors). Overfit models perform well on development data but poorly on new data.
Automated selection procedures include forward selection (adding predictors sequentially), backward elimination (removing predictors sequentially), and stepwise methods (combining both). These procedures can guide model selection but require careful validation.
Domain knowledge should guide predictor selection. Theory-driven predictors are often more reliable than data-driven selections. Including relevant confounders is essential for unbiased estimates.
Model Diagnostics
Model diagnostics evaluate whether regression assumptions hold and whether the model adequately represents the data. Diagnostic findings guide model improvement and affect interpretation validity.
Residual Analysis
Residuals (observed - predicted values) reveal model shortcomings. Plots of residuals against predicted values check assumptions. Patterns in residuals indicate specification problems.
Heteroscedasticity (non-constant variance) appears as residual spread changing with predicted values. This violates assumption and affects inference. Transformations or weighted regression might address this issue.
Non-linearity appears as curved patterns in residual plots. This indicates non-linear relationships not captured. Adding polynomial terms or transforming variables might improve specification.
Outlier and Influence Analysis
Outliers are observations with unusual Y values given their X values. They might indicate measurement errors, special circumstances, or important extreme cases. Detection uses residual statistics.
Influence measures assess how much each observation affects fitted results. Cook's distance combines residual and leverage information. Highly influential points warrant careful examination.
Handling outliers depends on their cause. Data errors should be corrected if possible. Genuine unusual cases might warrant separate analysis or robust methods.
Assumption Testing
Formal tests complement visual diagnostics. The Breusch-Pagan test evaluates heteroscedasticity. The Shapiro-Wilk test evaluates normality of residuals. The Durbin-Watson test evaluates autocorrelation.
Tests have limitations. With large samples, minor violations achieve statistical significance without practical importance. With small samples, tests lack power to detect meaningful violations.
Categorical Predictors
Regression handles categorical predictors through coding schemes that translate categories into numerical variables suitable for modeling.
Dummy Coding
Dummy coding creates binary (0/1) variables for categorical predictors. For a variable with k categories, k-1 dummy variables represent the categories. One category serves as the reference.
The coefficient for a dummy variable represents the difference between that category and the reference category. Interpretation compares each category to the reference.
Example: If region has categories (North, South, East, West), three dummy variables represent three categories compared to the reference category (say, North). Coefficients show how each region compares to North.
Interaction Effects
Interactions model how the effect of one predictor depends on another predictor's value. They enable more nuanced relationships than main effects alone.
Interaction terms multiply predictors to create new variables. The coefficient measures how the relationship between one predictor and outcome changes across values of another predictor.
Interpretation requires examining simple effects at different levels. Graphical displays help visualize interaction patterns. Complex interactions can be challenging to interpret.
Nonlinear Relationships
When relationships are not linear, various approaches capture non-linear patterns while maintaining regression framework.
Polynomial Regression
Polynomial regression adds squared, cubed, or higher-order terms of a predictor. This approach can fit curved relationships while retaining linear regression estimation and interpretation structure.
Degree selection involves tradeoff between fit and overfitting. Higher degrees fit data more closely but may capture noise rather than signal. Cross-validation helps select appropriate complexity.
Polynomial terms can approximate many smooth relationships but might perform poorly near boundaries or for highly non-smooth functions.
Log Transformations
Log transformations of outcomes or predictors can linearize relationships. Log-log models interpret elasticities directly. Log-linear models interpret growth rates. Logit models handle binary outcomes.
Interpretation changes with transformations. Coefficients represent proportional rather than absolute effects. Exponentiation transforms coefficients to interpretable scales.
Transformations might create new problems. Log of zero is undefined. Back-transforming predictions requires care to get appropriate intervals.
Regularization Methods
Regularization addresses overfitting by constraining coefficient estimates. These methods are particularly valuable with many predictors or when multicollinearity exists.
Ridge Regression
Ridge regression adds a penalty term to least squares: minimize sum of squared residuals + λ × sum of squared coefficients. The penalty shrinks coefficients toward zero, reducing variance.
The shrinkage parameter λ controls penalty strength. Larger λ produces more shrinkage. Optimal λ selection uses cross-validation to minimize prediction error.
Ridge handles multicollinearity by producing stable estimates when predictors are correlated. It cannot set coefficients exactly to zero, so all predictors remain in the model.
Lasso Regression
Lasso adds a penalty using absolute values: minimize sum of squared residuals + λ × sum of absolute coefficients. This penalty can set coefficients exactly to zero, performing variable selection.
Lasso is useful when many predictors might be irrelevant. It selects a subset of predictors while estimating their effects. This produces more interpretable models.
Elastic net combines ridge and lasso penalties, capturing benefits of both approaches when predictors are correlated.
Key Takeaways
- Regression provides tools for understanding relationships and making predictions
- Simple linear regression examines one predictor, multiple regression extends to multiple predictors
- Model diagnostics evaluate assumptions and identify problems
- Categorical predictors require appropriate coding schemes
- Regularization addresses overfitting with many predictors
- Interpretation requires careful attention to model form and coefficients