Model Ensembling

Ensemble Methods

Combining multiple models improves performance.

Voting Classifier

VotingClassifier([('lr', lr), ('rf', rf), ('svm', svc)], voting='soft').

Soft voting uses predicted probabilities. Hard voting uses class predictions.

Stacking

StackingClassifier([models], final_estimator=LogisticRegression()).

Base model predictions become features for meta-model.

Bagging and Boosting

BaggingClassifier uses bootstrap sampling. RandomForest is bagging with random feature selection.

Boosting builds sequentially, correcting errors. AdaBoost, GradientBoosting implement boosting.

Key Takeaways

1. Voting averages predictions from multiple models
2. Stacking uses predictions as features for meta-model
3. Boosting sequentially corrects errors