Advanced Ensemble Techniques
More sophisticated ensemble approaches.
Gradient Boosting Algorithms
XGBoost: regularized objective, tree pruning. LightGBM: histogram-based, GOSS sampling.
CatBoost: ordered target encoding. All implement gradient boosting with improvements.
Hyperparameter Tuning
n_estimators: more trees, better performance. learning_rate: lower needs more trees.
max_depth: deeper trees, more complex. subsample: row sampling prevents overfitting.
Feature Importance
Gain-based: improvement from splits. Cover-based: samples affected. SHAP-based: consistent feature importance.
Key Takeaways
- XGBoost, LightGBM, CatBoost are advanced boosters
- Key: n_estimators, learning_rate, max_depth
- Multiple importance types available