Advanced Feature Engineering

Feature Creation

Creating informative features often matters more than algorithm choice.

Polynomial Features

PolynomialFeatures creates interaction and higher-order terms. include_bias=False omits constant term.

degree=2 creates squared terms and interactions. degree=3 adds cubic terms.

This increases dimensionality; use with regularization.

Binning

Binning converts continuous features to categories. KBinsDiscretizer with encode='ordinal'.

Strategy: 'uniform' (equal-width bins), 'quantile' (equal-frequency), 'kmeans' (cluster-based).

Bin boundaries might need domain knowledge.

Feature Selection

SelectKBest selects top k features. SelectFromModel uses model importance. RFE recursively eliminates features.

Key Takeaways

1. Polynomial features capture non-linear relationships
2. Binning handles non-linear effects discretely
3. Feature selection reduces dimensionality and improves models