Feature engineering and feature selection are critical processes in building effective machine learning workflows. This article explores interaction terms, Recursive Feature Elimination (RFE), Lasso L1 regularization, and leak-proof Scikit-Learn pipelines.

1. Feature Engineering: Interaction Terms and Polynomials

Creating polynomial features and interaction terms allows linear models to capture non-linear relationships and complex feature interactions.

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True)
X_poly = poly.fit_transform(X)

2. Feature Selection Methods

Recursive Feature Elimination (RFE)

RFE greedily removes the least important features recursively until the desired number of features is reached.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
selector = RFE(estimator=LogisticRegression(), n_features_to_select=5)
X_selected = selector.fit_transform(X, y)

Lasso L1 Regularization

Lasso penalty forces coefficients of non-essential features to zero, effectively performing feature selection.

from sklearn.linear_model import LassoCV
lasso = LassoCV().fit(X, y)
selected_features = X.columns[lasso.coef_ != 0]

3. Preventing Data Leakage with Pipelines

To avoid data leakage, preprocessing steps must be fitted solely on training splits using Scikit-Learn Pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier())
])
pipeline.fit(X_train, y_train)