Features and Pipelines

Introduction

In real projects, most of the work and most of the gains come from the data, not the algorithm. Raw data is messy: different scales, text categories, missing values. Models need numbers in a suitable form.

Scaling numbers

Many models, such as logistic regression, work poorly when features have very different sizes, for example age (0 to 100) and income (0 to 1,000,000). Standardisation rescales each feature to a mean of 0 and a spread of 1.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

On the breast cancer data, the same logistic regression scored 0.986 test accuracy with scaling and 0.958 without it, in the same run. Tree-based models do not need scaling.

Encoding categories

Models need numbers, so text categories must be converted. One-hot encoding makes a yes/no column for each category:

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown="ignore")

Do not simply number categories 1, 2, 3 unless they have a real order, because the model would treat 3 as bigger than 1.

Missing values

Options include dropping rows with gaps, or imputing, which fills them with a typical value such as the median:

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")

Sometimes the fact that a value is missing is itself informative, so consider adding a "was missing" column.

Data leakage: the silent killer

Leakage is when information from outside the training data sneaks into the model, making scores look great and then failing in real use. A common cause: scaling or imputing using all the data, including the test set, before splitting. The test set has then "leaked" its statistics into training.

Other leaks: a feature that would not exist at prediction time (such as "refund issued" when predicting if a customer will complain), or duplicates present in both train and test.

Pipelines: do it safely

A Pipeline chains steps so that every step is fitted only on training data, and the same steps run for new data. It also keeps cross-validation honest, because preprocessing is redone inside each fold.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

numeric = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
categorical = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

prep = ColumnTransformer([
    ("num", numeric, ["age", "income"]),
    ("cat", categorical, ["city", "plan"]),
])

model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
# model.fit(X_train, y_train); model.predict(X_new)

The column names here are examples for a table with age, income, city and plan columns.

Feature engineering

Creating new features from existing ones often helps more than switching algorithms: ratios (price per square metre), dates broken into day of week and month, counts, and text turned into numbers. Use domain knowledge, and test each idea with cross-validation.

Try it yourself

Take a small table with one numeric column with gaps and one text column. Sketch (on paper) the pipeline that imputes, scales or encodes each, then feeds a classifier.

Summary

• Scale numeric features, one-hot encode categories and impute missing values.
• Data leakage inflates scores. Always fit preprocessing on training data only.
• Pipelines bundle the steps, prevent leakage and keep cross-validation honest.