Ensembles and Tuning

Introduction

A single decision tree is easy to understand but unstable and prone to overfit. A crowd of different trees, voting together, is usually far more accurate. Combining models is called an ensemble.

The simple explanation

Many weak or varied models, combined, make better decisions than one model alone, because their individual mistakes tend to cancel out.

Random forest

A random forest trains many decision trees, each on a random sample of the rows and a random subset of the features at each split. The final answer is the majority vote (or the average for regression).

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest.fit(X_train, y_train)
print(round(forest.score(X_test, y_test), 3))

On the iris split from the first lesson, the single tree scored 0.895 and the random forest 0.921. Forests also give feature importances, showing which inputs mattered most: forest.feature_importances_. Treat them as a hint, not proof of cause.

Boosting

Gradient boosting builds trees one after another, each trying to fix the errors of the ones before. It is often the most accurate method for tables of data, and libraries such as XGBoost, LightGBM and scikit-learn's own GradientBoostingClassifier implement it. It is more sensitive to settings and to overfitting than a random forest.

Hyperparameters

Parameters are learned from data, such as the slope of a line. Hyperparameters are settings you choose before training, such as the number of trees or the maximum depth. They control how flexible the model is:

• Too flexible: it memorises the training data (overfitting).
• Too simple: it misses real patterns (underfitting).

Grid search with cross-validation

Grid search tries every combination from a list, scoring each with cross-validation:

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    {"n_estimators": [50, 100], "max_depth": [3, None]},
    cv=3,
).fit(X_train, y_train)

print(grid.best_params_, round(grid.best_score_, 3))

On the breast cancer training data, this run reported:

{'max_depth': None, 'n_estimators': 50} 0.962

The differences between settings were small, which is common: tuning helps, but better data and features usually help more. For big searches, use RandomizedSearchCV, which samples combinations.

Rules for honest tuning

1. Tune using cross-validation on the training data only.
2. Evaluate the final chosen model once on the untouched test set.
3. If you tune against the test set repeatedly, you leak information and the score becomes optimistic.
4. Record what you tried, and keep random seeds fixed so results can be reproduced.

Choosing a model

Start with a simple baseline, such as logistic regression or a small tree. Then try a random forest or gradient boosting. Pick the simplest model that is good enough, and consider speed, explainability and how often it must be retrained.

Try it yourself

On the iris split, train a random forest with 5 trees, then 50, then 200. Compare test accuracy and note how much improvement more trees give.

Summary

• Ensembles combine many models. Random forests vote, and boosting corrects errors in sequence.
• Hyperparameters are set before training, and cross-validated grid search tunes them.
• Tune on training data, test once at the end, and prefer the simplest good model.