Regression and Classification Models

Introduction

This course assumes you have read Machine Learning Basics and know basic Python. The code is shown for reading. To run it, install the libraries with pip install scikit-learn numpy. The outputs here come from real runs with scikit-learn 1.9, and your numbers may differ slightly with other versions.

Two kinds of prediction

• Regression predicts a number: a price, a temperature, a delivery time.
• Classification predicts a category: spam or not spam, which species of flower, whether a customer will leave.

The scikit-learn pattern

Nearly every model has the same steps:

1. Create the model: model = SomeModel(options)
2. Learn from examples: model.fit(X, y)
3. Predict for new data: model.predict(new_X)

X holds the features, one row per example. y holds the labels or numbers to predict.

Regression example

Suppose the true rule is "answer = 2 × input + 1":

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([3, 5, 7, 9, 11])

model = LinearRegression().fit(X, y)
print(model.coef_[0], model.intercept_)   # the learned slope and intercept
print(model.predict([[6]]))

2.0 1.0
[13.]

The model found the slope 2 and intercept 1 from the examples, and predicted 13 for an input of 6.

Classification example: iris flowers

scikit-learn includes small practice datasets. The iris dataset has measurements of 150 flowers of three species. We keep 25 percent aside for testing, then train two models:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

logreg = LogisticRegression(max_iter=1000).fit(X_train, y_train)
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)

print("logistic regression test accuracy:", round(logreg.score(X_test, y_test), 3))
print("decision tree test accuracy:      ", round(tree.score(X_test, y_test), 3))
print("decision tree TRAIN accuracy:     ", round(tree.score(X_train, y_train), 3))

logistic regression test accuracy: 0.947
decision tree test accuracy:       0.895
decision tree TRAIN accuracy:      0.964

• random_state=42 makes the split repeatable. stratify=y keeps the species mix the same in both parts.
• .score reports accuracy, the fraction of correct predictions.
• The tree scores higher on the data it trained on (0.964) than on new data (0.895). That gap is a sign of mild overfitting.

The models in one line each

• Linear regression draws the best straight line (or plane) through the data.
• Logistic regression is, despite the name, a classifier. It estimates the probability of each class.
• Decision tree asks a series of yes/no questions about the features, like a flowchart. Easy to explain, and prone to overfit if allowed to grow deep.

Always test on unseen data

Judge models only on the test set. Score on training data tells you how well it memorised, not how well it will work.

Try it yourself

Change max_depth of the decision tree to 1, then to 10, and compare train and test accuracy. Which setting underfits and which overfits?

Summary

• Regression predicts numbers and classification predicts categories.
• Every scikit-learn model follows create, fit and predict.
• Compare models on a held-out test set, and a big gap between train and test scores signals overfitting.