Metrics and Cross-Validation

Introduction

"Accuracy" is the first number people reach for, and often the most misleading. If 99 of 100 patients are healthy, a model that always says "healthy" is 99 percent accurate and useless. You need metrics that show what kinds of mistakes a model makes.

We use a real dataset bundled with scikit-learn: 569 tumour measurements labelled benign or malignant. It is a teaching dataset, not medical advice. The outputs below come from a real run.

The confusion matrix

For a yes/no problem, every prediction falls into one of four boxes:

A false positive is a false alarm. A false negative is a miss. Which is worse depends on the problem: a missed disease is worse than a false alarm, while a wrongly blocked genuine email may be worse than letting a bit of spam through.

Precision, recall and F1

• Precision = of everything the model flagged as yes, how much was truly yes? It measures false alarms.
• Recall = of everything that was truly yes, how much did the model find? It measures misses.
• F1 is a single score that balances both, using their harmonic mean.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
]).fit(X_train, y_train)

pred = model.predict(X_test)
print("accuracy ", round(accuracy_score(y_test, pred), 3))
print("precision", round(precision_score(y_test, pred), 3))
print("recall   ", round(recall_score(y_test, pred), 3))
print("f1       ", round(f1_score(y_test, pred), 3))
print(confusion_matrix(y_test, pred))

accuracy  0.986
precision 0.989
recall    0.989
f1        0.989
[[52  1]
 [ 1 89]]

The matrix rows are the real classes, and the columns are the predictions. Here 52 and 89 were correct, with one false positive and one false negative.

Thresholds and trade-offs

Classifiers often produce a probability, and a threshold (usually 0.5) turns it into yes or no. Lowering the threshold finds more positives (higher recall) but raises false alarms (lower precision). Choose the balance that fits the real cost of each mistake.

One split can mislead

A single train and test split gives one score, and a lucky or unlucky split can distort it. Cross-validation repeats the process. With 5-fold cross-validation, the data is cut into five parts. Each part takes a turn as the test set while the other four train the model, giving five scores.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5)
print(scores.round(3), round(scores.mean(), 3))

[0.982 0.982 0.974 0.974 0.991] 0.981

The average, 0.981, is a steadier estimate, and the spread shows how much it varies. Use cross-validation to compare models and settings, and keep a final untouched test set for the last check.

Metrics for regression

For numeric predictions, common metrics are mean absolute error (the average size of the error), mean squared error (which punishes big errors) and R squared (how much of the variation the model explains).

Try it yourself

Change the threshold: use model.predict_proba(X_test)[:, 1] > 0.3 to make predictions, and recompute precision and recall. Which went up, and which went down?

Summary

• Accuracy can mislead. Use the confusion matrix, precision, recall and F1.
• Precision measures false alarms and recall measures misses. Choose by the cost of each.
• Cross-validation gives a steadier score than a single split.