Neural Networks and Gradient Descent

Introduction

Neural networks power image recognition, translation and modern AI assistants. They sound mysterious, but the core idea is simple: a big function with many adjustable numbers, tuned automatically to reduce its mistakes.

The simple explanation

A neural network multiplies inputs by weights, adds them up, bends the result, and repeats in layers. Training adjusts the weights to make fewer errors.

A single neuron

A neuron takes several inputs, multiplies each by a weight (how much it matters), adds a bias, then passes the total through an activation function:

output = activation( w1 × x1 + w2 × x2 + ... + bias )

The activation function adds non-linearity. Without it, stacking layers would still be just one straight-line calculation. Common choices:

• ReLU: outputs 0 for negative numbers and the number itself otherwise. Simple and widely used.
• Sigmoid: squashes any number to between 0 and 1, good for probabilities.
• Softmax: turns a list of scores into probabilities that add up to 1.

Layers

Neurons are arranged in layers: an input layer (the data), one or more hidden layers, and an output layer (the prediction). A network with many hidden layers is deep, hence "deep learning". Early layers tend to learn simple patterns, and later layers combine them into richer ones, such as edges, then shapes, then objects.

Loss: measuring how wrong

The loss function turns the model's errors into one number. Lower is better.

• Mean squared error for predicting numbers
• Cross-entropy for classification, which penalises confident wrong answers heavily

Gradient descent: learning by small steps

Imagine standing on a foggy hillside, trying to reach the valley. You feel which way the ground slopes down and take a step that way, repeatedly. In training:

1. Make predictions with the current weights.
2. Compute the loss.
3. Compute the gradient: how the loss would change if each weight changed slightly.
4. Move each weight a small step against the gradient.
5. Repeat many times.

The step size is the learning rate. Too large and you overshoot and bounce around. Too small and learning crawls. It is one of the most important settings.

A tiny sketch of the idea, shown for reading. It learns the best number w so that w × x matches y, where the true answer is w = 2:

x = [1.0, 2.0, 3.0]
y = [2.0, 4.0, 6.0]
w = 0.0
lr = 0.05

for step in range(100):
    # gradient of the mean squared error with respect to w
    grad = sum(2 * (w * xi - yi) * xi for xi, yi in zip(x, y)) / len(x)
    w -= lr * grad

print(round(w, 3))   # close to 2.0

Backpropagation

A real network has millions of weights in many layers. Backpropagation is the efficient method of working out each weight's gradient. It applies the chain rule of calculus backwards from the output, passing the error signal through the layers. Frameworks such as PyTorch and TensorFlow do it for you automatically (called automatic differentiation).

Training in practice

• Data is used in small groups called batches, and one full pass over the data is an epoch.
• Watch the loss on a validation set too. If training loss keeps falling and validation loss rises, the network is overfitting.
• Fighting overfitting: more data, dropout (randomly switching off neurons during training), weight decay, early stopping, and smaller networks.
• Good optimisers such as Adam adapt step sizes for you.
• Training big networks benefits from GPUs.

When to use neural networks

For tables of data, tree ensembles are often as good or better and far simpler. Neural networks shine with images, audio, text and other unstructured data, and when there is a lot of it.

Try it yourself

In the gradient descent sketch, change the learning rate to 0.5 and then 0.001, and the number of steps. Describe what happens to w in each case.

Summary

• A neuron weights its inputs, adds a bias and applies an activation. Layers of them form a network.
• Training reduces a loss using gradient descent, with gradients computed by backpropagation.
• Watch validation loss to detect overfitting, and use neural networks mainly for unstructured data.