Deep Learning Architectures

Introduction

A plain network of layers treats every input number separately. For images, text and audio, structure matters: nearby pixels belong together, and word order matters. Specialised architectures use that structure.

Convolutional neural networks (CNNs)

CNNs are designed for images. Instead of connecting every pixel to every neuron, they slide small filters across the image. A filter acts like a pattern detector, for edges, textures or shapes.

Convolution layers apply filters, and the same filter is reused across the whole image, so far fewer weights are needed.
Pooling layers shrink the image, keeping the strongest signals.
Early layers detect edges, middle layers shapes and later layers whole objects.

Uses: image classification, object detection, medical imaging, reading handwriting.

Quick quiz

Which architecture family is traditionally designed for image data?

Sequence models

Text, speech and time series are sequences. Older designs, such as recurrent neural networks (RNNs) and LSTMs, process one step at a time and carry a memory forward. They struggle with very long sequences and are slow to train because steps run one after another.

Transformers

The transformer, introduced in 2017, replaced recurrence with attention: each element of the sequence looks directly at all the others and learns how much each matters. This has big advantages:

It handles long-range relationships, such as a pronoun and the noun it refers to many words earlier.
It can be trained in parallel, which makes very large models practical.

Transformers underlie modern language models, and are also used for images (vision transformers), audio, and code. Large language models are transformers trained to predict the next token.

Transfer learning and fine-tuning

Training a large model from scratch needs huge data and compute. Transfer learning reuses a model already trained on a big dataset:

Start from a pre-trained model, which has learned general features.
Replace or add a final layer for your task.
Fine-tune: train on your smaller dataset. You may freeze the early layers and train only the later ones at first.

This often reaches good accuracy with only hundreds or thousands of examples. Check the model's licence and what data it was trained on.

Choosing an approach

Problem	Good starting point
Table of numbers and categories	Gradient boosting or random forest
Images	A pre-trained CNN or vision transformer, fine-tuned
Text	A pre-trained transformer, or a language model with good prompting
Time series	Statistical models first, then specialised networks
Little data	Transfer learning, simpler models, or data collection

Start with the simplest thing that could work, and measure.

Practical advice

Use a framework such as PyTorch or TensorFlow, and use high-level tools where they exist.
Check data quality first. Better data beats a fancier model.
Track experiments with seeds, settings and results, so you can reproduce them.
Mind the cost: large models need powerful hardware and energy. Smaller models may be enough.
Respect licences and privacy of data and pre-trained models.

Try it yourself

Pick a problem you care about, such as classifying photos of plants. Write which architecture family you would start with, which pre-trained model you could fine-tune, how many labelled examples you would need to try, and how you would test it.

Summary

CNNs suit images, and transformers using attention suit text and many other data types.
Transfer learning fine-tunes a pre-trained model, so you need far less data and compute.
Pick the simplest architecture that fits the data type, and measure results.