A Reproducible Pipeline

Introduction

A notebook is great for exploring, but cells run in the order you clicked, hidden state creeps in and results are hard to repeat. A pipeline is a series of steps that always run in the same order on the same inputs. It makes analyses reproducible: anyone can get the same answer from the same raw data. This final course assumes the earlier ones and basic Python.

The shape of a pipeline

Four stages:

1. Extract: read raw data from files or a database.
2. Clean: fix and validate it.
3. Analyse: calculate the results.
4. Report: write tables and charts to an output folder.

Each stage is a function that takes data in and returns data out, with no hidden global state.

import pandas as pd

def extract(con):
    return pd.read_sql_query(
        """SELECT l.*, b.genre, m.city
           FROM loans l
           JOIN books b ON b.id = l.book_id
           JOIN members m ON m.id = l.member_id""",
        con, parse_dates=["loaned_on", "returned_on"])

def clean(df):
    df = df.copy()                              # never change the input in place
    df["city"] = df["city"].str.strip().str.title()
    df["days"] = (df["returned_on"] - df["loaned_on"]).dt.days
    return df

def analyse(df):
    by_genre = (df.groupby("genre")
                  .agg(loans=("id", "count"), avg_days=("days", "mean"))
                  .round(1)
                  .sort_values("loans", ascending=False))
    by_month = df.groupby(df["loaned_on"].dt.to_period("M")).size().rename("loans")
    return by_genre, by_month

def report(by_genre, by_month, folder="output"):
    import os
    os.makedirs(folder, exist_ok=True)
    by_genre.to_csv(f"{folder}/loans_by_genre.csv")
    by_month.to_csv(f"{folder}/loans_by_month.csv")

def main(con):
    df = clean(extract(con))
    by_genre, by_month = analyse(df)
    report(by_genre, by_month)
    return by_genre, by_month

Running main(con) on the library data gives, for the genre table:

         loans  avg_days
genre                   
Crime       11      19.7
Fiction      8      15.9
Food         6      12.8
Home         5      18.2

The loan counts and average days match the earlier SQL results, which is a useful cross-check: two methods agree.

Principles

• Raw data is read-only. The pipeline creates everything else.
• Pure functions: the same input gives the same output, with no hidden state.
• Small steps are easy to test and replace.
• Configuration such as file paths and thresholds lives in one place, not scattered in code.
• One command runs everything, for example python run_pipeline.py.
• Random seeds are set where randomness is used.
• Dependencies are recorded in a requirements.txt file with versions.
• Version control: keep code in Git. Large raw data usually is not committed, but its source is documented.

Logging and failing loudly

Print what the pipeline does: row counts after each step, and fail clearly if a check does not pass.

def run():
    df = extract(con)
    print(f"extracted {len(df)} rows")
    df = clean(df)
    assert df["id"].is_unique, "duplicate loan ids"
    print("clean ok")

A pipeline that fails with a clear message is better than one that silently produces a wrong answer.

Scheduling and automation

Once reliable, a pipeline can run on a schedule, such as a nightly job, or when new data arrives. Tools range from simple scheduled tasks to workflow systems. Start simple, and add complexity only when the need is clear.

Try it yourself

Take your analysis from the earlier courses and split it into extract, clean, analyse, report and main. Add print statements for row counts, run it twice and check that the output files are identical.

Summary

• A pipeline runs extract, clean, analyse and report steps in a fixed order from raw data.
• Write small, pure functions, keep configuration in one place and never change raw data.
• Log row counts, fail loudly on bad data and provide one command to run everything.