Quality, Speed and Analysis Projects

Introduction

Most of a real analysis is not clever modelling. It is getting the data clean, keeping the code fast enough, and being able to repeat and explain every step. This final lesson covers those professional habits. The small outputs shown come from real runs.

Cleaning text

Inconsistent text is the commonest reason that groups and joins fail:

names = pd.Series([" Ana ", "BEN"])
print(names.str.strip().str.lower().tolist())

['ana', 'ben']

The .str methods apply to a whole column: strip, lower, replace, contains and split. Standardise case and spaces, spelling variants and codes before you group or merge.

Missing values and duplicates

print(pd.Series([1, np.nan, 3]).isna().sum())          # 1 missing value

dup = pd.DataFrame({"id": [1, 1, 2], "v": ["x", "x", "y"]})
print(dup.drop_duplicates().shape)                      # (2, 2)

Decide deliberately how to treat gaps: drop rows, fill with a sensible value, or keep them and flag them. Write down the decision, since it changes the results. For duplicates, decide what makes two rows "the same", using subset= to compare only some columns.

Validate your assumptions

Add quick checks that fail loudly when data is not as expected:

assert df["customer_id"].notna().all(), "Missing customer ids"
assert df["amount"].ge(0).all(), "Negative amounts found"
assert df["order_id"].is_unique, "Duplicate order ids"

These small checks catch problems when new data arrives.

Speed: vectorise

pandas is fast when it works on whole columns at once, called vectorised operations. It is slow when you loop over rows in Python.

# Slow: row by row
df["size"] = df["a"].apply(lambda v: "big" if v > 2 else "small")

# Fast: vectorised
df["size"] = np.where(df["a"] > 2, "big", "small")

Both give the same result:

   a   size
0  1  small
1  2  small
2  3    big
3  4    big

On large tables, the vectorised version can be orders of magnitude faster. Avoid iterrows for calculations, and prefer built-in methods, groupby and merge.

Memory: categories and types

Columns of repeated text, such as a country or status, can be stored as a category, which keeps one copy of each distinct label:

x = pd.Series(["a", "b", "a", "c"] * 1000)
print(x.memory_usage(deep=True) > x.astype("category").memory_usage(deep=True))

True

The category version uses less memory, and categories can also have a defined order, which fixes the sorting problem you met with month names. Also choose smaller number types where safe, and read only the columns you need with usecols=. For data too large for memory, process in chunks, or use tools built for big data.

A reproducible project

1. Question: write the question and what decision it supports.
2. Raw data: keep the original files untouched, with a note on where they came from and when.
3. Cleaning script: code that turns raw into clean data, with the checks above, and a log of what was changed or removed.
4. Analysis: code that makes the tables and charts. Set random seeds where they apply.
5. Results: the figures with clear titles, plus the numbers behind them.
6. Report: a short write-up with the answer, the evidence, the limits and the next steps.
7. Environment: record the Python and library versions (a requirements file), and keep everything in Git.

Someone else, or you in six months, should be able to run it from start to finish and get the same results.

Communicating

• Lead with the answer, then the evidence.
• Use the clearest chart or table for each point, with honest axes.
• State uncertainty and limitations, and what you did not check.
• Tailor detail to the reader. A manager needs the decision, and a colleague needs the method.

Try it yourself

Take a messy CSV from an open data source. Write a cleaning script with at least three assert checks, document each cleaning decision in comments, and produce one chart and a three-sentence conclusion that states a limitation.

Summary

• Standardise text, handle gaps and duplicates deliberately, and add assertions to catch bad data.
• Vectorise calculations, and use categories and smaller types to save memory.
• Keep raw data, code, checks and results organised so the analysis can be repeated and explained.