Asking Good Questions with Data

Introduction

Numbers feel objective, but conclusions from them are not automatically correct. Many wrong decisions come from sound-looking data and faulty reasoning. This lesson gives you the most important warning signs.

Correlation is not causation

Two things are correlated if they tend to move together. That does not mean one causes the other.

A classic example: ice cream sales and sunburn cases both rise in summer. Ice cream does not cause sunburn. A third factor, hot sunny weather, drives both. This hidden factor is a confounder.

When two things are linked, there are several possibilities:

1. A causes B.
2. B causes A.
3. A third thing causes both.
4. It is a coincidence.

To show cause, you generally need a well-designed experiment, such as randomly assigning people to two groups and comparing them, or very careful analysis. Be cautious with words like "causes", "leads to" or "prevents" when you only have observational data. "Is associated with" is safer.

Bias in samples

Your data is usually a sample from a larger group. If the sample is not representative, conclusions about the whole group may be wrong.

• Sampling bias: an online poll about internet use only reaches people who are online.
• Self-selection: people who choose to answer a survey are often those with strong opinions.
• Survivorship bias: looking only at the winners. If you study successful companies, you miss the many that failed with the same habits.
• Small samples: asking five people tells you little, and results can swing by chance.

Ask: who is in the data, and who is missing?

Other traps

• Cherry-picking: showing only data that supports your view.
• Averages hiding groups: an overall figure can hide very different results for different groups.
• Percentages without numbers: "a 50 percent increase" from 2 to 3 cases is tiny. Always ask for the actual numbers.
• Finding patterns in noise: if you search through enough data, something will look striking by pure chance.
• Confirmation bias: noticing evidence that fits what you already believe.
• Using data without context: a number means little without something to compare it with.

A simple analysis routine

1. Start with a clear question. "Do customers who join in January spend more?"
2. Check the data: where it came from, how it was collected, what is missing.
3. Clean it, and write down what you changed.
4. Explore: summaries and charts, looking for patterns and surprises.
5. Test your explanation: what else could cause this? What would show you are wrong?
6. Report honestly: what you found, how sure you are, the limits, and the numbers behind it. Include findings that do not fit your story.
7. Be ready to be wrong, and to update your view.

Ethics

• Do not present guesses as facts.
• Respect people's privacy in the data.
• Think about who could be harmed by a decision based on your analysis.

Where to go next

If you enjoyed this, continue to Data Analysis with Python and pandas, where you will use code to explore real tables, and then to the advanced course on combining and reshaping data.

Try it yourself

Find a headline that claims one thing causes another. List two other explanations, and write the one question you would ask to find out who was in the data.

Summary

• Correlation does not prove causation, because hidden factors and coincidence can create patterns.
• Biased samples, survivorship bias, small samples and cherry-picking lead to wrong conclusions.
• Start with a question, check and clean the data, test alternatives, and report honestly with the limits.