Research Methods and Metrics

Introduction

No single method answers every question. Strong teams combine ways of knowing: watching and listening to understand why, and measuring to understand how many and how much. This lesson helps you pick and combine methods, and avoid common errors with numbers.

Qualitative and quantitative

Qualitative research (interviews, observation, usability tests, diary studies) explains why people behave as they do. It uses small samples, and finds problems and ideas.
Quantitative research (surveys, analytics, experiments) measures how many, how often and how much. It uses larger samples, and finds patterns and sizes.

Use qualitative methods to discover and quantitative methods to verify and size. Each covers the other's blind spot.

Choosing a method

Question	Useful methods
What problems do people have?	Interviews, field observation, diary studies
Can people use this design?	Usability testing
Where do people struggle in the live product?	Analytics, session replay (with consent), support data
Does the structure make sense?	Card sorting, tree testing
How many people agree, or how satisfied are they?	Surveys
Which version performs better?	A/B tests
How do we compare with before?	Benchmarking studies with the same tasks

Ask first: what decision will this research inform? If it will not change a decision, do not do it.

Quick quiz

You want to know why users abandon a checkout. Which method is most suited to finding out why?

Surveys that work

Keep them short.
Use neutral wording, and avoid double questions ("Was it fast and easy?").
Offer balanced answer scales and an "other" option.
Pilot test them.
Be clear about privacy and how answers will be used.
Remember that survey respondents are self-selected, and people may not recall accurately.

Standard questionnaires, such as the System Usability Scale (SUS), allow comparison over time. Use them as designed.

Metrics for experience

Useful measures include:

Task success rate: did people complete the task?
Time on task and error rate
Satisfaction: short ratings after tasks or experiences
Adoption and retention: do people start using it, and keep using it?
Conversion: the share completing a goal, such as signing up
Support contacts and common problems reported

Teams sometimes use frameworks that group metrics, such as happiness, engagement, adoption, retention and task success. Pick a small set tied to goals, and avoid vanity numbers that go up without meaning, such as page views alone.

Running experiments carefully

An A/B test shows two versions to different, randomly chosen users and compares a metric.

Start with a hypothesis: "Showing delivery costs earlier will reduce abandonment."
Change one thing, or a clearly defined bundle.
Decide the primary metric and the sample size and duration in advance.
Do not stop early when you see a promising difference. Chance can create early swings.
Run for full weekly cycles, so day-of-week effects balance out.
Check that groups are comparable, and that nothing else changed.
Consider side effects: a change that raises sign-ups but also raises complaints is not a win.
With low traffic, you can only detect large effects. Use qualitative methods instead.

Remember that an experiment tells you what happened, not why.

Combining methods in practice

A good cycle might be: analytics shows a drop-off at step 3, interviews and usability tests reveal why, you design a fix, an A/B test measures the effect, and follow-up research checks for new problems.

Synthesis and communication

Combine findings into clear insights with evidence and a recommendation.
State confidence and limits.
Share highlights such as short clips and quotes, with consent, to build empathy.
Make findings findable: keep a research repository.
Track whether recommendations were acted on and what happened.

Ethics and privacy

Get informed consent, protect personal data, be careful with recordings, pay participants fairly and avoid manipulating or harming people through experiments. Follow the laws and policies that apply.

Try it yourself

Pick a product problem. Write your research question, the decision it informs, two qualitative and two quantitative methods you would use, and the metric you would track. Note one risk of misreading the results.

Summary

Qualitative methods find reasons and problems, and quantitative methods measure how many and how much.
Choose methods by the decision they inform, and keep surveys short and neutral.
Use a few meaningful metrics, run experiments with a pre-set plan and combine methods, with ethics and privacy throughout.