Spread and Standard Deviation

Introduction

Two classes can have the same average test score and very different results: one where everyone scored near 70, another with scores from 20 to 100. The spread of data matters as much as its centre. The most common measure of spread is the standard deviation.

Calculating it by hand

Data: 2, 4, 4, 4, 5, 5, 7, 9

1. Mean: (2 + 4 + 4 + 4 + 5 + 5 + 7 + 9) / 8 = 40 / 8 = 5.
2. Deviations from the mean: -3, -1, -1, -1, 0, 0, 2, 4.
3. Square them (so negatives do not cancel): 9, 1, 1, 1, 0, 0, 4, 16.
4. Add: 9 + 1 + 1 + 1 + 0 + 0 + 4 + 16 = 32.
5. Divide by n (8) for the population variance: 32 / 8 = 4.
6. Square root gives the population standard deviation: 2.

The standard deviation has the same units as the data, which makes it easier to interpret than the variance.

Population versus sample

• If your data is the entire population, divide by n.
• If it is a sample used to estimate a larger population, divide by n - 1 (Bessel's correction). This corrects a tendency of samples to understate spread.

For the data above as a sample: variance = 32 / 7 = 4.571, and standard deviation = about 2.138. Spreadsheets have both: STDEV.P for populations and STDEV.S for samples.

The normal distribution

Many measurements, such as heights, cluster around a mean in a bell-shaped curve. For a normal distribution, roughly:

• 68 percent of values lie within 1 standard deviation of the mean,
• 95 percent within 2,
• 99.7 percent within 3.

Suppose adult heights have a mean of 170 cm and a standard deviation of 10 cm. About 95 percent fall between 150 and 190 cm. Not all data is normal. Incomes, for instance, are strongly skewed, so check the shape before applying the rule.

Z-scores

A z-score tells how many standard deviations a value is from the mean:

z = (value - mean) / standard deviation

A height of 185 cm gives z = (185 - 170) / 10 = 1.5. This allows comparing different scales: an exam score of 82 where the class mean is 70 and the standard deviation 8 gives z = 1.5 as well, so these are equally unusual in their own groups.

Spread in decisions

• Variability matters in quality control, risk and reliability.
• Outliers can inflate standard deviation. Consider the median and interquartile range for skewed data.
• Small samples give uncertain estimates of spread.
• Reports of averages without spread can hide important differences.

Try it yourself

Calculate the population and sample standard deviation of 10, 12, 14, 16, 18 by hand and check with a spreadsheet. Find the z-score for 18.

Summary

• Standard deviation is the square root of the average squared deviation, in the data's own units.
• Divide by n for a whole population and n - 1 for a sample.
• For roughly normal data, about 68, 95 and 99.7 percent lie within 1, 2 and 3 standard deviations, and z-scores compare unlike values.