Summary statistics and one straight line

The first five numbers

mean is the average and median is the middle value; when they disagree by a lot, the data has a long tail and the median is usually the more honest single number. sd measures the spread — roughly, how far a typical value sits from the mean.

min and max give the range. summary(values) prints all of these at once, and on a table it does it column by column, which is the quickest way to meet a dataset you have never seen. On a factor column summary reports counts instead of quartiles, which is the right answer for a category and a reminder that the function looks at each column’s type.

A line through the points

lm(score ~ hours, data = study) fits the straight line that sits closest to the points, using the same tilde as aggregate: the thing you are explaining on the left, the thing doing the explaining on the right. coef(fit) hands back the two numbers of the line, the intercept first.

The slope is the useful one. If it is 5.56, one more hour is worth about five and a half more points, on average, in this data. The intercept is the value the line predicts when the explaining variable is zero, and it is often meaningless on its own — a score at zero hours of study is not a real case. coef(fit) is a named vector, so coef(fit)[["(Intercept)"]] and coef(fit)[[1]] reach the same number; the names are worth knowing when you print the whole vector.

How much of it the line explains

summary(fit)$r.squared is a number between zero and one. Near one, the points sit close to the line and the two columns move together; near zero, the line is a poor description and there is little to read from it. cor(hours, score) measures the same closeness with a sign, so it tells you the direction as well. In a simple fit, cor is the signed square root of r squared, so a correlation of -0.9 and an r squared of 0.81 describe the same line with opposite slopes.

A fitted line describes the data you gave it. Using it outside the range you measured is a different act, and one worth doubting.

Describe, then fit

mean(study$score)
median(study$score)
sd(study$score)
summary(study)

fit <- lm(score ~ hours, data = study)
coef(fit)
summary(fit)$r.squared

fitted(fit)
resid(fit)

Reading the five numbers together

A mean and a median are answers to the same question with different weights. The mean uses every value, so one very large reading pulls it up; the median is the middle position, so it barely moves. When the two disagree, print summary and look at the quartiles before deciding which one to quote.

sd is the standard deviation, and its denominator is n - 1 rather than n. That is Bessel's correction: the sample spread is measured around a mean estimated from the same data, so dividing by n would underestimate the spread of the wider population. For eight points the difference is small, but it explains why two tools can disagree in the second decimal place.

Reading a fit honestly

lm chooses the line that makes the squared vertical distances from the points as small as possible. fitted(fit) is the line at each measured x, resid(fit) is the gap between the point and the line, and the residuals always sum to zero. A pattern in the residuals — a curve rather than a random scatter — says a straight line is the wrong description, even when r squared looks healthy.

The slope has units: points per hour, dollars per unit, minutes per kilometre. The intercept is the prediction at zero of the predictor, and it is often outside the data, which is why it can be meaningless on its own. predict(fit, new) will happily answer for a value far outside the measured range, but the answer is an extrapolation rather than a result.

Try these next

• Add an outlier to the study table and compare the mean, median and slope before and after.
• Print summary(fit) and read the residual quartiles, then compare them with sd(resid(fit)).
• Predict at a value inside the measured range and at one far outside it, and explain which you trust.