Linear regression

The model and the loss

Linear regression predicts a number as a weighted sum of features plus an intercept. The intercept is the starting value, the prediction when every feature is zero. With one feature the model is a line. With several it is a plane or a hyperplane. The weights are the parameters, and the intercept lets the line sit away from the origin without changing its slope.

The loss is mean squared error. That is the average of the squared differences between prediction and label. Squaring punishes large misses more than small ones. It also makes the loss smooth and convex, meaning it has a single bowl-shaped minimum. Under a Gaussian noise assumption, meaning the errors follow a bell curve, minimising squared error is also maximising likelihood. That is a tidy reason for a choice that is often made for convenience.

The intercept is easy to forget and important to report. It is the prediction when every feature is zero. That point may be outside the data and may be meaningless. Centring the features, which means subtracting their means, makes the intercept the prediction at the mean. That is often easier to interpret. The slope does not change, because a shift in x is absorbed by the intercept.

The closed form

For one feature, the least-squares slope is the covariance of x and y divided by the variance of x. Covariance measures how the two move together. Variance measures how x spreads around its mean. The intercept is the mean of y minus the slope times the mean of x. In matrix form the same answer is (XᵀX)⁻¹Xᵀy. That form is exact and needs no iteration.

The closed form is fast and reproducible, but it has conditions. Two columns are collinear when they carry nearly the same information. Then XᵀX is nearly singular and the weights become unstable. A small change in the data can swing them wildly. Regularisation adds a penalty on the weights and is the usual fix. For very wide data, an iterative method is often cheaper than forming the matrix.

The model in scikit-learn

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

model = LinearRegression().fit(X_train, y_train)
print(model.coef_, model.intercept_)

predictions = model.predict(X_test)
print(mean_squared_error(y_test, predictions))

Gradient descent

The gradient of mean squared error with respect to a weight is the average of the prediction error times the feature. A gradient points uphill, so you move each weight against it by a small step. Repeat, and the loss goes down. The learning rate controls the size of the step. Too small and training crawls. Too large and the loss jumps around or diverges.

For a linear model the loss surface is a bowl. Gradient descent then converges to the same optimum as the closed form. The exercise is still worth doing, because the loop generalises to models where no closed form exists. Feature scaling matters here too. Unscaled columns make the bowl long and thin, and a single learning rate has to suit both directions.

The gradient loop also makes the role of the epoch visible. One epoch is one pass over the data. A step is one update. With full-batch gradient descent the two are the same. With stochastic gradient descent, which uses a small sample per step, there are many steps per epoch. The loss curve is a function of steps, not epochs. So two training runs with different batch sizes cannot be compared by epoch count alone.

What the line assumes

The model assumes the expected value of the label is linear in the features. It also assumes the errors are independent and have roughly constant variance. When the spread grows with the prediction, the standard errors are wrong. The fitted line can still look reasonable. Outliers have a large influence, because the loss squares their distance.

A coefficient is the change in the prediction for a one-unit change in that feature, holding the others fixed. That last clause is doing a lot of work. Correlated features share their effect. A coefficient can shrink or change sign when another column is added. A fitted association is not a causal effect, however clean the p-value looks.

R² is a ratio of explained variance. It is not a measure of whether the model is appropriate. It can be high for a curved relationship that the line approximates well over the observed range. It can be low for a relationship that is real but noisy. A residual plot, which shows the errors against the prediction, is more informative than the single number. It shows whether the errors have structure. It shows whether their spread changes, and whether a few points are doing all the work.