EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

cold start
The problem of recommending for a new user or a new item that has no interactions yet.
collaborative filtering
Recommending items by finding users with similar taste, or items that similar users liked, without reading item content.
explicit feedback
A signal where the user states a preference, such as a rating or a like, rather than one inferred from behaviour.
implicit feedback
A signal inferred from behaviour, such as a click, watch or purchase, that says something happened but not how much it was liked.
matrix factorisation
A method that gives each user and item a short list of learned numbers, so their match is the sum of products.
nDCG
A ranking score that gives more credit to relevant items near the top, then divides by the score of the best possible order.
popularity baseline
A list ordered by how many interactions each item has, used as the simple result every model must beat.
precision at k
The share of the first k items that are relevant, where k is the number of slots shown.
ranking
The second stage that orders a short list of candidates for one user, usually with a slower model.
recall at k
The share of all relevant items that appear in the first k positions.
relevance
How likely one user is to engage with one item; a ranking needs it per user, not as a global score.
retrieval
The first stage that cuts a large catalogue down to a few hundred candidates cheaply.
slot
One position on the screen that can hold an item; the fixed count of them limits how long a list can be.
two-tower retrieval
A design with one network for users and one for items, so the item side is computed before a request.

Recommender systems from the ground up

Online evaluation

Offline metrics filter bad models; online tests decide what users actually do. Randomisation, a pre-registered metric and a fixed horizon are what make the answer trustworthy.

Part 9 of 10

Advanced18 min readBack to Courses

Offline is a filter, online is the test

An offline metric is computed on logged data from the old policy. It can reject a bad model cheaply. It cannot tell you what users would do with a different list. Online evaluation runs the new policy on real traffic and compares it with a control. That is the only way to see the effect on behaviour.

The default design is an A/B test. Randomly assign each user to control or treatment. Keep everything else fixed. Compare a pre-registered metric over a fixed window. The randomisation unit should match the way users affect each other. Users in the same household or the same social graph can change each other’s behaviour. A session-level split can leak the treatment across a user’s visits.

Not every change needs an A/B test. A bug fix, a latency fix or a legal change can ship behind a flag and be watched. A ranking change that is meant to move engagement needs a test. The offline number has already been shown to disagree with the online one often enough to matter.

Metrics and guardrails

Pick one main metric and a small set of guardrails (numbers that must not get worse). The main metric is the thing the change is for: clicks per session, watch time, successful searches. The guardrails are latency, error rate, unsubscribe rate, complaint rate, and the share of traffic that ends in an empty list.

Define the metric before the test, including the denominator and the window. Clicks per user and clicks per session tell different stories when a change also changes session length. A ratio with a moving denominator can look like a win while every part of it is flat.

Practical significance is separate from statistical significance. A one percent lift with a million users is easy to detect and may not pay for the upkeep. Decide the smallest effect worth shipping before the test. Treat a result below it as a no, even when the p-value is small. The reverse is also true. A large effect in a tiny segment is a reason to test it in that segment, not to ship it everywhere.

Sample size with SciPy

from scipy import stats

baseline, mde, alpha, power = 0.10, 0.01, 0.05, 0.80
p1, p2 = baseline, baseline + mde
z_alpha = stats.norm.ppf(1 - alpha / 2)
z_beta = stats.norm.ppf(power)
n = (
    z_alpha * (2 * p1 * (1 - p1)) ** 0.5
    + z_beta * (p1 * (1 - p1) + p2 * (1 - p2)) ** 0.5
) ** 2 / (p2 - p1) ** 2
print("users per arm:", round(n))

An A/B test with a guardrail

Run the test and read the effect, interval and p-value. Then raise MIN_RELATIVE_LIFT to 0.10 and confirm the same result is now a no on practical grounds.

Output
Click "Run" to execute your code

Power, peeking and variance

An A/B test needs enough traffic to detect the effect it is looking for. The sample size depends on the baseline rate, the smallest effect you care about, the significance level and the power. Halving the effect you want to detect roughly quadruples the users you need. Small teams test big changes for that reason.

Looking at the result every day and stopping when it is significant inflates the false positive rate. Peek often enough, and a test with no effect will cross the threshold eventually. Fix the horizon in advance. Use a sequential test built for continuous monitoring, or correct the threshold for the number of looks.

Variance reduction makes a small test act like a larger one. CUPED uses a pre-period covariate (a measurement from before the test, usually the same metric) to explain part of the noise and shrink the variance of the estimate. It costs nothing at serving time. It can cut the required sample by a third or more when the pre-period predicts the outcome well.

Stopping the test the first time the p-value crosses 0.05 is not a shortcut. It is a different test with a much higher false positive rate than the one you planned.

Interleaving and the long view

Interleaving is an alternative to A/B testing for ranking changes. Two rankers are blended into one list. A click on an item credits the ranker that contributed it. Because every user sees both rankers, the comparison needs far fewer users than a between-subject test. The catch is that it only measures preference between two lists. It does not measure the absolute effect on revenue or retention.

Novelty and primacy are the two time effects that ruin a short test. A new interface gets attention because it is new. A familiar one gets clicks because users know where to look. Run long enough for both to settle. Keep a long-term holdout for changes that are meant to compound.

Finally, log what you need to analyse the test later: the assignment, the policy version, the propensities and the exposure time. A test that cannot be re-read is a number in a slide. The result should include the effect, the interval, the guardrails, the segment that moved most and the decision, in that order.

The cost of peeking

Run the simulation and compare the any-look false positive rate with the rate at the final look. Both arms have the same true rate, so every significant result is noise.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What can an online test measure that an offline metric cannot?