EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

cold start
The problem of recommending for a new user or a new item that has no interactions yet.
collaborative filtering
Recommending items by finding users with similar taste, or items that similar users liked, without reading item content.
explicit feedback
A signal where the user states a preference, such as a rating or a like, rather than one inferred from behaviour.
implicit feedback
A signal inferred from behaviour, such as a click, watch or purchase, that says something happened but not how much it was liked.
matrix factorisation
A method that gives each user and item a short list of learned numbers, so their match is the sum of products.
nDCG
A ranking score that gives more credit to relevant items near the top, then divides by the score of the best possible order.
popularity baseline
A list ordered by how many interactions each item has, used as the simple result every model must beat.
precision at k
The share of the first k items that are relevant, where k is the number of slots shown.
ranking
The second stage that orders a short list of candidates for one user, usually with a slower model.
recall at k
The share of all relevant items that appear in the first k positions.
relevance
How likely one user is to engage with one item; a ranking needs it per user, not as a global score.
retrieval
The first stage that cuts a large catalogue down to a few hundred candidates cheaply.
slot
One position on the screen that can hold an item; the fixed count of them limits how long a list can be.
two-tower retrieval
A design with one network for users and one for items, so the item side is computed before a request.

Recommender systems from the ground up

Ranking metrics

Precision at k, MAP and nDCG answer different questions about a list. Pick the one that matches what the interface promises.

Part 6 of 10

Intermediate16 min readBack to Courses

Cut at k

Every ranking metric starts with a cutoff. Precision at k is the share of the first k items that are relevant. Recall at k is the share of all relevant items that appear in the first k. The first is about the slot budget. The second is about how much of the user’s taste the list covers.

The cutoff is not a detail to tune after the fact. A model that is good at ten and bad at three is the wrong model for a three-slot carousel. Fix k from the interface. Then choose the metric that describes what the interface promises.

Ties are a real problem at small k. Ten items may share the same score when only three slots exist. The metric then depends on the tie-break, which may be alphabetical or arbitrary. Fix a tie-break you can repeat and report it. Without one, the same model will produce different numbers on different runs.

Average precision

Precision at k treats the list as a set. Average precision (AP) treats it as a sequence. It averages precision at every relevant position, so a hit at rank one counts more than the same hit at rank ten. Mean average precision is the mean of AP over users. It rewards putting relevant items early.

AP has a useful property. It does not care where the irrelevant items sit, only where the relevant ones land. That makes it a good fit for search-like lists, where the user scans from the top and stops when satisfied.

AP is not comparable across users in the same way as a rate. A user with one relevant item can score at most 1.0. A user with ten has many chances to build up precision. Report AP per user, then average. Keep the distribution visible.

nDCG in scikit-learn

from sklearn.metrics import ndcg_score

relevance = [[3, 2, 0, 1, 0]]
scores = [[0.9, 0.7, 0.8, 0.4, 0.1]]
print(round(ndcg_score(relevance, scores, k=5), 3))

Precision, AP and MAP

Run the report and recompute MAP after moving u2’s relevant item from rank four to rank two. The set of hits is unchanged; only the order moves.

Output
Click "Run" to execute your code

nDCG and graded relevance

nDCG (normalised discounted cumulative gain) lets relevance be a grade rather than a yes or no. It discounts each position by a log term, so lower slots count for less. The normalised version divides by the ideal ordering. A user with many relevant items and a user with one then sit on the same scale.

The discount is the point. Moving a strong item from position two to position one changes nDCG. Swapping two irrelevant items at the bottom barely moves it. That matches how people read a list, and it is the reason nDCG is the default for graded relevance.

The gain function is a choice. Linear gains treat a grade of four as twice a grade of two. Exponential gains treat it as far more. The original form uses an exponential. That fits judgements where the top grade is worth far more than the next. Either is fine if it is stated and used the same way every time.

Always print the baseline’s metric next to the model’s. A number without a reference is a press release, not an evaluation.

Report the distribution

A mean over users hides the users the system fails. Report the median, the tenth percentile, and the share of users with an empty or useless list. A model with a better mean and a worse tail may be the wrong release for a product that promises something to everyone.

Also report coverage and novelty. A ranker that shows the same ten popular items to everyone can win a top-k metric and still be a bad product. The metric is a lens, not a verdict. The dashboard should show more than one lens.

Finally, report the evaluation itself. How many users, how many candidates, which split, which seed, which version of the catalogue. A metric without that context cannot be reproduced. A metric that cannot be reproduced is a number nobody can act on six months later. Store the report with the model artefact, so a later comparison uses the same split, the same candidates and the same tie-break.

The discount in nDCG

Run the three orderings and compare nDCG. Then swap two items at the bottom and confirm the score barely moves, which is the property the discount is for.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does precision at k measure?