Capstone: a recommender and its evaluation

Define the task before the model

Write down the user, the context, the candidate set, the cutoff and the metric. “Recommend songs” is not a task. “Given a user’s plays before day 30, rank ten unheard tracks from the catalogue by nDCG at 10” is a task. That sentence decides the split, the baseline and the number that counts as success.

The candidate set is part of the task. Ranking the whole catalogue is a different problem from reranking forty candidates that a retrieval step produced. Most production systems do both. The offline evaluation should match the stage being changed.

The metric and the cutoff should be chosen together. nDCG at 10 and precision at 3 can disagree about which model is better. Neither is wrong; they describe different interfaces. Write the pair down. Treat a change to either as a change to the experiment, not as a detail of the report.

Split by time

A random split lets the model see the future. Use the last window as the test set. Train on everything before it. Never let a later interaction shape an earlier feature. For a new item, that means its features must have existed at training time, not only at evaluation time.

A temporal split is harder and more honest. It also exposes the cold-start problem that a random split hides. The test window naturally contains items and users that the training window barely saw.

Leakage is the failure to hunt down. A feature computed from the full history lets the future into training. So does a normalisation fitted on the test window, or a join that picks the latest value. The symptom is an offline score that is too good and an online score that is ordinary. The cause is almost always in the pipeline, not the model. Rebuild the feature from the raw events with an explicit as-of time. Then rerun the evaluation and expect the number to fall. That fall is the honest score.

Ranking evaluation with implicit

from implicit.als import AlternatingLeastSquares
from implicit.evaluation import ndcg_at_k, precision_at_k

model = AlternatingLeastSquares(factors=64, regularization=0.1)
model.fit(train)
print("precision@10:", round(precision_at_k(model, train, test, K=10), 3))
print("nDCG@10     :", round(ndcg_at_k(model, train, test, K=10), 3))

Baselines and hybrids

Popularity is the floor. Item-item CF is the next step. Content similarity covers items with no interactions. A hybrid blends them with weights chosen on a validation window, not on the test window. Or it reserves a slot for a cold item and accepts the cost in precision. On a small split, the floor can be the ceiling. That is exactly the result the report should show.

The blend is a product decision as much as a modelling one. A slot kept for exploration lowers today’s click metric and buys coverage tomorrow. Write the trade down, then measure both sides of it.

The blend weights belong to a validation window that sits between train and test, never to the test window itself. Tune the weight there, freeze it, and report the test number once. If the weight is chosen after looking at the test set, the test set has become training data with extra steps.

Report and iterate

Report the metric, the baseline, coverage, the per-user distribution and a handful of example lists. Then read the failures: empty lists, repeated items, the same head items for everyone, and users whose whole test set was cold. The next model is chosen from those failures, not from the leaderboard.

Finally, remember what offline evaluation cannot see. It cannot measure position bias (the pull of a slot rather than an item), surprise, or the long-term effect of what you chose to show. It is a filter for bad releases, not a prediction of product success.

The report should fit on one page: the task, the split, the metric, the baseline, the model, coverage and the three worst lists. If a reader cannot tell from that page what to try next, the report is a scoreboard rather than an evaluation. Name the next experiment in the last line.