Monitoring and drift

Four layers

Service metrics cover latency, errors and saturation. Input metrics cover schema violations, missing fields and the distribution of each feature. Output metrics cover the score distribution and the prediction rate. Outcome metrics cover clicks, conversions and the labels that eventually arrive. Each layer answers a different question, and the first three are available immediately.

The layers fail in order. A schema change breaks the input, the input shifts the output, the output moves the outcome, and only then does the business metric fall. Watching the earliest layer is how you find the cause before the symptom reaches a dashboard someone checks weekly.

Dashboards and alerts are different products. A dashboard is for exploration during an incident; an alert is a decision that someone should stop what they are doing. Keep the dashboard rich and the alerts few, each with a runbook that says what to check first and what to do if the check fails.

Data drift and concept drift

Data drift is a change in the input distribution: a new device, a new country, a feature that stopped updating. Concept drift is a change in the relationship between inputs and the target: the same inputs now mean something different. The first is visible without labels; the second usually is not.

A pandemic, a price change or a competitor’s launch can all change what a click means. The model has not moved, but the world it was fitted to has. That is concept drift, and no amount of input monitoring will find it on its own.

Seasonality is the common false positive. Retail traffic changes every December, and a detector calibrated on November will call it drift. Compare the live window with the same window a year ago, or detrend the series before testing, or the alert will fire on the calendar rather than on the model.

A two-sample test with SciPy

from scipy.stats import ks_2samp

statistic, p_value = ks_2samp(reference_scores, live_scores)
print(f"KS={statistic:.3f} p={p_value:.4f}")
if p_value < 0.01:
    print("the distributions differ")

Detect with a statistic, not a hunch

Population stability index over binned values, a Kolmogorov–Smirnov test on a numeric feature, or a simple mean shift with a threshold. Each has assumptions; none is a substitute for looking at the plot. Alert on a sustained shift, not on a single window, or every quiet Sunday will page someone.

Choose the threshold from the cost of a false alarm and the cost of missing a real change. A feature that is cheap to recompute can tolerate a looser threshold than one that feeds a credit decision. The statistic is a tool; the threshold is a risk decision.

Univariate tests miss joint shifts. Two features can each look stable while their combination moves, because the correlation between them changed. A model’s own score distribution is a cheap multivariate check: if the inputs are stable and the scores are not, the interaction between features is the place to look.

Labels arrive late

Accuracy may be unknown for days. Use proxies — the prediction distribution, downstream conversion, human review of a sample — and label the proxy as a proxy. When the labels land, reconcile the proxy with the truth and recalibrate the thresholds.

The reconciliation is the useful part. If the proxy said healthy and the labels say otherwise, the proxy is wrong and should be replaced or reweighted. A monitoring system that never checks itself becomes a source of false confidence.

Label delay is itself a signal. If the labels that normally arrive in a day take a week, the pipeline that produces them may be broken, and the model’s accuracy is unknown rather than unchanged. Monitor the label pipeline with the same seriousness as the prediction pipeline. A label pipeline that silently stops is worse than a wrong model, because the dashboard keeps showing the last good number and nobody knows it is stale.