CI for models

What CI must catch

Broken code, broken data, a training run that no longer converges, a metric below the floor, a slice that regressed, an artefact that cannot load. Each is a different test with a different runtime. A pipeline that only runs the unit tests catches the first and ships the rest.

The order matters. Cheap checks run first: lint, types, schema and a tiny training smoke test. Expensive checks run later: the full evaluation, the slice report and the artefact build. A failure early should stop the pipeline before it spends twenty minutes on a model that cannot import.

The test pyramid still applies, with one extra layer. Unit tests cover transformations; integration tests cover the pipeline from raw row to prediction; evaluation tests cover the model’s behaviour on a fixture; and a deployment test loads the artefact the way production will. Each layer should be fast enough to run on every change, or it will be skipped.

Fixtures and seeds

Keep a tiny golden dataset in the repository. Train on it with a fixed seed and assert that two runs agree. A smoke test that takes seconds catches the import error, the shape bug and the non-determinism that a full training run would hide for an hour.

Determinism is a testable property, not a hope. Fix the seed, the data order and the library flags that control threading. If the result still moves, the test has found a real source of irreproducibility, which is exactly what it is for.

A fixture should be small, readable and adversarial. Include the missing value, the unseen category and the row that used to crash the parser. A fixture made only of happy rows tests the plumbing and none of the behaviour that incidents are made of.

A parametrised metric test

import pytest

@pytest.mark.parametrize("threshold", [0.3, 0.5, 0.7])
def test_recall_floor(threshold):
    predictions = model.predict(fixture, threshold=threshold)
    assert recall(labels, predictions) >= 0.6

Evaluation gates

A gate is a threshold with an owner. Compare against the current production model on the same split, not against last week’s number. Gate on the headline metric and on the slices that matter; a model that improves the mean and breaks a segment is not a release.

The gate should also cover cost and latency. A model that is two points better and three times slower may be a worse product. Put the budget in the pipeline next to the accuracy, so the trade is visible before the release rather than after the incident.

Gates need a margin. If the candidate is within noise of production, a strict “must be better” rule blocks every release and a loose one ships regressions. Set the margin from the variance you measured across seeds, and treat a result inside the margin as a tie rather than a win. A gate that has never blocked a release is either perfectly calibrated or not being read.

From CI to CD

The pipeline builds the artefact once, tags it with the commit and the data snapshot, and promotes the same artefact through staging and production. The release decision is a gate, not a rebuild. If staging and production build separately, the thing you tested is not the thing you shipped.

Continuous delivery for a model also needs a rollback path, a canary and a monitoring check after promotion. Those belong in the same pipeline, not in a runbook that is opened only when something is already wrong.

Retain the artefacts and the reports. Disk is cheaper than an investigation, and the ability to reload the model from three releases ago is what makes a rollback a pointer change. Prune on a schedule, not on a hunch, and keep the last known good bundle forever. Keep the evaluation report next to the artefact; a metric without its report is a rumour.