EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

artefact
A saved file a release needs, such as the model, the code that prepares its inputs or the exact package versions.
batch
A run that scores many rows at once on a schedule and writes the results to a table.
canary
A release shown to a small random slice of traffic first, so a bad model reaches few users.
drift
A change in the inputs a model sees, or in the link between those inputs and the outcome it predicts.
evaluation gate
A check before release that stops a model when its measured quality falls below a set bar.
feature contract
An agreed promise about one input column — its type, range, missing values, uniqueness, freshness and owner — checked as data arrives.
incident
An event that degrades a service beyond its agreed level and needs a coordinated response.
inference contract
The agreed inputs, outputs and error cases for a model's prediction service, written down and tested.
latency budget
The time a request may spend in the model path, after network and rendering are subtracted from what users will tolerate.
model card
A short document describing what a model is for, how it was trained, its limits and who owns it.
monitoring
Watching inputs, outputs, service health and later outcomes so a change is noticed before users report it.
rollback
Returning to the last known good version, using a saved copy and a command that has been rehearsed.
serving
Running a model to answer requests in production, either one at a time or in scheduled groups.
training-serving skew
When an input is computed one way for model fitting and another way at request time, so the model reads different data.

MLOps from the ground up

Data and feature contracts

A feature is a promise about a column: its type, its range, its freshness and who owns it. Training and serving must read the same promise.

Part 6 of 10

Intermediate16 min readBack to Courses

Contracts at the boundary

A data contract names the table, the columns, the types, the allowed nulls, the ranges and the freshness. It is enforced where data enters the pipeline, not documented in a wiki. A violation should stop the job or quarantine the rows, not silently become a zero.

The boundary is the right place because that is where the information is still cheap to recover. Once a bad row has been joined, aggregated and written into a feature table, the cause is a hundred transformations away from the symptom.

Decide what happens on a violation before it happens. Failing the job is right for a training run that would otherwise learn from bad data; quarantining the rows is right for a stream that cannot stop. Both are contracts; silently substituting a default is not.

The six clauses

Type, range, nullability, uniqueness, freshness and ownership. Each has a failure mode: a string in a numeric column, a negative price, a missing user id, a duplicate event, a table that stopped updating, and nobody to call when it breaks.

Ownership is the clause teams skip and the one that decides how fast an incident ends. A contract without an owner is a suggestion. Put a team and a channel next to the table, and make the pipeline fail loudly enough that the channel is used.

Contracts evolve. Adding a nullable column is usually safe; changing a type or removing a column is not. Version the contract, keep the old version readable for the length of the longest backfill, and announce the deprecation before the producer changes the table.

A schema with pandera

import pandera.pandas as pa

schema = pa.DataFrameSchema({
    "user_id": pa.Column(str, nullable=False),
    "amount": pa.Column(float, checks=pa.Check.ge(0)),
    "country": pa.Column(str, checks=pa.Check.isin(["GB", "US", "DE"])),
})
validated = schema.validate(frame)

Validate before the join

Run the validator and fix the rows one at a time. Then add a duplicate user_id and decide whether uniqueness belongs in the schema or in a separate check.

Output
Click "Run" to execute your code

Training-serving skew

The most expensive bug in ML is a feature computed one way offline and another way online. A batch job averages over the last thirty days; the service reads a rolling counter that resets at midnight. Both are called average spend, and they disagree.

The fix is a shared definition, not a shared intention. Compute the feature once in a library used by both paths, or serve the batch value from a feature store and accept the staleness. Test the two paths against the same fixture and assert that they agree.

A feature store is one answer, but it is not magic. It centralises the definition and the serving path, and it still needs a test that the offline and online values agree for the same entity and timestamp. The store makes the disagreement visible; it does not make it impossible. The two paths also drift in subtler ways: a default that changed, a time zone, a join that now includes a new table. The only test that catches them runs both paths on the same entity and compares the numbers. A contract is only as good as its last test.

A feature computed differently in training and serving is a different feature. The model will be accurate on the data it was fitted to and wrong on the data it receives.

Point-in-time correctness

A training row must use only what was known at the prediction time. A join that picks the latest value leaks the future and inflates offline metrics. Build the feature table with an event time and an ingestion time, and join on the event time.

This is easy to get wrong and hard to notice. The offline metric looks excellent, the online metric is ordinary, and the gap is blamed on the model. In practice the model was trained on a feature that included the answer.

Late-arriving events complicate the join. An order that happened yesterday but arrived today should be visible to a prediction made tomorrow, not to one made yesterday. Keep both the event time and the ingestion time, and make the as-of time an explicit argument rather than an accident of when the job ran. A feature that depends on when the job ran will behave differently in a backfill.

The same name, two features

Run the comparison and find the users whose batch and online features disagree. Then change the online window to five days and confirm both gaps close, because the window now covers the whole history.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Where should a data contract be enforced?