Serving and batching

Two modes

Online scoring answers one request at a time under a latency budget. Batch scoring writes predictions to a table on a schedule. The same model can do both, but the surrounding code, the feature access and the failure handling are different enough that they should be separate services or at least separate entry points.

A common pattern is batch for the long tail and online for the head. Precompute recommendations for every user overnight, then rescore the top of the list when the user opens the app. The precomputation buys coverage; the online call buys freshness.

The boundary is a freshness decision. If a prediction can be an hour old, batch is cheaper and more stable. If it must reflect the last click, only online will do. Most systems end up with both, and the interesting design work is deciding which parts of the page need which.

The latency budget

Start from the user’s patience, subtract the network and the rendering, and what remains is the model’s budget. That number decides whether you can fetch features synchronously, call a second model, or must precompute. Measure p95 and p99, not the mean: the mean hides the requests that people actually notice.

A latency budget is a constraint on the whole path, not just the model. A fast model behind a slow feature store is a slow service. Instrument each hop, and give the feature fetch its own timeout so a stalled dependency cannot consume the entire budget.

Measure the budget from the client, not from inside the service. A model that reports eight milliseconds while the page takes four hundred has measured the wrong thing. Server-side timing misses the network, the queue and the rendering, which are often the largest parts of the wait.

A BentoML service

import bentoml

@bentoml.service
class Recommender:
    @bentoml.api
    def recommend(self, user_id: str, k: int = 10) -> list[str]:
        return model.recommend(user_id, k)

Batching is a trade

Grouping requests amortises fixed costs and improves throughput, at the cost of waiting for the batch to fill. A micro-batcher with a short window is the usual compromise. The window is a latency decision disguised as a throughput setting, and it should be chosen from the p95 budget rather than from the benchmark.

Batching also changes the failure mode. One bad row can poison a whole batch if the code does not isolate it, and a batch that times out fails every request inside it at once. Validate before batching, and keep the batch small enough that a retry is cheap.

Hardware changes the arithmetic. A GPU is efficient on large batches and wasteful on one row, so the optimal batch is a property of the device as well as the model. Benchmark on the hardware you deploy, and remember that the queueing delay is part of the latency your users feel. The batch window is a promise about waiting, and the promise should be visible in the product: a spinner or a slightly stale list is honest, while a blank page is not.

Degrade, do not fail

Every dependency can be slow. A timeout, a cached prediction, a popularity fallback and a clear degraded flag keep the page useful. The fallback does not have to be good; it has to be fast and honest, and the response should say that it was used.

Track the degraded share as a first-class metric. A service that silently falls back for a third of its traffic looks healthy on error rate and is failing its users quietly. The fallback is part of the product, so it belongs on the dashboard.

A circuit breaker makes the fallback automatic. After a run of timeouts, stop calling the primary for a cooling period, serve the fallback, and probe again. This protects the slow dependency from a retry storm and keeps the degraded path from becoming the permanent path without anyone noticing.