Cost and capacity

The unit is a prediction

Cloud bills arrive per instance-hour, per gigabyte and per request, but a product decision is made per prediction. The bridge is a unit cost: the hourly cost of the serving fleet divided by the predictions it completes in that hour. That number is comparable across instance types, batch sizes and providers, and it is the one to put next to latency.

The denominator is not the theoretical throughput. It is the throughput at the latency the product needs, including the idle time between requests. A GPU that can do a thousand inferences a second and is asked for ten a second is a very expensive CPU. Utilisation belongs in the unit cost, not in a separate dashboard.

Cost has a shape over the day. A service sized for the evening peak runs mostly idle at four in the morning, and a service sized for the average misses the peak. The capacity plan is a statement about which of those you are willing to pay for, and the answer is different for a checkout path and a nightly batch job.

Batching, latency and the queue

Batching is the main lever on cost per prediction. A larger batch uses the hardware better, so the cost per item falls, and the latency of the last item in the batch rises. The right batch size is the one that meets the latency budget with the least idle hardware, not the largest batch the memory allows.

Queueing is where capacity plans go wrong. If requests arrive faster than the service can finish them, the queue grows without bound and latency with it. A system at ninety percent utilisation has a much longer wait than the same system at seventy, and the relationship is not linear. Headroom is the price of a predictable tail.

Little’s law ties the three numbers together: the average number of requests in the system equals the arrival rate times the average time in the system. It is not an approximation; it is an identity for a stable system. Use it to check a capacity plan, because a plan that claims a short queue at a high arrival rate is claiming one of the other numbers is different.

A cost table in pandas

import pandas as pd

rows = pd.DataFrame([
    {"component": "gpu", "units": 720, "unit_cost": 1.20},
    {"component": "cpu", "units": 1440, "unit_cost": 0.08},
    {"component": "storage", "units": 500, "unit_cost": 0.02},
])
rows["cost"] = rows["units"] * rows["unit_cost"]
print(rows.groupby("component")["cost"].sum().round(2))
print("total:", round(rows["cost"].sum(), 2))

Autoscaling and its delays

Autoscaling reacts to a signal: CPU, queue depth, requests per second or a custom metric. It then takes time to act. A new container has to be scheduled, pull an image, load the model and warm up before it serves. That delay is the reason a scale-up rule based on the current load is always late for a sudden spike.

Keep a floor of warm capacity for the baseline, scale on a leading indicator where one exists, and pre-scale for known events. A launch, a marketing email and a nightly batch are predictable, and paying for an hour of extra capacity is cheaper than dropping the traffic they bring.

Scale-down deserves the same care. An aggressive scale-down saves money and produces a cold start on the next request, which is a latency spike the average hides. Set the cool-down from the model’s warm-up time, and measure the p99 after a scale event rather than the mean.

Where the money actually goes

Serving is usually the visible cost and training is the surprising one. A training run that searches a hundred configurations can cost more than a month of inference, and the artefacts, logs and feature tables that make it reproducible are billed too. A cost model that only counts the inference fleet is a partial view.

Storage and egress are quiet. Model checkpoints are large, feature stores keep history, and logs are retained for compliance. Egress is charged when data leaves a region, which is exactly what a multi-region deployment does. Put a retention policy and a region policy next to the model card.

Finally, attribute the cost. A team that sees its own inference, training and storage lines will make better trade-offs than a team that sees one shared bill. Chargeback does not need to be exact to change behaviour; it needs to be stable, visible and reviewed in the same meeting as the quality metrics.