Capstone: a resilient client

The shape of the client

A request goes through a small pipeline: build the body, check the local rate limiter, send with a deadline, classify the failure, retry or fail, validate the response, record the tokens and the cost. Each stage is ordinary code and each one is testable on its own. The pipeline is also where you put the product rules, such as “this feature may spend at most two cents per request”.

The transport is the seam. Put the network behind an interface and a scripted transport can inject a 429, a timeout and a malformed response without a provider, an API key or a flaky connection. The seam is what makes the failure paths testable at all; without it, every test of a retry is really a test of the network.

The pipeline is also where a product rule becomes code. “Never spend more than two cents on a chat turn” is a check before the call and a counter after it, not a sentence in a design document. The same is true of “never retry a charge” and “always show a partial answer as partial”: each is a branch in the client.

Failure is the normal path

A client that only works when the provider is healthy is a demo. The resilient version has a bounded number of attempts, a deadline that covers all of them, and a clear final error that says what was tried. The caller should never see a raw socket exception. The error should carry the last status, the number of attempts and the time spent, because that is what the incident review will ask for.

The same applies to partial success. A stream that dies halfway, a tool call that returns an error, a JSON body that fails validation: each is a state the caller can handle if the client names it, and an outage if the client throws something generic. Naming the states is most of the design work; the retry policy is comparatively small.

Let the SDK do the boring part

from openai import OpenAI

client = OpenAI(
    timeout=20.0,
    max_retries=3,          # retries 408, 409, 429 and 5xx
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=messages,
    max_tokens=400,
)

Measure the client, not the model

Success rate, attempts per success, p95 latency and cost per success are client metrics. They tell you whether a change to the retry policy helped, which a model benchmark cannot. A retry that raises the success rate from 97 to 99.5 percent may also double the cost, and only the client can see both. The four numbers belong on one chart, because improving one at the expense of another is the usual trade.

Run the scripted transport in CI with a fixed seed. A chaos test that is deterministic is a regression test; one that is random is a demo that sometimes fails. The script should include the ugly cases: a 429 followed by a 500, a stream that ends early, and a body that parses but fails the schema.

What is still missing

A production client also needs observability, a circuit breaker, a queue for work that can wait, and a budget that spans a user session rather than a single request. Each is an addition to a stage that already exists. None of them requires a new architecture, which is the point of building the pipeline first.

The point of the capstone is that none of it is mysterious. A timeout, a bounded retry, a limiter, a schema and a cost counter are five small pieces of ordinary code, and together they are the difference between calling an API and running a service. The model is the part you rent; the client is the part you own.

The client is also the part you can test without a provider, which makes it the part worth investing in first. A model upgrade is a config change; a client that handles timeouts, partial streams and bad JSON is the thing that keeps the product working when the upgrade is not the only thing that changed.