Retries, timeouts and idempotency

Three timeouts, not one

A connect timeout bounds how long you wait to reach the server. A read timeout bounds how long you wait for the next byte. A total deadline bounds the whole operation, including retries. A single timeout number conflates them and usually ends up too generous for connect and too tight for generation. Separate values are not over-engineering; they are the difference between a slow network and a slow model.

Set the total deadline from the caller’s budget and derive the rest. A chat request that must answer in ten seconds cannot spend eight of them on a connect attempt, and a batch job with an hour can afford a much longer read. The deadline should be visible in the logs, because “it timed out” is not an explanation until you know which clock ran out.

The three clocks also fail differently. A connect timeout means the request never reached the model; a read timeout means it may have. That difference decides whether a retry is safe, and it is invisible if you collapse every timeout into one number. Log which clock expired, not just that one did.

Retry only what is retryable

A 429, a 500, a 502, a 503, a 504 and a timeout are worth another attempt. A 400, a 401, a 403, a 404 and a 422 are not: the request is wrong, and sending it again produces the same wrong request. Retrying a 401 turns a configuration error into a lockout. The status code is a contract, and treating every failure as transient is how a small bug becomes an incident.

The distinction lives in the status code and, for streaming, in whether the stream started. A failure before the first token is a normal retry; a failure after it is a partial result, which is a different problem. A client that retries a started stream duplicates text the user has already read, which is worse than an error.

One deadline, split by phase

import httpx

timeout = httpx.Timeout(connect=3.0, read=60.0, write=10.0, pool=3.0)

with httpx.Client(timeout=timeout) as client:
    response = client.post(
        url,
        json=body,
        headers={"Idempotency-Key": key},
    )
    response.raise_for_status()

Idempotency makes a retry safe

A retry is only safe if repeating the request has the same effect as sending it once. A completion has no side effect, so repeating it is harmless apart from the cost, but a tool call that charges a card is not. If the first attempt succeeded and the response was lost, the retry charges again. The window is small, and it is exactly the window that produces duplicate charges.

An idempotency key is a client-generated id that the server stores with the result. Send the same key on the retry and the server returns the original result instead of performing the action twice. The key must be generated before the first attempt and reused, not regenerated per try. It should also be stored with the request so a process restart does not lose it.

Deadlines propagate

If a request has two seconds left, every downstream call should know it. A retry loop that starts a fresh sixty-second timeout inside a two-second budget has turned a deadline into a suggestion. The remaining time is a parameter, not a global; passing it explicitly is how the whole call graph agrees on when to stop.

Pass the remaining time down, and let the inner call fail fast when there is not enough of it. The worst outcome is not an error; it is a request that succeeds after the caller has given up, having spent the money and written the row. A late success is a side effect nobody is waiting for, and it is often the one that is hardest to undo.

Deadline propagation is easiest when the deadline is an absolute timestamp rather than a duration. A duration passed down the call graph starts counting again at every hop, which is how a two-second budget becomes a thirty-second request. An absolute time is comparable everywhere, and the remaining budget is one subtraction.