Rate limits and backoff

Limits have several dimensions

A provider may limit requests per minute, tokens per minute, concurrent requests and tokens per day, and the binding limit changes with the workload. A batch of short prompts hits the request limit; a batch of long documents hits the token limit with far fewer calls. If you only watch requests per second, the day the prompts get longer is the day the client starts failing for no obvious reason.

The response usually says which one you hit, in a header or in the error body, and how long to wait. Read it. A client that backs off by a fixed second when the server asked for thirty is a client that will be limited again immediately. The header is the server telling you the schedule; your backoff is only a fallback for when it does not.

The dimensions also interact. A retry that succeeds after a 429 consumes another request slot and another batch of tokens, so a client near the token limit can trip the request limit while recovering. The safe design leaves headroom on both, and the headroom is a number you choose rather than a number you discover during an incident.

Backoff with jitter

Exponential backoff doubles the wait after each failure: half a second, one, two, four. Jitter adds randomness so that a fleet of clients that failed together does not retry together. Without jitter, the retries arrive in a thundering herd and the limit is hit again on the same schedule. The herd is worst right after a deploy, when every instance starts at once.

Cap the delay and the number of attempts. A retry policy with no ceiling turns an outage into a queue of doomed requests, each holding a connection and a worker while it waits. The cap should be a number the product can tolerate: a user who has waited eight seconds is usually gone, no matter what the ninth second would have brought.

Backoff as a decorator

from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential_jitter,
)

@retry(
    retry=retry_if_exception_type(RateLimitError),
    wait=wait_exponential_jitter(initial=0.5, max=8),
    stop=stop_after_attempt(5),
)
def complete(prompt):
    return client.complete(prompt)

Pace before you are paced

A client-side token bucket keeps you under the limit instead of discovering it. Tokens refill at the allowed rate, each request takes one, and a request with no token waits. The queue is visible and bounded, which is better than a burst followed by a wall of 429s. The bucket also makes the cost of a burst explicit: capacity is how much you are willing to spend at once.

The bucket should be per provider and per model, because limits usually are. A single global bucket either throttles a cheap model to protect an expensive one or lets the expensive one through and trips its limit. The same logic applies per tenant: one noisy customer should not consume the quota that everyone else is waiting on.

Shed load deliberately

Retries and queues both add work during an incident. At some point the correct behaviour is to reject: return a clear “busy, try later” to the caller, keep the queue short, and let the system recover instead of amplifying the load. A queue is a buffer, not a strategy; past a certain length it only changes the shape of the failure.

That is a product decision as much as an infrastructure one. A queued request that eventually succeeds after ninety seconds may be worse than an immediate error the user can act on, especially when the user is waiting in a chat box. Decide the maximum wait before you build the queue, and let the queue reject at that boundary.

A token bucket is also a useful teaching object because it makes the trade visible in one place. Capacity is the burst you allow, rate is the sustained load you pay for, and the queue length is the latency you accept. Changing any one of them changes the product, which is why the numbers belong in config rather than in a constant.