Async and concurrency

Concurrency is waiting better

A model call spends almost all its time waiting: the request travels, the provider queues it, tokens arrive one at a time. Async I/O lets one process wait on hundreds of sockets without a thread each, which is why the Python SDK ships an async client.

It does not make the model faster and it does not create capacity. Throughput is set by the provider’s rate limit and by how many requests you are allowed to have in flight; concurrency only decides how well you use that allowance. A queue of ten thousand waiting requests is not throughput; it is memory, timeouts and a bill that arrives after the incident.

The event loop is single-threaded, so one blocking call stops everything. A synchronous database driver, a CPU-heavy parse or a time.sleep inside a coroutine turns a concurrent service into a queue. Keep blocking work in a thread or a separate process.

Bound the fan-out

The naive version is gather over every prompt. Ten thousand prompts become ten thousand simultaneous requests: the provider rate-limits most of them, the client buffers ten thousand responses, and every timeout fires at once. The failure looks like a provider outage and is really a client that never chose a limit.

A semaphore is the smallest fix. Pick the limit from the quota and the latency: with a hundred requests a second and one second per request, about a hundred in flight keeps the pipe full. Little’s law is the arithmetic, and a load test is the confirmation.

One global limit is a blunt instrument. Give interactive traffic its own pool, keep bulk jobs in a queue with a lower share, and let a tenant’s limit be smaller than the process limit. The queue depth is the signal to shed load before the provider does it for you. The queue depth is also the metric to alert on, because it moves before the latency does.

A semaphore around the async client

import asyncio

from openai import AsyncOpenAI

client = AsyncOpenAI()
in_flight = asyncio.Semaphore(8)


async def ask(prompt: str) -> str:
    async with in_flight:
        response = await client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{"role": "user", "content": prompt}],
        )
    return response.choices[0].message.content


answers = await asyncio.gather(*(ask(prompt) for prompt in prompts))

Cancellation, timeouts and partial results

Every awaited call needs a deadline, and the deadline should be smaller than the caller’s. asyncio.wait_for cancels the coroutine when the budget expires; the cancellation must reach the HTTP client so the socket closes rather than lingering until the provider answers. The deadline belongs in the client configuration, not in a comment next to the call site.

gather with return_exceptions=True gives a partial result: the successes come back as values and the failures as exceptions, in input order. A task group is stricter and cancels siblings on the first failure, which is right when the batch is all-or-nothing and wrong when it is a list of independent questions. A batch of independent questions is the common case, and partial success is usually more useful than a clean failure.

Cancellation is a feature, not an error path. When the user closes the tab, cancel the request and stop paying for tokens nobody will read. Catch CancelledError only to clean up, then re-raise it; swallowing it leaves the task alive and the bill running.

Batching and backpressure

Batch endpoints trade latency for price: a batch may finish in minutes or hours, and it is the right tool for backfills and evaluations, not for a chat reply. Batch by token budget rather than by count, because a hundred short prompts and a hundred long ones are different jobs. A batch job that is ninety percent done is not resumable without a record of which items completed.

Backpressure belongs before the provider. A bounded queue that rejects when full, a token bucket shared by every worker, and a retry that sleeps outside the semaphore slot are the three pieces that keep a spike from turning into a retry storm.

Measure the queue, not just the call: wait time, in-flight count, p95 latency and the rate-limit headers. A dashboard that only shows successful requests hides the minute a request spent waiting for a slot, which is the part the user experienced.