Anatomy of a request

The request is small

Every chat API is a variation on one shape: a model name, a list of messages, and a handful of parameters. Each message has a role — system, user, assistant, tool — and content. The system message sets behaviour, the user message carries the task, and the assistant messages are what the model said before. The order matters: the model reads the list as a transcript, and the last message is the one it is answering.

The knobs are few. temperature controls how much randomness the sampler applies, max_tokens caps the output, and stop ends generation early. Everything else is provider-specific sugar on top of that shape. A parameter you cannot explain is a parameter you should leave at its default until you can.

The request shape is also the portability layer. Every provider maps the same four roles onto its own API, so a client that builds this object once can switch models by changing a name and a base URL. The differences that remain are the ones worth reading the docs for: tool-call formats, cache controls and the exact meaning of a finish reason.

The server remembers nothing

A completion request is stateless. The model does not remember your last call, the conversation does not exist on the server, and there is no session unless the provider added one as a product feature. If you want the model to know what was said, you resend it. That single fact explains most of the cost and most of the bugs in a chat product.

That is why context is a client-side problem. The history you send is the model’s entire memory of the conversation, and every turn you drop is a fact it no longer has. The bill for a long chat is the bill for resending the transcript on every turn. A feature that “remembers” is a feature that stores, retrieves and reinserts, not a model that learned.

The shape, with the official SDK

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "You are terse. One sentence."},
        {"role": "user", "content": "What is a token?"},
    ],
    temperature=0.2,
    max_tokens=120,
)

print(response.choices[0].message.content)
print(response.usage)

The response tells you what happened

A response carries the message, a finish_reason and a usage block. The finish reason is the part people ignore: stop means the model finished, length means max_tokens cut it off, and tool_calls means it wants to call something. A truncated answer presented as a complete one is a bug you can see in the response. The same field also tells you when a safety filter ended generation instead of the model.

Usage is the other half. Input tokens, output tokens and, on some providers, cached tokens are what the bill is computed from. Log them per request from the first day; reconstructing cost later is guesswork. The usage block is also the cheapest way to notice that a prompt grew, a retry loop doubled the calls, or a cache stopped hitting.

The SDK is a thin wrapper

The official SDKs build a JSON body, sign an HTTP request, parse the response and raise typed errors. There is no magic and no hidden context. When something behaves unexpectedly, the fastest diagnosis is usually to log the body you sent. If the body is right and the answer is wrong, the problem is in the prompt or the model, not in the library.

That also means you can use the HTTP API directly. It is worth doing once, with curl or httpx, to see the shape without the SDK in the way. After that the SDK is a convenience rather than an article of faith. Knowing what it sends is what lets you switch providers without rewriting the product.

That is also why the body is worth logging in a structured form rather than as a string. A JSON object with the model, the roles, the token estimate and the request id can be searched, counted and diffed; a pretty-printed blob in a log file cannot. The log is the only record of what the model actually saw.