LLM APIs from the ground up
Requests, retries and bills you can reason about
- Beginner
- 10 chapters
- 151 min read
What actually goes over the wire when you call a model: messages and roles, tokens and context budgets, streaming deltas, timeouts and retries, idempotency keys, rate limits and backoff, schemas and tool calls, prompt caching and cost, and a capstone client that survives a scripted outage.
What You'll Learn
- Describe the request and response shape every chat API shares
- Budget tokens across input, history and output
- Assemble a streamed answer and handle a partial result
- Choose timeouts and retry only the failures worth retrying
- Make a retried side effect safe with an idempotency key
- Pace requests with backoff, jitter and a token bucket
- Get validated structured output and run a tool-call loop
- Measure and reduce cost, and build a client that survives failure
- Run concurrent requests with a bounded fan-out, timeouts and cancellation
- Move between providers behind an adapter without rewriting the product
Prerequisites
- Comfortable with Python functions, dictionaries and exceptions
- No API key is needed — every example runs offline against scripted responses
- No prior experience with a model provider is assumed
Vocabulary
The terms the chapters assume, in plain words.
- backoff
- Waiting longer between each retry, so a service that is struggling gets room to recover.
- context window
- The most text, counted in tokens, that a model can consider in a single request.
- cost
- The money a request is billed for, usually based on how many input and output tokens it uses.
- idempotency
- The property that repeating the same request has the same effect as sending it once.
- jitter
- A small random amount added to a wait, so many clients do not retry at the same moment.
- latency
- The time from sending a request to receiving the full answer, including time spent waiting in a queue.
- prompt caching
- Reusing the stored copy a service keeps of unchanged leading text so later requests cost less and start faster.
- rate limit
- A cap on how many requests you may send in a period, after which the service refuses more.
- schema
- A written description of the fields a response must contain and the type of each value.
- streaming
- Receiving an answer in pieces as it is produced instead of waiting for the complete response.
- structured output
- A response constrained to a schema, so every field and type can be checked and used by code.
- timeout
- A limit on how long a request may wait before your code gives up and treats it as failed.
- token
- A small unit of text, such as part of a word, a whole word or punctuation, that a model reads and bills for.
- tool call
- A request from a model for your code to run a named function with arguments, with the result sent back.
Course Content
- 01Anatomy of a requestNot started
An LLM request is a JSON object with a model, a list of messages and a few knobs. The server keeps no state, so everything the model knows about the conversation is what you send.
- 02Tokens and context windowsNot started
Models read tokens, not words, and the context window is a hard budget shared by input and output. Plan the budget before the request, because the API will not do it for you.
- 03StreamingNot started
Streaming sends the answer as it is generated, one small delta at a time. It changes perceived latency, not total latency, and it makes assembly and cancellation your problem.
- 04Retries, timeouts and idempotencyNot started
A network call can fail in three ways: it can be slow, it can fail before the server acts, or it can fail after. Timeouts bound the first, retries help the second, and idempotency keys make the third safe.
- 05Rate limits and backoffNot started
Rate limits are quotas per minute, per token and per connection. The server tells you when you have crossed one; the client’s job is to slow down in a way that does not make the problem worse.
- 06Structured output and tool callsNot started
Asking for JSON in the prompt is a request, not a guarantee. Schemas, validation and a repair step turn a hopeful parse into a contract; tool calls are the same contract with an action attached.
- 07Cost and cachingNot started
The bill is input tokens times a price, output tokens times a bigger price, and a cache that only helps when the prefix is stable. Measure cost per request before you optimise it.
- 08Async and concurrencyNot started
A model call is mostly waiting, so async lets one process wait on many requests at once. The work is in bounding the fan-out, cancelling what nobody needs and keeping the provider inside its quota.
- 09Provider portabilityNot started
“OpenAI-compatible” covers the common request shape and stops before the features you actually need. Own the types, normalise the errors and choose providers by capability and evaluation, not by brand.
- 10Capstone: a resilient clientNot started
One client that times out, retries with jitter, respects rate limits, validates output, accounts for cost and survives a scripted failure. Every chapter of the course, once.