EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

backoff
Waiting longer between each retry, so a service that is struggling gets room to recover.
context window
The most text, counted in tokens, that a model can consider in a single request.
cost
The money a request is billed for, usually based on how many input and output tokens it uses.
idempotency
The property that repeating the same request has the same effect as sending it once.
jitter
A small random amount added to a wait, so many clients do not retry at the same moment.
latency
The time from sending a request to receiving the full answer, including time spent waiting in a queue.
prompt caching
Reusing the stored copy a service keeps of unchanged leading text so later requests cost less and start faster.
rate limit
A cap on how many requests you may send in a period, after which the service refuses more.
schema
A written description of the fields a response must contain and the type of each value.
streaming
Receiving an answer in pieces as it is produced instead of waiting for the complete response.
structured output
A response constrained to a schema, so every field and type can be checked and used by code.
timeout
A limit on how long a request may wait before your code gives up and treats it as failed.
token
A small unit of text, such as part of a word, a whole word or punctuation, that a model reads and bills for.
tool call
A request from a model for your code to run a named function with arguments, with the result sent back.

LLM APIs from the ground up

Streaming

Streaming sends the answer as it is generated, one small delta at a time. It changes perceived latency, not total latency, and it makes assembly and cancellation your problem.

Part 3 of 10

Beginner14 min readBack to Courses

What streams

A streaming response is a sequence of server-sent events. Each event carries a delta: a few characters of content, a role announcement, a fragment of a tool call, or a finish reason. The client concatenates the deltas in order and the result is the message it would have received at once. The transport is plain text over HTTP, which is why a proxy that buffers can make a stream arrive all at the end.

The first token arrives much sooner than the last, so the user sees progress while the model is still working. Total time is unchanged, and for a long answer streaming can even be slightly slower because of the overhead per event. The benefit is psychological and real: a screen that fills is easier to wait for than a spinner that does not move.

The protocol is deliberately boring: lines of text, a blank line between events, a data prefix and a JSON payload. That is why a stream can be proxied, logged and replayed, and why a bug in the client is usually a bug in the assembly rather than in the transport. Read one raw stream once and the abstraction stops being magic.

You are the assembler

Nothing in the stream is a complete message. The content is a list of fragments, the tool call arguments arrive as a JSON string in pieces, and the finish reason may come in a separate event. If you display each delta as it arrives, you are showing the model’s draft, not its answer. The draft is useful for a chat UI and dangerous for anything that parses the result.

Accumulate first, render second. Keep the partial text for the UI and a separate buffer for the final message, and only persist the message when the stream ends with a finish reason rather than a dropped connection. The buffer is also what you validate, count and bill; the UI copy is disposable.

Read the deltas, keep the whole

stream = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=messages,
    stream=True,
)

parts = []
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        parts.append(delta)
        print(delta, end="", flush=True)

message = "".join(parts)

Assemble a message from deltas

Run the parser, then drop the DONE line and confirm the loop still ends on the finish reason. Add a second choice to one event and see what the parser does.

Output
Click "Run" to execute your code

Errors arrive mid-stream

A stream can fail after it has started: the connection drops, the provider returns an error event, or a proxy times out. At that point you have already shown the user half an answer, and the HTTP status was 200 when the request began. Error handling that only looks at the status code will call this a success.

The client needs a state for “partial”. Either retry from the beginning, which duplicates the visible text, or mark the message incomplete and offer a continue action. Silently keeping the partial text as if it were finished is the one option that is always wrong. A label costs one field and turns a mystery into a support answer.

A stream that ends without a finish reason is not a completed message. Treat a dropped connection as an error even though the text looks plausible — the missing half is often the part that matters.

Cancellation is a feature

Users stop answers. A stop button should abort the HTTP request, not just hide the output, or you keep paying for tokens nobody will read. Most SDKs expose the stream as a context manager or an object with a close method for exactly this reason. Closing the stream is also what frees the connection for the next request.

Cancellation also applies to your own timeouts. If the caller has a two-second budget and the model is still generating at three, the stream should be closed and the partial result discarded or labelled. The deadline belongs to the caller, not the provider. A stream that outlives its caller is work that nobody will read and somebody will pay for.

Cancellation also has a cost profile. Aborting early saves output tokens but not the input tokens already sent, and some providers bill the generated prefix anyway. That is still cheaper than finishing an answer nobody wants, but it is not free, which is why a stop button should be a deliberate product feature rather than a side effect of closing a tab.

Stop reading, keep the partial

Run the collector, then raise max_chars to 40 and confirm the stream finishes on its own. Add a final chunk after the cutoff and check that it is never read.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does streaming change?