Tokens and context windows

Tokens are pieces

A token is a chunk of text the tokenizer learned: a word, part of a word, a punctuation mark, a byte sequence for an emoji. English prose runs about four characters per token; code, JSON and non-Latin scripts run higher because they tokenize less efficiently. The ratio is not a property of the language, it is a property of the tokenizer, which is why a model upgrade can change your bill without changing a prompt.

That ratio is an estimate, not a rule. Counting with the model’s own tokenizer is exact and cheap; estimating with characters is fine for a budget and wrong for a hard limit. The difference matters at the boundary, which is exactly where limits are hit. Leave a margin of ten percent and the estimate is usually enough; leave none and the longest request is the one that fails.

The practical consequence is that token counts are not proportional to how much a passage means. A table of numbers, a stack trace or a paragraph of German can cost several times what the same number of English words costs, and the difference shows up in the bill before it shows up in the latency. Count the material you actually send.

The window is shared

A context window is the maximum number of tokens the model can attend to in one call: input plus output. A 128k window with a 4k output cap leaves 124k for the prompt, and the prompt includes the system message, the history, the retrieved documents and the tool definitions. Tool definitions are the line item people forget: ten tools with detailed schemas can cost more than the conversation.

Long context is not free attention. Models degrade on material buried in the middle, latency grows with input length, and the price is per token. A window is a limit, not a target. The best prompt is usually the shortest one that contains the evidence, and retrieval is how you keep it short without guessing.

Count with the model’s own tokenizer

import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")
tokens = encoding.encode(prompt)

print(len(tokens), "tokens")
print(encoding.decode(tokens[:40]))

Truncation is a policy

When the history does not fit, something must go. Dropping the oldest turns is simple and loses the setup. Summarising the old turns costs a model call and loses detail. Retrieving the relevant old turns costs an index and keeps what matters. Pick the policy by what the product needs to remember, not by which one is easiest to write.

The system message should never be the thing that falls off, and neither should the current user message. A truncation function that walks backwards and stops when the budget is full will drop both unless you pin them. Pin them explicitly, then trim the middle, and log when a trim happens so a quality complaint has a timestamp.

Budget for the answer

Reserve output tokens before you fill the input. A request that uses the whole window for context leaves no room for an answer, and max_tokens then truncates the response mid-sentence. The reservation is a product decision: how long should the answer be allowed to be? A support reply and a code review have different answers, and neither is “as long as the window”.

Track the ratio of input to output tokens over time. A retrieval system whose prompts grow every week is a system whose cost grows every week, and the growth is usually a history that was never trimmed. The chart to watch is not total spend; it is tokens per request and requests per user, because those two explain the total.

A useful rule for a new feature is to write the budget down before the prompt: system message, retrieved context, history and answer, each with a number. The numbers will be wrong, but they make the trade visible, and the first week of real traffic will show which one was optimistic. A budget nobody wrote down cannot be defended or improved.