Structured output

Why a schema and not a format request

“Reply in JSON” produces JSON-shaped text: sometimes fenced, sometimes with a sentence in front, sometimes with a key renamed because the model preferred a synonym. A schema is stricter — it names the keys, their types and the allowed values.

The difference matters at the boundary. A parser can extract an object from almost anything; only a validator can tell you the object is usable. Both are needed, and they are different jobs.

At the API level, some providers can constrain generation to a schema, which removes the syntax errors but not the judgement errors. The model still chooses the values, and a value can be legal and wrong. Others only accept a request for JSON, which is a hint rather than a guarantee. Knowing which one you are using tells you whether the parser is a formality or the actual boundary.

Validate, then retry with the error

When validation fails, the useful move is to send the reply and the failure back: “the key confidence was missing” or “sentiment must be one of positive, negative, neutral”. A model that sees the specific complaint usually fixes it in one turn.

A blind retry is worse than no retry. It costs the same and has the same chance of failing, because nothing about the request changed. The error message is the change.

A schema the model can read

from pydantic import BaseModel, Field

class Verdict(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float = Field(ge=0, le=1)
    reason: str = Field(max_length=200)

judge = model.with_structured_output(Verdict)
verdict = judge.invoke(prompt)   # a Verdict, or an exception

Errors are instructions

A validation error is only useful if it names the field, the value and the rule. “Invalid output” tells the model nothing it can act on; “confidence must be a number between 0 and 1, got the string 0.9” tells it exactly what to change. The retry message should contain the original request, the previous reply and that sentence, in that order.

The symptom of a vague error is a retry loop that repeats the same failure. The model is not being stubborn; it has been told that something is wrong without being told what. If two attempts produce the same invalid object, stop and surface the raw reply — a third identical request is just a more expensive way to learn nothing.

The failure modes are predictable

A missing key, a value outside an enum, a number sent as a string, a list where one item was expected. Those four cover most failures, and each has a clear error message that a model can act on.

The fifth is subtler: a valid object with the wrong content. The schema cannot catch it, because the shape is correct. That is a job for evaluation, not validation — and it is why a passing schema is not a passing answer.

The fifth failure has a signature: the reply validates, the downstream code runs, and the answer is confidently wrong. A schema cannot help, because the shape is exactly what you asked for. The fix is a small evaluation set with known answers, run whenever the prompt or the model changes. Validation tells you the object is usable; evaluation tells you it is true.

Where the boundary belongs

Validate once, at the edge where the model’s text becomes your program’s data. After that, pass the typed object. Re-validating in three places means three definitions of correct, and they will disagree.

Record validation failures as a metric. A rising rate means the prompt and the schema have drifted apart, which is a prompt bug rather than a model problem.

Log the raw reply next to the validation result, not instead of it. When a failure rate rises, the raw text tells you whether the model changed its format or the schema changed its expectation. Without it you have a number and no cause, which is the least useful kind of metric.