Structured output and tool calls

Prompted JSON fails at the edges

A model asked for JSON usually returns JSON, until it returns a sentence before the JSON, a trailing comma, a markdown fence, or a field with the wrong type. The failure rate is low per call and certain over a million calls, which is why “usually” is not a schema. The failures also cluster: a prompt change that helps one model can make another wrap the object in prose.

The fix is to make the shape part of the request. JSON mode constrains the output to valid JSON; a JSON schema constrains the fields and types as well. Providers differ in how strictly they enforce it, so validation stays on your side either way. Treat the schema as the contract and the prompt as the hint that makes the contract likely to be honoured.

The same applies to numbers and booleans. A model may return “3” instead of 3, or “true” instead of true, because the JSON is generated token by token and the quotes are cheap. A validator that checks types catches this; a parser that only checks syntax does not, and the bug surfaces later as a comparison that is always false.

Validate, then repair once

Parsing is not validation. A parsed object can still be missing a required field or carry a string where a number belongs. Validate against the same schema you sent, and on failure make one repair attempt with the error message in the prompt. The error message is the useful part: “priority should be int” tells the model what to change, while “invalid” tells it nothing.

One attempt, not a loop. A model that failed the schema once will often fix it when shown the validation error, and a model that fails twice is usually failing for a reason a third call will not change. Return a typed error and let the caller decide. A repair loop that runs until success hides a prompt problem and turns a bad request into an unbounded bill.

A schema the SDK can enforce

from pydantic import BaseModel
from openai import OpenAI

class Ticket(BaseModel):
    title: str
    priority: int
    tags: list[str]

client = OpenAI()
ticket = client.beta.chat.completions.parse(
    model="gpt-4.1-mini",
    messages=messages,
    response_format=Ticket,
).choices[0].message.parsed

A tool call is a protocol

A tool call is not the model doing something. It is a message with an id, a function name and a JSON string of arguments. Your code parses the arguments, decides whether to run the function, and appends a tool message with the same id and the result. The model then continues. The id is what ties the result to the request, so it must be preserved exactly.

The loop is yours, which is the point: validation, permissions and the actual execution all happen in code the model cannot edit. The model proposes; the program disposes. That separation is also what makes the tool layer testable, because a fake model can emit a scripted call and the dispatcher can be tested without a provider.

Parallel calls and strict schemas

A model can request several tool calls in one turn. Run the independent ones together, return all the results in one user turn, and let the model reconcile them. Serialising them costs latency for no gain. Calls that depend on each other are the exception: those belong in separate turns, because the second argument may depend on the first result.

Strict schemas make the arguments predictable: no extra fields, no missing required ones, types enforced. They also make the tool definition longer, which costs tokens on every request. Define the schema once, keep it small, and reuse it for validation on the server side. The schema is both the instruction to the model and the check on the model, which is why it should have exactly one home.

Tool definitions are also a prompt, and a long one. Every tool costs tokens on every request, whether or not it is called, so a registry of thirty tools is a tax on every conversation. Group related actions behind one tool with an action field, or load the definitions for the task at hand, but measure the effect on selection quality.