Output validation and moderation

Output is input

The model’s output leaves the model and becomes input to something else: a browser, a shell, a SQL client, another model, a colleague’s inbox. Every one of those has its own injection surface. Markdown that renders HTML is cross-site scripting; a URL fetched by a server is SSRF; a string concatenated into a query is SQL injection; a command run with the user’s privileges is remote code execution.

The mental model that works is a form submission from an untrusted client. You would not render a stranger’s HTML, follow a stranger’s URL or execute a stranger’s command because it arrived in a JSON field. The model is that stranger, however helpful its tone.

The sink matters as much as the text. The same summary is harmless in a log and dangerous in innerHTML, harmless in a chat window and dangerous in a shell. Classify the sink before you decide how much to trust the output, because the sink is what turns text into an effect.

Shape first

Validate the shape before the content. A schema check catches the wrong type, the missing field, the unexpected property and the value outside an enum, and it does so deterministically. Parse into a typed object, not a dictionary you index later. Then check the domain: the id exists, the amount is within the cap, the date is in the future.

When validation fails, you have three options: repair, retry or refuse. Repair means a deterministic fix such as trimming or filling a default; retry means asking the model again with the error message; refuse means stopping. Retry once, then refuse. A loop that retries until the JSON parses is a denial of wallet with a nice name.

Structured output helps, but it is not validation. A model can return valid JSON with a negative quantity or an id from another tenant; the schema only says the shape is right. The domain check is still yours, and it still runs after the parse.

Parse, then sanitise

import bleach
from pydantic import BaseModel, Field, ValidationError


class Answer(BaseModel):
    summary: str = Field(max_length=800)
    confidence: float = Field(ge=0, le=1)
    sources: list[str] = Field(max_length=5)


try:
    answer = Answer.model_validate_json(raw)
except ValidationError as error:
    print(error.errors())
    answer = None

if answer is not None:
    safe = bleach.clean(answer.summary, tags=[], strip=True)

Moderation is a classifier, not a judge

A moderation model returns a score per category. It is a signal with a threshold, not a verdict. Choose thresholds per category and per product: a children’s app and a security research tool should not share one. Measure the false-positive rate on benign traffic, because over-blocking is the failure mode users notice first.

Keep the decision record: category, score, threshold, action. That is what lets you explain a block and tune it later. Never treat the moderator’s label as the policy itself; the policy is yours, and the classifier is one input to it.

Moderation also has a language problem. A classifier trained mostly on English will miss the same content in another language, and the gap is invisible in an aggregate score. Measure per locale, or the average will hide the users you are failing.

URLs, HTML and code

URLs need an allow-list of schemes and hosts. https only, no userinfo, no redirects to a private address, and a DNS check at fetch time rather than at validation time. HTML needs a sanitiser with an allow-list of tags, not a block-list of script tags. Code needs a sandbox, not a regex.

The same rule applies to file paths and shell arguments: build them from a known base and a validated component, never by concatenation. If the model supplies a path, resolve it and check that the result is still inside the directory you meant.

For code, the only safe default is not to run it. If the product needs execution, use a sandbox with no network, a time limit and a disposable filesystem, and treat the sandbox as the control rather than the prompt. A regex is not a jail.