EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

constraint
A limit on the answer that a program can check, such as a length, an allowed label or a required citation.
context
The background material the model needs to answer well, kept separate from the thing it must act on.
continuation
The text a model writes next, given everything that came before it.
delimited block
Text wrapped by matching markers, such as triple backticks, so its edges are unmistakable.
distribution
The pattern of how often each possible value appears, or how likely each one is.
few-shot example
A worked input and answer pair placed in the prompt so the model can copy the pattern.
instruction
The sentence that names the task, usually starting with a verb.
JSON
A text format for structured data built from names, values, lists and nested braces.
model
The trained program that reads text and predicts what comes next.
output contract
The part of a prompt that states the shape of a valid answer, in words a checker can enforce.
parser
The code that turns the model's text into a value your program can use.
prompt
Everything sent to the model in one request: instructions, background, the input and the answer's required shape.
prompt injection
Text that tries to give the model orders, either typed by a user or hidden inside a document it reads.
register
The level of formality and the kind of wording suited to a particular audience or setting.
schema
A named set of fields with their types and allowed values, used to check an answer's shape.
system message
The opening instruction the model treats as standing policy for the whole exchange.
temperature
A setting that controls how much randomness the model adds when choosing each token.
token
A chunk of text — often part of a word — that the model reads and bills for.
user message
A turn written by the person using the app, treated as data rather than standing policy.

Prompt engineering from the ground up

Capstone: a prompt harness

Three prompt variants, one set of cases, and a table that says which one to ship. The whole course, applied to the decision people actually argue about.

Part 10 of 10

Advanced20 min readBack to Courses

The question a harness answers

Every prompt change invites the same argument: someone prefers the new wording, someone else prefers the old. Without a set of cases, the argument is about taste and the loudest voice wins.

A harness replaces the argument with a comparison: these cases, these checks, this pass count, these failures. It does not make the decision for you, but it makes the decision about evidence.

A case is three things: an input, the property the answer must have, and a check that decides whether it does. “Refund question, must mention five business days, checked by substring” is a case. Twenty of those are worth more than two hundred opinions, because they run in seconds and give the same verdict to everyone. The harness is not a benchmark; it is your own traffic, written down.

Three variants, one contract

The variants must share an output contract, or the comparison is meaningless. If one prompt returns JSON and another prose, the checker is measuring two different products.

Hold the model, the temperature and the cases fixed, and change one thing at a time. A variant that changes the instruction, the examples and the format together cannot tell you which change helped.

The shape of the run

for name, prompt in VARIANTS.items():
    results = [check(run(prompt, case), case) for case in CASES]
    print(name, sum(not r for r in results), "/", len(results))

Three prompts, one table

Run the comparison, then add a fourth variant that ignores the length limit. The table should show it failing the same cases the baseline passes.

Output
Click "Run" to execute your code

Cases are the asset

The cases matter more than the variants. Start with ten real inputs that the current prompt handles, and five that it fails. Write the check as a property rather than an exact string: “mentions the refund window” survives a rewording, while “equals this sentence” fails on a comma and teaches you nothing about quality.

A common mistake is a check so literal that a variant passes by echoing the case. If the case says “must contain five business days” and the prompt is allowed to repeat the question, a lazy variant can score well without answering. The symptom is a variant that wins the harness and disappoints in production; the fix is a check that requires the answer to be derived, not copied.

Read the failures, not just the score

Two variants can pass the same number of cases for different reasons. One fails the long inputs, the other fails the refusals. The score says they are equal; the failure list says which one is safer for your traffic.

That is why the harness prints a row per case. A summary alone hides the only information that distinguishes a lucky prompt from a robust one.

Keep the case ids stable and print them beside the failures. “Case 7 failed” is actionable when case 7 is the gift-card refusal; “two failures” is not. Stable ids also let the gate compare this run with the last one without diffing prose, and they are the only part of the harness that must not change casually.

Keep every variant that ever shipped, with its score. The history is how you learn which kinds of edits actually help on your task, rather than in general.

From harness to habit

The first harness is an afternoon. The habit is what compounds: every production failure becomes a case, every prompt edit runs the set, every release compares against the last.

That is also the point where the evaluation course picks up — confidence intervals, judges and gates are the same harness with more rigour around the edges.

Keep the harness small enough to run on every change. A suite that takes twenty minutes is a suite people skip, and a skipped suite is worse than no suite because it looks like coverage. Twenty cases that run in ten seconds will be run; two hundred that need a GPU will be run once, at the launch, and then forgotten.

Start with the cases you already have: the bug report from last week, the question the support team keeps asking, the input that made the model refuse. Each one is a case with a known-good property, and each one costs a few lines. The harness grows from the work rather than from a taxonomy someone invented in advance.

Fail the release on a regression

Compare a candidate against the baseline and print the cases that regressed. Then fix one and confirm the gate passes.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does a prompt harness replace?