Feedback loops and labels

Feedback is an event, not a column

A thumbs-down is not a field on a request; it is an event that arrives later, sometimes from a different service, and it should be appended rather than overwriting the original record. Link it by trace id, timestamp it, and record the source: explicit rating, correction, accepted edit, retry or abandonment. The same request can collect several signals over time, and they can disagree. A rating from an agent and a rating from a user are different evidence, even when the number is the same. Recording the source is also the only way to compare feedback across channels.

Appending keeps the history. A trace that was marked helpful and then corrected a week later tells a different story from one that was never rated. The event log also lets you change the label schema without rewriting the traces. When the definition of a failure changes, the old events still say what the user did, which is the part that does not change. A correction is often more informative than a rating: it says what the right answer was.

Explicit feedback is biased

People who respond are angry or delighted; silence is not satisfaction. A helpful rate computed from thumbs is a rate among the people who chose to press a button, not among users. Implicit signals are noisier but broader: did they copy the answer, edit it, retry the same question, or leave? Each is a hint, none is a verdict. A user who leaves without rating may be satisfied, confused or gone; the absence is not a zero. Treat silence as unknown, not as approval.

Use both, and state the denominator. If the helpful rate is 70 percent of 40 ratings, the number is not the same as 70 percent of 4,000 requests. The first is a signal to investigate; the second is a product metric. A dashboard that shows the percentage without the count invites a decision the data cannot support. Segment the rate before comparing versions, or a change in the mix of users looks like a change in quality.

Attach a score to a trace

from langfuse import Langfuse

langfuse = Langfuse()
langfuse.create_score(
    trace_id="8f2c1a",
    name="helpful",
    value=0,
    comment="said thirty days; the policy says five",
)

Labels need a definition

A label without a definition is an opinion with a database column. Write a rubric: failure mode, severity, expected answer, and examples of each. Two labelers should agree on a sample before you trust the labels; if they do not, the rubric is the thing to fix. Agreement is not bureaucracy, it is the difference between a dataset and a collection of moods. The rubric should include a catch-all class, so a labeler is not forced to choose between two wrong categories.

Keep the label schema small enough to remember. Five failure modes that people apply consistently beat twenty that nobody can distinguish. Version the rubric, because a label from last quarter may mean something different from the same label today. The version belongs on every labeled case, next to the trace id. A label that requires a paragraph of explanation is usually two labels that have been merged.

Close the loop

Negative feedback becomes useful only when it changes something. Deduplicate, redact, and turn the case into an eval item with the question, the context and the expected answer. Then fix the prompt or the retriever and rerun the set. The loop is complete when the case is in the regression set, not when the ticket is closed. The expected answer is the part most teams skip, and the part that makes the case runnable.

A feedback dashboard that nobody acts on is a museum. The weekly question is not how many thumbs-down arrived; it is which case was added and which change made it pass. That is the difference between listening and measuring, and it is the only part of the loop that compounds. The review should end with a named change and a date, not a general intention to improve.