Build a feature on a language model
Treat a model as an unreliable dependency: validate what it returns, measure it with an evaluation set, and bound what each call may cost.
BY THE END, YOU CAN
- Treat a model’s output as untrusted input and validate its structure
- Build an evaluation set that catches regressions when the prompt or the model changes
- Bound the cost and latency of a model call, and plan for its failure
Diagram description: Error message → Limits; Limits → Model; Model → Validate; Validate → Use it (valid); Validate → Fallback (invalid); Evaluation set → Model (measures)
The model is an unreliable dependency
Suppose Relay asks a language model to read a failed job’s error message and return a category (network, data, or code), whether a retry makes sense, and a one-line summary.
The model returns likely text, not checked text. NIST’s Generative AI Profile calls confidently stated false content confabulation; the model can also break the format you asked for, and the same input can produce different answers. Design for that, as you would for a flaky external service whose answers are suggestions.
Ask for JSON with a fixed set of fields; some model APIs can hold the output to a schema, but validate it anyway: required fields, types, values from a closed list, and lengths. When validation fails, retry once or fall back, for example to the category unknown and a person who reads it. Never pass model output to a shell, eval, a SQL string, or a web page without escaping: OWASP’s advice on improper output handling is to treat the model like any other user.
Text in the input can carry instructions, and they can reach the output.
import json
CATEGORIES = {"network", "data", "code"}
def parse_triage(text):
try:
data = json.loads(text)
except ValueError:
return None
if not isinstance(data, dict) or data.get("category") not in CATEGORIES:
return None
if not isinstance(data.get("retry"), bool):
return None
summary = data.get("summary")
if not isinstance(summary, str) or not 0 < len(summary) <= 200:
return None
return {"category": data["category"], "retry": data["retry"], "summary": summary}Evaluate before and after every change
A prompt that worked on three examples has been tried, not tested.
NIST’s profile warns against judging a generative system from narrow, anecdotal assessments. Build an evaluation set instead: inputs that look like real traffic, each with its expected result, including hard cases and hostile ones, such as an error message that contains instructions, an empty message, and a very long one. Grade with code wherever you can: the category matches, the format is valid.
Keep human review or a written rubric for free text such as the summary. Run the whole set on every change to the prompt, the model, or the provider’s version, the way you run a regression suite, and compare the pass rate per category and the list of failing cases, not one overall number. Because answers vary, run each case several times and count a case as passed only when every run passes.
Keep part of the set aside and never tune against it, and rerun the set on a schedule, since a hosted model’s behavior can change over time.
Bound the cost and the wait
Every call costs money and time, and both grow with the text you send and the text that comes back.
Without limits, one user, or one attacker, can run up the bill or slow the service for everyone; OWASP calls this unbounded consumption. Cap the size of the input and the length of the output, set a timeout on each call, and give each user a rate limit and a quota. Cache the answer for identical inputs when the same question comes back often.
Keep slow calls out of the request path: Relay can triage a failure in the worker, after the job is recorded, instead of while a client waits. Decide what happens when the model is down or too slow, and prefer a degraded answer to a failed job: the category unknown, routed to a person. Log the tokens and the latency of every call to watch cost and speed over time.
A smaller, cheaper model may be good enough; the evaluation set is how you find out.
COMMON FAILURE MODE
Shipping a prompt change because a few examples looked better. Without an evaluation set, nobody knows which cases it broke.
TRY IT YOURSELF
Write twelve evaluation cases for Relay’s triage: four ordinary, four hard, and four hostile, one of them with instructions inside the error message. For each, write the expected category and whether a retry makes sense.
Make the call in the laboratory.
The two-question checkpoint, your notes, and your saved progress are in the laboratory. Free, with no account.
Open in the laboratory ↗CONNECT IT TO RELAY
Add failure triage to Relay’s worker: call a model with a timeout and a size limit, validate the JSON with parse_triage, fall back to unknown, and log tokens and latency. Write an evaluation set of at least twenty cases and run it in CI whenever the prompt or the model changes.