Every AI feature has a definition of good output somewhere. Sometimes it is written down. More often it lives in the head of whoever set up the evaluation, which usually means an engineer, under a deadline, checking for the things that are easiest to check. Writing that definition is design work, and designers should own it.
An eval, for anyone who has not met one, is a set of inputs and a set of criteria. The feature runs on the inputs, the output is checked against the criteria, and the whole thing runs again every time the instructions the feature is given (the prompt), the model or the data changes. The machinery that does the running is engineering. The criteria are judgment: a list of what this feature must do, must never do, and is allowed to do badly.
Itamar Medeiros makes the same split in a piece on evals for designers and product managers (opens in a new tab): engineers build and run the machinery, but the inputs that make it mean anything are design and product inputs, and when engineers define the criteria alone, the eval drifts towards what is easy to measure. His line for an eval without that work is a good one: "An eval without good criteria is a ruler with no units."
The same judgment as a design review
In a design review I look at a screen and say something like: this decline message does not tell the person whether they were charged. That sentence is a criterion. It is said once, about one screen, to the people in the room, and it is gone by the next release unless someone remembers it.
An eval is that sentence written down so it applies to every output the feature will ever produce. Nothing about the judgment changes. What changes is that it stops depending on who was in the room, and it stops expiring.
That is why I do not accept the framing of evals as a testing concern that design hands over to engineering. A team that lets someone else write the criteria has let someone else run the design review, permanently, on every output, without a designer present. The judgment layer does not go away because the output is generated. It just has to be written down, or it does not happen at all.
Payments makes the criteria concrete
I design payment systems at Uber, and payments is a good place to learn this, because "good enough" has a price. A wrong amount is a wrong amount. A charge that failed and reads as succeeded becomes a support ticket, then a refund, then a customer who does not come back. A decline message that leaves someone unsure whether money moved is not a tone problem, it is the whole problem. I have written about the states payment flows forget, and every one of them is a place where a generated answer can be confidently wrong.
That pressure is useful, because it makes vague criteria impossible to defend. Imagine an assistant that answers the question "why was I charged twice?" A criterion like "the answer is helpful and accurate" checks nothing. These check something:
- It never states an amount that is not in the transaction data it was given.
- It tells money the bank has only reserved apart from money that has actually been taken, and says which one this is.
- It says what happens next and when, or says plainly that it does not know.
- It never promises a refund, because it cannot issue one.
Each of those is something a checker, a person or a program, can mark as pass or fail on a single output. That is the test for whether a line belongs in an eval.
What an eval written by a designer contains
Pairs of good and bad output
Not a definition of good, which nobody can apply consistently, but examples: this answer passes, this one fails, and here is the sentence that explains the difference. The bad examples matter more than the good ones, because they mark the edge. A good answer shows what you hoped for. A bad answer that looks almost right shows where the line is.
The situations
The same list a designer already keeps for a screen. The input is empty. The input is in another language. The data is missing a field. The payment is still pending. The customer asks something the feature was never meant to answer. The model will meet every one of these in its first week; the eval should have met them first.
The unusual cases that cost something
Not every unusual case deserves the same attention. The ones worth writing first are the ones where a wrong answer costs money, trust or a support contact. In payments that list writes itself. In other domains it takes longer, and working it out is design research.
Tone, as rules that can be checked
"Friendly" is not a criterion. "Does not blame the customer", "apologises at most once", "names the next step in the first two sentences" are. Most of what makes an error message work translates directly, because a generated answer that tells someone something went wrong is an error message, whatever the feature calls it.
Which failures block a release
Some criteria are hard: a single wrong amount fails the release. Some are soft: a slightly stiff phrasing is tolerable at a low rate. Deciding which is which is a decision about the cost of being wrong, and that is the same decision a designer makes when choosing which state to design first.
How it differs from a specification
A specification, the document that tells a team what to build, describes one thing. It is read by a person, mostly once, and then the built product replaces it as the source of truth. It can afford gaps, because the engineer reading it fills them, ideally in conversation with the designer.
An eval describes a range of acceptable outputs, because a generative feature never produces the same answer twice. It is read by a machine, on every change, for as long as the feature exists, and nothing ever replaces it, because the output is never finished. And it cannot afford gaps. A criterion that a checker cannot apply to a single output is not a criterion, it is a hope.
So the craft is different. A specification designs an answer. An eval designs the boundary of a space of answers, and decides what happens at its edges.
Who holds the pen
Engineers should build the machinery, choose the tools and run it. Designers should write the criteria and the examples, decide with product how many failures of each kind are acceptable, and read the failures, because a failing output is a design review waiting to happen.
I already work this way with my coding agents, in a smaller form: I give them written standards because I am not watching while they work, and anything I want to hold has to be written somewhere they will read it. An AI feature in production is the same situation at a larger scale. Nobody is watching each output. The criteria are the only part of the designer that is in the room.
A design review protects one release. Written as an eval, it protects every one after it.