I was the product manager, and I wrote the stories. For the AI features they read like this: something like this, for something like that. No strict positive cases, no negative ones. The developers built what they understood from it. The client rejected what we shipped.
That happened more than once. What got us out was replacing the rules in the story with examples.
Why the old acceptance criteria stopped working
A normal feature can be specified as a rule. Given this input, return exactly this. A developer can write a test for it, and "done" means the test passes.
An AI feature doesn't fit that shape. The same input can give a different answer each time. Most answers are not right or wrong in a way an equality check can see. A summary can be correct and useless. An answer can be well written and come from the wrong version of a document.
So the stories went vague, because the rules we knew how to write didn't apply. And vague stories gave us features that demoed well and fell flat on real user data. The client's review of one of them was a single line: "It just does not work as expected."
The team was reluctant to change how we worked. What changed our minds was the client not accepting anything we shipped. After that we started defining each request in much more detail.
Acceptance criteria as examples
What we ended up with was a set of examples instead of a set of rules. For each feature:
- Real inputs. Questions users actually ask, documents they actually upload. Not ones we made up to look tidy.
- For each input, what a good answer looks like.
- For some inputs, what a bad answer looks like, and why it is bad.
- A bar. The feature gets at least this many right before we ship.
"Done" then meant passing that set. Not passing a demo.
This is close to what Chip Huyen calls evaluation-driven development in AI Engineering: before you build, write down how this particular application will be judged. She also calls evaluation the biggest bottleneck holding back AI adoption. From the product side, I'd agree. Our bottleneck was that nobody had written down what correct meant.
Hamel Husain's guide on using an LLM as a judge adds two details that made our examples better. First, label each example pass or fail, not 1 to 5, because a 3 out of 5 gives nobody anything to act on. Second, write a short critique for each label: why it passed, why it failed. The critique is where the real requirement lives.
The questions I started asking clients
The examples only work if the client helps make them. So the first meetings changed. These are the questions I never asked for a normal feature and now ask every time:
- Show me real examples of what your users will ask. Not a description of the users. The actual messages, files, search terms.
- What should happen when the AI doesn't know? Say so, hand over to a person, or give its best guess with a warning. Each is a product decision, and each needs examples.
- What does a wrong answer cost you? A wrong policy answer to an employee is not the same as a wrong number in a contract. This sets the bar.
- Who checks the output before it matters? If a person reviews every answer, the bar can be lower and the review screen matters more. If nobody does, the bar goes up.
The answers to these four questions become the first rows of the example set.
How to say "good enough"
The hardest conversation for a PM on an AI feature is the one where the client says it isn't good enough and the team says it is.
The fix is to have that conversation before building. You agree on the examples and the bar together. Then it isn't my opinion against theirs. It is the score on their own examples.
Husain makes a related point in Your AI Product Needs Evals: the pass rate you aim for is a product decision, not always 100%. That is exactly the PM's job. Someone has to decide whether, say, 27 out of 30 is fine for a search assistant, and whether anything that writes to a customer record needs all 30. That someone should do it in writing, before the sprint, with the client in the room.
And show the failures, not just the wins. We show the client the answers that failed and why. It tells them we know where the feature breaks, which a demo never can, because a demo only shows the wins.
The list will change, and that is fine
You won't get the examples right the first time. That is not a failure of planning.
There is a research name for it. Shankar and others call it criteria drift in Who Validates the Validators?: people need criteria to grade outputs, but grading outputs is what helps them define the criteria. In practice it looks like this: a client reads twenty answers and only then says the feature should never show an internal ID. Nobody would have written that rule up front. Now it's a row in the set.
Some of those new rows can be checked by code, not judgement. Husain's evals post describes a team that added a simple check to make sure internal IDs never showed up in replies to users. Once a rule like that is written down, the developers can test it on every change, and the PM doesn't have to remember it.
So plan for the set to grow. Every bug a user reports becomes an example. Every argument about whether an answer is acceptable ends with a new labelled row.
What a PM needs to understand about evals
The test set is the spec.
If a PM can't say what good looks like for 30 real examples, the developers can't build it. They will build what looks good in the demo, because that is the only target they were given.
This doesn't mean a PM has to write graders or know how an LLM judge is calibrated. The engineers can own that. The PM owns the part nobody else can: which inputs are real, what a good answer is, what a bad answer costs, and where the bar sits. For a normal feature those lived in the story. For an AI feature they live in the example set.
If you have an AI feature that demoed well and is now stuck between "almost there" and "the client won't sign off", that gap is what we work on in prototype to production.
One thing to do this week
Pick one AI feature on your board. Before the next planning meeting, write down 10 real inputs from users. For each, write one line on what a good answer must contain and one line on what would make it unacceptable.
Bring that page to the meeting instead of the story. The argument you have about it is the spec you were missing.