Most AI projects are judged by a demo. Someone asks ten questions, the answers look good, and the system is declared ready. Weeks later a customer asks the eleventh question. An evaluation set replaces that impression with a measurement, and it is cheap to build if you do it first.
What is an evaluation set?
It is a fixed list of inputs the system will face, each with a description of the right outcome. For a knowledge assistant that means questions with the correct answer and the document it should come from. For a classifier it means cases with the correct label. The list is scored the same way every time, so two runs can be compared.
Why write it before the build?
Because a bar set after you have seen the results is not a bar. Teams that write criteria late tend, without meaning to, to describe what the system already does. Writing the set first also forces the useful arguments early: what counts as correct, which mistakes are tolerable, and who decides.
What goes into a good set?
| Type of case | What it tests | Example |
|---|---|---|
| Common questions | Everyday accuracy | What is the notice period for this policy? |
| Hard but answerable | Reading across sources | Which exceptions apply to both products? |
| Unanswerable | Whether it declines | A question the documents do not cover |
| Out of scope | Whether it stays in its lane | A request for legal or medical advice |
| Awkward phrasing | Robustness | Typos, lowercase, mixed languages |
Take the common questions from real logs, tickets or emails, not from imagination. Ask the people who do the work today what they are asked most and what trips up new staff.
Why include questions it cannot answer?
A system that answers everything will score well on a set made only of answerable questions, and it is the most dangerous kind to deploy. The failure that hurts is the confident, fluent answer to something the sources never covered.
So a meaningful share of the set should be cases where the right behaviour is to say “I don’t have that” or to pass the request to a person. Check that it refuses in substance, not only that it uses the words: a reply that begins with a disclaimer and then invents an answer is a failure.
How do you set the pass criteria?
- Score more than one thing: is the answer correct, does it cite the right source, and did it decline when it should?
- Name the failures that block release outright, such as an invented figure or a disclosed personal detail, however good the average looks.
- Agree the threshold for everything else with the business owner, and write down why.
- Record who signs off. It should be the person accountable for the outcome, not the team that built it.
What happens after launch?
The set becomes a regression test. Re-run it whenever the model version, the prompt, the retrieval settings or the underlying documents change. Any of those can move results in either direction.
Two habits keep it honest. First, hold back a portion of the cases that the people tuning the system never see, and use that portion for the release decision. Second, add new cases from real use every month, especially the ones that went wrong.
A one-week way to start
- Collect real questions or cases from the last few months.
- Have two people who know the work write the expected outcome for each.
- Add unanswerable and out-of-scope cases.
- Agree the blocking failures and the threshold with the owner.
- Set a portion aside, unseen, for the release decision.
An evaluation set does not make a system correct. It tells you how it behaved on those cases on that day, which is the most a careful team can claim.
