Write the acceptance bar first: AI evaluation sets

How to build a test set for an AI system before the build starts: real questions, unanswerable ones, pass criteria, sign-off and re-running after changes.

Most AI projects are judged by a demo. Someone asks ten questions, the answers look good, and the system is declared ready. Weeks later a customer asks the eleventh question. An evaluation set replaces that impression with a measurement, and it is cheap to build if you do it first.

What is an evaluation set?

It is a fixed list of inputs the system will face, each with a description of the right outcome. For a knowledge assistant that means questions with the correct answer and the document it should come from. For a classifier it means cases with the correct label. The list is scored the same way every time, so two runs can be compared.

Why write it before the build?

Because a bar set after you have seen the results is not a bar. Teams that write criteria late tend, without meaning to, to describe what the system already does. Writing the set first also forces the useful arguments early: what counts as correct, which mistakes are tolerable, and who decides.

What goes into a good set?

Type of caseWhat it testsExample
Common questionsEveryday accuracyWhat is the notice period for this policy?
Hard but answerableReading across sourcesWhich exceptions apply to both products?
UnanswerableWhether it declinesA question the documents do not cover
Out of scopeWhether it stays in its laneA request for legal or medical advice
Awkward phrasingRobustnessTypos, lowercase, mixed languages

Take the common questions from real logs, tickets or emails, not from imagination. Ask the people who do the work today what they are asked most and what trips up new staff.

Why include questions it cannot answer?

A system that answers everything will score well on a set made only of answerable questions, and it is the most dangerous kind to deploy. The failure that hurts is the confident, fluent answer to something the sources never covered.

So a meaningful share of the set should be cases where the right behaviour is to say “I don’t have that” or to pass the request to a person. Check that it refuses in substance, not only that it uses the words: a reply that begins with a disclaimer and then invents an answer is a failure.

How do you set the pass criteria?

  • Score more than one thing: is the answer correct, does it cite the right source, and did it decline when it should?
  • Name the failures that block release outright, such as an invented figure or a disclosed personal detail, however good the average looks.
  • Agree the threshold for everything else with the business owner, and write down why.
  • Record who signs off. It should be the person accountable for the outcome, not the team that built it.

What happens after launch?

The set becomes a regression test. Re-run it whenever the model version, the prompt, the retrieval settings or the underlying documents change. Any of those can move results in either direction.

Two habits keep it honest. First, hold back a portion of the cases that the people tuning the system never see, and use that portion for the release decision. Second, add new cases from real use every month, especially the ones that went wrong.

A one-week way to start

  1. Collect real questions or cases from the last few months.
  2. Have two people who know the work write the expected outcome for each.
  3. Add unanswerable and out-of-scope cases.
  4. Agree the blocking failures and the threshold with the owner.
  5. Set a portion aside, unseen, for the release decision.

An evaluation set does not make a system correct. It tells you how it behaved on those cases on that day, which is the most a careful team can claim.

Common questions

What is an evaluation set for an AI system?

A fixed list of realistic inputs with the expected outcome for each, scored the same way every time so you can compare versions and decide whether a system is ready.

How many test cases does an AI evaluation set need?

Enough to cover the common questions, the hard ones and the cases it should refuse. It is better to start with a modest, well-checked set and grow it from real use than to wait for a large one.

Who should sign off an AI system before release?

The person accountable for the business outcome, using criteria agreed before the build. The team that built the system should not be the one to approve it.

How often should an AI system be re-tested?

After every change to the model, prompt, retrieval settings or source documents, and on a regular schedule in between, since provider models and your own data both change over time.

Talk to us

Want this applied to your organization?

Tell us the business area you have in mind. We aim to reply within one business day with how we would approach it and whether an assessment fits.

Take the AI readiness check

We use these details only to reply to this request. No newsletters, no list. We aim to reply within one business day. See our Privacy Policy.

Book a callTalk to our team