Write the evaluation suite before the first prompt

The AI projects that reach production share one habit: they measured answer quality from the first week. The ones that stall share the opposite habit. They built a demo, showed it to leadership, and then spent months arguing about whether it was good enough, with nothing to argue from.

What an evaluation suite is

A set of real questions from the people who will use the system, each with a reference answer written by someone whose judgment the business trusts. For a dispatch assistant, that was 340 questions from dispatchers with answers from senior dispatchers. For a policy assistant, it is questions from staff with answers from the policy owner.

Each change to the system runs the suite and reports:

• Retrieval recall. Did the right source document appear in the top results?
• Faithfulness. Is every claim in the answer supported by a retrieved source?
• Relevance. Does the answer address the question asked?
• Refusal correctness. For questions the system should not answer, did it decline?

Scores are compared to the previous run. A change that improves faithfulness but drops recall is a conversation, not a release.

Why it has to come first

Three reasons.

The reference set is the specification. Writing it forces the business to say what a correct answer looks like, which is the conversation most projects skip. It surfaces the questions nobody can answer, which are the ones the system should refuse.

It makes every later decision cheap. Chunk size, embedding model, retrieval filters, prompt wording, model choice: each is an experiment with a number attached rather than an opinion with a demo attached.

It is the acceptance test. When the suite scores above the threshold the business set, the system ships. When a change drops the score, it does not. Legal and security teams can read a suite report. They cannot read a demo.

How to build one in two weeks

Collect 50 to 300 real questions from logs, tickets or interviews. Have the domain experts write short reference answers and mark the questions that should be refused. Store them in version control beside the code. Run them with an open framework such as Ragas or a few hundred lines of your own, and put the scores on a dashboard the business can see.

The suite is also the cost model. Every run reports tokens per query, so the cost ceiling is a number you track, not a number you discover on the first invoice.

The result

The dispatch assistant reached 92 percent faithfulness on its suite before launch and the number is still reviewed monthly. The vendor demo it replaced scored 71 percent on the same questions. Nobody had measured it before, which is why it was never deployed.