How to check whether an AI assistant answers correctly?
A fluent answer is easy to notice. A reliable answer needs a test you can repeat. Build a small evaluation set that exposes mistakes before users have to find them.
8 min · Reading & exerciseFree lesson · No registration
Define observable acceptance criteria, separate different types of failure and design a first evaluation set for your own use case.
Before you start
You have tried an AI assistant. No coding experience is required; the examples use a fictional internal support assistant.
01 / Define success
Write the test before improving the prompt.
Start with the decision the answer enables. For a support assistant, success might mean giving the applicable procedure with its source and stating any missing information. Define those conditions before comparing models or prompts. Otherwise, a more confident tone can look like a quality improvement.
Fictional teaching example
The access policy assistant
Our invented company, Northstar, has three documents: an employee onboarding guide, a contractor access policy and a retired access policy. The current contractor policy permits access for 30 days after sponsor approval. This dataset is illustrative; it is not a client project or a measured result.
Question: Can a contractor receive access for 90 days without approval?
Acceptance criteria: Say that the current policy does not permit it; give the 30-day limit and sponsor approval requirement; cite the current contractor policy. Do not borrow a rule from the employee guide.
A reference answer can be a checklist of facts rather than one exact sentence. This lets the evaluation accept different wording while still rejecting the wrong rule.
02 / Build the dataset
Include cases where the right answer is a question.
Use representative requests, known failures and boundary cases. Include questions with sufficient evidence, ambiguous requests and questions the documents cannot answer. Write the expected behavior beside each case. Keep a separate set for checking changes so you do not repeatedly tune on every example you report.
Case
Expected behavior
Failure to catch
Current contractor access
Answer from the current policy with a citation.
Applying the employee rule.
“How long can I get access?”
Ask whether the requester is an employee or a contractor.
Guessing the requester’s status.
Access rules for a missing country
State that the available documents are insufficient.
Inventing a local exception.
Retired and current policies both found
Use the applicable current version.
Quoting a superseded limit.
Document the user role, relevant source version and evaluation date. A test without that context may have several defensible answers. Keep personal or confidential data out of a shared teaching set.
03 / Score separate dimensions
A citation is a starting point for verification.
Check whether the linked passage actually supports the claim. An assistant can attach a valid document to an unsupported answer. Judge factual correctness, support from sources, coverage of required facts and appropriate handling of uncertainty separately.
Correctness: Does the answer apply the contractor rule to the contractor?
Grounding: Does the cited passage support the 30-day limit and approval requirement?
Completeness: Does the answer include both conditions the user needs?
Uncertainty: Does it request missing context or admit missing evidence?
Use deterministic checks where the rule is exact. For open explanations, use a written rubric and human review. A model can help grade answers, but calibrate its judgments against examples reviewed by someone who understands the domain. Record disagreements instead of hiding them in an average score.
04 / Observe the result
For an agent, inspect the business state.
If the assistant can create an access request, its final message is only one piece of evidence. Verify that a request exists for the correct person, scope and duration; that approval was required; and that no duplicate was created. Keep the tool results and final state available for review.
For a failed answer, inspect which documents were retrieved and what the model received. For a failed action, inspect authorization, tool arguments, responses and the resulting state. This separates a retrieval fault, reasoning fault and integration fault so the next change addresses the cause.
05 / Compare changes fairly
Track reliability alongside time and cost.
Run the same cases against the old and new configurations, with the same source snapshot. Repeat important cases because model behavior can vary. Record the model, prompt version, retrieval configuration and date so the comparison can be interpreted later.
Report results by case category and failure severity. A better overall score can still hide a newly broken authorization case. Measure task completion, human corrections, response time and cost together. Agree the release criteria with the people who own the process; there is no universal pass percentage for every application.
Your exercise / About 5 minutes
Create a five-case evaluation sheet.
Choose one narrow task your assistant should support.
Write two ordinary cases, one ambiguous case, one missing-evidence case and one important boundary case.
For each, record required facts, acceptable uncertainty and the source or system state that proves success.
Run the cases and classify each failure before changing anything.
Review your work
Another person should be able to use your criteria and reach a similar verdict. If they need to guess what “good” means, make the rule more concrete. For an action, include an observable state check; for an answer, include supporting evidence.
Knowledge check
Sources & further reading
Teaching examples and exercises on this page are fictional. The following primary source develops the evaluation principles used here.