Evaluating AI outputs: how to test an assistant before launch
A practical guide to testing an AI assistant before launch: build a test set from real questions, define pass marks, check for made-up answers and retest after every change.
The Gihonix team · · 4 min read
Key takeaways
- Test with real questions and known answers, written before you see the assistant perform.
- Include hard cases: missing information, trick questions, requests it should refuse.
- Set pass marks in advance, review failures by cause, and retest after every change.
To test that an AI assistant is accurate before launch, build a test set of realistic questions with known correct answers, run the assistant against it, score the results against pass marks you set in advance, fix the causes of failure, and repeat. A demo that looks good is not a test; a scored set of real cases is.
This guide works for chat assistants, document assistants and agents connected to your systems, for example through an MCP server.
Step 1: What does good look like?
Before testing, write down what the assistant is for and what counts as a good answer. For a support assistant, "good" might mean: correct, consistent with policy, polite, quotes the source, and hands over to a human when unsure. Different tasks need different measures:
| Task | Key measures |
|---|---|
| Answering from documents | Correct, cites the right source, says "I don't know" when the answer is absent |
| Looking up records | Exact values correct, right tool used |
| Drafting text | Accurate facts, correct tone, no promises you cannot keep |
| Classifying or routing | Right category, with errors counted |
Step 2: How do you build a test set?
- Collect 30 to 100 real questions or tasks from emails, tickets and staff. Start smaller if you must, and grow it.
- For each, write the expected answer, or the expected action and source, before you test.
- Have someone who knows the subject check the expected answers.
- Keep the set in a file under version control so that it can be re-run.
Do not write the test only from questions you know the assistant handles well.
Step 3: Which hard cases should you include?
Add on purpose:
- Questions with no answer in your data. The assistant should say so, not invent one.
- Ambiguous questions. It should ask for clarification.
- Outdated information traps. Does it use the current policy?
- Out-of-scope requests. It should decline politely or hand over.
- Permission tests. Asking for another customer's data must fail.
- Manipulation attempts. Text inside a document or email saying "ignore your instructions" must not be obeyed.
- Edge cases. Misspellings, short messages, other languages your customers use.
Step 4: How do you score answers?
Use a simple scale per answer: correct, partly correct, wrong, or unsafe. Define each in a sentence so that different reviewers agree. For high-stakes areas, have two people score a sample and compare.
Automated checks help for exact values, such as order numbers or totals. Some teams use a second AI model to grade answers at scale, but it makes mistakes too, so keep human review of a sample. Review the method and the results with someone who understands the subject.
Step 5: What pass marks should you set?
Decide before testing, and let risk set the bar:
- For anything that could cause harm if wrong, such as medical, legal or financial content, expect a person to review every output regardless of score.
- For low-risk lookups, set a high bar and zero tolerance for made-up facts.
- Treat any unsafe output or permission failure as a launch blocker.
We do not suggest a universal percentage, because the right bar depends on your use case and the cost of an error.
Step 6: How do you analyse failures?
For each failure, record the cause:
- wrong tool chosen, or unclear tool description
- missing or outdated source data
- retrieval pulled the wrong passage
- instructions unclear or conflicting
- model limitation
Most fixes are cheap: better descriptions, cleaner documents, clearer instructions. See MCP tools, resources and prompts for writing tool descriptions, and RAG explained for document-based answers.
Step 7: Run a limited pilot
After the test set passes, release to a small group with a visible "report a wrong answer" option and logging, as in MCP server testing and monitoring. Review real conversations weekly, and add every real failure to the test set.
Step 8: Retest after every change
Re-run the set whenever you change the instructions, documents, tools, model or provider. Models and products are updated by their vendors, so behaviour can shift even when you changed nothing. Keep results over time.
A pre-launch checklist
- Purpose and success measures written
- Test set from real cases with checked answers
- Hard cases included: no answer, ambiguity, permissions, manipulation
- Pass marks set before testing
- Failures categorised and fixed
- Human review in place for high-stakes output
- Pilot group, feedback button and logging
- Retest process after every change
- Risks recorded in an AI risk register
Work with Gihonix
Gihonix is a software, AI and web development company based in Coimbatore, India. We build test sets and review processes into every project through our AI solutions and QA testing services. Contact us to test an assistant you already have.
Frequently asked questions
How many test questions do I need?
A useful start is 30 to 100 realistic cases, grown over time with real failures. Quality and coverage matter more than the exact number.
Can an AI grade another AI's answers?
It can help at scale, but it is fallible. Keep human review of a sample, especially for important topics.
What if the assistant fails some tests?
Find the cause, fix it, and re-run. Treat unsafe outputs and permission failures as launch blockers.
Do I need to test again after launch?
Yes. Retest after any change to instructions, data, tools or the underlying model, and review real conversations regularly.