Evaluating AI Systems

Introduction

"It worked when I tried it" is not enough for software that gives different answers on different days. AI features need evaluation (often called evals): a repeatable way to measure whether the system is good enough, and whether a change made it better or worse.

The simple explanation

An eval is a set of test cases with a way to score the answers, run every time you change a prompt, model or tool.

Build an evaluation set

1. Collect real examples of the inputs your system will see, including the messy, rare and tricky ones.
2. For each, record what a good answer looks like. This may be an exact answer, key facts that must appear, or a description of the right behaviour.
3. Include negative cases: inputs where the right answer is to refuse, say "I don't know", or ask for clarification.
4. Start with 30 to 100 cases, and add every failure you meet in real use.
5. Keep a separate set you never tune on, so you can check you have not just fitted your prompt to the tests.

Ways to score

• Exact or rule-based checks: is the JSON valid, does the answer contain the right number, does the code pass its tests? These are cheap and objective, so use them wherever possible.
• Human review: people rate answers on clear criteria such as correctness, helpfulness and tone. It is the gold standard but slow and costly, and reviewers need guidelines and should check each other's agreement.
• Model-as-judge: another model grades the answers against a rubric. It scales well, but judges have biases, such as favouring longer answers. Calibrate them against human ratings and spot-check them.

Choose metrics that fit

• Accuracy for classification, with precision and recall when mistakes differ in cost
• Task success rate for agents: did it finish the job correctly?
• Faithfulness for grounded answers: is every claim supported by the sources?
• Safety: how often does it produce harmful, private or disallowed content?
• Latency and cost per request, because a great answer that is too slow or too expensive may not be usable.

Report the numbers with the number of cases, because small tests are noisy.

Run it often

Run the evals whenever you change the prompt, the model, retrieval or tools, and in your automated build pipeline. Compare to the previous version. Models change over time too, so rerun them periodically.

Red-teaming

Red-teaming means deliberately trying to make the system fail: confusing inputs, attempts to bypass rules, prompt injection, requests for private data, unusual languages. Test before release, and fix what you find. Use diverse testers.

Monitor in production

• Log inputs and outputs, respecting privacy rules.
• Track user feedback such as thumbs up and down, and sample conversations for review.
• Watch for drift: changes in what users ask, or in model behaviour.
• Provide a way to report problems, and a plan to fix or roll back quickly.

Try it yourself

For an AI feature you know, write ten test cases: seven normal, two tricky and one where it should refuse or say it does not know. Decide how you would score each, and what pass rate you would need before launching.

Summary

• Evals are repeatable tests with real cases, clear expected behaviour and held-out data.
• Prefer rule-based checks, add human review, and use model judges carefully.
• Red-team before release, rerun evals on every change, and monitor in production.