The LLM Evaluation Lifecycle
Evaluation involves identifying failure patterns and measuring how often they occur.
Testing agents is hard. Outputs change from run to run, from test to test, and there is no formal spec. Sometimes answers look ok but are subtly wrong, other times they sound plausible but miss the point. ML systems have clear outputs and labels; agentic systems are open-ended and often produce subjective results.
Pre-launch evaluation and production monitoring catch failures, but they are different. One catches anticipated failures; the other catches those that only appear under real input distributions and user behavior.
How do we define evals?
Evaluation involves identifying failure patterns and measuring how often they occur. Applications need several evaluations integrated in four ways:
- background monitoring for drift or degradation;
- guardrails in the critical path — block, retry, or fall back;
- improvement mechanisms such as labeled data or few-shot examples;
- model migration, comparing model versions to decide when to switch.
Evaluation throughout the LLM lifecycle
Pre-training uses stats like perplexity and scaling laws. These show how well the model acquires language patterns, not whether it will be useful later.
Post-training uses instruction tuning with hand-picked examples and preference data. Newer methods include DPO, preference reward models, and RLVR. All are measured with public benchmarks. The method you pick shapes the errors you see later.
Most engineers work at the application stage, where public benchmarks rarely match real-world tasks like contract review or medical note summarization. The application team sets success criteria, designs metrics, and keeps monitoring.
Three problems show up at this stage: goals that only become clear after seeing outputs, incomplete instructions, and models that change behavior with real data.
Design starts with clean, ideal data. Real data is messy. Forwarded messages, quoted threads, paste-ins. Outputs are just as complex. You can't check every output, so you need to spot patterns at scale. A few good examples can mislead you. The model fills in the gaps, so outputs vary.
Even with clear prompts, models still make mistakes. Sometimes they pull the public figure's name from the message body instead of the sender's address. These errors persist.
The Analyze–Measure–Improve lifecycle
- AnalyzeRead traces, find failure modes
- MeasureBuild evaluators, quantify failure rates
- ImproveFix prompts, architecture, or model
Analyze
Read traces, find failure modes
↓
Measure
Build evaluators, quantify failure rates
↓
Improve
Fix prompts, architecture, or model
↺Note. Adapted from Evals for AI Engineers: Systematically Measuring and Improving AI Applications (Part I, Chapter 1: Foundations), by S. Shankar and H. Husain, 2026, O'Reilly Media.
Analyze by reading traces, the full record of what the agent did. Talk to users and domain experts about what counts as "correct." List all the ways things can go wrong. Identify them, but don't measure yet. This mostly covers comprehension, but also surfaces issues with specification and abstraction.
Quantify generalization problems by checking how often each failure happens. Decide if the rate is low enough to release. Relying on impressions leads to mistakes. Build automated evaluators; these can be code rules or LLMs as judges.
To improve, close specification gaps by tightening the prompt or agent logic. For generalization failures, use stronger fixes: more examples, better retrieval, fine-tuning, or a system redesign.
When something fails, check what kind of gap it is. For comprehension gaps, review more traces before changing anything. For specification gaps, make requirements explicit. For generalization gaps, measure the rate first, then try bigger fixes. Feed the results into the next analysis phase.
References
Shankar, S., & Husain, H. (2026). Evals for AI engineers: Systematically measuring and improving AI applications. O'Reilly Media.