WTF is Evals?
Test suites that measure how well an AI system performs on the tasks you actually care about.
Also called: evaluations, llm evals
TL;DR
You can't improve what you can't measure. Evals are like unit tests for AI behaviour: a set of example inputs plus a way to score outputs, so you know whether a prompt change, model switch or new tool made things better or worse.
How it works
Layers usually include deterministic assertions (format, required fields, no forbidden content), model-graded checks (LLM-as-judge with a rubric), and human review of samples. For agents, evals look at final outcomes and at trajectories (did it use the right tools, how many steps, cost). The hard part is usually building a representative dataset from real traces, then keeping it current as the product changes.
Origin
- Coined / introduced by
- ML practice, popularised for LLM apps
- When
- 2023
- Story
- Evaluation is core ML practice; in LLM app development the term became a rallying cry through practitioners such as Hamel Husain.