How to evaluate an LLM system

A team without evals ships a prompt change, tries five examples, sees them look better and deploys. Two weeks later a user reports something broken that used to work. Nobody can say when it broke, because nothing was measuring it.

This is the default state of most LLM products, and it is why teams plateau. Every change is a coin flip whose result is invisible. The first evaluation set a team builds usually finds a regression that has been live for a month.

The deeper cost is organisational. Without a number, disagreements about quality are settled by whoever is most senior or most confident, and the team loses the ability to make a decision it can later defend.

It is a versioned collection of cases with something checkable about each expected result. Not necessarily an exact expected string — often a property: contains this figure, cites a source, refuses this request, stays under this length, returns valid JSON.

The cases should come from reality. Synthetic examples invented at a desk miss the messy inputs that break systems. Pull from real usage — the awkward phrasings, the ambiguous requests, the ones that previously went wrong.

Include the boring cases as well as the interesting ones. A suite composed entirely of edge cases tells you nothing about whether the ordinary path still works, and the ordinary path is the overwhelming majority of what users experience.

• Cases drawn from real traffic, including past failures
• A checkable property per case rather than an exact match
• Coverage of the boring majority, not only edge cases
• Version controlled and reviewed like source code
• Large enough that a single fluke does not move the number

Prefer deterministic checks wherever the property can be stated precisely. They are fast, free and unambiguous. A surprising share of what matters is checkable this way: valid structure, presence of a required field, absence of a forbidden claim, length, latency.

Where the property is genuinely a judgement, model-graded evaluation is the practical option — another model scores the output against a rubric. It is useful and imperfect, and it needs its own calibration: periodically check the grader against human judgement on the same cases.

Human review stays necessary for a sample. It is slow and expensive, which is exactly why it should be spent on calibrating the cheaper methods rather than on grading everything.

• Deterministic — schema, keywords, numbers, structure, latency
• Model-graded — rubric scoring for tone, relevance, faithfulness
• Human — a small sample, used to keep the other two honest

The value compounds when evals run automatically on every change to prompts, context assembly, retrieval or model version. Treated as a one-off pre-launch exercise, they tell you about a system that no longer exists.

Wire them into the same pipeline as your tests, keep the history, and look at the trend rather than a single run. A change that improves the average while breaking three previously passing cases is usually not the change you want, and only the per-case view shows that.

Keep the runs cheap enough that nobody is tempted to skip them. A suite that takes forty minutes and costs meaningfully per run gets disabled during a busy week, and a disabled suite protects nothing.

The blocking belief is usually that a proper evaluation framework is needed first. It is not. Thirty cases in a file, a script that runs them, and a printed score is a working evaluation suite, and it will find something within a day.

Collect the cases from what already exists: the bug reports, the awkward questions from your own testing, the three examples someone pasted in a channel when they said the output looked wrong. Those are real failures and they are already written down.

Add a case every time something goes wrong in production. That single habit is what turns a small suite into a good one over a few months, and it costs nothing at the moment of discovery when the example is in front of you anyway.

Quality alone leads to decisions that are technically correct and commercially unviable. A change that raises accuracy by one point and doubles cost per request needs both numbers visible to be judged properly.

The useful dashboard is small: pass rate on the eval set, cost per request, latency at the tail, and refusal or fallback rate. Four numbers that move together tell you more than twenty that nobody reads.

Refusal rate is the one most often omitted and the most diagnostic. A system that quietly becomes more willing to guess looks better on every other metric while getting worse in the way users actually notice.

Evaluation work is unglamorous and decisive, which is a reliable recipe for specialisation. Teams are now hiring people whose primary responsibility is knowing whether the system is getting better — designing eval sets, calibrating graders, investigating regressions.

It is an unusually good entry point. It needs judgement and rigour more than deep ML background, and it puts you at the centre of every shipping decision. If you want into AI engineering without a research CV, this is the most accessible serious door.

It also builds unusual leverage. The person who can say whether a change helped is consulted on every release, learns the whole system quickly, and accumulates the context that later makes them the obvious choice to own a larger part of it.