How to Evaluate an LLM Feature Without Fooling Yourself

Most LLM features ship after the same ritual. Someone writes a prompt, tries it on five or six examples, likes the answers, and says it looks good. Then a model version changes, or someone tweaks the prompt to fix one complaint, and nobody knows whether things got better or worse overall.

Traditional software has unit tests because humans are bad at noticing regressions by feel. LLM features need the same discipline, adjusted for the fact that outputs are not deterministic and "correct" is often a matter of degree.

Start with a test set, not a metric

Before you argue about scoring methods, collect examples. Fifty well-chosen cases beat five thousand random ones.

Pull them from real usage if you have it, and write them by hand if you do not. Make sure the set includes:

• the common, boring requests that make up most of your traffic
• the edge cases you already know are hard
• requests the feature should refuse or hand off
• examples that broke in the past, so they stay fixed

For each case, write down what a good answer must contain or must avoid. You do not need a perfect reference answer. A short list of checkable expectations ("mentions the refund deadline," "does not invent an order number," "asks a clarifying question") is often more useful.

Version this set alongside your prompts. When a user reports a bad answer, the fix is not done until that case is in the test set.

Use the cheapest check that works

Many expectations can be checked with plain code. Is the output valid JSON? Does it include a required field? Does it stay under a length limit? Does it contain a phrase it should never contain? These checks are fast, free, and never disagree with themselves. Use them wherever you can.

Some qualities are hard to check with code: whether an answer is faithful to a source document, whether it actually answers the question, whether the tone fits. That is where people reach for an LLM as a judge.

LLM judges are useful and biased

Using one model to grade another's output works better than you might expect, and it scales in a way human review does not. It also has known failure modes.

A 2023 paper by Zheng and colleagues, which introduced the MT-Bench benchmark, studied this directly. They found that strong models used as judges agreed with human preferences at rates similar to how often humans agree with each other. They also documented biases: a tendency to favor whichever answer is presented first, a preference for longer answers, and a tendency to favor outputs from the judge's own model.

That gives you a practical checklist:

• Swap the order when comparing two answers, and only count a win if it holds both ways.
• Grade against criteria, not general quality. "Does the answer state the refund deadline from the policy? Yes or no" is far more reliable than "rate this answer from 1 to 10."
• Check the judge against people. Have a human grade a sample, compare, and adjust the rubric until they mostly agree. Repeat when you change judge models.
• Watch for length creep. If scores rise every time answers get longer, the judge may be rewarding verbosity.

Run it like a test suite

The point of all this is to make changes safely. Run the evaluation every time you change a prompt, switch a model, adjust retrieval, or upgrade a dependency. Compare against the previous run and look at the cases that changed, not only the overall score. An average can stay flat while a few important cases quietly break.

Because outputs vary from run to run, a single run can mislead you. For important decisions, run each case more than once and look at consistency. A feature that answers correctly four times out of five has a problem that one lucky run will hide.

Keep watching after launch

Offline evaluation tells you whether a change is safe to ship. It does not tell you how the feature behaves on the questions real users ask next month. Sample production traffic regularly, review it, and feed the interesting failures back into your test set. That loop, more than any single metric, is what makes an LLM feature get better over time instead of drifting.

None of this is glamorous. It is the same habit that makes any software reliable: write down what correct looks like, check it automatically, and do not trust your own impression of five examples.