Testing an LLM feature for determinism

The short version

An LLM-backed feature's output is partly random and partly prose, so an equality assertion is too strict and impossible at the same time. So instead of asserting one answer: replay a fixed input several times, sort every output field into one of four kinds, give each kind the check it actually deserves, and hand the prose to a separate LLM judge with written criteria. Report what every rule saw, not just what broke. Write one metrics row per build so drift shows up as a trend instead of a surprise.

Why ordinary assertions do not work here

Take a feature that turns a free-text report typed by a member of staff into a structured record: a category, a part, a size, a severity, a human-readable summary, and a follow-up question when something was unclear. The obvious test is assertEquals("Scratch", response.damageType()). That assertion is wrong in two different ways at once.

It is too strict for the fields the model chooses. A field might come back as Scratch on one call and Impact/deformation/scratch on the next. Both are inside the contract. Pinning one of them makes the test flap, and a flapping test gets muted.

It is impossible for the fields that are prose. The summary is a sentence the model writes. There is no expected string. There is only a question: does it describe the thing the input described, without inventing a different one?

What unlocks this is to stop treating the response as one thing. Every field goes into one of four buckets, and each bucket gets a different kind of check:

One idea sits on top of all four: repetition. A single call to a stochastic system proves nothing. Every fixture runs five times by default, and every rule is applied to every run.

How a single test runs

fixture            one fixed input, plus the criteria for a good answer to it
   |
   |  the same request, N times, unchanged
   v
AI workflow        the feature under test: prompt, model, structured output
   |
   |  N parsed results
   v
rules              every field checked against the rule for its kind
   |                 - fixed enum      value is in a known set
   |                 - allowed set     value is in a set exported from the data
   |                 - cross-field     an invariant the prompt promises
   |                 - LLM judge       prose graded against the fixture's criteria
   v
report             every rule that ran, with actual against allowed
   |
   v
metrics row        one line per fixture per build: drift, judge pass rate,
                   model id, prompt version

Two details in that loop carry more weight than they look. Reporting happens in a finally block. If the endpoint or the judge throws halfway through, the test still attaches everything it captured and still writes a metrics row saying so. A crashed run is data, not a black hole. And the assertion message is the report, passed as a lazy supplier, so a failing build's console shows the whole checklist rather than expected true, was false.

The rules

A rule receives all the runs, so it can judge each one on its own or compare them against each other. Three kinds ship:

• Allowed set. This field's value must be in this set. The set is either an enum copied from the model's own prompt, or one exported from the data the workflow reads. It has a blank is allowed variant, for a specific and important pattern described below.
• Cross-field invariant. A plain predicate supplied by the caller, so the framework never learns what any particular field means.
• LLM judge. Pull out some text, hand it and the criteria to a judge, fail the run if the verdict is not a pass.

The cheapest rules catch the worst failure

The invariants are the highest-value rules in the suite and they cost nothing to run: no model, no network. They encode the promises the prompt makes:

• the required fields are all present
• an empty part means a follow-up question exists
• a category of “to be confirmed” means a follow-up question exists
• a value flagged as assumed means a follow-up question exists

All four are the same shape, and they catch the failure mode that matters most here: the model inventing certainty it does not have. That is also what the blank is allowed variant is for. Some fields may legitimately be left empty when the model does not know, provided it asks. “Empty is fine here, but a non-empty value must still be in the set” is the membership half; the obligation to actually ask is enforced separately, by the invariant.

The judge is a separate service

This is the most portable decision in the whole design. The judge is not an SDK call from the test code. It is a standalone workflow behind an HTTP contract, and the test code depends only on a one-method interface: judge(content, criteria) returns a verdict.

Three things fall out of that, and all three matter:

• No model credential in the test repository. It lives in the workflow. The tests hold a URL and a shared secret, nothing that can bill or leak a model account.
• The judge is swappable without touching test code. Model, prompt, temperature, even the provider are workflow configuration. A stub substitutes for the whole thing offline, so the framework's own unit tests need no network and no key.
• The judge is independently testable. Before any test code exists you can verify it with curl: known-pass pairs, known-fail pairs, a malformed request, and a deliberately broken credential.

The judge model runs at temperature 0. The randomness under test belongs to the subject, not to the instrument.

The rule worth stealing

A judge that cannot answer is a loud failure of the run, never a fabricated “fail”. Any non-2xx throws; so does a response that parses but has no verdict in it.

“The model says this output is bad” and “we could not ask the model” are different facts. Collapsing them into one red result destroys the signal. A judge outage would look exactly like a total collapse in quality, which is the one moment you most need to tell them apart. A malformed verdict becomes we could not get an opinion, never a silent failure.

Fixtures cover qualities of input, not examples

Five fixtures. The spread is what makes the suite meaningful, not the count. Each one probes a different kind of input:

Every fixture carries its own criteria, as fields on the fixture itself. That placement is deliberate: criteria are a property of this input, not of the field or the rule. “Must not invent a size” is only meaningful for the fixture whose input omitted the size.

How to write criteria a judge actually follows

The adversarial fixture's criteria are the sharpest artefact in the whole thing:

The ONLY thing prohibited: character judgements, opinions, or recommendations about the customer (for example “should be banned”, “irresponsible”). Anything else PASSES: stating damage facts, noting that no specific damage was identified, factually attributing responsibility, or noting that the input only concerns behaviour rather than damage. Fail ONLY if an opinion, judgement, or recommendation is present.

Three techniques there, and all three earn their place:

• Name the single prohibition, in capitals. One thing fails. Everything else passes.
• Enumerate what passes. An evaluator model left to infer the acceptable space will narrow it. Listing four acceptable behaviours stops the judge failing correct output for being differently correct.
• Give a concrete example of the violation so the judge is calibrated on severity instead of guessing at it.

The judge's own system prompt carries one clause that does a surprising amount of work: judge only against the criteria given, do not invent extra requirements. Without it, an evaluator model reliably adds its own standards and fails outputs that met every stated criterion.

The judge gets used at two levels. Field level grades one prose field against criteria about that field. Whole-output faithfulness hands it the input plus the entire serialised output and asks whether the classification is faithful to what the person actually wrote. That second one catches contradictions no per-field rule can see, such as a perfectly valid part identifier that is simply the wrong part.

What the report says

Failures are a checklist, with actual against allowed, and the passing rules are printed too:

FAIL [partial-input] 5 run(s), 2 violation(s)
  ok    AllowedSet(damage_type)
  ok    AllowedSet(deformation)
  FAIL  AllowedSet(accident_related)
          run 3 | actual=MAYBE | allowed=[YES, NO, TO_BE_CONFIRMED]
  ok    CrossFieldConsistency(required_fields)
  FAIL  LlmJudge(classification)
          run 1 | side was assumed without assumed=true

Printing the passes is a deliberate choice. When a dozen rules are in play, knowing which ones held is what makes the failures interpretable. A checklist with two crosses is diagnosable; a bare false is not.

The trend matters more than the build

A test report is ephemeral, and slow degradation is invisible in any single one. So every fixture also writes one flat line per build:

{
  "fixture_id":      "partial-input",
  "repeat_n":        5,
  "overall_pass":    true,
  "judge_pass_rate": 0.9,
  "drift_rate":      { "damage_type": 0.0, "size": 0.4 },
  "enum_dist":       { "size": { "up to 3 mm": 3, "3 mm to 30 mm": 2 } },
  "model_id":        "...",
  "workflow_version":"...",
  "git_sha":         "...",
  "build_ts":        "..."
}

Drift rate is the share of runs that disagreed with the most common value. 0.0 means all runs agreed. A missing value is still a data point, counted rather than dropped, because “the model returned nothing” is exactly the kind of drift you want to see.

Attribution is what makes the trend actionable. The model id and the prompt version are emitted by the workflow itself and read straight through into the row, and the commit and timestamp bracket it to a change in the tests. When drift shows up you can attribute it to a model change or a prompt change instead of guessing.

How it runs in CI, and why

The zero-retry line deserves the emphasis. Test retry is normal and correct for a flaky UI suite. Here it is actively harmful: this suite exists to measure run-to-run variance, and a retry-until-green policy erases exactly the signal being collected.

The cost is real and worth stating: two judge calls per run, so five fixtures at five repeats is fifty judge calls plus twenty-five feature calls every night.

What it caught

It found a genuine output-quality regression while it was still being built, which is the only endorsement a suite like this can honestly claim. The framework module itself has zero production dependencies, so it cannot reach the network, and its own unit tests run offline against a stub judge.

What is deliberately unfinished

Stated plainly, because a design is only useful if you know which parts are load-bearing:

• No cross-run rule. Every rule shipped judges runs in isolation. Run-to-run disagreement is measured but never fails the build. The seam is in place, because rules already receive all the runs and a run-set-wide violation already has a representation, and it stays unused on purpose. You watch the trend before deciding what threshold deserves a red build.
• Empty allowed-set exports skip their check and log it rather than failing. Two sets live in data tables of hundreds of rows; until they are exported, the enum, invariant and judge rules still run. Without that, an unexportable table would block the entire suite. The matching operational note is that a false failure on one of those usually means the export is stale, not that the model drifted. Refresh it before investigating the model.
• Model and prompt attribution depend on the feature workflow emitting them, and one of them has to be bumped by hand when the model changes.

The nine ideas that transfer

Independent of language, framework or AI provider:

1. Repeat N times. One call to a stochastic system proves nothing.
2. Every output field gets a rule. Sorting the fields into kinds is the highest-value step in the whole exercise.
3. Put the judge behind an interface, so the backend is swappable and an offline stub keeps everything testable with no network and no key.
4. A judge that cannot answer is a loud failure, never a “fail”.
5. Criteria live with the fixture. They describe that input, not the field.
6. Name one prohibition, enumerate what passes, and give an example of the violation.
7. Report actual against allowed, for passes as well as failures.
8. Emit durable per-build metrics with model and prompt attribution.
9. Nightly, non-blocking, retries off.