ADR 0006 — The model bake-off is a screen, not a ranking

2026-08-29 · Status: accepted**

Context

poc/bakeoff.py replays 14 recorded calls through 8 candidate LLMs against an unchanged dialogue layer. The first run produced a clean table and an obvious conclusion: the incumbent wins. An adversarial methodology review dismantled it.

• The corpus is off-policy and the incumbent generated it. Every vendor utterance is a human's reaction to something gemma-3-27b said.
• Three of six flags could not fire. rate_words is empty on all 69 model turns, so dropped was impossible by construction and the counter_once stance — the only turn where money is at stake — never occurs.
• The english flag had a 0% true-positive rate across 26 firings, and was deciding places 2–7.
• Zero of seven comparisons survive Holm correction at n=69. Separating 98.6% from 94.2% needs ~59 calls.

Decision

Keep it, and label it. It is a screen for gross failures — it reliably catches things like one model repeating a sentence verbatim across four different questions. It does not choose a model.

Choosing a model requires on-policy runs where each candidate generates its own trajectory, plus human ratings for fluency.

Consequences

• The broken flags were deleted rather than repaired. Replacing a broken metric with a plausible one is how the first version happened.
• The table prints its own caveats so a reader cannot take the numbers neat.
• We still ship gemma-3-27b — not because it won, but because nothing beat it.

Revisit when

A corpus exists that reaches the counter_once stance, or the eval harness can drive candidates on-policy.