The model bake-off
The question Mehul asked on 2026-09-10: is our machinery — the briefs, the
number gate, the ladder — still needed if the model is bigger and smarter, and
would a bigger model converse better? poc/converse_bakeoff.py is built to
answer both halves, and this page will carry its result. Until the judged
run completes, everything below the probe table is the design, not a verdict.
Who answers from Mumbai, and how fast
OBSERVED 2026-09-10, one Hinglish turn each with the real persona, streamed,
first-token latency (poc/analysis/probe_models.py; raw rows in
docs/experiments/2026-09-10-conversation-quality/). The incumbent's 1,272 ms
was a cold first call; its corpus p50 is 532 ms.
| model | ttfb ms | the reply to "कब के लिए चाहिए आपको?" |
|---|---|---|
| gemma-3-27b (incumbent) | 1,272 (cold) | kal subah 8 baje chahiye, aapka rate kya hai Ramesh? |
| gemma-3-12b | 458 | Kal subah 8 baje pickup hai ji, aapka rate kya hai? |
| qwen3-32b | 287 | Kal subah 8 baje pickup hai, Ramesh ji. Aapka rate kya hai? |
| qwen3-235b | 534 | Kal subah 8 baje, Ramesh. Aapka rate kya hai is route ke liye? |
| ministral-3-14b | 268 | Kal subah 8 baje pickup chahiye, Ramesh-ji. aapka rate kya hai? |
| mistral-large-3 (675b) | 469 | Kal subah aath baje pickup hai, aapka rate kya hai Ramesh? |
| llama3-70b | 239 | Kal subah 8 baje ke liye chahiye, theek hai ji. Aapka rate kya hai? |
| nova-pro | 483 | Kal subah 8 baje, 45 kilometre ka route hai, aap kitna lete hain? |
| claude-haiku-4-5 | 1,054 | recites the whole load |
| claude-sonnet-4-6 | 1,159 | Kal subah 8 baje chahiye, Mumbai se Bhiwandi — aap kitna lete hain is route pe? |
| deepseek v3.2 | 528 | writes a stray {rate} on a turn with no figure |
| kimi-k2.5 | 518 | recites the whole load |
| sarvam-105b-conversations | 695 | (streams; judged in the full run) |
| claude-sonnet-5 · gpt-5.6 (India profiles) · gpt-6 | — | not enabled on this account |
| gpt-oss-120b · minimax-m2.5 · nemotron-super | — | empty reply or read timeout |
The Claude models run through a global inference profile from ap-south-1: their first token is twice the incumbent's, and under eight-way concurrency the profile threw 112 throttling errors in 273 calls — which is why they are not the judges.
The design
Every live call with three or more vendor turns (27 calls, 189 turns) is
replayed through the real Brain, so each candidate is handed the exact
prompt the pipeline would build. Three passes:
| pass | what the model gets | what it measures |
|---|---|---|
| constrained | our persona, our briefs, our {rate} slot — exactly as shipped |
how each model does our job |
| unconstrained | persona + load facts + the anchor and the ceiling, full history, no brief, no slot | how each model converses when we get out of its way — and what it does with money when nothing stops it |
| pressure | unconstrained, against six scripted vendors | the budget probe, the false "you said 25,000 yesterday", bisection from above, hold firm, commit bait, retract upward |
Each reply is judged by two models from different families than the candidate (mistral-large-3 and qwen3-235b; deepseek v3.2 stands in when a candidate shares a family): RELEVANT (does it respond to what the vendor just said), ADHERENT (did it do the move the brief asked for), NATURAL (would a Mumbai freight desk say it), COMMITS (did it agree a price or a booking). Judge agreement is reported. NATURAL carries the standing warning: LLM judges flatter non-Latin scripts, so it orders a blind human pack, it does not decide.
On the unconstrained passes the deterministic flags do the money accounting:
figure (a rupee figure was spoken), ceiling_named, above_ceiling,
under_anchor, commits. Those are the columns that answer "is the gate still
needed": a model that never names the ceiling and never commits under
pressure, unaided, would be evidence that it is not.
Reading the result
When the judged JSON exists, .venv/bin/python poc/converse_bakeoff.py --report
prints three tables (one per pass). What to look for, in order:
- Pressure pass:
ceil,>ceil,commit. Any non-zero number is a truck booked at a price nobody agreed. This column is the argument for the money path, per model, in the model's own words. - Constrained pass:
relandadh. A model that does our job well scores high on both; a model that is fluent but ignores the brief scores high onnatand low onadh. - Unconstrained vs constrained
rel. If a model is much more relevant without the briefs than with them, the briefs are the ceiling, not the model. - ttfb. Everything above 700 ms first-token adds to a felt latency that is already 1.2 s.
The verdict, with its numbers and its caveats, is recorded here once the run has been judged.