Everything we tried, and what it bought

Every experiment that changed a decision, with its numbers. Ordered by subsystem, not by date.

SOURCED throughout — the running experiment log kept OUTSIDE this repo (strategy/, not on GitHub) — the record kept outside the repo. Numbers re-verified against code where the log could have gone stale; those rows are marked OBSERVED.

⚠ Every latency figure in this file excludes telephony. The PHONE channel is a band-pass filter — no codec, no jitter buffer, no packet loss. A real carrier leg adds ~100–200 ms each way that nothing here models.

1. Speech to text

Scored on rupees and plates recovered, never WER. A transcript that gets every word right and the rate wrong is a failed booking.

The recognisers, on phone-channel audio

SOURCED — providers_phone.json

⚠ sec/clip includes each vendor's retry backoff and AssemblyAI's polling loop. It is not a latency comparison.

AWS Transcribe was disqualified outright: अठारह पाँच came back as ₹5. Four wrong rates out of seven ends the conversation.

⭐ Sarvam beats Deepgram on plates 8/9 to 4/9 and is 5× faster on batch. Deepgram was chosen for its streaming socket, which Sarvam did not offer at the time. If plates ever become the binding constraint, this row is where to look.

The tie that never broke

SOURCED — Experiment 26, on 10 rate-bearing clips at G.711 µ-law 8 kHz

⚠ Our corpus is Bulbul TTS — Sarvam's own voice — reading text we wrote. It is phonetically canonical, which flatters whoever generated it. Voice of India measured real telephonic speech and reports the opposite ordering by a wide margin.

The synthetic ceiling says Deepgram. The published real-audio floor says Sarvam by 2.6×. Only real recordings break the tie.

PROPOSED — this is still the largest open question in the stack.

Endpointing

SOURCED — realtime_deepgram.json

Deepgram Flux — rejected, then adopted

SOURCED — flux.json, first pass

Rejected on that table — Flux produced a wrong rate where nova-3 produced none.

It was adopted anyway, for a reason the table cannot show. Flux accepts µ-law 8 kHz natively. Every other candidate needed resampling, and a resampling mismatch is our most dangerous failure mode:

⭐ The strongest argument for Flux is not its score. It is that the 8 kHz mismatch is our most dangerous failure mode and Flux is the only candidate that does not have it.

OBSERVED — the shipped config is model="flux-general-multi", language_hints=["hi","en"], numerals=True, eot_threshold=0.8 (poc/pc-web/server/ori_bot.py:492-560).

The India endpoint

OBSERVED — ori_bot.py:522-526

api.deepgram.com      TCP p50 257 ms
api.in.deepgram.com   TCP p50  38 ms   ws handshake 303 ms

⚠ Undocumented — absent from Deepgram's docs, changelog and status page. No SLA. One env var (ORI_DEEPGRAM_URL) reverts it.

Numerals in Hindi — the vendor table is wrong for our case

OBSERVED — ori_bot.py:552-560, measured on 14 of our own clips

Deepgram's feature matrix says numerals are unsupported for Hindi. True — but our speakers say numbers in English, where numerals works:

numerals=false   "...सत्रह feet twenty eight हज़ार five hundred..."
numerals=true    "...सत्रह feet 28 हज़ार 500..."

The second is parseable. The first is not.

2. Text to speech

Same round trip for all: text → voice → phone channel → nova-3 → our parser.

SOURCED — the TTS bake-off

⛔ And this metric missed the thing that mattered

A native speaker listened and said the Hindi sounded like "an English speaker trying to speak Hindi" — while every number above said the voice was fine.

Cause: of 28 replies, zero contained a single Devanagari character. The prompt said "Hinglish", the model wrote it romanised, and a hi-IN voice handed Latin script has only English letter-to-sound rules to apply.

Intelligible to a machine and acceptable to a carrier are different properties, and only the first was ever measured.

Blind A/B, same sentences, same judge:

Sarvam bulbul:v2 · DEVANAGARI  ==  Murf Falcon-2 namrita · DEVANAGARI
    >  Murf Gen2 shweta · DEVANAGARI
(neither Latin rendering ranked at all)

⭐ The variable is SCRIPT, not vendor. Two vendors' Devanagari tied. So the voice stayed and what we feed it changed.

Reading a plate aloud

SOURCED — read-back encoding test

⭐ The comma is load-bearing. Space-joined scores 2/5 on Murf; comma-joined 5/5. The comma forces a prosodic break — what a person does spelling over a bad line.

⛔ SSML made it worse: interpret-as="spell-out" on Polly rendered MH as M8.

3. The language model

Gate: 28 turns (22 adversarial). Fatal = emits a figure, claims a booking, invents a load fact, or denies being a machine.

Latin-script output

SOURCED

Devanagari-script output

Amazon Nova — tested for one reason, lost on both axes

SOURCED — 56 turns each

Tested specifically because Bedrock prompt caching supports Nova and not gemma/glm. Caching cannot close a 220 ms gap on a ~900-character system prompt, where the quoted 30–60% gains assume ~1500 tokens.

⚠ Bedrock access trap: Nova and Claude are not callable on-demand in ap-south-1. They need the apac. inference-profile prefix. apac.amazon.nova-pro-v1:0 and nova-lite work; nova-2-lite and every Claude model do not, even with the prefix.

Rejected on structure, not score

• sarvam-105b (not -conversations) — emitted 1,199 reasoning chunks and zero content in 13.8 s. Any model with reasoning on by default is unusable for a user-facing turn.
• claude-3-sonnet as a judge — capped at 2 req/min, silently returned None for 77 of 90 lines, leaving a 13-sample "judge" that looked like a result.
• glm-4.7 as a judge — it is also a candidate. A model grading itself is not a second opinion. The judge is now deepseek.v3.2.

The founding measurement

SOURCED — the reason ADR 0003 exists

sarvam-105b returned {"rate_inr": 1805} for ₹18,500 — at temperature 0, with a perfect JSON schema. The schema was flawless. The number was not.

⭐ The failure does not reproduce on the shipped model: on the end-of-call JSON block, zai.glm-4.7 scored 0 fatal / 4 serious and no model put a rate in the object. The four faults were an under-specified outcome enum in our prompt, not model error.

PROPOSED — that it does not reproduce today is not a reason to remove the control. The control costs nothing; the failure costs a truck.

4. Techniques, and what each bought

SOURCED

The denoiser: a recommendation that was backwards

⛔ A comment in bench_providers.py claimed --denoise took wrong rates 3→1. The saved data says the opposite: every un-denoised arm has zero wrong rates; every denoised arm has one or two.

5. Transliteration — the 411 ms

Devanagari costs 2–4 tokens per character; multilingual tokenizers run 4–8 tokens per word on Indic against 1.4 for English.

SOURCED

⚠ It is not output length. Measured on our own field: glm-4.7 writes the shortest replies (49 chars) and is the slowest (818 ms); gemma-3-27b writes 66 chars in 453 ms. No amount of prompt trimming fixes it.

So the model writes Latin and a dictionary converts before TTS in 0.0042 ms.

The lexicon, in three tiers — and why order is the whole design

⛔ The bigger, human-annotated dataset is worse for this domain if used naively. Dakshina disagrees with tier 1 on 82 of 356 shared words (23%), and the disagreements go the wrong way:

bata  -> बाटा   the SHOE BRAND, not बता ("tell")
ab    -> एबी    the LETTERS A-B, not अब ("now")
karna -> कर्ण   the Mahabharata CHARACTER, not करना ("to do")
baad  -> बाड़   "fence", not बाद ("after")

Dakshina maps native Wikipedia words to romanisations; inverting it asks "which native word is most often written this way in an encyclopedia". 128 of our 484 forms are absent entirely — karungi, confirm, mxl, and the client name itself.

It still earns tier 2: 47,611 forms, and it got kam → कम and to → तो right — two errors we had to find by hand.

⚠ Licence: CC BY-SA 4.0 is share-alike. It does not touch source code, but a derived lexicon shipped as data to a customer needs attribution and the same licence.

⚠ The honest limit: the lexicon is the pronunciation, and no automatic check can hear it. Seven entries were checked by hand; 477 have not been.

6. Turn detection — a full cycle spent on a model we cannot use

SOURCED — Experiments 27 and 28

LiveKit's semantic end-of-utterance model was tested standalone, honestly, and then discarded:

⛔ The licence forbids it. A full experiment cycle spent measuring something unusable — avoidable by reading a licence first, five minutes of work deferred and labelled "before shipping".

⭐ But the finding survived the model: numbers and sentences fail differently, and one model will not catch both. That is why turn detection is still two components, and why swapping LiveKit for Flux changed the parts list without changing the architecture.

Two further traps from the same experiments:

• ⭐ Use the "gpu" ONNX file even on CPU — it halves Hindi's error.
• ⛔ It silently misreads our audio — hardcoded to 16 kHz. This is the exact class of bug that made Flux's native µ-law support decisive.
• ⛔ The first test set was rigged, and we wrote it.

Narrowband mitigations — what actually works

SOURCED — published work, not our measurement

⚠ No famous leaderboard tests telephone audio or Hindi. HuggingFace Open ASR is English + European, wideband. Artificial Analysis is English-only mic audio. Pipecat's benchmark is 16 kHz English. All three are irrelevant to us.

7. Live calls

The first always-listening conversation

SOURCED — Experiment 29

⛔ The conversation was not recorded and is gone. No audio, no transcript, no JSONL. The single most informative event to that point, unreproducible. That is the finding that changed what we built next — session recording is now unconditional.

Two faults it surfaced:

• ⛔ The agent did not know who it was.
• ⛔⛔ It gave away the ceiling, unprompted, on the first try.

The 23× region

SOURCED — Experiment 30

⛔ We were calling Bedrock in the wrong continent.

OBSERVED — CLAUDE.md, and it is now a documented trap

⛔ A second fault in the same session: telemetry in the critical path silenced the agent.

Five money leaks, all found by review

SOURCED — the bake-off and review round

⛔⛔ Five money leaks, all found by adversarial review, all now closed. None were found by the test suite.

⛔ The headline latency filter has never once fired.

⛔ The bake-off ranks nothing, and saying otherwise was the mistake. The corpus was generated by the incumbent model, and n=69 separates nothing after correction. It is a screen for gross failures — ADR 0006.

8. Measurement mistakes, and what each cost

Kept because every one produced a number that looked reasonable and was wrong.

SOURCED

⭐ The pattern: a check tested only against inputs that should PASS looks perfect. Every grader in this project has been wrong at least once, and each time it changed a verdict about a model. All are now pinned in both directions — a grader that cannot fail is indistinguishable from no grader.

9. What the log listed as open, and where it stands now

The experiment log's §10 is partly stale. Re-checked against the code:

OBSERVED — .venv/bin/python, calling hear_rate() directly

Still genuinely open:

• Real call recordings. Everything above is synthesised speech. The single largest unknown.
• 477 lexicon entries have had no human eye.
• Kannada — pack wired, checks are script-only, needs a native reader.
• Barge-in, dual-channel, TTS phrase caching — designed, never built.
• English number-words glued to a Hindi scale — twenty eight हज़ार five hundred returns ambiguous (OBSERVED), which is safe but costs a turn.