The voice architecture — what runs, and why it is shaped this way

Ori phones an Indian transporter and negotiates a freight rate. Every design choice below answers to one asymmetry:

A missed rate costs one more question. A WRONG rate books a truck at a price nobody agreed.

That is not a slogan. It is the reason the system looks strange in places, and the tie-breaker whenever two designs are otherwise equal.

1. The founding measurement

A 105-billion-parameter model, at temperature 0, with a perfect JSON schema, returned 1805 for ₹18,500.

Not a hallucination in prose — a wrong integer in a structured field it had been told exactly how to fill. And the model gets roughly four in ten Hindi number-to-words conversions wrong, always on the same near-homophones: chhiyasi (86) for arsath (68), unsath (59) for unhattar (69).

So: a language model is never allowed to choose, state, or convert a rupee figure. Everything else follows from that.

2. The one rule

The parser owns the number. The model owns the prose.

Two halves, with a hard boundary:

	owns	never touches
Deterministic code	every rupee figure, the ladder, intent routing, what may be spoken	tone, word choice, sentence shape
The model	one short Hinglish sentence per turn	any digit, any threshold, any decision about money

⛔ The model is not asked to avoid numbers and trusted. It is never given one. A model cannot leak what it was never told. The first prompt on this project put four internal figures in the model's context — opening anchor, market anchor, low-quote threshold, walkaway ceiling — guarded by the words "never reveal". That is the design this replaced.

3. The pipeline, in order

  microphone
      │
      ▼
 transport.input()          WebRTC audio in
      │
      ▼
 DeepgramFluxSTTService     speech → text          p50 679 ms  ← the biggest slice
      │
      ▼
 OriBrain  ◄── dialogue.classify()  deterministic intent (20 intents, keywords)
      │    ◄── Brain.decide()       chooses the MOVE and the authorised figure
      │    ◄── Negotiation.after()  the ladder decides accept / counter / stop
      │
      │  replaces the transcript with a BRIEF: "your move is X". No numbers.
      ▼
 user_aggregator → AWSBedrockLLMService     prose only      p50 503 ms
      │
      │  the model writes: "…{rate} mein ho jayega ji?"
      ▼
 SlotFill                   substitutes the ONE authorised figure, in words
      │
      ▼
 FigureGate                 ⛔ LAST GUARD. Scans the outgoing sentence; any
      │                     unauthorised figure → speak a written fallback
      ▼
 MurfTTSService             text → speech          p50 110 ms
      │
      ▼
 transport.output() → recorder → assistant_aggregator
      │
      ▼
  the vendor's ear                          end-to-end felt latency p50 ~1,190 ms

⚠ Order is the safety property. FigureGate sits immediately before the voice because that is the last place a figure can still be stopped. Move it earlier and anything generated after it goes out unchecked.

4. The ladder — where every rupee comes from

app/domain/sourcing/ceiling.py. Fixed before anyone picks up, so a vendor who is better at this than we are cannot argue it upward mid-call:

anchor        we open at the contracted rate
concession    exactly ONE move, and only if he is above us
ceiling       contract + the policy cap. A hard stop.

⛔ The ceiling is never quoted. Saying "my limit is 27,000" turns the limit into the price — every vendor then asks for 27,000. Above the ceiling the agent does not haggle harder; it stops and asks a person.

⭐ And the concession must not be a fixed fraction of the gap. It used to be the midpoint. We say the anchor out loud (we open on it) and we say the concession out loud (it is our one move) — so two of the three numbers are public and the third was a function of them:

concession = (anchor + ceiling) / 2   ⇒   ceiling = 2·concession − anchor

Exactly, on every ladder. A vendor who heard both figures recovered the hard limit with one subtraction. Worse, "split the difference" is the oldest move in bargaining, so an experienced negotiator did not need the arithmetic — the midpoint is the tell.

The concession fraction is now drawn from a wide band using a per-load seed the vendor cannot know. Reproducible for audit (same load, same ladder, forever), no longer predictable. The residual leak is a bound, not the number.

5. The two controls

Both exist because a prompt instruction is not a control.

CLAUDE.md: "If you find yourself protecting this with a prompt instruction, you are one bug away from losing it — make it a control."

SlotFill — the model writes the literal token {rate}. SlotFill substitutes rupees_in_words(figure), rendered by the same exhaustive table the read-back uses. The model therefore cannot state a wrong figure, because it states none.

FigureGate — reads the finished sentence in both scripts (the spoken Devanagari and the written Latin, because transliteration can destroy a figure and invent one) and refuses anything the ladder did not authorise. A blocked line is replaced with a written fallback, and the block is recorded.

⚠ Neither is optional decoration. FigureGate was once "widened" by replacing its frame check rather than extending it — 912 tests stayed green while the gate was entirely switched off.

6. The dialogue layer

dialogue.classify() — ~20 intents, deterministic keyword matching, no model in the routing path. Latency and predictability are the reasons; a router that sometimes changes its mind is not a router.

⚠ This is the weakest part of the system and we know it. Ordering is load-bearing and invisible, and adding a phrase to one list silently captures utterances meant for a later branch. It has caused, in production:

मसाले (spices) matched साले → ABUSE → hung up on a customer
a load question before the pitch → 90 seconds of reciting the brief
"पिछली बार कितना price था?" → PROBE, beating the branch added for exactly it
"अभी फोन मत करना, शाम को करना" (call me this evening) → ABUSE → hung up

Mitigations now in place: word-boundary matching in both single and multi-word phrases, and all 133 real vendor utterances pinned as a golden corpus so a change that silently reroutes any of them fails CI.

7. What is recorded, and why

Every call writes a stereo WAV and a JSONL of every decision, tagged with who ran it. The rule, from telemetry.py:

"The stereo WAV is the ground truth; this is the index into it."

⚠ And the standing warning attached to it: every number in the log can look healthy while the agent concedes ₹39,000. That is not hypothetical — it is what Experiment 29 did. Telemetry tells you what happened, never whether it was good. A person listening is still the best evaluator available.

8. What this architecture is bad at, stated plainly

Routing — §6. Keyword lists do not scale and every new intent is a risk.
No mid-call lookups. She cannot answer a question the brief does not contain. She can now say so, which is new; before, she answered with a different fact.
The ceiling is a deterministic oracle across calls. Anchor and ceiling do not jitter — only the concession does. Five calls locate the ceiling to 0.32%.
No reopen policy. A vendor descending 50,000 → 20,000 is refused all four times, ending ₹4,000 under our ceiling. Three reopen designs were tried and all three leaked; the safe direction is currently the expensive one.
Live partial transcript is unavailable — Deepgram Flux emits no interim frames, so text cannot appear as the vendor speaks.

9. The rule this project keeps re-learning

A fix is not done because the tests pass.

Eleven adversarial reviews, and every round of fixes contained a bug worse than the one it closed — found only by something actively trying to break it. Two changes were reverted rather than shipped (Marathi numerals, the reopen policy) because the safe direction was the expensive one.

⭐ Adversarial review is not a quality step here. It is the only step that has ever found the serious bugs. Budget for it like a compiler, not like a code review.

10. Verified against outside practice — 2026-09-08

Two research agents checked this architecture against published work, current frameworks and the leading observability tools. Everything below is either SOURCED (a URL they fetched) or verified by running our own code.

The money path is right, and better-evidenced than we knew

⭐ Our pattern has a name and prior art. {rate} + substitution is delexicalized NLG with relexicalization — standard task-oriented dialogue generation, and it survived into the LLM era intentionally. FigureGate measures what the literature calls slot error rate. We can stop describing this as a bespoke invention.

⭐ Someone published almost exactly this, for freight. "Anchor-and-Resume Concession Under Dynamic Pricing for LLM-Augmented Freight Negotiation" (arXiv 2604.20732): a deterministic engine owns price dynamics and concession logic, the LLM writes justifications, and the system extracts numerical values from the LLM response to verify consistency with the deterministic module. That is SlotFill plus FigureGate, in our industry.

⭐ And the stance vocabulary is Cicero's shape. Meta's Diplomacy agent couples a planning engine to a controllable dialogue model, passing an intent from planner to generator. ask_why / counter_once / last_ask is the same control signal. ⚠ The difference: Cicero fine-tuned for controllability; we prompt. If the model ever drifts off-stance, that is the answer — not a better prompt.

⚠ TERMS-Bench (13 negotiation agents) found the cue penalty is negative for all 13: warm cues induce over-concession, pressure cues cause brittleness. A friendly transporter is an attack surface. It also found the simplest fixed-concession baseline beat GPT-4o-mini. The money path is the right call.

Two urgent findings, both checked and both clean

The architecture reviewer raised these as its highest-priority concerns and could not see our code. Both were verified by running it:

⭐ The gate reads number WORDS, not just digits. All six forms caught: digits, Devanagari digits, Hindi words, romanised words, the elided trader form ("attharah hazaar paanch sau"), and mixed. The feared hole does not exist — parse_numbers is doing the work.
⭐ The context aggregator receives GATED text. LLMTextFrame subclasses TextFrame and the gate mutates frame.text in place, so what reaches the aggregator is what was spoken. On a block the original is DROPPED entirely, so an unauthorised figure never enters the conversation record. Confirmed across 7 real blocks: every one is followed by a working turn.

Where we are wrong

⛔ The intent router. The field has moved and we have not. Rasa — the most determinism-loving vendor in the space — abandoned intent classification entirely in CALM and routes with an LLM. LiveKit recommends LLM intent detection and calls rigid predefined paths the IVR failure mode. Pipecat Flows transitions on function calls. Vapi retired its Workflows builder. Bland's deterministic graphs are the acknowledged maintenance trap, with the reported failure at ~50 nodes being exactly ours. ⭐ The cheapest fix specific to our shape: we already pay 503ms for an LLM turn — ask that same call to emit a route label alongside the prose. Marginal latency ~0. Cost: the clean separation of routing from generation. ⛔ And one non-negotiable: the abuse → hang-up path must not be single-signal keyword. Hanging up on a paying customer is our highest-cost error and it sits behind our lowest-precision mechanism.

⛔ "Exactly ONE concession" is probably costing money. The freight paper finds multiple concessions beat single, and that concession count should be adaptive to the spread — narrow spreads need quick concessions to secure coverage, wide spreads let you hold firm. Ours is a safety rule doing a strategy job. Making count and depth a deterministic function of the spread keeps every property we care about — no LLM in the money path, hard ceiling intact, fully auditable — while recovering surplus.

⚠ gemma-3-27b is a generation behind (Gemma 4 reached Bedrock June 2026), and it arrives via Marketplace rather than the first-party path — which is why Bedrock-native constrained decoding likely does not apply to us.

Observability: three things are already computed and thrown away

The end-of-turn decision. Pipecat attaches the entire raw Flux TurnInfo dict to every transcription frame (flux/base.py:783). We discard it. It says whether the turn ended on confidence or timeout, and whether an eager end-of-turn was later contradicted. A turn that fired 400ms early and a correct one look identical in our UI today.
STT confidence. _calculate_average_confidence() runs at base.py:711, is used only for a threshold, and is never attached to the frame. It is what separates "the model misread the brief" from "Flux heard ३२ as ३८".
Time-to-first-AUDIO. TTFAMetricsData(ttfa, ttfb, leading_silence) is computed in the base TTS service by real speech-onset detection. It separates Murf thinking from Murf padding silence, which are indistinguishable in a TTFB waterfall and have different fixes.

⚠ There is no OTel convention for voice at all — no STT, TTS, VAD, endpointing or barge-in attributes exist. Any voice spans we emit are our own invention regardless. The LLM half does have a standard, and gen_ai.evaluation.* is the natural home for the gate's verdict.

⚠ Latency verdict: we are above median. Industry-published for this pipeline shape is 1.4–1.7s p50; we are at 1,190ms. The "feels natural" threshold is ~800ms and our entire gap to it is STT.