Conversation quality — what the calls show, and the architecture that fixes it

2026-09-10 · the applied-science pass before the v0.0.2 code changes. Tags: OBSERVED = measured on this machine today, with the command; SOURCED = from a document; PROPOSED = inference, not yet measured.

Mehul's two complaints, verbatim: "it is not conversing very well generally" and "it is answering its own thing". His recorded voice note adds a third: "she interrupts me mid-sentence, then forgets what I said". This document takes the recorded calls apart to find the mechanisms behind those three sentences, then lays out the architecture that removes each mechanism rather than papering over it with another prompt line.

1. What the corpus says

OBSERVED — poc/analysis/mine_calls.py over 43 live sessions (63 Pipecat files minus eval runs), and poc/analysis/replay_calls.py replaying the 22 calls with three or more vendor turns through today's Brain:

The negotiation core almost never runs. A vendor quotes, she asks why, and the call dies in the exchange that follows.

2. The mechanisms, one per complaint

2.1 "Answering its own thing" — the deferral attractor

OBSERVED — call pipecat-20260910-080020 (3.0 stars). Six consecutive model turns; five are the same sentence:

t4  ठीक है, ok.                          -> affirm  "ab main aapko rate confirm karne ke liye vapas call karti hoon"
t5  पर आप मुझसे rate तक पुछ लीजिए        -> other   "haanji, main confirm karke vapas call karti hoon"   (+ stray {rate})
t7  तो मेरा rate लगभग   [cut off]        -> other   "theek hai Ramesh ji, main confirm karke vapas call karti hoon"
t9  हां जी actually petrol बहुत ज़्यादा महंगा हो गया  -> affirm "theek hai ji, main confirm karke vapas call karti hoon"
t11 Petrol महंगा हो गया इसलिए            -> other   "theek hai ji, main confirm karke vapas call karti hoon"

He asked her to ask for the rate (t5); he gave the rate (t8); he justified it (t9, t11). She said the wind-down line to all of them.

Three mechanisms compound:

1. The persona teaches the line as the default move. persona.py says "When a price needs settling, say ONCE that you will confirm and come back to them." The model cannot count "once", and reaches for the one sentence the prompt marks as always safe whenever the brief is vague.
2. The briefs describe an intent class, not a phase. AFFIRM → "They agreed. Acknowledge it warmly and move the call forward" — before any rate exists, "forward" is undefined, so the model winds down. OTHER → "Answer what they actually said" — for a justification of a price it has no instruction at all.
3. She has no memory of what she said. Every spoken line is pushed with append_to_context=False, so the assistant side of the LLM context is empty. The model sees a sequence of briefs and none of its own replies; the only memory is the one-line "do not repeat" hint. From the model's side each turn is the first turn, and the safe first move is the deferral.

2.2 "Not conversing well" — the ladder stalls where a person would push

OBSERVED — call pipecat-20260910-102849 (4.5 stars):

t5  अभी इसका price होगा 23400          -> rate / why_so_high   "Itna zyada kaise ji? Pichhli baar to isse kam mein ho gaya tha."
t6  Diesel mango here, actually        -> other                "Diesel ka arrangement aapko khud karna padega Ramesh ji."   ⛔ invented a term
t7  और 3rd party का भी price ज़्यादा    -> other                "theek hai ji, main confirm karungi aur vapas call karti hoon."

₹23,400 against a ₹23,100 ceiling is ₹300 over. She asks why — correct. But brain.py had already fed the quote to Negotiation.after(), which returned close_out_of_range and closed the ladder before he answered. His answer therefore has nowhere to go: probe_answered() refuses on a closed negotiation, the turn falls to the OTHER brief, and the model — given no move — invents one ("you will have to arrange the diesel yourself"). Nothing in the pipeline checks for an invented commercial term; FigureGate reads figures only.

The same shape in pipecat-20260910-094121: two justification turns (t5, t6) land on OTHER with empathetic non-moves, then his goodbye (t7, "हां जी ठीक है फिर धन्यवाद") was read as affirm and became the trigger for counter_once — she countered on a farewell.

2.3 "Interrupts me, then forgets" — the guard sees the cut and we answer anyway

OBSERVED — same 08:00 call, t7: guard = "ends on a dangling लगभग — nothing to attach it to". The deterministic endpoint guard correctly identified a truncated turn. The decision layer logged it and replied anyway (with the deferral). He had to quote the rate again on t8.

Flux ended the turn on his pause before the number. eot_threshold was raised 0.7 → 0.8 the same day — a documented direction, not a measured one; the bench corpus is clean single utterances and cannot reproduce a mid-sentence pause (strategy/22, 2026-09-10).

2.4 "How she speaks" — the speller, not the voice

OBSERVED — poc/analysis/mine_words.py: 196 distinct Latin words across 153 model lines on live calls. The transliterator's last-resort speller produced these for words the model actually wrote:

Mehul's note on the 09:41 call — "महंगा की spelling और pronunciation ग़लत" — is this class. The speller's own docstring says it "cannot be made good and is not trying to be"; it reaches 1% of tokens on the corpus it was built from and a much larger share of what the model writes on a live call, because the model writes English.

2.5 The call does not end when the business has ended

OBSERVED — 09:41 call: last_ask (canned), he says "ठीक है", she says the wind-down, and the line stays open; Mehul: "call भी cut अपने आप नहीं हुई". last_ask is not a terminal stance, so _deal_done never set and the wrap-up never fired. The wind-down sentence — "I will confirm and call you back" — is, to the vendor, the end of the call. To the code it was nothing.

3. The architecture

Each item names the mechanism it removes. Every one is a control, not a prompt line. Ordered by what it buys.

A. The call has phases — CallPhase in Brain

stateDiagram-v2
    [*] --> OPENING: greeting spoken
    OPENING --> PITCHED: right person confirmed → load pitched
    PITCHED --> RATE_WANTED: any turn without a figure
    RATE_WANTED --> PROBING: figure heard → ask why
    PROBING --> COUNTERED: justification heard → counter_once
    PROBING --> RATE_WANTED: new figure heard (re-evaluate)
    COUNTERED --> LAST_ASK: he holds or raises
    COUNTERED --> DEFERRED: he accepts / inside policy
    LAST_ASK --> DEFERRED: anything but a new lower figure
    RATE_WANTED --> DEFERRED: at or under anchor (accept)
    DEFERRED --> CLOSING: acknowledgement or goodbye
    CLOSING --> [*]: goodbye spoken, audio finished
    OPENING --> CLOSING: wrong number / voicemail / callback / abuse×2

Today the phase is implied by nine booleans (_load_pitched, have_rate, _deal_done, _wrap_offered, _squeezed, _asked_why_high, _identity_challenged, _closing_seen, _riders_asked). They become one enum, and briefs are keyed by (phase, intent). The two rules that matter:

• PROBING expects a justification. The next vendor turn is the answer to "why so high?" unless it is a new figure, a question, a hearing problem or a farewell. That is a phase rule, not a content heuristic, so _has_content, _trails_off and the exclusion list in brain.py:786-792 go away.
• DEFERRED is terminal. Once the wind-down line has been said, the next acknowledgement ends the call. That closes 2.5.

Removes: 2.1(2), 2.2 (the stall), 2.5.

B. Ask why before the ladder closes

On an out-of-range quote the brain currently calls Negotiation.after() first (which closes) and asks why second. Reverse it: ask why, hold the quote, and feed the ladder only after the answer — a justification-and-hold closes close_out_of_range as now; a lower figure is evaluated fresh. One chance to come down, no reopen mechanics, the ceiling is never named, and the binary-search attack stays impossible because the second out-of-range quote closes immediately.

PROPOSED, to be simulated before it ships: every quote sequence of depth four over the six demo ladders, asserting the ceiling is never revealed and agreed_rate never exceeds it (the existing property tests in tests/domain/test_stance.py extend to this).

Removes: 2.2.

C. A transcript window in the prompt, figures stripped

The prompt gains the last three exchanges as plain text — what he said, what she said — with _strip_figures applied to her lines. Deterministic, bounded, no rupee figure can cross. The model can then answer "what did I ask you?" and stop restarting the call every turn. This is Mehul's "context देना" ask, done as a control rather than by trusting the context aggregator.

Removes: 2.1(3).

D. The wind-down is canned and the model may not write it

close_defer already has canned variants. The persona line "say ONCE that you will confirm" comes out; every non-money brief says plainly "do not say you will call back or confirm"; and a repeat guard replaces any model reply whose word set overlaps her last line above 0.8 with the phase's canned line. The model loses the attractor and gains nothing to fall back on but the brief.

Removes: 2.1(1).

E. Hold the turn on a dangling figure

When utterance_may_continue() fires, OriBrain holds the fragment for up to 1.5 s and merges the next transcript into it before deciding. If nothing arrives, the decision runs on the fragment — but the brief for a held fragment is "ask him to finish the figure", not the generic move. The knob (eot_threshold) stays where it is; the guard is deterministic and already right.

Removes: 2.3.

F. PolicyGate beside FigureGate

The offline quality gate already has _claims_booking, _INVENTED and the commitment patterns (bench_llm_quality.py:233-320). They run in a benchmark and never on a live line — the same situation unauthorised_figures was in before FigureGate existed. Promote them: a reply that books, agrees, or asserts a commercial term ("aapko khud karna padega", "hum denge", "advance mil jayega") on a turn whose brief did not authorise it is replaced by the phase's safe line and recorded, exactly as a figure block is.

Removes: 2.2 (the invented term).

G. Unknown English words stay in Latin

translit._spell is replaced for words the lexicon does not carry: leave them in Latin. A hi-IN voice reading "arrangement" says a recognisable English word; "अररनगेमेनत" says nothing. The acronym A/B (speak.py) already showed Latin wins where no Hindi spelling exists. Plus lexicon entries for the model's real vocabulary — the 196 mined words become a test: zero speller outputs on the words the model writes.

Removes: 2.4.

H. Shorter lines, measured

Non-money briefs ask for one sentence under twelve words. Ori speaks ~2.3 words a second; 13 words is 5.7 s of air, and the corpus shows ~9 interruptions per call. Words per turn is already in the session record, so the change is measurable on the next ten calls without a listening session.

Reduces: interruptions (2.3, the other half).

I. The model — measured, not assumed

gemma-3-27b was chosen on safety and latency (0 fatal, 428 ms); fluency was never scored. OBSERVED today from Mumbai, one Hinglish turn each, first-token latency (the incumbent's 1,272 ms was a cold first call; its corpus p50 is 532 ms):

The rest of the list — Claude Haiku 4.5 / Sonnet 4.6 / Sonnet 5, GPT-5.6 on the in. India profiles, Sarvam 105B — is being probed now, and then all of them go through the conversation bake-off in §4. The question is not "bigger" but "which model, given our briefs, answers the vendor rather than itself, and does it within the latency we have".

4. The conversation bake-off (poc/converse_bakeoff.py)

Built on bakeoff.py's replay, which reconstructs every live call through the real Brain and hands each candidate the real prompt. Added:

• Two passes. Constrained: our persona, briefs and slot, exactly as shipped. Unconstrained: persona and load facts only, full dialogue history, no brief — how well the model converses when we get out of its way. A model that is good unconstrained and poor constrained tells us the briefs are the ceiling; poor both ways, the model is.
• Three judged criteria, yes/no with a reason: RELEVANT (does the reply respond to what the vendor just said), ADHERENT (does it do the move the brief asked for), NATURAL (would a freight desk say it in Hindi). Two judges from different families than the candidate under test; both verdicts reported with their agreement rate. NATURAL carries the standing warning: LLM judges flatter non-Latin scripts, so it ranks candidates for a blind human pack, it does not decide.
• The deterministic flags stay (leak, stray slot, dropped slot, parrot).
• Latency from Mumbai, first token, per candidate, on the same turns.

Candidates: the incumbent, gemma-3-12b, qwen3-32b, qwen3-235b, ministral-3-14b, mistral-large-3, Claude Haiku 4.5, Claude Sonnet 4.6, GPT-5.6 (India profile), Sarvam 105B conversations. DeepSeek and Kimi drop out on the probe above unless the full run says otherwise.

5. What the rest of the field ships — and what it changes here

A research pass on 2026-09-10 (25 fetches; report in the session log) against the Pipecat docs and changelog, LiveKit, Vapi, Retell, Bland, ElevenLabs and Cartesia. The parts that move a decision:

• SOURCED — Pipecat's EndWorkerFrame is the sanctioned graceful end and "downstream is correct: queued frames flush" (pipecat/learn/pipeline-termination). It flushes the queue; it does not wait for the speaker. CallEnd (A above, built) is the missing half.
• SOURCED — DeepgramFluxSTTService.Settings(eager_eot_threshold=…) turns on eager end-of-turn, pushed as InterimTranscriptionFrames; eot_timeout_ms (500–60000, default 5000) is the other knob. PROPOSED: run the parser and router on the eager interim and pre-warm the LLM, cancelling on TurnResumed — most of the ~700 ms STT wait overlaps LLM time. Not in this release; needs the phase work first so a speculative decision is cheap to discard.
• SOURCED — user turn strategies exist as a pluggable pair: MinWordsUserTurnStartStrategy(min_words=…) stops "हां"/"हम्म" backchannels from counting as barge-ins; a custom BaseUserTurnStopStrategy could refuse to close a turn on a dangling figure. The hold in OriBrain does the second today at the processor level; the first is the candidate fix for the ~9 interruptions per call and is measurable on the next ten calls.
• SOURCED — Vapi retired its Workflows builder (2026-08-18) and documents that its LLM-extracted variables are "best-effort, not deterministic"; LiveKit routes with the model and gates side effects behind tools; ElevenLabs exposes end_call as a tool the LLM invokes. Nobody fetched runs a rule router in front of the model. Our split is unusual and the money half of it is what the freight paper and the E-Commerce Bench (arXiv 2608.30730, "every price, concession and accept or walk-away decision comes from a deterministic kernel") argue for. The intent half is where the field went the other way — which is why Step 4 hardens the router rather than growing it.
• SOURCED — MuRIL reaches 87.3% intent accuracy on Hindi-English code-mixed queries, beating IndicBERT/XLM-R/mBERT. PROPOSED: the semantic fallback under the rules, once 200–300 real labels exist.
• SOURCED — Pipecat 1.8.x adds repeat: and results.jsonl to the eval harness and TTFAT metrics; we are on 1.7.0. An upgrade is a separate, measured step after this release.

6. What this does not fix

• Real telephony audio. Every number above is browser audio.
• Turn-taking on a real pause profile. E removes the worst case; the knob itself stays unmeasured until real recordings exist.
• The register the model writes in. F and D bound what it may say; only the bake-off can say whether a different model writes better Hindi.