Incidents — what broke, what it cost, what guards it now
The code used to carry these as paragraph-long comments above the fix. The comments were the project's memory and they were also the reason a 1,000-line file was half prose. Each entry below is the story; the code keeps a one-line why and points here; the test named in the last column is the guard.
Entries are grouped by the part of the system they belong to. OBSERVED means the number was measured on this machine at the time.
Ending the call
| what happened | cost | guard now |
|---|---|---|
The pipeline ended on LLMFullResponseEndFrame — the model had finished writing, not speaking. Her goodbye was clipped: 4.4 s for a line needing 5.2 s, bot_stopped_speaking never fired (eval, 2026-09-10 10:39). |
A vendor hung up on mid-sentence | test_call_end.py::test_goodbye_audio_that_started_after_arming_ends_the_call |
The audio-based fix was written in SlotFill's idiom (self.b.…) and pasted into OriBrain, which has no b. Every BotStoppedSpeakingFrame raised AttributeError, caught as non-fatal and logged as service_error — 7 in one session — while the symptom read as "the fix didn't work". |
The call never ended; a day of misdiagnosis | test_no_routine_frame_raises_on_the_way_through |
on_pipeline_finished was registered with one argument; Pipecat passes two. It raised silently on every call end, the browser was never told, and "she hangs up herself" never actually worked until the eval harness caught it — the first defect on this project found by a test before a person. |
Weeks of "the stop button doesn't trigger" | The two-argument handler in ori_bot.py |
Canned goodbyes (voicemail, wrong number, abuse, callback, give-up) never ended the call: EndWorkerFrame was only pushed from the model path. |
She said she was hanging up and stayed on the line | Both paths now arm CallEnd; test_a_canned_goodbye_arms_the_end_and_speaks_first |
EndWorkerFrame pushed from the canned path is a system frame: it overtakes the queued TTSSpeakFrame, so "after the line" was not after the audio. |
The goodbye could be clipped on that path too | CallEnd waits for audio that started after arming |
The wind-down line ("I'll confirm and call back") was terminal to the vendor and nothing to the code, so after last_ask + "ठीक है" the line stayed open (live, 2026-09-10 09:41). |
Mehul: "call भी cut अपने आप नहीं हुई" | The DEFERRED phase (see the conversation-quality spec) |
The turn boundary
| what happened | cost | guard now |
|---|---|---|
Flux ended a turn on the pause before a number: "तो मेरा rate लगभग". Our guard flagged it (ends on a dangling लगभग) and the decision layer answered anyway. He quoted the rate again next turn. Mehul's note: "she interrupts me mid-sentence, then forgets". |
A wasted turn on the money turn | OriBrain holds the fragment and merges the continuation; test_a_cut_off_figure_is_held_and_merged_with_its_continuation |
| A 300-second idle timeout killed the first regression run before it connected: the harness synthesises audio up front, minutes on a cold cache. | A suite that failed with a connection error naming nothing | cancel_on_idle_timeout=(run_tag != "eval") |
The eval socket's 20 s pong timeout starved against a grader running Whisper and Kokoro on this CPU (gaps of 22.4 / 19.7 / 17.5 / 14.4 s). Every run died with 1011 keepalive ping timeout and was recorded as our agent failing. |
Days of a suite that could not pass | _eval_transport_params raises it to 180 s, eval transport only |
The pipeline and the page
| what happened | cost | guard now |
|---|---|---|
rate_words was a single slot on the transcription frame, read by SlotFill a whole response later from another task. Turn 2 overwrote it while turn 1 was streaming: turn 1 spoke ₹28,000 on a ₹22,000 ladder, invisible to FigureGate because the gate read the same object. |
An unauthorised figure, spoken | OriBrain._pending_rates — one entry per decision, popped once per response |
The pending-rate queue was appended on every decision, but canned turns never produce an LLMFullResponseStartFrame, so SlotFill never popped for them: the queue ran one behind for the whole call. counter_once spoke an empty {rate} hole and the concession surfaced a turn later on the goodbye. |
A counter with no number in it | Append only when the turn reaches the model |
A process-wide "selected scenario" global: with two testers, B's pick changed the load A was greeted with. The per-connection path had never worked from a browser — the JS sends requestData, FastAPI bound request_data. |
Wrong vendor name, wrong lane, silently | POST /start stashes the body per session |
The pipeline is built once per process; on_client_ready fires per connection. Under the eval transport the bot outlives a run, so the second run inherited the first one's brain — and the first caller's LLM context. |
A verdict that depended on run order; a context leak between callers | on_client_ready builds a fresh Brain, CallEnd and context |
A failed STT socket pushes a non-fatal ErrorFrame and then she is deaf for the rest of the call — silence identical to a vendor who never spoke. run.sh ends in exec with no log file. 1 in ~42 handshakes on the undocumented India endpoint. |
A dead ear with no trace | service_error row in the session and a red line on the page |
| The page learned its record id from one unacknowledged message at turn 1; lost, and feedback had nothing to attach to. Two real submissions were accepted and discarded. | Lost feedback | session message at connect; /api/feedback returns 404 rather than {"ok": false} with 200 |
UserBotLatencyObserver emits an empty breakdown on interrupted turns, not nothing; the page drew a bar-less waterfall indistinguishable from a real one. |
A wrong latency picture on exactly the turns worth studying | empty=True sent explicitly |
The session header claimed language_hints: ["hi"] while the service sent ["hi","en"], and recorded neither the endpoint nor the keyterms — across 53 sessions every call claimed the same STT config. |
Latency numbers that could not be compared | BotConfig.record() writes the header from the object the services are built from |
Speech and the model
| what happened | cost | guard now |
|---|---|---|
eot_threshold 0.7 → 0.8 after he was cut off mid-number. A documented direction, not a measured one: the bench corpus is clean single utterances and reports split = 0 at both settings. |
An unmeasured knob | Recorded here; the hold (above) is the deterministic fix |
| Numerals were off because Deepgram's matrix says "unsupported for Hindi". Her callers speak Hinglish and Flux renders those numbers in English, where it works: 7/8 → 8/8 rates recovered, 0 wrong either way (n=14, OBSERVED 2026-09-10). | Rates arriving as English words the parser reads as ambiguous |
numerals=True |
| Keyterm boosting of हज़ार/लाख/सौ/रुपये/रेट produced 14/14 byte-identical transcripts — and one hallucinated हज़ार next to a truck size turns "सत्रह फीट, 28 हज़ार 500" into ₹45,500. | Zero measured benefit against a catastrophic tail | No keyterms |
| Bedrock rejects a context that starts with an assistant message; a canned greeting made Ori speak first and every turn 400'd. | Silent calls | The developer seed message before the greeting |
| The India Deepgram endpoint: TCP p50 38 ms vs 270 ms global, handshake 303 ms vs 1,175 ms (n=8, OBSERVED). Absent from the docs, changelog and status page; certificate issued 2026-09-08, expires 2026-12-07. | No SLA on the biggest latency leg | ORI_DEEPGRAM_URL flips back to the global host with no code change |
The brain
| what happened | cost | guard now |
|---|---|---|
"I will confirm and call back" was the model's reply to five of six turns in one call (2026-09-10 08:00). The persona taught it as the safe move ("say ONCE that you will confirm"), the AFFIRM/OTHER briefs gave it no move, and with append_to_context=False it had no memory of having said it. |
Mehul: "it is answering its own thing" | The line is canned and the phase decides when it is said; no brief or prompt mentions calling back; test_phases.py::test_the_worst_recorded_call_no_longer_winds_down_five_times |
| An out-of-range quote (₹23,400 on a ₹23,100 ceiling) was fed to the ladder, which closed, and only then was he asked why. His answer had nowhere to go; the model invented a term ("diesel aapko khud karna padega"). | A lost load and an invented commitment | Brain._money asks first and holds the quote; test_an_out_of_range_quote_is_questioned_before_the_ladder_closes |
last_ask was not terminal: after it, "ठीक है", the wind-down, and the line stayed open (09:41). |
"call भी cut अपने आप नहीं हुई" | The DEFERRED phase ends on the next acknowledgement; test_an_acknowledgement_after_the_wind_down_ends_the_call |
| Two attempts to act on the answer to "why so high?" carried a remembered figure into the ladder and booked trucks at prices nobody said (₹22,677 against a man whose only stated price was ₹12,000). | Two reverts in one morning | Negotiation.probe_answered(standing_rate) takes the parser's current figure and refuses a mismatch; test_probe_answered.py |
| The load pitch, generated by the model, once asked "kya rate milega aapko" — what rate will YOU get. | Backwards Hindi on the pivotal turn | Brain.load_pitch() is spoken, not generated |
confirm_number asked him to repeat a figure the parser had read perfectly (₹10,500 looked implausible against ₹22,000). The first fix told the MODEL to read it back. |
A rupee figure written by the model | The read-back is rendered from the parsed figure and authorised for the gate |
| A spice load (मसाले) matched साले and ended the call as abuse; "अभी फोन मत करना, शाम को करना" ended it too. | Hung up on customers | Word-bounded matching, callback before abuse in the rule table, and abuse warns once before ending |
The router
| what happened | cost | guard now |
|---|---|---|
| Substring matching on Devanagari: "मसाले का लोड है क्या?" contained साले → abuse, hung up; "आप है" inside "आप हैदराबाद" → presence. | Hung up on a spice load | Word-bounded matching in both scripts; test_classify_corpus.py |
| "अभी फोन मत करना, शाम को करना" (call this evening) → abuse; "कृपया कुछ देर बाद फोन कर लीजिए" → voicemail. Both ended the call. | Hung up on two callbacks | Callback precedes abuse in the rule table; unmistakable recordings precede callback |
| A bare "नहीं" fell to UNCLEAR, whose brief is "ask them to repeat" — three times in one call. "हैलो" was absent while "हेलो" was present, so turn 1 was "say again?". | Three sessions ending in "I cannot hear you" with a perfect transcript | Yes/no before the catch-all; OTHER (answer normally) is the fallback, UNCLEAR is for under two words |
| Length gates: an 8-word "हां madam, मैं यश बोल रहा हूं. बोलिए." skipped the affirmative check; a 6-word farewell with thanks routed to AFFIRM and she replied to a goodbye with a price. | A bid at a man saying goodbye | Position carries meaning — head word for yes/no, what follows the thanks for a farewell |
| "नहीं, कब चाहिए आपको यह?" is a correction, not a refusal; read as DENY he asked the same question three times. The mirror bug on AFFIRM ("हां जी आप कहां से बोल रही है?") threw the question away — found because all eight bake-off models failed the turn identically. | Questions discarded | A negation or affirmation before a question word falls through to the question |
| "ना" (tag particle) and "बस" ("just") fired as refusals: "हां जी बिल्कुल हो जाएगा ना" → DENY. A trailing "जी" hid "नहीं" in "इतने में नहीं जी" → OTHER. "कोई दिक्कत नहीं" (no problem) → DENY. | Agreements read as refusals and refusals as small talk | Tag particles excluded; address terms stripped from the tail; the no-trouble idiom exempt |
| Inflections: बोल handled, बोलो not; दोगे handled, देंगे not. Four defects in one day of the same shape. | A terse driver told "say that again"; a polite probe answered with load details | Verb forms generated from one paradigm (forms()) |
| Flux joins the first two words of a turn ("नहींजी", "तीसहज़ार"): 29 of 133 pinned routings moved, 18 money-critical (OBSERVED 2026-09-10). | Refusals and rates lost | normalise() splits a joined token into two known words: 1 of 133 moves; test_router_stability.py pins the ceilings |
Flux writes English halves in Latin: "गलत number है" was OTHER (she asked whether anyone else at the firm could quote); "switch off … कृपया कुछ देर बाद प्रयास करें" was a callback. Found by poc/e2e.py, 2026-09-10. |
Wrong number and voicemail handled as people | Both scripts in the lists; test_flux_writes_english_halves_in_latin_and_the_router_reads_both |
| "पिछली बार कितना" sat in the PROBE list and beat HISTORY_Q; "पहले आप अपना रेट बता दीजिए" (you go first) became HISTORY_Q because bare पहले was a past marker. | Wrong premise on the pivotal turn | Past markers are compounds ("महीने पहले"); history before probe |
| She negotiated with whoever answered ("मैं यश बोल रहा हूं"); "रमेश" vs "Ramesh bhai" compared as different people across scripts. | A rate agreed with the wrong man | speaker_name / same_person compare cross-script and err towards a match |