Incidents — the ASR layer
What broke in app/domain/asr/, what it cost, and what guards it now. The
code keeps a one- or two-line why beside each rule and points here; the test
named in the last column is the guard. Where no test pins a rule the column
says so and names the function that does.
Every figure below was measured on this machine at the time it was written. None have been re-run for this ledger. Utterances are quoted as the recogniser wrote them.
asr/numbers.py — hearing the number
Unqualified test names are in tests/domain/test_asr_numbers.py.
Lexicon and spelling
| what happened | cost | guard now |
|---|---|---|
Saarika hears "पैंतीस आठ" (thirty-five eight, 35,800) and writes 35.8, because a decimal is what that sounds like in any domain but this one. Saaras leaves the words alone. The words are recoverable; 35.8 is not. Saarika's 13.05 for "तेरह पाँच" is the same shape with a padded zero; 26.23 for छब्बीस तीन is genuinely corrupted. |
A rate the parser could not read | _DECIMAL reads a one-digit or zero-padded fraction as hundreds and declines the rest; test_a_decimal_rate_is_an_elliptic_rate, test_a_corrupted_decimal_is_declined_rather_than_guessed |
| Marathi shares Devanagari, so unlike Kannada it cannot be refused: "एक लाख वीस हजार" with वीस absent parsed as ₹1,00,000 for a real ₹1,20,000, and "पाच सात आठ दोन" broke the digit run. Bhiwandi and Pune are core lanes in the corpus. | A parse that succeeds and is wrong | वीस/दोन in _HINDI (neither collides with Hindi; दोन is safer than दो, which is also "give"); test_marathi_numerals_that_have_no_hindi_spelling, test_rate_regression.py::test_marathi_rates_are_not_read_yet |
| saaras:v3 transcribed a vehicle number as "थर्टी एट ... नाइन थाउजेंड एंड ट्वेल्व" — English numerals rendered in Devanagari. | English numerals invisible | _ENGLISH table, first-class not fallback; test_hinglish_digits_and_hindi_multipliers_mix |
Only "thousand" and "k" were on the Latin side of _MULT. The model now writes romanised Hinglish, and "Bhai rate 18 hazaar 500 fixed hai" parsed as two unrelated bare numbers, 18 and 500. |
18,500 read as 18 | hazaar/hazar/hajaar/sau in _MULT; test_romanised_scale_words_compose |
| सवा was missing from the lexicon. "सवा लाख" fell through to a bare multiplier and returned a confident ₹1,00,000 instead of ₹1,25,000. | A quarter of a lakh, silently | _QUARTER_PREFIX; test_a_fraction_scales_the_operand_not_the_total (the "सवा लाख" row) |
| AssemblyAI wrote साड़े where the lexicon held साढ़े; the half was dropped and ₹33,500 became ₹33,000 — a silent 1.5% error with no flag. The aspirate fold was verified collision-free across all 228 numerals, 19 multipliers and 60 units before being switched on. | A wrong rate with no flag | _ASPIRATE in _spelling_variants; test_aspirated_and_unaspirated_retroflex_are_the_same_word |
| Deepgram wrote "बाइस" (short i) where the lexicon held "बाईस"; 22 never parsed and the only number left standing was a 32,000 the recogniser had hallucinated out of "बत्तीस MXL". The vowel-length fold was verified collision-free across all 231 numerals, 24 multipliers and 75 units, and between those lexicons. | A confident wrong rate from a missing vowel | _VOWEL_LENGTH; test_short_and_long_i_are_the_same_number, test_vowel_length_variants_across_the_lexicon, test_vowel_folding_did_not_merge_two_different_numbers |
| The lexicon held नाइंटीन (anusvara); saaras:v3 wrote नाइनटीन (full consonant). No match, and "nineteen thousand eight hundred" became nothing. Generating variants beats adding words one at a time — the same whack-a-mole that once made ten real emails parse to zero orders. Verified collision-free across all 161 numerals, 11 multipliers and 36 units. | ₹19,800 lost | _nasal_variants; test_a_nasal_written_two_ways_is_the_same_word, test_generating_variants_does_not_collide_two_numerals |
_STANDALONE_FRACTION was folded without _spelling_variants, alone among the lexicons: ढाई folded but its aspirate variant डाई did not, and "डाई लाख" (₹2,50,000) came back as ₹1,00,000. Invisible because the canonical spelling worked. |
Wrong by ₹1,50,000 | Folded with variants like every other table; test_a_fraction_scales_the_operand_not_the_total (the "डाई लाख" row) |
| Deepgram's realtime socket writes अट्ठारह where its batch endpoint writes अठारह; unfolded, that cost ₹18,500 on an otherwise perfect transcript. | ₹18,500 lost | spelling.fold reduces gemination; test_geminated_spelling_is_the_same_number |
| A fused fraction prefix — "साडेचौतीस हजार", the normal spelling in Marathi and routine ASR output in Hindi — was one unknown token. The 34 vanished and an older number won: the broker's own ₹32,000 was returned for a carrier asking ₹34,500. | Booked at our own anchor | _tokens splits a fraction prefix off a known number word; test_fused_fraction_prefix_splits |
Devanagari digits ०-९ sit inside _TOKEN's Devanagari alternative, so they matched before the comma-group alternative: "रेट ४२,५०० है" split into ४२ and ५०० and read ₹42,000; "१८,५००" read ₹18,000. The only wrong rate in the whole phone benchmark — the phone channel changed Sarvam's output formatting to Devanagari numerals and the parser had never seen one. |
A live wrong rate, ₹500 gone | _DEVA_DIGIT translation before tokenising; test_devanagari_digits_parse_like_latin_ones, test_rate_regression.py::test_a_figure_we_once_invented_never_comes_back |
Units — what a number is a number of
| what happened | cost | guard now |
|---|---|---|
| "पचास पैसे प्रति किलोमीटर" — a real way unions publish rates — was read as Rs 50,000 and answered with "itna zyada kaise ji?". | Fifty paise argued with as fifty thousand | पैसे/paise in _UNITS; no test — pinned by _UNITS |
Counted nouns were missing from _UNITS, so the number before them took the bare-thousands rule and became a confident rate the ladder accepted: "पंद्रह गाड़ी चाहिए" → ₹15,000; "पचास पेटी माल है" → ₹50,000; "बारह तारीख को गाड़ी भेजूँगा" → ₹12,000; "गाड़ी चालीस हज़ार किलोमीटर चली" → ₹40,000; "डीज़ल नब्बे रुपये लीटर" → ₹90,000. ₹15,000 sits inside a typical band, so respond_to accepted it with needs_confirmation False. |
Fifteen vehicles booked as a price | count/date/distance/volume units; test_a_counted_noun_is_not_a_rate |
Adding गाड़ी to _UNITS broke the commonest way a rate is quoted, because गाड़ी is the most frequent noun on the call: "पच्चीस हज़ार गाड़ी का लगेगा", "बीस हज़ार गाड़ी", "रेट अट्ठाईस हज़ार गाड़ी का है" all went absent. |
The fix for one wrong rate lost every right one | _SOFT_UNITS disqualify a bare number only — a multiplied one keeps its rate; test_a_counted_noun_after_a_multiplier_is_still_a_rate |
The romanised half of _UNITS was missing while FigureGate scans both scripts: "45 किलोमीटर" passed as not-money, "45 kilometre" was money and blocked. Every answer carrying a non-rupee number — pickup time, distance, weight — was replaced by "एक मिनट जी, मैं अंदर पूछ के बताती हूँ." Measured on two live calls; both testers read it as her stalling. |
She could not say when the load lifts or how far it goes | Romanised units in _UNITS (they only ever remove money-ness); test_units_both_scripts.py::test_the_same_fact_reads_the_same_in_either_script, test_a_romanised_load_fact_is_not_money |
| Deepgram wrote a 32-foot SXL as "बत्तीस अस एक्सल" — an unparseable "अस" between the number and the body word — so the 32 kept no unit, survived as a candidate, and a perfectly clear ₹1,10,000 in the same sentence came back ambiguous. | A clear rate refused | _unit_after looks two tokens out for body and length only (looking ahead for टन or बजे would re-break "नौ बजे तक पीण्या"); no test — pinned by _unit_after |
| "बत्तीस एस एक्सल … एक लाख दस हज़ार रुपए से कम मत बोलिए": the spaced body "एस एक्सल" was unknown, the 32 survived, and a clear ₹1,10,000 came back ambiguous. | A clear rate refused | Rule 2b — exactly one rupee-denominated candidate wins, two stay ambiguous; test_the_stated_denomination_wins, test_two_rupee_figures_stay_ambiguous |
| "नाइंटीन हज़ार एट हंड्रेड नाइन टन" read as one run became a single 19,809-tonne quantity and the rate vanished entirely. | ₹19,800 lost | A trailing bare number followed by a unit belongs to the unit; test_a_trailing_number_with_a_unit_is_not_part_of_the_rate |
| Deepgram's realtime socket split one utterance and rejoined it as "19 हज़ार 809 टन" — the 800 of ₹19,800 welded onto the 9 of "नौ टन". The tonnage carried a unit and was discarded, leaving a clean, confident, wrong ₹19,000. A 32-foot MXL carries about 25 tonnes; 809 is not a payload. Across all 11 archived runs the guard fires on 8 transcripts, converts 3 wrong rates into questions and costs 0 correct ones — the last wrong rate in the corpus. | A confident wrong rate | IMPOSSIBLE_TONNES (60) in hear_rate, which can only downgrade; test_an_impossible_tonnage_invalidates_the_rate, test_a_normal_tonnage_does_not_trip_it, test_the_guard_only_downgrades |
"डीज़ल नब्बे रुपये लीटर है" → ₹90,000 as the freight rate; "पचास रुपये प्रति किलोमीटर" → ₹50,000. The carrier's costs, read as the rate: रुपये sat between the number and लीटर, so _unit_after stopped there. Corpus check: the only " |
Diesel argued with as freight | A stated रुपये after a small number defeats the thousands shorthand; test_a_cost_the_carrier_cites_is_not_the_rate, test_explicit_rupees_defeats_the_thousands_shorthand, test_a_full_form_rupees_amount_still_reads |
| "दो हज़ार रुपये प्रति टन" is ₹2,000 a tonne — how much bulk freight is priced. As a ₹2,000 total it is an order of magnitude out on a 16-tonne load, and this module does not know the tonnage. | An order-of-magnitude error | status="per_unit" with (amount, unit) surfaced; test_a_per_unit_rate_is_surfaced_not_swallowed |
| Body types split into letters ("बत्तीस एम एक्स एल") left "बत्तीस" bare and readable as ₹32,000. | A truck size read as a rate | BODY_LETTERS_MIN; test_a_body_type_split_into_letters_still_shields_its_number |
Attribution — whose number it is
| what happened | cost | guard now |
|---|---|---|
The attribution filter worked and then gave its answer away: if ours: kept the attributed numbers whenever every candidate was attributed. "आपने पैंतीस कहा था ना?" parsed as theirs=True, the filter was skipped, and respond_to() accepted ₹35,000 at our own anchor. |
Booked at our own opening offer, reported as a win | No fallback — every figure ours means no vendor ask; test_the_vendor_quoting_our_own_offer_back_is_not_their_ask, test_our_own_offer_quoted_back_is_still_refused |
| Making the filter authoritative turned every polite opening into a dropped rate, because साहब and bare आप were attribution markers: "साहब, पच्चीस हज़ार लगेगा", "हाँ जी साहब, बत्तीस हज़ार", "आप बताइए, अट्ठाईस हज़ार में भेज दूँ?" all went absent. | Every polite opening lost | आपने stays unconditional; bare आप/तुम need a _THEIRS_AFTER verb; test_a_vocative_does_not_attribute_the_rate_to_us |
Bare दे sat in _THEIRS_AFTER since the module was written. It is the stem of "give" and cannot tell "you are giving" from "I will give", so once the fallback was gone "बाईस हज़ार में दे दूँगा।", "बीस हज़ार में गाड़ी दे देंगे।", "पैंतीस हज़ार में दे रहा हूँ।" were all attributed to us. lines.FALLBACK itself ends "…गाड़ी दे दीजिए?". The review that loosened the BEFORE side never looked at this one. |
Total loss of the commonest sentence in the trade | _THEIRS_AFTER_PHRASE — second-person "give" phrases only; test_the_vendor_offering_their_own_truck_is_their_ask |
| "अरे बयालीस पर क्यों अटके हो" is us being asked why WE are stuck at 42; their own ask, 44, came later in the sentence, and both survived so the turn went ambiguous. | A wasted read-back | अटके/अटक in _THEIRS_AFTER; test_second_person_attribution_resolves_what_used_to_be_ambiguous |
Self-correction and restatement
| what happened | cost | guard now |
|---|---|---|
नहीं lived in _CORRECTION: "बीस हज़ार से कम नहीं, अठारह हज़ार में तो नुकसान है" → heard ₹18,000, "speaker corrected themselves" — the figure the vendor had just refused. "से कम नहीं" is the single most common sentence on a freight call. |
The vendor's ask replaced by the one he refused | _CORRECTION_WEAK with _WEAK_CORRECTION_REACH = 1 — नहीं corrects only next to the figure it withdraws; test_ordinary_negation_is_not_a_self_correction, test_a_real_self_correction_still_takes_the_later_figure |
| "नहीं, तुमकुर का माल नहीं उठाएँगे…" opens with a refusal; treating any नहीं as a correction took the last number in the sentence, which was a lorry's number plate. | A plate read as a rate | The marker must fall between two candidates; test_a_refusal_at_the_start_is_not_a_correction |
The weak-correction reach was measured in tokens; , and ... are tokens, so "पचास हज़ार, अरे नहीं नहीं, पैंतालीस हज़ार" looked three tokens away and went ambiguous. |
A real correction refused | Reach measured in words; test_punctuation_does_not_eat_the_correction_window |
| Skipping every break to fix that skipped the danda, and the marker reached into the next sentence: "बाईस हज़ार लूँगा। नहीं तो अठारह हज़ार वाला काम मत दीजिए।" → ₹18,000 (truth ₹22,000); "पच्चीस हज़ार चाहिए। नहीं, बीस हज़ार वाली गाड़ी अलग है।" → ₹20,000 (truth ₹25,000). One fix, two new wrong answers, in the one class that costs money. | Two confident wrong rates | The window skips soft separators and stops at _SENTENCE_END; test_a_correction_marker_does_not_reach_across_a_sentence |
Treating the Latin full stop as a terminator stopped the window on "तीस हज़ार... नहीं भाई, अट्ठाईस हज़ार" — an ellipsis is three . tokens. In Hindi transcripts the danda ends the sentence; a Latin full stop is usually the recogniser's. |
A real correction refused | _SENTENCE_END is danda, ? and ! only; test_an_ellipsis_is_still_soft |
| "नहीं तो" is "otherwise", not "no": "बाईस हज़ार लूँगा, नहीं तो अठारह हज़ार वाला काम मत दीजिए" → ₹18,000 (truth ₹22,000). The danda spelling was pinned by a test and refused; the comma spelling — which the archive uses more often, 292 transcripts against 199 — returned the wrong figure. The "तो" lookup has to read the raw stream because the window is capped at the reach. | A condition read as a correction | The नहीं तो exclusion; test_a_correction_marker_does_not_reach_across_a_sentence |
scaled[-1] booked at the advance: "पैंतीस हज़ार नहीं, चालीस हज़ार, दस हज़ार एडवांस" → ₹10,000 (truth ₹40,000). "I misspoke — it is X, and the advance is Y" is the most ordinary sentence in freight bargaining. |
The advance booked as the rate | The first candidate after the marker wins; a different figure after it refuses; no test — pinned by _hear_rate |
मतलब was in both _RESTATEMENT and _CORRECTION, and the correction rule runs first: "रेट 45,000 है, यानी बयालीस पाँच" → ambiguous, "रेट 45,000 है, मतलब बयालीस पाँच" → heard ₹42,500. |
Two synonyms, two confident answers | मतलब only in _RESTATEMENT; test_a_restatement_that_disagrees_outranks_the_rate_cue |
| The restatement guard sat at step 5, after the cue rule, which almost always fired first: "45,000 है, यानी बयालीस पाँच" → ambiguous, but "रेट 45,000 है, यानी बयालीस पाँच" → heard ₹45,000. Adding the word "रेट" turned "ask a person" into a confident answer. | A guard that was unreachable on the utterances it was written for | Moved to 4b-i, above every tie-break; test_a_restatement_that_disagrees_outranks_the_rate_cue, test_a_restatement_that_agrees_is_confirmation |
| Tested over the whole utterance, the restatement rule fired on मतलब as an ordinary discourse filler ("मतलब देखो भाई, …"), which opens Hindi sentences constantly, and sent four perfectly clear rates to a human. | Four clear rates refused | Positional — only markers between the first and last figure count; test_a_restatement_marker_before_the_first_figure_restates_nothing |
| U-hi-040 ("हुबली से बैंगलोर, बत्तीस MXL, रेट बस बाईस"): Deepgram wrote the 32-foot vehicle as "बत्तीस हज़ार" and we returned a confident wrong ₹32,000; Sarvam kept it ambiguous and we returned nothing. | A wrong rate from one recogniser, none from the other | The rate cue breaks ties the speaker already broke, and only when exactly one value is cued; test_a_cued_rate_beats_a_bare_number_elsewhere, test_two_cued_rates_stay_ambiguous, test_the_cue_does_not_reach_across_a_clause |
| "रेट तेरह पाँच" is 13,500. AssemblyAI heard the head as "तेरा" — a real word meaning "yours", so nothing looked broken — and the lone "पाँच" scaled to a confident ₹5,000. | Wrong by 8,500, stated without hesitation | _looks_beheaded: a lone 1–9 after an unparseable word is refused, not guessed; test_a_lost_elliptic_head_is_refused_not_guessed, test_a_genuine_low_quote_after_a_known_word_is_not_refused |
Composition and the tokeniser
| what happened | cost | guard now |
|---|---|---|
BARE_THOUSAND_MAX was 999, so a bare 500 was multiplied to ₹5,00,000. On our own corpus all ten bare three-digit numbers are literal ("28 हज़ार 500 दोगे तो गाड़ी भेजता हूँ", "bas 500 ka fark hai"): bare 3-digit numbers that meant thousands, 0 of 10. The worst was "saadhe pachaas hazaar — bas 500 ka fark hai": the real rate 50,500 was in the same sentence and we returned ₹5,00,000 from the word for the gap between two prices. |
Ten times over, confidently | BARE_THOUSAND_MAX = 99; test_bare_three_digit_is_literal_not_thousands, test_two_digit_shorthand_survives |
| दो as an elliptic tail: "चौवालीस दो" is "give me 44,000", not ₹44,200, and reading it as a tail was a wrong answer on the corpus. Adding adjacent bare numbers turned it into 46. | A rate nobody said | दो excluded from ELLIPTIC_TAIL; adjacent bare numbers are never summed; test_the_verb_do_is_not_the_digit_two, test_adjacent_numbers_are_never_summed |
| Two numbers before one scale were summed: "पैंतीस चालीस हज़ार" → ₹75,000 for a carrier asking 35–40k; "बीस पच्चीस हज़ार" → ₹45,000 for 20–25k. Nearly double the ask, confident, a number nobody said. The explicit "पैंतीस से चालीस हज़ार" already went ambiguous; only the commoner bare juxtaposition was wrong. | The worst failure this module has had | Both readings emitted as form="range" so the turn is read back; test_a_range_is_read_back_not_added, test_a_hindi_pair_is_still_a_range, test_composition_still_composes |
| The range rule was wrong in English, which writes 21–99 as two words: "twenty two thousand से कम नहीं होगा" → a confident ₹20,000 (the 2 became 2,000, fell below the floor and was dropped). A real transcript from the phone benchmark; Deepgram nova-3 emits English numerals for code-switched speech routinely. | A confident wrong rate | A tens word plus a unit before a scale is one compound number; test_an_english_compound_numeral_is_one_number |
| A bare multiplier composed as 1 × itself, so any unrecognised word before it produced a confident ₹1,00,000: "सवा लाख" (सवा missing), "डाई लाख" (unfolded aspirate), "लाख कोशिश की" (the idiom "tried a lakh times"). | Wrong by ₹25,000, by ₹1,50,000, and a rate from a sentence with none | implied_one drops the candidate; a fraction prefix still licenses it; test_a_multiplier_with_nothing_in_front_of_it_is_not_a_number |
The fraction was added to the total after the multiplier — _compose(run) + half — and int() truncated it away: "साढ़े नौ हज़ार" → 9,000 (truth 9,500); "रेट साढ़े अठारह हज़ार" → 18,000 (18,500); "साढ़े तीन लाख" → 300,000 (350,000); "पौने दो लाख" → 199,999 (175,000). Only the bare "साढ़े तैंतीस" was in the suite — the one case the bug cannot reach. |
Silently wrong by up to ₹50,000 | The fraction scales the first operand; test_a_fraction_scales_the_operand_not_the_total |
| "डेढ़ लाख" (1,50,000) and "ढाई लाख" (2,50,000) both came back as a flat ₹1,00,000: the standalone-fraction branch emitted 1.5 and moved on, and the लाख parsed as its own number. | Two figures, the wrong one reported | The branch consumes a following multiplier; test_a_fraction_carries_its_multiplier, test_a_bare_fraction_is_unchanged |
| The danda (U+0964) sits inside the Devanagari block, and including it in the token class glued punctuation to the last word: "पाँच।" stopped being पाँच and the elliptic rate silently became half a rate. Five wrong answers before it was found. | Five wrong answers | _TOKEN excludes the danda; test_elliptic_is_hundreds_not_a_decimal |
The comma split "18,500" into 18 and 500 — two candidate rates, ₹18,000 and ₹500,000 — and the utterance went to a read-back. The ASR emits "₹18,500" and "₹1,57,500" routinely, and do_table.py had handled Indian grouping since day one; the two halves of the system disagreed about the same number. |
A clear rate read back | Grouped digits matched before plain digits, 2–3 digits per group so "1,2,3" stays a list; test_a_thousands_comma_is_not_a_sentence_break, test_a_comma_separated_list_is_still_a_list |
| Deepgram's smart_format turns spoken digits into a clock time: Polly says "अठारह सौ ... नौ दस" and the transcript reads "आठ सौ 09:10"; the colon was stripped, both halves joined the run, and ₹19,800 was reported as ₹19,819. The same formatter renders TN38EF9012 as "TN 3 8 EF 09:01 2". The barrier loses a genuine "साढ़े नौ बजे" pickup time, which is read from the words around it anyway. | A wrong rate invented by a formatter | A clock-shaped token ends a run; test_a_formatted_clock_time_does_not_join_a_rate, test_a_real_pickup_time_does_not_disturb_the_rate |
_TOKEN drops any script it does not match, so Kannada "ರೇಟ್ 8 ಹಜಾರ್ 500 ಸಾರ್" tokenised to ['8', '500'] and we answered a confident ₹8,000 — the right ballpark, the wrong number. Adding Kannada numerals would only move the problem to the next script. |
A confident wrong rate | unreadable_script refuses before parsing; test_a_script_we_cannot_tokenise_is_refused, test_hindi_and_english_are_still_read |
On a call a plate is spelled — "एम एच वन टू सी डी फाइव सिक्स सेवन एट" — and to a number parser that is the digits 1, 2, 5, 6, 7, 8 in the open; the Latin _PLATE regex cannot see it. Indian registrations are 9–11 characters; the longest digit-only rate ("पाँच पाँच — पाँच हज़ार पाँच सौ") is six digit-words with no letters and "4-2-5-0-0" is five. |
A plate read as a price | _spelled_out_plates with PLATE_RUN_MIN = 7, PLATE_LETTERS_MIN = 2; test_a_plate_spelled_out_character_by_character_is_not_a_rate, test_a_spelled_plate_does_not_hide_a_real_rate_beside_it, test_a_run_of_digits_that_is_not_a_plate_stays_visible |
The endpoint
| what happened | cost | guard now |
|---|---|---|
| An endpoint that fires while the carrier is still speaking a figure produces a clean, plausible, wrong rate: "अठारह हज़ार [pause] पाँच सौ" → ₹18,000 for a real 18,500; "एक लाख [pause] दस हज़ार" → ₹1,00,000 for a real 1,10,000. It cannot be detected after the fact — "अड़तीस हज़ार" is a complete, common rate and "अठारह हज़ार" is the truncation of a different one, the same shape. Hindi pauses after the thousands and before the hundreds, exactly where a silence endpointer fires. | Ten thousand rupees, gone silently | figure_may_continue returns a hold hint for the streaming layer; test_hold_the_endpoint_when_a_figure_may_continue, test_close_immediately_when_nothing_can_follow |
| The first version stripped the full stop and judged the word before it, so "अठारह हज़ार।" still asked to hold. | Latency on a sentence that had explicitly ended | _SENTENCE_END closes immediately; test_close_immediately_when_nothing_can_follow |
Experiment 27 built a set to expose the numeric guard's blind spot — "रेट है", "अठारह हज़ार और", "पैंतीस से", "हमारा बजट", "डीज़ल का" — and it scored 0/12. The author had scored the previous guard 21/24 on self-written cases; an adversarial set scored the same guard 8/20. So the dangling-word rule was derived from 605 unique real transcripts before it was written: से 289 medial / 0 final, के 101 / 0, में 79 / 0, का 61 / 0, तो 59 / 0, और 21 / 0; false positives 0 of 605. तक is excluded because it does end utterances ("…नेलमंगला तक", 3 occurrences) — a destination is a complete answer. तो is the weakest member: spoken Hindi ends turns with a discourse तो and the corpus is largely TTS-read text; it is the first thing to drop if live calls show latency on completed turns. utterance_may_continue catches 3 of the 12 (the function-word endings) and not "रेट है" or "हमारा बजट", which need a semantic model. See strategy/22-experiment-log.md §28. |
A carrier cut off mid-sentence | _DANGLING in utterance_may_continue; no test — pinned by utterance_may_continue |
Speaking a figure
| what happened | cost | guard now |
|---|---|---|
| sarvam-105b-conversations at temperature 0, asked to speak a rupee figure in Hinglish: 6 of 10 correct, every failure an irregular Hindi ten — 68,000 → "chhiyasi hazaar" (86,000, +26%); 69,000 → "unsath hazaar" (59,000, −14%); 67,000 → "saath hazaar saat sau" (60,700, −9%); 58,000 → "athwanja hazaar" (not a word). saath 60 / sarsath 67 / arsath 68 and unsath 59 / unhattar 69 are near-homophones, and in the read-back ("sahi suna maine?") a wrong conversion launders our error into the vendor's confirmation. | An offer the vendor confirms that nobody made | rupees_in_words is an exhaustive table and the model never converts; test_the_irregular_hindi_tens_the_model_gets_wrong, test_every_plausible_rate_survives_a_round_trip, test_indian_grouping_not_western |
asr/figures.py — does this reply speak a figure
Unqualified test names are in tests/scripts/test_quality_gate.py.
| what happened | cost | guard now |
|---|---|---|
The fatal check's Hindi half was entirely Devanagari (अठारह\|उन्नीस\|बीस\|इक्कीस\|बाईस\|तेईस\|चालीस\|बयालीस\|पचास\|हज़ार\|लाख\|सौ). Of 56 saved zai.glm-4.7 replies, zero contain a single Devanagari character — it answers in romanised Hinglish every time — so against the shipped model the check reduced to hazaar\|lakh\|thousand\|hundred\|rupees. "Atharah pe kar dete hain, aap bas haan boliye." and "Mera budget atharah tak ka hai, aap bataiye." both scored CLEAN; both quote ₹18,000 and the second reveals the ceiling. A hand-built pack of ten unacceptable replies — quoting prices, booking the truck, denying being an AI, inventing load facts, insulting the carrier — scored 9 of 10 completely clean while the harness printed ✅ CLEAN and ⭐ smallest that passes. "0 fatal" was evidence the model did not trip a check, not evidence it said no number. |
Prices quoted and the ceiling revealed, scored clean | TENS_AND_UP — romanised Hindi numerals ten and up, with every spelling seen; test_figures_are_fatal, test_a_romanised_hindi_figure_is_fatal |
saath is 60 (साठ) and also "with" (साथ): "aapke saath pehle bhi achha rate chala hai" fired as fatal: figure on the multi-turn harness and marked two correct sentences as the gravest failure the gate has. |
Two correct sentences marked fatal | _AMBIGUOUS counts only next to a multiplier ("saath hazaar"); no test — pinned by _AMBIGUOUS |
| A numeral before a unit is a quantity: "das minute mein call karti hoon" is ten minutes, and flagging it fires the fatal check on an ordinary, correct sentence — a check people learn to ignore. | An ignored check | NOT_MONEY, unless a scale word sits beside it; test_a_small_quantity_is_not_a_figure |
The Devanagari side flagged every numeral unconditionally while the romanised side required a multiplier for 1–9. एक is the indefinite article, and "कृपया एक निश्चित रेट बताइए।" ("please tell me a definite rate") became the gravest failure the gate has on the first run after the script change. |
A correct request for a rate scored fatal | _DEVA_ONES need a multiplier like ONES; test_the_devanagari_indefinite_article_is_not_a_figure |
SCALES had no Devanagari multipliers, so "दो लाख" licensed nothing and only the लाख was reported. The sentence still flagged, but the token list in the error message lied. |
A misleading error message | Devanagari scales in SCALES; test_a_devanagari_figure_is_still_fatal |
asr/plates.py — hearing the registration
Unqualified test names are in tests/domain/test_asr_plates.py. Several rows quote a measurement "against the pre-fix decoder, mean of twelve seeds × 800 plates" — a synthetic round-trip generator, not live calls.
| what happened | cost | guard now |
|---|---|---|
| Deepgram heard the plate perfectly and wrote the last block as English words: "ka zero one ab one thousand two hundred thirty four". The first decoder knew only Devanagari character names (sarvam:v3 "के ए ज़ीरो वन ए बी वन टू थ्री फोर", assemblyai "के A01, AB1234"), so a correct transcript scored 0/9 and the provider was blamed for our gap. | A correct recogniser scored zero | _english_number composes a four-digit block; test_every_provider_spelling_reads_the_same_plate, test_english_magnitude_composition_in_the_middle |
| The first decoder concatenated every token in the transcript, so "कल सुबह आठ बजे" three words after the plate appended an 8 and KA01AB1234 became KA01AB12348. Both models that got the plate right scored as wrong. | Two correct transcripts scored wrong | The plate is read from a bounded window; test_a_time_after_the_plate_does_not_append_a_digit, test_a_rate_before_the_plate_does_not_prepend_digits |
Any two letters passed as a state code, and ordinary freight sentences assembled confident registrations: "Delhi se Bombay 12 ton, rate 5678 rupaye" → DB12TON5678; "दिल्ली से गोवा 12 टन, आगरा वाला 5678 रुपये" → DG12A5678; "ok 12 ab 3456" → OK12AB3456; "Mumbai se Nagpur 16 ton, 8500 rupaye" → MN16TON8500. All four reproduce against the pre-fix decoder. An earlier draft also cited "एजेंट ज़ीरो वन..." → AJ01AB1234, which does not reproduce (एजेंट does not segment into letter names) and was removed. On the ten no-plate utterances checked into the repo the old decoder returned heard for 4 and this one for 0. |
40% of a realistic corpus | _STATE_CODES, 40 codes; test_ordinary_freight_talk_does_not_spell_a_registration, test_a_state_code_that_does_not_exist_is_not_a_registration |
| TG was missing from the whitelist. MoRTH approved the switch from TS to TG in 2024, Hyderabad is a top-five freight origin, and every Telangana plate issued since was silently refused the moment the whitelist landed. UA (Uttarakhand, 2000–2007) is superseded by UK, but a 2006 lorry is twenty years old and still working. | Every Telangana plate refused | Both codes listed; test_the_state_whitelist_does_not_refuse_real_states |
| District 0 / 00 is never issued; it is what a dropped digit looks like. | A plate with a digit missing accepted | _is_registration requires 1–99; test_district_zero_is_never_issued |
Python's str.isdigit() is True for '१२' and \d matches Unicode Nd, so the shape check passed and "MH१२CD५६७८" was returned with status="heard". The read-back speaks it correctly, the vendor says हाँ, and the stored string matches no RTO record, gate pass or e-way bill. |
A malformed value carrying a confirmation | _DEVA_DIGITS translation; test_devanagari_numerals_never_reach_the_emitted_plate |
| The three-character fallback let a unit spell a plate: "Mumbai se Nagpur 16 ton, 8500 rupaye" → MN16TON8500. MN is Manipur and 8500 is a real block, so the state check cannot catch it. | A registration from a tonnage | _NEVER_LETTERS; test_a_measurement_word_is_not_a_series_block |
_NEVER_LETTERS contained hr, which is Haryana, and lr, mt, kg, ft, all legal series blocks. The written-plate path usually rescued them, but not when the recogniser comma-separates the blocks, which corpus.json's own _meta says the synthesiser produces: "HR, 55, AB, 1234" → absent; "m h 12 lr 5678" → absent. lr/mt/kg/ft stay, as freight jargon far outweighs a series that spells one; hr is out. |
Every Haryana plate refused | test_the_state_whitelist_does_not_refuse_real_states |
The doubling guard lived on the phonetic branch alone and compared out[-1] != letter: in "एक्स से एक्स रे" the second एक्स took the _LETTER branch, which had no guard, and produced XX. Against the pre-fix decoder the doubling accounts for roughly 3% of failing phonetic read-backs; the 16-token window and the letter O are the large causes. The _EITHER_O clause in the same guard is not an observed failure but one the fix would otherwise create; an earlier draft called it observed and quoted "39 of 300", and neither is right. |
XX for X | _emit_letter on every branch; test_the_letter_is_not_doubled_by_its_own_phonetic_word, test_the_phonetic_read_back_round_trips |
| saarika:v2.5 writes ज़ीरो with a precomposed ज़ (U+095B) where saaras:v3 writes ज + nukta. Unfolded, the leading zero of KA01AB1234 did not match and four correct transcripts decoded to KA1AB1234 — a legal-looking registration for a lorry that does not exist. | Four correct transcripts, one nonexistent lorry | _LETTER/_DIGIT folded through spelling.fold; test_precomposed_and_decomposed_nukta_read_the_same, test_leading_zero_survives |
| A carrier saying "M for Madras" scored zero: the phonetic alphabets — the commonest way a plate is spoken on an Indian line — were unreadable. | The commonest spelling scored zero | _PHONETIC (NATO, the Indian telephone alphabet and its Devanagari renderings); test_indian_telephone_alphabet, test_nato_alphabet, test_the_same_in_devanagari |
say.py speaks Q/U/V/W as क्वीन/उत्तर/विक्टर/वाशिंगटन and only the Latin equivalents were in _PHONETIC. An earlier comment claimed those plates "could not be read back"; false — say_plate emits the letter name first ("क्यू से क्वीन"), which already decoded, and removing the four again leaves the round trip at 0.0%. They earn their place for a carrier who says only the phonetic word. X needed no entry: एक्स is a letter name, and its failure was the doubling bug. |
A false claim in a comment | The four Devanagari words in _PHONETIC; no test — pinned by _PHONETIC |
| Deepgram's smart_format renders the spoken TN38EF9012 as "TN 3 8 EF 09:01 2" — a leading zero and a colon invented out of four spoken digits. Split naively, that decoded to TN38EF0901. | A legal-looking plate for a lorry that does not exist | _CLOCK ends the run; test_a_formatted_clock_time_ends_the_registration |
| saarika:v2.5 writes the state code as one word ("टीएन 38 ईएफ 9012") and at worst the whole registration as "एमएच12सीडी5678"; a decoder reading only separated names scored them zero. A partial-match segmenter then let ordinary Hindi words donate letters: "naya vehicle hai" produced AI24BH5678. | Correct transcripts scored zero; a plate from a Hindi word | _segment consumes the entire token or declines; test_glued_state_code, test_fully_glued_registration, test_segmentation_does_not_invent_plates_from_ordinary_words |
| Deepgram wrote 9012 as "9 thousand 12", not "nine thousand twelve"; and "nine thousand twelve nineteen" — 9012 followed by "nineteen feet", the vehicle length — summed to TN38EF9031, a confident wrong plate from two unrelated facts. | A plate assembled from a truck length | Digits join the composition; a ones term cannot follow a ones term or a teen; test_digits_mixed_into_an_english_composition, test_composition_stops_before_the_vehicle_length, test_teens_do_not_chain |
| "zero one" composed to the value 1, losing the leading zero and everything after it. | The district code lost | Compose only when "hundred" or "thousand" is present; test_bare_english_digits_are_not_composed, test_a_bare_digit_group_is_not_composed_without_a_magnitude |
| Deepgram wrote the district of MH12CD5678 as the English word "twelve" ("mh twelve c d five six seven eight"); the digit fallback knew only single digits, so the 12 was dropped silently and the run never matched. | A plate that could not match | A lone tens word is its two digits; test_a_tens_word_is_two_digits_not_nothing |
ओ and o are keys in both _DIGIT ("0") and _LETTER ("O"), and _DIGIT was tested first, so every O decoded as zero. Our own read-back speaks O as ओ: AP48UO4893 spoken and heard back was AP48U0489 — a different plate, heard; OD01AB1234 came back absent, as did every Odisha plate. An earlier draft quoted "23.2%", the worst of twelve seeds. Re-measured against the pre-fix decoder, mean of twelve seeds × 800 plates: 7 states with a 2-letter series 21.3% (range 19.1–23.2); 10 states with 1–3 letters 17.0% (range 15.8–18.5). ~19 of 800 returned a different registration confidently; the rest refused. A plain round trip failed iff the plate contained an O, so the rate is P(O in the series) plus P(state code OD/OR). 0.0% now, on every generator tried. |
Every plate with an O | _EITHER_O marker resolved by _readings against the shape; test_the_letter_o_survives_our_own_read_back |
| The written-plate scan ran on a separator-stripped copy, so "naya vehicle hai, 24 BH 5678 AB" flattened and the tail of "hai" became the state code: AI24BH5678. | A plate assembled half out of a Hindi word | Scanned on the raw text with word boundaries; test_bharat_series, test_segmentation_does_not_invent_plates_from_ordinary_words |
| The written path returned on the first match: "पहले KA-01-AB-1234 भेजने वाला था, अब MH-12-CD-5678 भेज रहा हूँ" booked the OLD truck, and "TN38EF9012 nahi, TN38EF9013 bhej raha hoon" took the refused one. The spelled-out path already reported two readings as ambiguous. | The wrong truck booked | All written matches collected, two distinct ones are ambiguous; test_two_written_registrations_are_ambiguous_not_first_wins |
| A shortest-first window search found five legal plates in one utterance: "one thousand two hundred thirty four" cut short gives 1000, then 1002, 1200, 1230 — artefacts of the window, declared as ambiguity. | A correct plate refused as ambiguous | Longest window first, first hit per starting point; test_every_provider_spelling_reads_the_same_plate |
| The window was capped at 16 tokens, but a phonetic read-back of a three-letter series runs to 21 ("K for Kashmir, A for Agra, zero one, A for Agra, B for Bombay, C for Calcutta, one two three four"). Against the pre-fix decoder, mean of twelve seeds × 800: series length 1–3, 47.1% failed (44.8% refused, ~21/800 wrong); series length 2, 14.4% failed. The gap between those rows is the finding — a 2-letter series is 18 tokens and a 3-letter series 21, so the failure rate is mostly how often the generator crosses the cap. An earlier draft quoted "44% refused" from the 1–3 regime alone. | Nearly half of three-letter plates refused | _MAX_WINDOW = 24; test_the_phonetic_read_back_round_trips |
asr/say.py — speaking a value back
Unqualified test names are in tests/domain/test_asr_say.py.
| what happened | cost | guard now |
|---|---|---|
Measured by synthesising a read-back and running it back through a recogniser — same plate, same voice, only the written form differing: MH12CD5678 polly 0/5, murf 4/5; एम एच वन टू सी डी फाइव सिक्स सेवन एट polly 3/5, murf 2/5; एम, एच, वन, टू, सी, डी, फाइव, सिक्स, सेवन, एट polly 4/5, murf 5/5. A comma forces a prosodic break between characters — what a person does spelling over a bad line — and took Murf from 2/5 to 5/5 at no cost. The plain Latin block scores 0/5 on Polly because it reads it as a word. |
A read-back the recogniser could not hear | say_plate joins Devanagari character names with ", "; test_the_comma_is_load_bearing, test_what_we_say_is_what_we_can_read_back, test_a_plate_is_still_spelled_with_commas |
<say-as interpret-as="spell-out">MH12CD5678</say-as> on Polly renders as "M812CD5678" — the tag designed for spelling turns an H into an 8. interpret-as="digits" emits Hindi number words. |
SSML mangles the plate | No SSML; no test — pinned by say_plate |
| The measurement above is a machine listening to a machine — a proxy for intelligibility, not a person. The NATO and Indian city alphabets score badly through a recogniser and may well be better for a driver on a noisy handset, which is what they were invented for. | An unmeasured choice | phonetic=False by default until a human has listened; test_the_phonetic_form_also_round_trips |
A model asked to convert ₹18,500 into Hindi gets roughly four in ten wrong, always on the near-homophones (chhiyasi for arsath, unsath for unhattar); in a read-back a wrong conversion makes the carrier agree to a number neither side said. Figures in numbers.py, "Speaking a figure". |
An agreed number nobody said | confirm_rate renders from rupees_in_words only; test_a_rate_read_back_is_recoverable |
asr/spelling.py — one normalisation for Devanagari lexicons
| what happened | cost | guard now |
|---|---|---|
A nukta consonant is written precomposed (ज़, U+095B) or as base plus combining nukta (ज + ◌़, U+091C U+093C); identical on screen, unequal in Python. saaras:v3 writes ज़ीरो as ज + ◌़ + ी + र + ो, saarika:v2.5 as ज़ + ी + र + ो. Twice that difference destroyed a value: in numbers.py the lexicon held साढ़े, the recogniser wrote it the other way, the half was dropped and ₹33,500 became ₹33,000; in plates.py the lexicon held ज़ीरो, saarika wrote the precomposed form, the leading zero vanished and KA01AB1234 became KA1AB1234. The second happened after the first was fixed, because the fix lived inside numbers.py. |
Two silent wrong values in two modules | fold — NFD, drop the nukta, lowercase — imported by every lexicon; test_asr_numbers.py::test_the_nukta_is_not_load_bearing, test_asr_plates.py::test_precomposed_and_decomposed_nukta_read_the_same |
| Deepgram's realtime socket writes अट्ठारह where its batch endpoint writes अठारह; unfolded, that cost ₹18,500 on an otherwise perfect transcript. The first rule written matched "the same consonant twice" and never matched the motivating case, because a doubled aspirate is written unaspirated + aspirated (ट्ठ, not ठ्ठ). | ₹18,500 on a perfect transcript | _GEMINATE in fold; test_asr_numbers.py::test_geminated_spelling_is_the_same_number |