Testing
Four layers, each of which can fail on its own. The project's standing lesson is that green from a harness means nothing until the harness has been shown to run: an English TTS default, a skipped file, a text mode that bypassed the processor under test — each produced a passing result while measuring nothing.
flowchart LR
U["Unit
1,289 tests · 4 s"] --> C["Pinned corpus
133 real utterances
+ perturbation ceilings"]
C --> M["Mutation
scripts/mutate.py"]
M --> E["End to end
poc/e2e.py · real audio both ways
asserts on the session record"]
E --> H["A person listening
rate_call.py · the blind pack"]
1. Unit
.venv/bin/python -m pytest tests/ -q # 1,289 passed, 2 skipped, 4 xfailed — OBSERVED 2026-09-10
One interpreter for everything since v0.0.2. The Pipecat-dependent tests
(tests/poc/) run in the same suite; nothing importorskips any more, so a
test that skips is a test that failed to be collected, and it shows.
What the money path is held to: test_ceiling.py (33 cases: the ceiling is
never spoken, a counter is never above the quote, the concession is never the
midpoint), test_stance.py (24: a closed call stays closed, a raise after the
counter is not accepted), test_multiturn_allowlist.py (16: the gate catches a
figure recombined from authorised words), test_figure_gate_frames.py (7: the
gate as a frame processor, including the streamed-fragment leak).
What the call structure is held to: test_phases.py (17 cases, each named for
the live failure it pins), test_call_end.py (16), test_probe_answered.py
(15), test_brain_abuse.py, test_pitch_deadlock.py, test_no_verbatim_repeat.py.
2. The pinned corpus
tests/domain/calling/fixtures/routing_corpus.json holds every distinct
sentence a real vendor has said on a recorded call — 133 rows — with the route
the router gives it. A change that moves any of them fails CI; an intended move
is a diff in the same commit. It asserts stability, not correctness: the
hand-checked cases in test_classify_corpus.py are the ones asserted correct.
test_router_stability.py is the ratchet. Five perturbations a real call
applies constantly, over the 133 rows; the number of routes that move is
pinned and may only fall.
| perturbation | v0.0.1 | v0.0.2 | of which money-critical |
|---|---|---|---|
| leading हां | 10 | 10 | 2 |
| trailing जी | 8 | 7 | 0 |
| punctuation stripped | 5 | 4 | 0 |
| first two words joined | 29 | 1 | 1 (was 18) |
| nukta dropped | — | 0 | 0 |
OBSERVED with poc/analysis/perturb.py, 2026-09-10.
tests/scripts/test_rate_regression.py replays every saved recogniser
transcript through the parser and fails if a confident wrong rate reappears
(a range read as ₹75,000, a diesel price as ₹90,000). The data is gitignored,
so on a fresh clone it degrades to eight literal pins and says so.
tests/domain/fixtures/model_vocabulary.json is every Latin word the model
has written on a live call; every one must reach the voice as Devanagari or a
known acronym.
3. Mutation
.venv/bin/python scripts/mutate.py # or: make mutate
.venv/bin/python scripts/mutate.py ceiling stance --max 40
Each mutant flips one thing — a comparison, and/or, a dropped not, a
constant, a guard turned into if True: — writes the module in place, runs
the target's tests, restores the file. A mutant the tests do not kill is a
line the tests do not check. Survivors are listed with the mutation and the
line; each needs a test or a written equivalence note. Results for this
release are on the changelog.
4. End to end
set -a; . ./.env; set +a
.venv/bin/python poc/e2e.py # every scenario under poc/pc-web/server/evals/
.venv/bin/python poc/e2e.py hangup_vendor_goodbye regressions
pipecat eval run grades what she said — a Kokoro voice speaks the vendor's
Hindi, Flux hears it, Murf speaks her reply, Whisper transcribes it, a judge
model reads the transcript. poc/e2e.py wraps that and grades what the system
did: a fresh bot per scenario, then the session record is checked — who
ended the call, with which disposition, whether the last audio stopped before
the line dropped, no service_error, tagged eval so it never joins the live
corpus.
| scenario | what it proves |
|---|---|
regressions |
the seven utterances that broke live calls, in one conversation |
hangup_vendor_goodbye |
he says goodbye; she says goodbye and ends the call herself |
hangup_after_wrap_up |
a good quote is squeezed once, taken, and the next "ठीक है" ends the call |
hangup_callback |
"call me this evening" is acknowledged and ended, not treated as abuse |
hangup_wrong_number |
an apology, nothing about the load |
hangup_voicemail |
a switched-off recording is not a person |
hangup_browser |
the tester hangs up; the record says so and nothing raises |
Three things the harness cannot judge, recorded so nobody chases them as
agent bugs: Whisper garbles proper nouns ("ओरि" → "और ही"), so names get an
exact unit test instead; Kokoro cannot reliably say a one-syllable "हां."; and
the harness holds the socket ~10 s after the bot's EndFrame before the
recording is written.
5. A person
poc/rate_call.py scores a call by ear right after it happened; the 31-clip
blind listening pack settled the voice and the pace. No benchmark of spoken
Hinglish conversation quality exists, and LLM judges flatter non-Latin
scripts, so the judged bake-off orders candidates for a human ear rather than
deciding.
Comment trims are verified
scripts/comments_only.py <file> parses the committed and working versions,
drops every docstring, and compares the syntax trees. Every comment trim in
v0.0.2 printed ok:; the checker caught two accidental deletions on the way.