ADR 0002 — Prompts move to BAML

2026-08-13 · Status: accepted · Supersedes ADR 0001

Context

ADR 0001, written this morning, kept the prompt as a Python constant and explicitly declined BAML:

BAML — Real benefits — typed functions, compiler guarantees, a forgiving parser. But we already get type safety from Pydantic plus forced toolChoice, and we have recorded 0 schema violations across every run.

The trigger it named for revisiting was "schema violations start appearing". That is not what happened. Two other things did.

1. The reasoning-first finding. Within hours of 0001 we found that action was the first property in the tool schema and therefore the first token generated — before any reasoning. The model emitted hold_and_ask and then wrote eleven numbered steps concluding create_with_flag. Fixing it meant reordering the schema, which is a prompt change expressed as a data-structure change. ADR 0001 treated prompt and schema as separate concerns; they are not. Its own prompt_fingerprint() hashed only the prompt text and would have reported that fix as no change at all.

2. Prompt edits went missing. Two rules were reported as applied and were in no commit, because a str.replace() against stale text is a silent no-op. Both losses came from the prompt being an unremarkable string inside a 600-line script — nothing typed it, nothing validated it, nothing failed when it was wrong.

The decision to adopt BAML was the user's. These are the reasons it holds up.

Decision

The extraction prompt and its output schema live together in baml_src/extraction.baml. baml_client/ is generated and committed.

Why this is better here, specifically

Field order is declared, not incidental. In BAML the class definition is the output contract, and the order you write the fields is the order the model fills them. The single most valuable thing we learned about this prompt — reasoning first, action last — is now a visible property of a type declaration instead of a comment above a dict literal begging nobody to reorder it.

One artefact, one version. ADR 0001's fingerprint covered the prompt but not the schema. With both in one file the question "what produced this number" has one answer.

The schema-aligned parser earns its keep on the long tail. We recorded zero schema violations across the corpus — but the corpus is 25 emails we generated. Forced toolChoice gets strict JSON at the cost of the model having to emit exactly one shape. BAML parses a more forgiving surface and coerces it, which matters more on a public demo taking arbitrary pasted email than it did on a closed benchmark.

It is faster. Same model, same rules, on the same email: 55.6 s through forced tool-use, 40.8 s through BAML.

What ADR 0001 got right and we are keeping

  • The system prompt is a frozen prefix; the email goes in the user turn. Verified after porting: BAML renders one 10.8 KB system block with the rules and the output format, and puts only the email in the user message. The prompt-caching argument survives intact.
  • Provenance is the real problem, not file format. Still true. BAML gives a versioned artefact; it does not excuse us from stamping results.
  • domain/ stays pure. baml_client is an adapter-layer concern. The import-linter contract is unchanged and still passes.

Costs, honestly

  • A codegen step. baml-cli generate must run before packaging, or a stale baml_client ships yesterday's prompt. Wired into deploy/deploy.sh.
  • A 21 MB native dependency in the Lambda package. Fine here; it would not be fine somewhere with a tight cold-start budget.
  • Enum variants must be uppercase, so Create carries @alias("create") to keep the wire values the benchmark ground truth already uses.
  • bench_extraction.py still holds the old prompt and tool schema. Until it is cut over, two definitions of the prompt exist — precisely the drift ADR 0001 warned about, now real. Cutting the benchmark over is the next task and should not wait.

baml_src/extraction.baml · docs/extraction-benchmark.md findings 11 and 12 · BAML