ADR 0001 — Prompts stay in Python, versioned

2026-08-13 · Status: superseded by ADR 0002**

Superseded the same day. The trigger this ADR named for revisiting BAML was "schema violations start appearing". That is not what happened — instead the tool schema's field order turned out to change model behaviour more than any rule edit, which this ADR's prompt-only fingerprint could not even detect, and two prompt edits were silently lost. See ADR 0002.

Context

The extraction system prompt is ~77 lines of rules and the tool schema ~125 lines, both currently inline in app/scripts/bench_extraction.py. That is 39 % of a test script being product. We need to decide where they live before A2 builds the real extractor on top of them.

The field has four real answers in 2026:

Option What it is Adoption
Python constants A string in a module The default everywhere
Jinja2 .j2 Template files rendered at runtime Widely described as the standard for separating prompt from code
.prompty (Microsoft) Markdown + YAML frontmatter + Jinja body, with a VS Code extension and multi-language runtimes A genuine emerging standard
BAML A DSL where a prompt is a strongly-typed function with a compiler ~7,500 stars; strong reputation for production type safety
Langfuse / PromptLayer / Humanloop Platform: prompts stored, versioned and fetched at runtime The three most deployed prompt-management tools

Decision

Keep the prompt as a Python module-level constant, in app/domain/extraction/prompt.py, with an explicit PROMPT_VERSION and a content hash stamped into every benchmark result.

Not a template file, not YAML, not a DSL, not a runtime fetch — for now.

Why

1. We have almost nothing to template. The prompt is 77 lines of static rules with zero interpolation today. A template engine for a constant string is ceremony that buys nothing.

2. ⚠️ Templating the system prompt would break prompt caching. Caching is a prefix match — any byte change anywhere in the prefix invalidates everything after it. The moment we interpolate per-request data into the system prompt, the cache never hits. Dynamic content (retrieval few-shot, knowledge facts) belongs in the user turn, after the cache breakpoint — which means the system prompt should stay a frozen constant by design, not just by convenience.

3. domain/ must stay pure. The import-linter contract in pyproject.toml forbids I/O in the domain layer. A Python constant needs no file read; a .j2 or .yaml does.

4. YAML is the wrong container for prose. Seventy-seven lines of rules inside a YAML block scalar means indentation and escaping problems for zero gain. YAML is good for metadata, and our metadata is two fields.

5. The actual problem isn't the format — it's provenance. The prompt changed six times in one day (bookable fields, no_order vs hold, totals arithmetic, customer identity, gate-time descriptions, the multilingual repair) and not one result file records which version produced it. So "field accuracy went 98 % → 100 %" is currently unfalsifiable. That is fixed by a version constant, in any format. Changing file type without adding versioning would solve nothing.

What we are not doing, and when we would

Option Why not now Trigger to revisit
Jinja2 .j2 Nothing to template; would invite breaking the cache Genuine per-request templating that does not sit in the cached prefix
.prompty Couples model config to the prompt file, but we choose the model from a CLI flag and drive Bedrock directly. Another dependency and toolchain If a non-engineer starts editing prompts and wants the VS Code tooling
BAML Real benefits — typed functions, compiler guarantees, a forgiving parser. But we already get type safety from Pydantic plus forced toolChoice, and we have recorded 0 schema violations across every run Schema violations start appearing, or we want its parser's tolerance of malformed output
Langfuse prompt management Decoupling prompt edits from deploys matters when several people edit prompts. We have one engineer, and a runtime fetch adds a failure mode mid-booking Phase D, when Langfuse is already in for tracing — then prompt versions and traces link up

Consequences

Good

  • No new dependency, no toolchain, no codegen step.
  • The system prompt is a frozen byte-stable constant, so prompt caching works.
  • domain/ stays pure and importable from anywhere without I/O.
  • Version + hash in every result makes prompt-to-score attribution real.
  • Moving to any of the four alternatives later is mechanical — the prompt is already a single named constant with one import site.

Costs

  • Prompt edits require a deploy. Acceptable at one engineer; the trigger to revisit is written above.
  • Prose inside a .py file is marginally more awkward to read than a .md. Mitigated by keeping it in its own module with nothing else in it.
  • No web UI for editing. Deliberate — a UI for editing the rules that govern what a booking agent may say is a feature to design, not a side effect of a file format.

Implementation

# app/domain/extraction/prompt.py
PROMPT_VERSION = "2026-08-13.10"
SYSTEM = """You extract freight orders from ..."""

def fingerprint() -> str:
    """Covers the prompt AND the tool schema. Property order in the schema changes
    the model's behaviour as much as the prompt text does, so hashing the prompt
    alone is not provenance."""
    return hashlib.sha256(
        (SYSTEM + json.dumps(TOOL, sort_keys=True)).encode()
    ).hexdigest()[:12]

Every result file records {"prompt_version", "prompt_sha", "run_at"}. A run whose hash does not match its declared version is a warning, not a silent pass.

The schema is part of the prompt. Moving action from the first property to the last changed classification behaviour more than any rule edit did — the model generates tool JSON top-down, so a property can only be conditioned on the properties above it. reasoning first, action last, is therefore a prompt decision expressed in schema, and the fingerprint has to cover it. See docs/extraction-benchmark.md finding 12.

Prompt edits must assert they applied. Two rule changes were once reported as made and were never in the file: str.replace() on a stale target is a silent no-op. Any programmatic prompt edit asserts its match count before writing.

docs/how-the-benchmark-works.md §3 · the import-linter contract in pyproject.toml · Prompty · Langfuse prompt data model · Jinja2 for prompt management