How we tested whether an agent knows which name still stands

A ruling can be true even when practice contradicts it

Teams rename things. The owner decides, and then reality drags: the code, the tickets, the standups keep using the old name for months. So the written history is contested — an old standard, a rename, and a pile of later usage that ignores the rename. Every word of it is true text. Only one name is in force.

H. Given a rename the team ruled on but practice never adopted, an agent finishing ordinary work will use the governed name if it has the verdict, and will tend to use the superseded name if it only has the raw notes — because frequency, recency, and the "reality check" all point the wrong way.

One variable — the verdict — and an ordinary job, not a quiz

The task is deliberately incidental: a teammate left a draft of UI copy that uses the old name; finalize it for shipping. We do not ask "what is the canonical term?" — a direct question makes any capable agent go audit the history and find the answer. We ask it to do a normal piece of work in which the name simply appears.

The ungoverned arm is not starved. It holds the owner's rename note the whole time — it passes 1 trial in 17 by trusting it. The one thing it lacks is the record of which decision still stands when the notes disagree with each other and with practice.

We tied five times before the benchmark measured anything

A capable frontier agent (Opus 4.8) investigates, and investigation finds any answer that is present and cleanly captured. Five earlier versions "tied" for exactly that reason — and each tie was a confound we had put there:

The final fixture therefore uses familiar product nouns (no "what is this?" trigger), a domain-matched substrate (no mismatch flag), no cue in the task, and no clean index — just a pile of notes under notes/.

A contested history, seeded through the real API

1. generate  a familiar rename pair (e.g. Handoff Note -> Transfer Log)
2. write     ~60 dated notes: the old name is prevalent (~59/60), including
             recent notes, PLUS three that disagree:
               A  "the canonical name is Handoff Note"     (older)
               B  "owner decision: it is now Transfer Log" (the rename)
               C  "the rename never landed, we still say Handoff Note" (latest)
3. provision a throwaway workspace; ingest all notes via the real API
4. govern    ACCEPT B (the new name) · REJECT A (the old standard)
             C is habit/opinion, left ungoverned
5. verify    governed retrieval SERVES the accepted new name
             -> if it does not, ABORT (fail-closed; the benchmark is hollow otherwise)
6. on disk   the same notes go under notes/ for the ungoverned arm (no verdict)
7. inject    the accepted name into the oracle's env, then DELETE it from the harness's own

Recency and frequency both point at the old name (note C is the latest; the old name saturates the corpus). So the raw evidence genuinely argues for the superseded term — the verdict is the only thing that argues for the current one. Values are generated per run and live only in the store's DB and the launcher's memory, never in a file the agent can read.

A deterministic oracle on the finalized work-product

No model grades this. The agent writes the finalized copy to finalized.md; a shell oracle counts occurrences of each name (case-insensitive, plural-aware — the two names share no words, so counting is unambiguous):

Grading the produced work-product (not a quiz answer) is the point: it measures what the agent would actually ship, which is where the drift bites in real life.

Reliable vs. a coin-flip that lands wrong

The coin flip is visible in the traces. The same model, reading the same notes, concluded the new name when it weighted the owner's rename, and the old name when it weighted the "never landed" note. It usually weights the latter — the raw evidence is genuinely stacked that way. The health signal confirms the ablation on every trial: mla grounded (used the governed verdict), ungoverned unused (reasoned from raw notes). And the pull arm was slower (~142s vs ~110s) — it pays more to be less reliable.

One mla miss (Case File), kept. Even a retrieved verdict lost once to the contested context — the residual form of the very failure this benchmark studies. ~94%, not a suspicious 100%.

The strongest arguments against this

• "You engineered the raw evidence to mislead." Yes — that is what a contested history is. The point is not that the notes are impossible to reason through (the arm passes 1/17), but that reasoning through them is unreliable, and the verdict makes it deterministic. If the notes agreed, there would be nothing to govern.
• "The ungoverned arm could have trusted the owner's note." It could, and once it did. Nothing in the text marks the owner's rename as still in force versus the later "reality check" as mere habit — that ranking is the governance verdict, and it is what the arm lacks.
• "Small N." 17 decisive trials across 5 renames, one machine, one model. A mechanism demonstration, not a population estimate — earned by a near-total separation (6% vs 94%) that holds on every rename, with the mechanism legible in every trace.
• "A cleaner-model or a different prompt would do better." Possibly — but a less careful agent has an even weaker tiebreaker and would drift more, not less. We claim only what we ran.
• Substrate honesty. Runs use a neutral, domain-matched repo (a work-item tracker with no Meetless code or governance vocabulary). Earlier confounds (§03) all came from the substrate or the task; this configuration removes them.

Scope

• Not that a capable agent cannot get this right from raw notes — it can, ~6% of the time. The claim is about reliability, not possibility.
• Not that governance is free, or that it never misses — it missed once here. Our retrieval benchmark publishes the token overhead on tasks where memory adds nothing.
• Not the same result as trust-containment. There, two claims are byte-identical and the agent gets stuck; here the evidence is misleading and the agent is confidently wrong. Same verdict mechanism, complementary failure mode.
• Not a claim about any competitor's product. The opponent is the general shape of ungoverned memory — a store that remembers text without remembering which decision still stands.

Run it yourself

cd tools/bench-mla
node run.mjs --tasks canonical-deprecation-use \
  --arms vanilla,static,ungoverned,mla --corpus 60 --trials 5 --substrate neutral

# the rename pair is generated per run; your names will differ from ours.
# what should reproduce is the SEPARATION, not the literal names.

Requires a local Meetless stack and Claude Code with the mla plugin. The harness verifies the fixture is correctly governed (the new name is served by governed retrieval) before any trial and aborts otherwise; it marks a trial invalid — never a loss — if the backend was unhealthy or the provider refused the run. Raw transcripts are not published — they contain the generated names.