Our first benchmark beat an agent with no memory, and we said the credible opponent was a well-written CLAUDE.md snapshot. This is that fight. Same task, one difference: the decision changed after the snapshot was taken.
The result below is real and reproducible. But it proves something narrower than "governance handles change," and it is worth being precise about which.
The opponent here is a CLAUDE.md snapshot: a thing that structurally cannot update. It lost for being uninformed, not for being ungoverned. So we built the fight we had been avoiding — the same task against an ungoverned memory store holding both decisions, with perfect recall and timestamps (deliberately more than a real vector index gets):
| arm | correct | shipped the STALE value | median time | tokens |
|---|---|---|---|---|
| static — the snapshot (this page) | 0/5 | 4 | 419s | 16,655 |
| ungoverned — both docs, dated | 5/5 | 0 | 26s | 3,802 |
| mla — governed memory | 5/5 | 0 | 76s | 6,771 |
An ungoverned store solves plain supersession perfectly — three times faster than us, on half the tokens. It sorts by date and it is simply right. No governance required. Where governance buys nothing, its overhead buys nothing either, and that is our number to own.
The claim, stated narrowly and defensibly: governance earns its keep in exactly one place — when the newest document is not the authoritative one. A rejected proposal is newer than the decision it failed to overturn. A reverted change is newer than what it reverted. A draft is newer than the approved spec. A vector store sorts by time and hands you the wrong one with total confidence — and we measured that separately at 0/5 (trust-containment), where it did not go wrong so much as get stuck. "We handle change" was too broad. "We handle change when time doesn't tell you who's right" is true.
The decision rule that forced this correction was pre-registered before the fixture existed: if the ungoverned arm scores ≥4/5, raw retrieval with timestamps solves supersession, our "change" claim does not extend to it, and this page gets corrected. It scored 5/5.
The static arm holds a competent, accurate team decision record in CLAUDE.md — the kind a careful team really commits. It was simply written before the decision was revised. Governed memory tracked the revision; the file did not. mla returned the current value in all 5 trials, without a single filesystem search. The snapshot arm never got there: twice it confidently reported the stale number, and three times it searched until the clock ran out, unable to confirm whether what it held was still true.
Nothing here is authored or memorized. Each run invents two random values, seeds the first as a governed decision, then supersedes it — and verifies the store actually serves the new one before any agent starts.
| metric | mla | static |
|---|---|---|
| reported current value | 5/5 ✓ | 0/5 ✗ |
| retrieval | grounded | none |
| filesystem searches | 0 (every trial) | 7 (5–13) |
| agent time (median) | 59 s | 3/5 hit the cap |
Cliff's δ = −1.00 on searches — perfect separation, not a shifted average. mla never opened the repo; the snapshot arm never stopped looking.
| metric | mla | static |
|---|---|---|
| context tokens (median) | 200k | 705k |
| range | 164k–253k | 534k–1.54M |
| retrieve calls | 3–6 | 0 (no access) |
| separation | δ = −1.00 · p ≈ 0.01 | |
Uncertainty is not free. Unable to verify whether its note was current, the snapshot arm searched, re-read, and reasoned — spending 3.5× the context to arrive at the wrong answer, or at none.
Both failure modes are worth naming. Stale is the dangerous one — a confident, precise, wrong number an engineer would act on. No answer is the honest one — the agent sensed its note might be out of date and hunted for confirmation it could never obtain. Neither is a working memory.
The failure mode we feared was the opposite of a flattering one: an unfair win. If the store hadn't finished indexing the new value, mla would fail for a reason that has nothing to do with the claim. So the harness refuses to run until it has confirmed the store serves the current value — and refuses to score a trial whose backend was unhealthy.
The original static 0/5 was true, but only 2 of those 5 trials actually demonstrated the thing we said it demonstrated.
Two of the five had been stalled by the machine going to sleep mid-run — a suspended process runs no timers, so our cap never fired and the trial produced no answer at all. A third ran out the clock. Only two trials actually showed the static arm confidently returning the superseded value, which is the claim the page was making.
The error ran in our favour — it made the opponent look worse than it is — so we re-ran it rather than footnoting it. On the honest re-run, 4 of 5 static trials returned the stale value and the fifth timed out. The conclusion survived; the evidence for it is now real. Had it gone the other way we would have retracted the number.
| Originally published | Honest re-run | |
|---|---|---|
| static: returned the STALE value | 2/5 | 4/5 |
| static: no answer (stall / timeout) | 3/5 | 1/5 |
| static: task passed | 0/5 | 0/5 |
| mla: task passed | 5/5 | 5/5 |
# reproduce cd tools/bench-mla node run.mjs --tasks governed-freshness --arms static,mla --trials 5 node analyze.mjs # runs/ → results/summary.json