Everything below is accurate. The experiment was incomplete. Both arms here are informed vs uninformed: the static arm holds a snapshot, which structurally cannot reflect a later decision. So it lost for being uninformed, not for being ungoverned โ and we were reading the win as evidence that governance is what lets an agent track change.
We added the missing third arm: ungoverned โ a plain memory store holding both decisions, with perfect recall (every document, every query, no ranking, no cutoff) and timestamps. Deliberately more than a real vector index would have.
| arm | what it holds | correct | median s | tokens |
|---|---|---|---|---|
| static | the OLD decision only, as a CLAUDE.md rule | 0/5 | 419 | 16,655 |
| ungoverned | BOTH decisions, perfect recall, dated | 5/5 | 26 | 3,802 |
| mla | both, with the supersession recorded | 5/5 | 76 | 6,771 |
Raw retrieval with timestamps solves plain supersession โ faster and cheaper than governed memory does. The decision rule that forced this correction was pre-registered before the fixture existed: if the ungoverned arm scores โฅ 4/5, our "change" claim does not extend to it and the page gets corrected. It scored 5/5.
The corrected claim: governance earns its keep only where the newest document is not the authoritative one โ a rejected proposal, a reverted change, a draft. Measured separately at ungoverned 0/5 in trust-containment.
Fixture discipline โ the first attempt at this correction was invalid three ways, and we caught all three. (1) The superseding document was titled "(updated)" โ a single parenthetical that hands the answer to any arm that can read. (2) We dated the static snapshot at the same time as adding the new arm โ two variables at once โ which flipped static from confidently stale to stalling, and would have silently broken comparability with this page. (3) The results directory held 21 stale rows from earlier runs, with different values and an older grader, which a naive read would have averaged in and called a clean 5-trial result.
Everything needed to attack this result: the hypothesis, the fixture generator, the isolation model, the grading, the statistics, and the things we explicitly do not claim.
H1. When a team decision is revised, an agent relying on a written snapshot will report the superseded value (or fail to answer), while an agent with governed memory will report the current one.
The null we tried to hold. If governed memory offered no advantage, both arms would land the same way โ either both current, or both stale. A single stale answer from mla, or a single current answer from static, is evidence against us.
| arm | what it has | why it's here |
|---|---|---|
| static | An accurate, well-formed team decision record in CLAUDE.md, written at T0. Sandboxed from the store. | The steelman. This is what a careful team actually does โ and it is exactly what our first benchmark said we owed the reader. |
| mla | Governed memory via the plugin's MCP server. No snapshot in CLAUDE.md. | The system under test. |
Everything else is pinned identical: same model, same prompt, same repo at the same commit, same permission mode, same timeout, same machine, and the same generated fixture within a run โ both arms answer the same question about the same decision. The vanilla (no-memory) arm from the previous benchmark was dropped here: it establishes a floor we have already measured, and its timeouts dominate the cost of a run.
A benchmark with hard-coded facts invites the strongest objection there is: the model was trained on your answer, or your answer leaked into the repo. So there are no facts in this repository. Each run manufactures them:
1. generate n0, n1 = two distinct random 3-digit values (n1 != n0 enforced)
2. provision a throwaway workspace via the real API
3. kb add "the current export API rate limit is n0 rpm" -> born PENDING
4. accept -> n0 is now a governed, accepted decision (T0)
5. kb add same document, value n1 -> new revision
6. accept -> n1 SUPERSEDES n0; the prior revision retires (T1)
7. poll /v1/ask/retrieve until the store demonstrably serves n1
(up to 90s; if it never does, RE-SEED a fresh workspace, up to 3x;
if it still doesn't, ABORT the run)
8. write the T0 document into the static arm's CLAUDE.md โ accurate, and stale
9. inject n1 into the oracle's environment, then DELETE it from the harness's own
Step 7 is the one that matters. Claim indexing is asynchronous and LLM-backed, so its latency varies. Without that gate, a slow index would make mla fail for a reason unrelated to the hypothesis โ and we would have published a number that understated our own system while looking rigorous. A fixture that cannot prove it serves the current value never reaches an agent.
The values live in exactly two places: the store's database and the launcher's process memory. They are in no file, no commit, no environment variable the agent can read. Randomization per run also means a published result cannot contaminate a future one.
The governing invariant: the current value must be reachable only through governed retrieval. Every path an agent could take to it is a leak, and agents are relentless about finding them โ in the previous benchmark the "no memory" arm located the mla binary by absolute path, read its config, and queried the live knowledge base over localhost. It was not told to. It simply tried.
| leak path | closed by |
|---|---|
| Benchmark's own task/oracle files in the worktree | Stripped from the worktree before launch |
| git show <sha>:oracle.sh โ the answer key in history | .git unlinked; the worktree is a plain directory |
| Answer literal in the grading script | Injected via environment at launch; never written to disk |
| Agent runs env and reads the expected value | Deleted from the harness's environment before the agent spawns |
| Inherited MCP server / mla on $PATH | Both stripped from the non-mla arm |
| Reconstructing access via ~/.meetless config + localhost | sandbox-exec: deny file-read on the config and repo tree; deny network to localhost; fake empty HOME |
| The stale snapshot accidentally containing the new value | The static document is generated from T0 only โ it cannot contain T1 |
The static arm's transcripts confirm the seal holds under pressure: it located the retrieval service, attempted to reach it, and reported it as unreachable from its environment โ then fell back to the file, or to searching.
No model grades this. The agent writes answer.txt containing RATELIMIT: <number>. A shell oracle checks that the number equals the current value (T1) with digit-boundary matching. Reporting the superseded value fails. Reporting nothing fails. There is no partial credit and no interpretation.
One known weakness, disclosed: the oracle checks that T1 appears in the answer line, so a contrived answer like RATELIMIT: 106 (superseded by 872) would pass when it should not โ a false pass that would over-credit mla. We audited every stored oracle line from this run: no trial produced that shape. The static arm wrote exactly RATELIMIT: 106; mla wrote 872. Tightening the oracle to parse the first number is queued for the next run.
Partway through an earlier run, the retrieval service died. Every retrieve_knowledge call returned "intel unreachable". The agent retried nine times, inserted its own sleep calls hoping the service would come back, hit the timeout, and the row was written as success=false.
Nothing about that trial says anything about governed memory โ it measures a service being off. Left alone it would have become a published number claiming mla lost. This is the most dangerous bug class in a benchmark: infrastructure failure wearing the costume of a result.
Did the agent report the current value? mla 5/5, static 0/5.
Searching is behavior, not a reward. An arm that grepped 13 times and still answered wrong spent that effort, so effort metrics include failed trials. Scoring them winners-only would have made the losing arm's cost invisible โ flattering precisely the arm that did worst.
A trial killed at the timeout emits no final usage event, so its output tokens are unrecoverable. The per-message usage in the stream is a start-of-message snapshot (2โ6 tokens even for long messages) and summing it would be fiction. We therefore record output as null โ never 0, which would silently make the hardest-working trials look free โ and compare cost on context tokens (input + cache), which is exact for every trial including the killed ones.
| metric | mla (median) | static (median) | Cliff's ฮด | p (approx) |
|---|---|---|---|---|
| reported current value | 5/5 | 0/5 | โ | โ |
| filesystem searches | 0 | 7 | โ1.00 | โ0.01 |
| context tokens | 200k | 705k | โ1.00 | โ0.01 |
| agent time | 59 s | 3/5 hit cap | โ | โ |
On the statistics. Mann-Whitney U and Cliff's ฮด, nonparametric, N=5 per arm. At this N a p-value is directional only โ we report it for completeness, not authority. The load-bearing number is the effect size: ฮด = โ1.00 means every mla trial beat every static trial, with zero overlap in the distributions. That is a separation, not a shifted mean, and it is what a 5/5-vs-0/5 split looks like from the inside.
cd tools/bench-mla node run.mjs --tasks governed-freshness --arms static,mla --trials 5 node analyze.mjs # runs/ -> results/summary.json # every run generates its own fixture, so your numbers will differ from ours. # what should reproduce is the SEPARATION, not the literal values.
Requires a local Meetless stack (control + retrieval) and Claude Code with the mla plugin. The harness self-checks: it verifies the store serves the current value before starting, and aborts rather than scoring a trial against an unhealthy backend. Raw transcripts are not published โ they contain the generated answer and a real credential.