A head-to-head on the capability that matters: does the agent know the team's settled decisions β and what does that cost when it doesn't need them?
The 0/5 below is not a measurement of the opponent. It is a measurement of our own question. The task prompt said, verbatim: βthe answers β¦ are NOT in this repository β they live only in that knowledge base. Look up the knowledge base β¦β
Any agent without a knowledge base reads that, concludes the task is impossible, and writes UNKNOWN. We proved it: we gave a static arm all three facts in its CLAUDE.md β sitting in its own context window β and under that prompt it still scored 0/5, explaining βI could not query the knowledge base in this environment.β It never looked at the file. Under a neutral question it scores 5/5. So does an ungoverned vector store.
| arm | under the published prompt | under a neutral question |
|---|---|---|
| static β the three facts in CLAUDE.md | 0/5 | 5/5 |
| ungoverned β the three facts in a plain store | β | 5/5 |
And the task was the wrong question to ask us anyway. Two of its three facts are a codename and an owner β naming and organisational facts that governed memory deliberately does not store. Our extractor's own contract is βdoes the statement BIND the product, or merely DESCRIBE it?β A codename binds nothing. It is supposed to be skipped.
For raw fact recall from a document, a CLAUDE.md or a vector store will always beat governed memory. They hand back text; we hand back governed decisions. We are not a document store. We are a decision store. A benchmark that asks us to be a document store is one we should lose β and it should not be on this site claiming the opposite. Why this page is withdrawn rather than corrected
A rebuilt version will ask about decisions and requirements β the things we actually claim to govern β with a prompt that names no mechanism, and a lint that refuses to run any task whose wording presupposes one. The claims that survive our own adversarial re-testing are provenance (a file answers βwho approved thisβ 0/5, structurally) and contested-canonical. (An earlier version of this note also cited trust-containment as a survivor; we retracted that one on 2026-07-17 β it graded the opponent against a verdict we never put in its corpus.)
Asked for three settled facts that exist only in the team's knowledge base β nowhere on disk, in git, in env, or reachable over the network. The facts are randomly generated on every run, so no model could have been trained on them. mla retrieved them every time (grounded, 0 filesystem searches in all 5 trials, ~31 s). Vanilla failed every time β it grepped, guessed, and gave up, burning 7Γ the context and 4.7 minutes to still be wrong. This isn't faster-vs-slower; it's right vs. wrong.
| metric | mla | vanilla |
|---|---|---|
| passed | 5/5 β | 0/5 β |
| retrieval | grounded | none |
| greps (median) | 0 (every trial) | 9 (6β15) |
| work tokens | 5.4k | 21.2k (β€24k) |
| context tokens | 163k | 1.12M (6.9Γ) |
| agent time | 31 s | 4.7 min (β€7.2) |
Vanilla's numbers are its failure cost β it spent 7Γ the context and produced the wrong answer. Cliff's Ξ΄ = β1.00 on both searches and context: perfect separation, not a shifted average. Efficiency here collapses to a single fact β only one arm can answer at all.
| metric | mla | vanilla |
|---|---|---|
| passed | 5/5 β | 5/5 β |
| retrievals | 0 | 0 |
| work tokens | 5.8k | 3.7k |
| agent time | 31 s | 14 s |
| overhead | +2.1k tok (+58%) Β· +17 s | |
A fully-specified task needing no team knowledge. mla loses here, and by a perfectly separated margin (Cliff's Ξ΄ = 1.00 on tokens β every mla trial cost more than every vanilla trial). Note it made zero retrieval calls: the cost is the grounding floor, not reflexive lookups. This is the honest counterweight, and it goes on the page.
A benchmark is only as honest as its isolation. The "vanilla" arm was given every chance β including access to mla's own installation β and each way it leaked the answer was found by forensics and sealed. The result above is what's left after all seven.
Sealing the leaks answers "could the agent have found it?" It does not answer the harder question: could the model already know it? A benchmark with hand-written facts can always be dismissed as memorized from training data or leaked into a public repo. So we stopped shipping facts and shipped a generator.
# reproduce cd tools/bench-mla node run.mjs --trials 5 # tasks Γ arms Γ trials β runs/ node analyze.mjs # runs/ β results/summary.json node report.mjs # β results/report.md + .html